Sweeping weight decay across four decades on a wearable-accelerometer activity classifier, how do you pick the value?

## Why a four-decade logarithmic sweep The decay coefficient enters the update multiplicatively and useful values on real problems span several orders of magnitude, so the only sensible search is a logarithmic one. Start with one run per decade across four decades, look at the shape, then refine with a factor-of-three grid in the one or two decades that matter. A linear grid over the same range spends almost every run in a regime where decay does nothing at all, and will report — correctly and uselessly — that the knob has no effect. ## The two curves and what each one is for Run the sweep on a fixed recipe: same architecture over the six sensor channels, same window length, same learning-rate schedule, same number of epochs, same seed policy. Record both a training metric and a held-out metric. The **training metric** is a sanity instrument, not a selection instrument. It should fall monotonically as decay rises. If it does not move at all across the whole sweep, decay is not reaching the parameters you think it is — a grouping bug, or values so small they are swamped by the gradient. If it collapses immediately at the smallest nonzero value, your grid starts too high. The **held-out metric** is what you select on, and its shape has three regions: - **Flat left region.** The penalty is too weak to move the equilibrium weight norm perceptibly, so the run is indistinguishable from no decay. Every point here is the same model. - **Rising middle region.** Decay is now suppressing weights the 20,000 windows do not support, and held-out performance improves while training performance slips. This is the regime you want. - **Falling right region.** Both curves now decline together. That joint decline is the diagnostic signature of underfitting: the penalty is suppressing structure the data genuinely supports. There is no ambiguity here — regularising too hard and overfitting look nothing alike on a two-curve plot. ## Choosing a point Take the flat top, not the argmax. On a dataset of this size, run-to-run variation from initialisation and data ordering routinely exceeds the gap between adjacent grid points, so the single best number in a sweep is partly luck. A value sitting in a broad plateau will reproduce; a spike that beats its neighbours by a hair usually will not. If you can afford it, run two or three seeds at the two or three candidate values and choose on the mean, preferring the slightly stronger decay when the means tie, since that is the side that degrades gracefully. ## What invalidates the result The decay coefficient is not a property of the architecture. It is calibrated against a particular amount of data, a particular number of update steps and a particular learning-rate schedule, and changing any of these moves the right value. - **Data volume.** More data does more of the regularising by itself, so the best explicit penalty falls as the dataset grows. - **Run length.** Decay acts once per update, so total shrinkage depends on the number of steps. Halving the epochs weakens the effective regularisation at a fixed coefficient. - **Learning-rate schedule.** In the usual update the shrinkage applied per step is scaled by the learning rate, so a change to the schedule changes how much decay is actually applied. On layers followed by normalization the coupling is stronger still, since the two together determine the effective step size. - **Other regularisers.** Adding augmentation or dropout after tuning decay means part of the job is now being done twice. Any of these means the sweep is stale and needs re-running, at least coarsely. ## Sweeping decay together with other knobs A joint search over learning rate and decay is more honest than a sequential one, because the two interact, but it costs a product of runs rather than a sum. A workable compromise is a coarse joint grid — three learning rates by four decay values — to locate the region, then a one-dimensional refinement of decay at the winning learning rate. State the compromise explicitly rather than pretending the sequential search was principled; interviewers are listening for whether you know the knobs interact. ## What a weak answer looks like Selecting on the training metric, sweeping linearly, tuning once and never revisiting, or treating more decay as inherently safer. The last is the most common: candidates internalise that regularisation prevents overfitting and conclude that erring high is conservative. On the right-hand side of the curve it is simply a worse model, and unlike overfitting it will not announce itself as a train-versus-held-out gap.