Why are biases and normalization scale and shift parameters usually exempt from weight decay?

## What the parameter groups actually are A trained network's parameters fall into a few structurally different kinds. There are the multiplicative weights — the entries of dense weight matrices and convolution kernels — which multiply incoming activations. There are bias terms, one per output unit, which add a constant offset. And where normalization layers are used there is a learned scale and a learned shift, typically one of each per channel or per feature, applied after the layer has standardized its input. Weight decay is conventionally applied to the first group only. The other parameters are placed in a group with the decay coefficient set to zero. Understanding why is a good test of whether a candidate knows what decay is actually for. ## Argument one: they barely exist Decay is a budget on the squared L2 norm of the parameters. A dense layer mapping 512 inputs to 512 outputs has 262,144 weights and 512 biases — the biases are two-tenths of a percent of the parameters in that layer, and their contribution to the norm penalty is correspondingly negligible. The same holds for normalization scales and shifts, which are per-channel and therefore vanishingly few next to the kernels feeding them. Whatever regularisation pressure you hoped to apply, applying it to these groups contributes almost none of it. The capacity of the model lives in the multiplicative weights, so that is where a capacity constraint belongs. ## Argument two: shrinking them does not constrain the right thing The reason large weights are dangerous is that they let a layer amplify small differences in its input into large differences in its output — that amplification is what makes it possible to fit noise. A bias does not amplify anything. It translates a unit's pre-activation by a constant, shifting where a nonlinearity's threshold sits relative to the data. Shrinking every bias toward zero does not reduce how sharply the network can bend; it just forces every unit's operating point toward a particular location, which is a systematic constraint on where the model can sit rather than on how complex it can be. On small networks this can measurably hurt, because the bias is often exactly the degree of freedom the layer needs. The normalization scale is worse. The layer standardizes its input to roughly zero mean and unit variance, then computes `scale * normalized + shift`. That scale is the layer's output gain. Decaying it pulls the gain toward zero, which uniformly attenuates everything the layer passes on. This does not simplify the function in any useful sense — the next layer's weights simply grow to compensate, and now you have decay fighting to shrink one parameter while the gradient inflates another. Decaying the shift toward zero similarly biases the layer toward zero-mean output whether or not the data supports that. ## How the failure shows up If you drop every parameter into one decayed group and watch the per-group norms across epochs, the picture is diagnostic. The weight norms find their equilibrium as expected. The normalization scales, however, are being pulled down by decay and pushed up by the loss, which needs a working gain; the tug of war shows up as scales that sit systematically lower than in a correctly grouped run, with downstream weight norms drifting upward to compensate. Held-out accuracy is usually a little worse, and the effect gets larger the stronger the decay coefficient — which is exactly the regime where you were relying on decay to do real work. Be honest about the magnitude, though. On large networks with moderate decay, the practical difference between decaying everything and decaying only the weights is often small, and plenty of results have been obtained with sloppy grouping. The reason to get it right is that it costs nothing, it removes a confound from your decay sweep, and the error grows precisely as you turn decay up. ## The rule to state in an interview Decay the multiplicative parameters — dense weight matrices and convolution kernels. Exempt the additive and gain parameters — biases, normalization scales, normalization shifts. The justification is that decay is a magnitude budget for the parameters that control how strongly the network can amplify its input, and the exempt groups neither amplify anything nor contribute meaningfully to the norm, while one of them directly controls a layer's output gain and reacts badly to being shrunk.