In Adam, what changes when weight decay is applied directly to the weights instead of added to the gradient?

### The two places a decay term can enter Weight decay is a small pull of every weight toward zero on each optimizer step. There are two structurally different places to inject that pull, and once the optimizer rescales the gradient per parameter, the two stop being the same operation. **Form 1 — an L2 term added to the gradient.** Before the optimizer runs, you form the sum `g <- dL/dw + wd * w`. The optimizer never learns that part of `g` came from a penalty rather than from the data. It builds its running first- and second-moment estimates from this summed vector and then takes its usual step, `w <- w - lr * m / (sqrt(v) + eps)`. **Form 2 — decoupled decay.** The adaptive step is computed from the loss gradient alone, and a separate term is subtracted afterwards: `w <- w - lr * m / (sqrt(v) + eps) - lr * wd * w` That second line is AdamW. Written side by side on a whiteboard the diff is one term moving from inside the fraction to outside it, and that is the whole idea. ### Why the denominator changes the meaning of the coefficient The quantity `sqrt(v)` in the denominator is, roughly, the root-mean-square magnitude of that parameter's recent gradients. Dividing by it is what makes the optimizer adaptive: a parameter with tiny gradients still gets a step of usable size, a parameter with huge gradients gets its step damped. In form 1 the injected term `wd * w` rides through that same division. The shrinkage actually applied per step is approximately `lr * wd * w / (sqrt(v) + eps)` which is *inversely proportional to how big that parameter's gradients have been*. In form 2 the shrinkage is `lr * wd * w` for every parameter, so each weight is multiplied by roughly `(1 - lr * wd)` per step regardless of its gradient history. There is a second, subtler effect in form 1: the penalty contaminates the moment estimates themselves. The running averages now describe gradient-plus-penalty, so the direction of the adaptive step is altered, and the penalty's influence is smeared across many steps by the averaging rather than applied cleanly on the step where it was added. ### What this looks like in a real network Take one network with two very differently scaled layers — an early block whose gradients have RMS around 1e-1 and a late block whose gradients have RMS around 1e-4. Under form 1, one single coefficient produces shrinkage that differs by a factor of about a thousand between them: the high-gradient layer is barely regularized while the quiet layer is squeezed hard. Nothing in the configuration says this; it falls out of the denominator. Under form 2 the same coefficient means the same relative shrinkage everywhere, and the amount of regularization a layer receives is no longer an accident of its gradient scale. This is why practitioners describe the coupled form as regularizing exactly the wrong parameters: the weights doing the most work, which receive the largest gradients, are the ones the penalty touches least. ### Reading the coefficient The number labelled "weight decay" means two different things in the two forms. In the coupled form it is the coefficient of a term that the optimizer will divide by a per-parameter estimate you never see. In the decoupled form it is, directly, the fraction of itself each weight loses per step (times the learning rate). A value tuned under one form carries no guarantee under the other, and the mapping between them is not a single constant — it varies per parameter and drifts over training as gradient magnitudes change. ### Common confusions worth clearing up - *Thinking decoupled decay is still minimizing an explicit L2 objective.* It is not. Under form 2 there is no term in the loss whose gradient the optimizer follows; the shrinkage is a modification of the update rule itself. - *Thinking the difference is just a rescaling of the learning rate.* A rescale would affect all parameters by the same factor. The coupled denominator affects each parameter by a different factor. - *Thinking the epsilon in the denominator makes this negligible.* Epsilon is there to prevent division by zero; it is far smaller than typical values of `sqrt(v)` and does not flatten the per-parameter spread. ### The short version to say out loud Coupled means the penalty goes through the adaptive rescaling and every parameter gets a different real decay. Decoupled means the decay is applied straight to the weights after the adaptive step, so the coefficient means one thing across the whole network. That is the change AdamW makes.