Why does an exponential moving average of a network's weights often evaluate better than the live weights?

## What is being averaged Weight averaging averages the **parameters** produced at successive training steps - not gradients, and not predictions. You keep a second copy of the parameter vector, usually called the shadow or averaged copy, and after every optimizer step you fold the current weights into it. Two forms are standard. **Uniform (Polyak) averaging** takes the plain mean of the iterates over some window: `w_bar = (1/T) * sum_t w_t`. This is the classical form, and the classical result is a variance-reduction claim: for a suitably decaying step size the averaged iterate converges at a better asymptotic rate than the last iterate, because the random component of each step is independent enough to partly cancel. **Exponential moving average (EMA)** is the practical form: `shadow <- d * shadow + (1 - d) * w` after each step, with `d` close to 1. The weight given to a step `k` updates ago is proportional to `d^k`, so the effective averaging window is about `1 / (1 - d)` steps: 100 at `d = 0.99`, 1,000 at `d = 0.999`, 10,000 at `d = 0.9999`. Initialize the shadow to the current weights rather than to zero, otherwise the early shadow is dragged toward zero and needs a bias correction. **Stochastic weight averaging (SWA)** is the coarse-grained cousin: instead of every step, you collect one checkpoint at the end of each cycle of a cyclical or high constant learning rate over the last portion of a run - say eight checkpoints over the last quarter of a 200-class fine-grained bird-species run - and take their **equal-weight** mean. Fewer points, spread much further apart. ## Why the average is better than any single iterate A mini-batch gradient is a noisy estimate of the full-data gradient. With a fixed, non-tiny learning rate, the parameter sequence does not settle onto a point; it reaches a stationary distribution and wanders inside a region around the minimum whose radius grows with the learning rate and with the gradient noise. Each iterate is therefore `minimum + offset`, where the offset is a random draw. Averaging many iterates averages many offsets, and because they are only weakly correlated across a long enough window, the average of the offsets is much smaller than a typical single offset. The averaged parameter vector sits nearer the centre of the region the run was orbiting, and that point usually has lower loss on held-out data than the point the run happened to stop at. A second, purely practical effect: the averaged copy is far less jumpy across evaluations. Validation curves computed on the shadow copy are smooth, which makes model selection and early-stopping decisions much less noisy than validation on the live weights. ## Choosing the decay and the window The decay is the one real knob, and it is a bias-variance trade in time. Too small a decay (short window) and you barely reduce the noise. Too large a decay and the shadow is a stale mixture that still contains weights from a much earlier, much worse stage of training - on a short run an EMA at `d = 0.9999` may never catch up before training ends. Two rules of thumb: size the window so it is a meaningful fraction of the steps you plan to spend at the current learning rate, and start averaging only after the run has stopped making large systematic progress, typically in the last quarter of the budget. Averaging from step zero at a high decay is the classic way to ship a model that is worse than the baseline. The averaging and the schedule interact. If the learning rate is annealed all the way to near zero, the iterates stop moving and the average converges to the same point as the live weights - there is nothing left to cancel, so the gain shrinks toward nothing. This is why SWA deliberately keeps the rate high or cyclical: it wants the iterates spread out so the mean of them is a genuinely different point. ## What it is not Averaging weights is not ensembling. An ensemble keeps several models and averages their **outputs**, paying several forward passes at inference; weight averaging produces one parameter vector and costs exactly one forward pass. It is also not momentum: momentum averages **gradients** into a velocity and feeds that back into the update, changing the trajectory. The weight average is write-only - it never feeds back, so it cannot destabilize training and can always be evaluated alongside the live weights and discarded if it loses. Finally, one operational caveat that catches teams: if the network contains layers that carry running statistics estimated during training, the averaged weight vector has no matching statistics, and evaluating it without recomputing them can look catastrophic. Always evaluate the averaged copy through the same procedure you will ship.