After averaging weights, why must you re-estimate a network's normalization running statistics?

## The mismatch Some normalization layers cannot compute their statistics at inference the way they do at training, because at training they use statistics of the current batch and at inference you may be scoring one example. Those layers solve it by accumulating a running estimate of the activation mean and variance during training and using the frozen estimate afterwards. That estimate is a **buffer**, not a learned parameter: it is not updated by the optimizer and it is not part of the gradient computation, but it is absolutely part of the model. Weight averaging updates parameters only. The averaged vector `w_bar` is a point the run passed near but never actually occupied, so **no forward pass was ever executed at `w_bar`**, and no activation statistics were ever collected there. The buffers you inherit are whatever the last live weights left behind - or, if you averaged checkpoints, whatever the last checkpoint happened to carry. Why does that matter so much? Because a normalization layer's output is `gamma * (x - mu) / sqrt(var + eps) + beta`, and `mu` and `var` at inference are the stored estimates. If the averaged weights produce activations whose true mean and scale differ from the stored ones, every downstream layer receives inputs that are systematically shifted and mis-scaled. The error compounds through depth. In practice this shows up as an averaged model scoring near chance while the live weights score fine - a result that looks like the averaging destroyed the model, when in fact only the buffers are stale. ## The fix Run one extra pass over training data with the averaged weights loaded: 1. Put the network in the mode where normalization layers compute statistics from the current batch and update their running estimates. 2. Reset the running estimates first, so you are re-estimating rather than blending into the stale values. 3. Forward only. No loss, no gradients, no optimizer step - nothing may change the averaged parameters. 4. Iterate over training data with the same preprocessing you will use in production, in batches large enough that per-batch statistics are not wildly noisy. This is cheap: a few hundred batches is usually plenty, since you are estimating a mean and a variance per channel, not fitting anything. It is a fixed post-processing step of the averaging procedure, and it should live in the same script - not in a runbook that someone may skip. A few details that matter in practice: - **Use training data, not validation data.** Re-estimating on the evaluation split leaks the evaluation distribution into the model and invalidates the measurement. - **Be consistent about augmentation.** Statistics gathered under heavy augmentation describe a different activation distribution than statistics gathered on clean inputs. Whichever you choose, apply it consistently and check the result on validation. - **Batch size matters a little.** Very small batches give noisy per-batch statistics; the running average over many batches smooths this out, but it is one more reason to use a reasonable batch size for the pass. - **Not all normalization needs it.** Layers that compute their statistics per example over features at inference time carry no running buffers at all, so they are unaffected by averaging. A network built entirely from those needs no extra pass - which is exactly why some teams hit this problem on one architecture and never on another, and wrongly conclude that averaging is unreliable. ## The EMA variant With a step-by-step exponential average you have a choice: keep an exponential average of the buffers alongside the parameters, or recompute them at the end. Averaging the buffers is cheap and usually works, because the buffers move slowly and the averaged weights are close to the recent live weights. Recomputing is strictly more correct, because it measures the actual activation statistics of the model you are shipping rather than assuming that an average of statistics matches the statistics of an average - which is not true in general, since the map from weights to activation variance is nonlinear. With coarse-grained checkpoint averaging, where the averaged point can be a long way from any individual checkpoint, recomputation is not optional. ## How to catch it Make the acceptance test structural rather than a matter of memory. Evaluate the averaged model through exactly the serving code path before it can be promoted, and compare it against the live weights on the same split. A near-chance score with sane-looking weights is the signature of stale buffers; a small, plausible score change is the signature of an averaged model that simply did not help. Confirming it takes one run of the re-estimation pass.