Why freeze a backbone's batch normalization statistics when fine-tuning at 2 images per device?

## The statistics are estimates, and small batches estimate badly A batch normalization layer does not know the true mean and variance of a channel's activations. It estimates them from whatever samples are in the mini-batch. The sampling error of a mean estimated from m independent samples shrinks like `sigma^2 / m` in variance, so going from 256 samples to 2 multiplies the variance of that estimate by roughly 128. The variance estimate is worse still, since it is a second moment computed from the same tiny sample and is itself divided into every activation. For a convolutional layer the effective count is `m * H * W`, not m, because spatial positions are pooled in — which is why tiny batches hurt convolutional networks less catastrophically than the raw batch size suggests, and why the problem worsens deeper in the network where the feature map has shrunk to a few positions per side. In the deepest blocks of a detection backbone at two images per device, a channel's statistics may be estimated from a few dozen numbers. ## What the noise actually does Three separate harms compound. **Forward noise.** Each sample is divided by a scale that jitters from step to step for reasons unrelated to the sample. A modest amount of this is the regularizing effect that makes batch normalization useful; at two samples it is no longer a mild regularizer but a corruption of the signal. **Gradient coupling.** The normalization ties the samples in a batch together, so each sample's gradient depends on the others. With two samples, one is effectively normalized against the other, and the gradient carries a large component that says nothing about the loss surface. **Buffer drift.** The running statistics are a moving average of these noisy batch estimates. Averaging does reduce the noise over many steps, but only if the underlying quantity is stationary — and during fine-tuning the activations are moving because the weights are moving. The result is buffers that trail a moving target while being noisy about it, so evaluation-mode behaviour diverges from training-mode behaviour and your fine-tuning metric becomes unreadable. ## Why the inherited statistics are the better answer The pretrained backbone shipped with running statistics that were estimated on a large batch over a large dataset. If the fine-tuning data resembles the pretraining data — the usual case when you take a general-purpose backbone to a new detection task — those numbers are an estimate of the same underlying quantity, computed from vastly more samples. Replacing a good estimate with a bad one is a strictly worse trade, and that is exactly what leaving the statistics unfrozen does. Freezing means two things together, and candidates often name only one: 1. The layer uses the stored running mean and variance in **both** training and evaluation, so the two modes agree by construction. 2. The moving-average update is switched off, so the stored values do not drift. The learned scale and shift are a separate question. They are ordinary parameters and can keep training — the layer then behaves as a fixed standardization followed by a trainable affine map, which is well-conditioned and adds useful capacity. Freezing everything, statistics and parameters alike, turns the layer into a constant affine transform, which is what you want if the backbone is fully frozen and only a head is being trained. ## When freezing is the wrong call Freezing bets that the pretraining and fine-tuning activation distributions are close. If they are genuinely different — medical or satellite imagery against a natural-image backbone, a different sensor, a different preprocessing pipeline — the inherited statistics are simply wrong for the new data, and the network will run with badly scaled activations no matter how well the head trains. The repair is not to unfreeze and hope. It is to estimate the statistics properly and then freeze them: hold the weights fixed, run many forward passes over the fine-tuning data, accumulate the mean and variance across all of them, write those into the buffers, and only then train. A statistic accumulated over hundreds of forward passes is a large-sample estimate even when each pass sees two images. The other route is to stop relying on the batch axis for statistics at all and use a normalization scheme whose statistics come from within each sample — a different layer family with its own tradeoffs. ## Related failure: a batch with no diversity The same fragility shows up whenever a batch is not a random draw. If a data pipeline builds batches by grouping — sorting by class, by source file, by scene — then every sample in a batch may share a label. The per-channel batch mean then carries class information, and centering subtracts precisely the signal that distinguishes that class. Worse, samples in the batch can effectively see each other: training accuracy can look healthy because the batch composition itself is a hint, while evaluation, where statistics are fixed and no hint exists, collapses. The check is trivial — confirm the loader shuffles — and the symptom, a large train/eval gap that scales with how correlated the batches are, is worth recognizing on sight. ## The answer to give At two images per batch the statistics are noise, the inherited large-batch statistics are a strictly better estimate of the same thing, and freezing removes the train/eval discrepancy for free. If the domain has genuinely shifted, re-estimate the statistics over many forward passes with the weights held fixed, then freeze those instead.