Why do batch-statistics normalization layers not see the full effective batch under gradient accumulation?

## Where the two mechanisms live Gradient accumulation is a statement about **gradients**: run several backward passes, add their results, step once. Normalization over the batch dimension is a statement about the **forward pass**: for each feature, compute a mean and a variance across the samples currently being processed, and standardise the activations with them. Those are different phases of the step, and accumulation happens strictly after the forward pass has already committed. When you split an effective batch of 32 into 8 micro-batches of 4, each forward pass computes its statistics from 4 samples. There is no mechanism by which micro-batch 7's forward pass could use micro-batch 3's samples — micro-batch 3's activations were freed long before. So the honest statement is: **accumulation reproduces the gradient of a large batch, not the forward computation of one.** For a model with no batch-coupled operations these coincide exactly. For a model with batch-coupled operations they do not, and the gap is not a rounding error — it is a different function being differentiated. ## What the gap costs Two distinct effects, worth separating: 1. **Noisier standardisation during training.** A mean and variance estimated from a handful of samples are high-variance estimates. Every activation in the forward pass is divided by a noisy scale, which injects noise into both the forward values and the gradients — over and above the sampling noise the loss already has. With very small micro-batches this can dominate, and it is the mechanism behind small-batch normalization degradation generally. 2. **The statistics carried into inference are estimated from small samples.** Layers of this kind accumulate running estimates during training to use when no batch is available at inference time. Those estimates now come from micro-batch-sized samples. They still converge over many steps, but they are estimating the statistics of a 4-sample activation distribution, which is not identical to the 32-sample one, and any distribution shift within a micro-batch — for example if micro-batches are not shuffled and correlate with a source, a sensor or a class — is baked in. The lidar case makes it concrete. Suppose a point-cloud detector where a single scene barely fits, and the published recipe calls for sixteen scenes per update. Accumulating sixteen single-scene micro-batches gives you the published effective batch in the optimizer's eyes. But every normalization layer in the backbone is now standardising one scene's activations by that scene's own statistics — an entire training run where normalization is effectively per-scene. Reproduction failures traced to "we matched the batch size" usually come from exactly this. ## What generalises beyond normalization Anything the forward pass computes **across** samples, rather than per sample, has the same problem. The clearest case is a loss whose value for one sample depends on the other samples present — for example an objective that treats the other members of the batch as negatives to push away from. Its behaviour depends directly on how many samples are in the same forward pass, so accumulating micro-batches gives you an average of several small-negative-pool losses, not the large-negative-pool loss the recipe intended. Similarly, any per-batch normalisation of a quantity, or any pairing or ranking done within the batch, is defined by the micro-batch. Dropout and similar per-sample stochastic layers are **not** affected in this way: their randomness is drawn independently per sample, so splitting a batch changes nothing about the distribution of masks. The dividing line to state in an interview is per-sample versus across-sample operations. ## What to do about it Within the training-loop design, the options are: - **Use a normalization that does not read the batch dimension.** Schemes that normalise within each sample — over the feature dimension, over channel groups, or per channel of a single sample — give identical results for any micro-batch size, which makes accumulation exactly equivalent again. This is the cleanest fix and it is a modelling decision made up front. - **Make the micro-batch large enough** that its statistics are acceptable, accepting fewer accumulation steps or a smaller effective batch; often the micro-batch, not the effective batch, is the number that matters for these layers. - **Freeze the normalization layers** and use fixed statistics, common when fine-tuning a pretrained backbone at a micro-batch size far below what the original training used. - **Report it.** If you are reproducing published numbers, say that you matched the effective batch by accumulation and that the batch-coupled parts of the forward pass saw the micro-batch. It explains gaps that would otherwise be blamed on data or initialisation. ## The line to say out loud "Accumulation makes the optimizer see a big batch. It does not make the forward pass see one. Anything computed across the batch dimension still sees only the micro-batch." Then name the two things in the model that are computed across the batch dimension, and say which of them your model contains.