Why does a dropout layer placed just before batch norm hurt evaluation accuracy?

## The two moments behave differently Inverted dropout multiplies an activation `a` by a Bernoulli mask and divides survivors by the keep probability `keep`. The first moment is preserved: `E[out] = keep * (a / keep) = a`. The second is not: `E[out^2] = keep * (a / keep)^2 = a^2 / keep`. So the mask leaves the mean alone and inflates the second moment by a factor `1 / keep` — a factor of two at a drop rate of 0.5. At evaluation the mask disappears, so that inflation vanishes with it. Anything downstream that cares only about the average of its inputs is unaffected. A layer that measures and stores the *spread* of its inputs is not. ## What batch norm stores Batch norm standardises each channel of its input to roughly zero mean and unit variance, then applies a learned scale and shift. During training it uses the mean and variance computed over the current batch, and separately maintains running estimates of those statistics, updated as training proceeds. At evaluation it stops using batch statistics and uses the stored running estimates instead, so that a prediction depends only on the example in front of it and not on whatever else happens to be in the batch. That switch is the crux. The running estimates are a summary of *training-time* inputs. If training-time inputs and evaluation-time inputs have different distributions, the stored statistics are simply wrong for the inputs they are applied to. ## The mismatch Put dropout immediately before a batch-normalized layer and exactly that happens. Throughout training, the normalization sees activations whose variance carries the mask's `1 / keep` inflation, and its running variance converges to that inflated value. At evaluation, dropout is an identity and the incoming activations have their un-inflated variance. Dividing by a standard deviation that is too large shrinks the normalized activations toward zero relative to what the learned scale and shift were fitted for. Two things make the effect worse than it sounds. It compounds: a shrunken output feeds the next block, whose own stored statistics were also estimated under different conditions, and the discrepancy accumulates through depth. And it is invisible in the place you normally look — the training loss is computed with batch statistics, which are recomputed on the fly and therefore always correct for the inputs at hand. The damage appears only in the evaluation-mode forward pass. ## What you observe The signature is a model whose training-mode behaviour looks fine while its evaluation-mode accuracy is worse than the loss curves led you to expect, with the gap growing as you raise the drop rate or add depth. It does not look like overfitting — more regularisation makes it worse rather than better, which is the tell that separates it from a genuine generalisation gap. Freezing normalization statistics or reducing the rate visibly moves the number, which is a cheap confirmation. ## How to avoid it The simplest structural fix is placement: keep dropout out of the normalized part of the network entirely and apply it only after the last batch-normalized layer, which in a typical classifier means just before the final linear layer. That is also where dropout does the most good, since the classifier head is where parameters concentrate. Lowering the drop rate shrinks the inflation factor — at a rate of 0.1 the variance is inflated by about 1.11 rather than 2 — which reduces the mismatch without removing it, and is a reasonable compromise when you want some noise inside the stack. Choosing a normalization scheme that computes its statistics per example rather than accumulating them over training also sidesteps the problem, because there are no stored statistics to go stale; that is a different design decision with its own consequences. ## Why the ordering intuition matters The general lesson generalises beyond this one pairing: any training-only perturbation that changes the *distribution* rather than just the mean of a layer's inputs is dangerous in front of a component that estimates and freezes distributional statistics. Reason about it by asking what the downstream layer measured during training and whether that measurement still describes what it will see at evaluation. Dropout before normalization fails that test; dropout after the last normalization passes it.