Why does BatchNorm's backward pass make one sample's input gradient depend on the whole batch?

### The graph is not elementwise Per channel, BatchNorm computes over a batch of `N` values: ``` mu = (1/N) * sum_j x_j v = (1/N) * sum_j (x_j - mu)^2 xhat_i = (x_i - mu) / sqrt(v + eps) y_i = gamma * xhat_i + beta ``` Look at where `x_i` appears. It appears directly in its own numerator, and it appears inside `mu` and inside `v` — and `mu` and `v` feed *every* `xhat_j`. So a single input node has an edge to every output node in the batch. Backprop must therefore come back along all `N` of those edges, and the gradient at `x_i` inevitably contains sums over the whole batch. Nothing about this is an implementation choice; it is what the forward graph says. ### The gradient Let `g_i = dL/dy_i` and `ghat_i = gamma * g_i`. Differentiating through the three paths and collecting terms gives ``` dL/dx_i = (1 / (N * sqrt(v + eps))) * ( N*ghat_i - sum_j ghat_j - xhat_i * sum_j ghat_j * xhat_j ) ``` The parameter gradients are simple batch sums, because the learned scale and shift are broadcast across the batch: ``` dL/dgamma = sum_i g_i * xhat_i dL/dbeta = sum_i g_i ``` ### Reading the two correction terms Divide through by `N` and the expression reads as `ghat_i` minus its batch mean, minus its projection onto `xhat`, all scaled by `1/sqrt(v + eps)`. Those two subtractions are not arbitrary. BatchNorm's output is invariant to shifting its input by a constant across the batch, and invariant to scaling its input by a positive constant across the batch — both are absorbed by re-centring and re-scaling. A function that is exactly invariant along a direction has exactly zero derivative along that direction, so the input gradient is *forced* to be orthogonal to the constant-shift direction and to the `xhat` direction. The backward pass is, quite literally, projecting the upstream gradient out of the two directions the layer cannot see. One useful corollary: the `1/sqrt(v + eps)` factor means the gradient reaching the input is rescaled by the inverse of the activation's own spread. If a channel's activations blow up, the gradient flowing back through that channel is damped in proportion, which is a real part of why the layer stabilises training and is invisible if you model BatchNorm as an elementwise rescale. ### Consequence 1: there is no per-sample gradient A sample's gradient is not a function of that sample alone. That breaks any procedure that needs per-example gradients — per-example gradient clipping, influence-function analyses, and differentially private training all assume you can attribute a gradient to a single record, and with BatchNorm in the graph that quantity does not exist. Normalizations computed within one sample, such as LayerNorm or GroupNorm, restore it, which is precisely why those are the standard substitutes in that setting. ### Consequence 2: micro-batch accumulation is not a large batch Splitting a batch of 256 into eight micro-batches of 32 and summing the parameter gradients does **not** reproduce a single batch of 256. Each micro-batch normalizes with its own mean and variance, so the forward activations differ, and the coupled backward differs too. What you accumulate are parameter gradients computed under eight different normalizations. The loss curve will not match, and a hyperparameter set tuned at one micro-batch size does not transfer to another. This is one of the most common real surprises when a team enables gradient accumulation to fit a bigger effective batch. ### Consequence 3: batch size 1 is degenerate, not merely noisy With `N = 1` and no spatial axis to normalize over, `mu = x`, so `xhat = 0` and `y = beta` regardless of the input. Substituting `N = 1` into the gradient formula gives `ghat - ghat - 0 = 0`: the input gradient is **exactly** zero, and the layer blocks all gradient flow to everything below it. It is not that the statistics become noisy — the layer stops being a function of its input at all. A convolutional BatchNorm at batch size 1 is different: its statistics are taken over the spatial positions as well, so the effective count is `H*W` and the layer remains non-degenerate, though its statistics now come from a single image. ### Consequence 4: do not treat the statistics as constants A tempting shortcut is to compute `mu` and `v`, detach them, and backpropagate as if the layer were an elementwise affine map. That removes both projection terms, so you are optimizing with a gradient that is not the gradient of the network you are actually running. Training often still moves, which is what makes it insidious; it typically shows up as instability at higher learning rates and as a model whose evaluation behaviour diverges from its training behaviour in ways the usual explanations do not cover. ### Consequence 5: batch composition leaks Because each sample's gradient contains sums over its batch-mates, gradients depend on how batches were assembled. Sorted, grouped or class-clustered batches make the coupling systematic rather than random, and in setups where samples in a batch are related by construction, information flows between them through the shared statistics. The backward pass is where that coupling becomes a training-dynamics problem rather than a curiosity. ### What the backward needs to keep The formula needs `xhat` (or `x` together with `mu` and `v`) and `gamma`. That is retained activation memory proportional to the layer's output, which is why normalization layers are frequent targets for recomputation strategies in memory-tight training.