In a convolutional network, what does a batch normalization layer compute its statistics over?

## The axis question Every normalization layer is defined by one thing: which axis its statistics are averaged over. Batch normalization averages over the **batch axis**. What varies is which other axes come along for the ride. For a fully-connected layer with activations shaped `[N, F]` (N samples, F features), the layer computes F means and F variances, each averaged over the N samples. Feature 3 is normalized using how feature 3 behaved across the batch, and never mixed with feature 4. For a convolutional layer with activations shaped `[N, C, H, W]`, the feature is the channel, and the spatial positions are treated as extra samples of that same feature. So the layer computes C means and C variances, each averaged over `N * H * W` numbers. Channel 3's mean is taken over every sample and every pixel of channel 3. That spatial pooling is a deliberate design choice, not an implementation shortcut. A convolutional filter applies the same kernel at every location, so its output at every location is a sample of the same feature. Normalizing each pixel with its own statistics would break that shared-weight logic and make the transform depend on position. ## The full computation Per channel c, over the mini-batch: ``` mu_c = mean over N, H, W of x[:, c, :, :] var_c = mean over N, H, W of (x[:, c, :, :] - mu_c)^2 xhat = (x[:, c, :, :] - mu_c) / sqrt(var_c + eps) y = gamma_c * xhat + beta_c ``` `eps` is a small constant inside the square root. It exists purely for numerical safety: a channel that happens to be nearly constant across the batch has a variance near zero, and dividing by the square root of zero would blow the activations up or produce a non-finite value. It is not a tuning knob in any interesting sense. ## What the layer stores For C channels, the layer keeps four vectors of length C: - `gamma` and `beta` — trainable, updated by the optimizer, `2C` parameters in total. - running mean and running variance — buffers, updated by a moving average of the batch statistics, `2C` more numbers that ship with the model but are never touched by the optimizer. So a normalization layer following a 256-channel convolution stores 1024 numbers, half of them trainable. This is tiny compared with the convolution itself, which is why the layer is essentially free in parameter count while being far from free in memory traffic. ## Why gamma and beta exist If the layer only standardized, it would force every channel to zero mean and unit variance forever, which is a real constraint on what the network can represent. Consider a channel feeding a saturating nonlinearity: pinning it to unit variance fixes how much of that nonlinearity's saturating region it can reach. The learned scale and shift give the parameters back. Crucially, the pair can recover the identity — if setting `gamma_c` to the batch standard deviation and `beta_c` to the batch mean is what minimizes the loss, the optimizer can find it. Normalization is therefore a reparameterization of the activation, not a restriction on the function class. A useful consequence: the bias of the layer immediately before the normalization is redundant. Any constant added to a channel is added to that channel's mean and subtracted right back out by the centering step, so the bias has no effect on the output — `beta` already plays that role. Dropping it costs nothing and removes a parameter whose gradient is structurally zero in effect. ## Where it sits in a block The conventional convolutional block is convolution, then normalization, then the nonlinearity. Placing normalization before the nonlinearity is what controls the input distribution the nonlinearity sees, which is the whole point; putting it after normalizes the already-rectified activations, whose distribution is one-sided, and is used far less often. The block also drops the convolution's bias for the reason above. ## Common confusions worth naming **'It normalizes each sample.'** It does not. Batch normalization's statistics are shared across the batch, and each sample's output depends on its neighbours. Normalization schemes that work within a single sample exist and are a different family with a different axis. **'It normalizes over the training set.'** Not during training. The mean and variance come from the current batch. Only the buffers, used at evaluation time, are a whole-training average, and even they are an exponentially weighted one. **'It has no learnable parameters.'** It has `2C` of them, and they matter: `gamma` in particular is the handle people inspect when a channel is effectively dead, and initializing the final normalization layer of a residual block with `gamma` at zero is a known trick for making very deep networks start close to the identity. ## The one-line answer One mean and one variance per channel, taken over the batch and all spatial positions; then a per-channel learned scale and shift on top.