What does RMSNorm drop compared with LayerNorm, and what does that assume?

## Two normalizers side by side For one sample's activation vector `x` with `H` components, LayerNorm removes the vector's mean and divides by its standard deviation, then applies a learned gain and bias: ``` mu = (1/H) * sum_i x_i var = (1/H) * sum_i (x_i - mu)^2 y_i = g_i * (x_i - mu) / sqrt(var + eps) + b_i ``` RMSNorm keeps the divide and throws the centring away: ``` rms = sqrt( (1/H) * sum_i x_i^2 + eps ) y_i = g_i * x_i / rms ``` There is no `mu`, no subtraction, and usually no bias term -- just a per-feature gain. Both are computed inside a single sample, so both are equally indifferent to batch size and equally free of stored statistics. ## The invariance that is given up LayerNorm is invariant to two transformations of its input vector. Rescale the whole vector by a positive constant `a` and the normalized output is unchanged; add the same constant `c` to every feature and the output is again unchanged, because the shift moves `mu` by exactly `c` and cancels in `x_i - mu`. These are called re-scaling invariance and re-centring invariance. RMSNorm keeps only the first. Scale the vector and `rms` scales with it, so the output is unchanged. Add a constant to every feature and the output *does* change, because a shift moves the root mean square without cancelling anywhere. The bet RMSNorm makes is that re-scaling invariance is the property that made normalization work in the first place, and that re-centring was along for the ride. On many trained models that bet holds: quality is comparable and the layer is cheaper. ## What the saving actually is The saving is small per element but the layer is invoked constantly. Dropping the mean removes one reduction and one subtraction over the feature dimension in the forward pass, simplifies the corresponding backward terms, and removes one parameter vector (the bias) per normalization site. Because these layers are limited by how fast activations can be moved rather than by arithmetic, shaving passes over the vector is worth more than the operation count suggests. The honest framing in an interview is "a small constant-factor saving on a layer that runs at every site, plus fewer parameters" -- not a transformative speedup. ## When the missing mean bites There is one clean piece of arithmetic worth carrying into the room. The mean square decomposes: ``` (1/H) * sum_i x_i^2 = mu^2 + var ``` So `rms = sqrt(mu^2 + var + eps)`. When the vector's mean is small compared with its spread, `rms` is essentially the standard deviation and RMSNorm behaves almost identically to LayerNorm. But when a layer's pre-normalization activations carry a large shared offset -- every feature sitting at roughly the same large value, with the interesting variation riding on top -- then `mu^2` dominates and the divisor is set mostly by the offset. The informative variation gets divided by a number that has little to do with it, and the normalized vector arrives at the next layer squashed and still off-centre. That is the concrete answer to "when would you not use it": when you have reason to believe the representation at that point carries a large common-mode offset that the network never learns to remove. In practice architectures that adopt RMSNorm are built so this does not happen, and the learned gain can absorb a stable mismatch, but a per-sample offset that varies with the input cannot be absorbed by a shared parameter. ## Why the bias usually disappears too If you drop the mean subtraction it is natural to drop the additive bias as well, and most designs do. The reasoning: a bias immediately after the layer is redundant with the bias of the linear map that follows it, so it costs parameters and gradient traffic while adding nothing the next layer could not express. It is not a rule of nature -- a design can keep a bias -- but "gain only" is the common form, and being able to say why is a good signal. ## What to say if asked to choose If you are debugging training instability, the mean subtraction is not usually the culprit, so switching between these two is not a fix for a diverging run. Treat RMSNorm as an efficiency and simplicity decision made at design time on a large model where the layer is called at every site, and treat LayerNorm as the safer default when you have no evidence either way and the layer is not on your critical path. Whichever you pick, both give you the property that motivated leaving the batch axis behind: a statistic defined on one sample, identical in training and serving, well-defined at batch size 1, and immune to whatever the padded positions of a ragged batch happen to contain.