Why can a global gradient norm in 16-bit come out infinite when every gradient is finite?

## Reductions are where formats break first Elementwise arithmetic keeps values in the same neighbourhood: multiply two order-1 numbers and you get an order-1 number. A **reduction** — a sum over many elements — does not. Its output magnitude grows with the number of elements, and a training run reduces over tensors with millions of them. So the format constraint that matters is not "can this format hold a gradient" but "can this format hold a sum of a hundred thousand of them". A global gradient norm is the canonical case: `sqrt(sum over all parameters of g_i^2)`. ## The overflow arithmetic A 16-bit float has 5 exponent bits, and its largest finite value is `65504`. Suppose 100,000 elements with gradients of magnitude around 1. Each square is around 1 and entirely unremarkable. The running sum passes `65504` after roughly 65,000 of them and becomes `inf`. The remaining additions leave it there, `sqrt` of it is `inf`, and anything computed from the norm inherits the infinity. The diagnostic trap is that this looks exactly like an exploding gradient. Every instinct says to hunt for the one huge value. There is no huge value: the maximum element is 1. What overflowed is the accumulator, and the two situations demand opposite responses — one is a modelling or stability problem, the other is a numerics bug in a reduction that would have been fine at any element magnitude if the element count were smaller. A useful reflex: for a sum of `n` squares to stay under `65504`, the mean square must be under `65504/n`. At `n = 100,000` that is a mean gradient magnitude of about `0.8`. That is not an exotic condition — it is an ordinary one. ## The mirror failure: a sum that stalls The same accumulator fails in the opposite direction without ever producing an infinity. Precision in floating point is relative, so the spacing between representable values grows with the running total. Once a 16-bit sum has reached about 2000, one ulp is about 2; any addend below about 1 is invisible. If you are summing a long tail of small squares — say `1e-4` each — after the total has grown large, every one of them rounds away and contributes exactly nothing. The result is a norm that is finite, plausible-looking, and systematically too small. That is arguably more dangerous than the infinity, because an infinity announces itself and a quiet underestimate does not: anything that consumes the norm as a threshold or a scale then behaves as though the gradients were smaller than they are. Ordering matters here in a way it does not for overflow. Summing smallest-first lets small values combine with each other before meeting the large partial sum, which recovers some of the lost mass — but it is a mitigation, not a fix, and it does nothing about the ceiling. ## Machine epsilon as a budget A compact way to reason about drift: each rounded addition costs at most one unit roundoff of relative error — `2^-11` ≈ `4.9e-4` for 16-bit, `2^-24` ≈ `6e-8` for 32-bit. Chain `n` sequential additions and worst-case relative error grows on the order of `n` times that (random-walk cancellation usually makes it closer to `sqrt(n)` times in practice). A 16-bit running statistic accumulated over a few thousand sequential steps — a moving average over a long time series, for instance — has therefore burned its entire relative-precision budget, while the same chain in 32-bit is still five orders of magnitude away from mattering. It is the same budget question as the norm, spread over time instead of over elements. ## Fixes **Accumulate wider than you store.** The elements can live in a narrow format; the running total should not. This is a standard hardware design point rather than an exotic trick: matrix units multiply narrow inputs and accumulate the products in a wider format precisely because the reduction, not the multiply, is what loses information. **Rescale before squaring.** Compute `M = max |g_i|`, then `M * sqrt(sum (g_i / M)^2)`. Every squared term is now at most 1, the sum is at most `n`, and the ceiling is only reached if the element count itself is enormous. This is the classical scaled-hypotenuse trick and it costs one extra pass over the data. **Tree or blocked reduction.** Summing in pairs or in per-block partials keeps the partial sums closer in magnitude to the elements, which reduces both overflow exposure and drift — and it is what parallel hardware does naturally anyway, since a parallel reduction is a tree. ## What to take away The question "is this value representable?" is the wrong one for a reduction. The right question is "is every intermediate value in this reduction representable, and does each addend survive being added to the partial sum it meets?" Overflow and drift are the two ways the answer is no, and both are fixed at the accumulator, not at the data.