Why can a gradient that is nonzero in 32-bit floats round to exactly zero in 16-bit?

## The layout, and which field does what Every IEEE floating-point number is three fields: a **sign** bit, an **exponent** field, and a **mantissa** (also called the significand or fraction). The value is roughly `sign * 1.mantissa * 2^(exponent - bias)`. Two of those fields do two completely different jobs, and confusing them is the most common mistake on this topic: - The **exponent width decides range** — how large and how small a magnitude the format can reach at all. - The **mantissa width decides relative precision** — how finely spaced the representable values are within a given magnitude. Underflow to zero is purely a *range* failure. It has nothing to do with how many mantissa bits you have. ## The numbers A 16-bit float uses 1 sign bit, 5 exponent bits and 10 stored mantissa bits. Five exponent bits give a very short ladder of magnitudes: - Smallest positive **normal** value: `2^-14`, about `6.1e-5`. - Largest finite value: `65504`. A 32-bit float uses 1 sign bit, 8 exponent bits and 23 stored mantissa bits: - Smallest positive normal value: about `1.2e-38`. - Largest finite value: about `3.4e38`. That is the whole story of underflow. A per-parameter gradient of, say, `1e-8` is an unremarkable number in 32-bit — it sits about thirty orders of magnitude above the floor. In 16-bit it is below the floor. ## Subnormals: a soft landing, not a rescue Between the smallest normal value and zero, IEEE formats do not jump straight to zero. They switch to **subnormal** (denormal) encoding: the implicit leading `1.` of the significand becomes `0.`, the exponent is pinned at its minimum, and the number is represented as `0.mantissa * 2^-14`. This buys extra range downward, but it is paid for one mantissa bit at a time. By the time you reach the smallest subnormal — `2^-24`, about `5.96e-8` — only a single significant bit is left. Below half of that, round-to-nearest has nowhere to go but zero. So the descent looks like this: above `6.1e-5` you have full 11-bit relative precision; between `6.1e-5` and `6e-8` you have progressively fewer significant bits; below about `3e-8` you have zero. ## What it does to a training step The update rule is `w <- w - lr * g`. If `g` has flushed to zero for a parameter, the update term is exactly `0`, the weight is written back bit-for-bit unchanged, and the parameter is frozen for that step. If the gradients for that parameter are chronically in the underflow region, it is frozen for the whole run — it contributes whatever its initialised value contributes and never learns. Two properties make this nasty in practice: 1. **It is silent.** There is no exception, no infinity, no NaN. The IEEE underflow flag exists in hardware, but nothing in a training loop reads it by default. 2. **It is partial.** Only the parameters whose gradients happen to be tiny are affected. The loss keeps going down, driven by the parameters whose gradients survived, so the curve looks healthy while a slice of the model is dead. The usual tell is a histogram of gradient magnitudes with a large spike at exactly zero, or a fraction of parameters whose values are bit-identical between checkpoints. ## Why the direction matters Gradients are the exposed tensor, not activations. Activations in a well-normalised network sit around order 1 — comfortably mid-range for any format. Gradients are routinely several orders of magnitude smaller than the activations that produced them, and a small learning signal makes them smaller still. That asymmetry is why the *backward* values are the ones that hit the floor first while the forward pass looks fine. ## The two-sided view Range has a ceiling as well as a floor. The same 5 exponent bits that stop at `6e-8` below also stop at `65504` above; a value larger than that becomes `inf`, and once an infinity enters an arithmetic chain it propagates (and `inf - inf` or `0 * inf` produces NaN). Underflow and overflow are the same fact about exponent width seen from opposite ends. And note what underflow is *not*. A gradient rounding to zero is a range failure. A perfectly representable small update disappearing when it is added to a much larger weight is a *precision* failure — a different mechanism, with a different fix. Keeping those two apart is the whole reason to know the field layout in the first place.