Your half-precision run needs a tuned loss scaler; why does bf16 let you delete it?

## Two different problems wearing one name A 16-bit float can fail a training run in two unrelated ways. **Range failure.** The value's magnitude falls outside what the exponent field can express: too small and it becomes zero, too large and it becomes infinity. This is what kills gradients in a narrow-exponent 16-bit format, and it is what loss scaling exists to work around. **Precision failure.** The value is inside the range but the significand is too short to hold the detail — most importantly when a small quantity is added to a large one and simply disappears. Loss scaling addresses **only the first**. Understanding that is the whole question, because bf16 changes the first and slightly worsens the second. ## What bf16 changes Both formats are 16 bits wide; they divide those bits differently. fp16 spends 5 bits on the exponent and 10 on the mantissa. bf16 spends **8 bits on the exponent — the same as 32-bit float — and only 7 on the mantissa**. The consequences follow directly: - The **range** of bf16 is essentially fp32's. Gradients that would have collapsed to zero in fp16 are represented fine. Values that would have overflowed to infinity do not. - The **precision per value** is lower than fp16's, because there are three fewer mantissa bits. Each individual number is stored more coarsely. So the scaler goes away. You delete the constant, the overflow inspection, the halve-and-double policy, and the skipped steps. That last one is worth naming explicitly: with no discarded steps, every optimizer update is applied, so an epoch's applied-update count no longer depends on numerics and the run becomes easier to reproduce step for step. ## What bf16 does not change This is where candidates overclaim. Deleting the scaler does not make the rest of the mixed-precision arrangement unnecessary. **Master weights still stay in 32-bit.** The update `w <- w - lr * g` adds a small number to a large one, which is a precision problem, and bf16's shorter mantissa makes it *worse*, not better — the gap at which an update vanishes into a weight is narrower than in fp16. Dropping the master copy because "bf16 is more robust" produces the classic silent plateau where parameters freeze exactly. **Long reductions still accumulate in 32-bit.** Summing thousands of terms with a 7-bit mantissa loses small contributions quickly. The matrix units accumulate in 32 bits regardless of the input format, and softmax denominators, log-sum-exp and normalization statistics are kept out of the reduced format for the same reason as before. **Per-value rounding noise is somewhat larger.** In practice, well-conditioned training absorbs this — the noise is small compared to the stochasticity of minibatch gradients — but it is not nothing. Anything that depends on fine resolution within a narrow dynamic range is where you would notice: a running statistic held in the reduced format, a value whose meaningful variation is in its last few bits, or a sum of many nearly-cancelling terms. ## The scenario this comes from The classic story is a large recommendation model that trained cleanly in full precision and fell apart when switched to fp16: some embeddings received gradients so small they underflowed to zero and their rows stopped moving, while a handful of rare, high-magnitude features occasionally overflowed. Tuning a fixed scale to satisfy both ends at once was impossible — anything large enough to save the tail pushed the head into overflow — and a dynamic scaler spent the run oscillating. Moving to bf16 dissolved the problem, because both ends of that very wide gradient distribution fit in the range. The cost was accepting coarser per-value resolution, which the model's already-noisy gradients absorbed without a visible difference. That is the shape of the tradeoff in general: **bf16 buys range, which is what training dynamics usually need, at the cost of resolution, which training usually has to spare.** ## How to answer the "which do I use" version Be careful not to turn this into an abstract format comparison. The training-side answer is operational: if your platform supports bf16 for the operations you care about, you get the same throughput win with a strictly simpler recipe — no scaler, no skipped steps, no calibration run, fewer moving parts to explain when something goes wrong. You keep fp16 with a scaler when bf16 is not available end to end, or when you have already validated an fp16 recipe and the resolution difference has been shown to matter for your model. A candidate who says "bf16 is just better" has skipped the reason. A candidate who says "bf16 keeps fp32's exponent range, so underflow stops being the failure mode and the scaler has nothing to do; the mantissa is shorter, so everything I kept in 32 bits for precision reasons stays in 32 bits" has answered it. ## Interview framing Expect the follow-up "so can you drop the 32-bit master weights too?" It is the trap. The answer is no, and the reason — the update is a precision problem and bf16 is *less* precise — is what demonstrates you have separated range from precision rather than memorized a preference.