Why does half-precision training multiply the loss by a large scale factor before backpropagation?

## The problem A 16-bit floating-point format with a narrow exponent field can only represent magnitudes down to a certain point; below it, values become zero. Gradients are not comfortably above that point. In a deep network, gradient magnitudes span many orders of magnitude across layers and parameters, and a substantial fraction of them — often the majority by count, concentrated in the early layers and in small-magnitude weights — fall below the threshold. In 32-bit they are tiny but real. In the narrow 16-bit format they are **exactly zero**, and the affected parameters receive no update at all. The failure is quiet. The run does not crash, the loss still goes down (the large gradients still work), and the model just ends up worse than the same recipe in full precision, for no visible reason. This is why loss scaling exists. ## The trick Backpropagation is linear in the loss. If you compute gradients of `S * L` instead of `L`, the chain rule multiplies every partial derivative in the entire graph by the same constant `S`: ``` dL/dw -> S * dL/dw for every parameter w ``` So one multiplication at the very top of the backward pass shifts the whole gradient distribution up by `S`, moving the underflowing tail into representable range. Nothing about the model, the loss, or the geometry changes — only which numbers survive the format. Then, before the optimizer does anything, each gradient is divided by `S`. The update that lands is the same one you would have computed in full precision, minus the gradients that were still too small even after scaling. ## The ordering rule that gets missed **Unscale before anything that looks at gradient magnitude.** Gradient clipping compares a norm against a threshold; if the gradients are still scaled by 1024, the norm is 1024 times too big and the clip fires on every step. Weight decay applied inside the update, adaptive optimizers' second-moment statistics, and any logging of gradient norms are all in the same boat. The canonical order is: backward on the scaled loss, check for overflow, unscale, clip, step. Getting this order wrong produces a run that trains but behaves as if the clipping threshold were absurdly tight, which is a genuinely confusing bug. ## Choosing the scale: static A **static** scale is a single constant chosen once — 1024 is a common pick — often by running a few hundred steps, histogramming the gradient magnitudes, and choosing a power of two that lifts the bulk of the distribution above the underflow threshold with headroom before the maximum. Powers of two are used because multiplying and dividing by them is exact, adding no rounding error of its own. Static scaling is free at runtime and has no surprises. Its weakness is that it is calibrated to the gradient distribution *at the moment you measured it*. Gradient magnitudes shift over a run — typically shrinking as the model converges, but also spiking at a learning-rate warmup boundary, a schedule change, or a new data stage. A scale that was correct at step 500 can be too small at step 50,000 (underflow returns silently) or too large after a spike (overflow, and now the run produces infinities). Both failures are invisible unless you are watching for them. ## Choosing the scale: dynamic A **dynamic** scaler adapts. The loop is: 1. Backward pass on the scaled loss. 2. Inspect the gradients for any infinity or not-a-number. 3. If found: **skip the weight update entirely** and halve the scale. 4. If clean: unscale, clip, step, and count the clean step. After some number of consecutive clean steps (a couple of thousand is typical), double the scale. This walks the scale up until it overflows, backs off, and repeats — keeping it just below the ceiling, which is exactly where you want it, because that is where the underflow tail is smallest. **Skipped steps are the designed cost, not a bug.** On a fresh run the scaler starts far too high on purpose, so the first steps overflow and are discarded while it calibrates down; a few percent of early steps being skipped is normal and the rate drops to near zero once it settles. A structure-prediction model trained this way might discard around 3% of its early steps and almost none later. What *is* a signal is a persistently high skip rate, or a scale that keeps ratcheting downward: that says the gradients themselves are misbehaving, and the scaler is reporting a training problem rather than causing one. ## Static versus dynamic in practice Dynamic is the default because it is robust to distribution shift and needs no calibration run. Static is defensible when you need every step to be bit-reproducible and identically shaped — skipped steps make step counts depend on numerics — or when the overflow check's cost matters, or when you already know the distribution because you have run this exact recipe before. The honest comparison is: static trades robustness for determinism and zero overhead; dynamic trades a few discarded steps and a per-step check for never having to think about the constant again. ## Interview framing The answer has three parts: *why* (underflow to zero, silently), *how* (linearity of the backward pass, one constant, unscale before the step), and *which constant* (static calibrated once versus dynamic adapting by halving and doubling, with skipped steps as the expected cost). A candidate who says "it stops gradients from vanishing" without distinguishing format underflow from the vanishing-gradient problem of deep architectures has not understood the question — these are different phenomena that happen to share a word.