Which parts of a mixed-precision training step must stay in single precision, and why?

## The single rule behind the whole list Every item on the "keep this in 32-bit" list is an instance of one problem: **a small quantity has to survive being combined with a much larger one**. Floating point stores a value as a signed significand scaled by an exponent, and an addition first aligns the two operands to a common exponent. If the operands differ enough in magnitude, the smaller one shifts entirely out of the significand's width and the addition returns the larger operand unchanged. A 16-bit format has far fewer significand bits than a 32-bit one, so the gap at which this happens is much narrower. Wherever a training step deliberately adds something small to something large, that gap is the failure. That single rule generates the whole list. ## Master weights and the optimizer step This is the item interviewers care most about. The optimizer keeps a **32-bit master copy** of every parameter. Each step casts it down to 16 bits for the forward and backward passes, but the update `w <- w - lr * g` is applied to the 32-bit master, and the next step's 16-bit weights are cast fresh from it. Why: late in training a weight might be of order 1 while a single update is of order 1e-7. In 32-bit that update lands. In 16-bit the two magnitudes are far enough apart that the add is a no-op, and the weight stops moving entirely — not noisily, but *exactly*. The model appears to plateau for no visible reason. Keeping the master copy costs memory but is non-negotiable, and the optimizer's own state (momentum buffers and second-moment estimates, for optimizers that keep them) is kept in 32-bit for the same reason: those accumulators are long-running sums that must not be truncated. A useful way to say it in an interview: **the forward and backward passes are where you spend precision; the update is where you must not**. ## Reductions A reduction is any operation that collapses many values into one by summing. The danger is the same alignment problem, applied repeatedly: once the running total has grown, each new small term contributes nothing, and the error compounds over the length of the sum. The reductions that matter in practice: - **The accumulator inside a matrix multiply.** The matrix units are built to take reduced-precision inputs and accumulate into a 32-bit register precisely so a long dot product does not decay. This one is handled by the hardware, not by you, but knowing it is what separates a real answer from a memorized list. - **Softmax and log-sum-exp.** The denominator is a sum of exponentials. Beyond precision, the exponential itself can leave the reduced format's range on a large logit, so this is a range hazard as well as a precision one. - **Normalization statistics.** A normalization layer computes a mean and a variance over some axis — both reductions — and then divides by the square root of the variance. A variance computed badly can come out near zero or negative from cancellation, and dividing by it amplifies the damage into every downstream activation. ## The loss The loss is a reduction over the batch, usually fed by a softmax or a log. It is a single scalar computed once per step, so keeping it in 32-bit costs essentially nothing and removes a whole class of range and cancellation problems at the point where the entire backward pass begins. A wrong loss corrupts every gradient behind it, so this is the cheapest safety in the entire arrangement. In a translation model trained in half precision, for example, the usual arrangement is exactly this: the bulk projections run reduced, while the softmax, the log-sum-exp reduction behind the loss, the cross-entropy itself and the normalization statistics all stay in 32-bit. The reduced-precision fraction of the step is still the overwhelming majority of the arithmetic, because those excluded operations are cheap. ## What is safe in 16 bits Stating the complement is what shows you understand the rule rather than a list. Safe: the dense multiplies (with 32-bit accumulation), the activations flowing between layers, elementwise nonlinearities, and the gradients as they propagate — all of these are values of comparable magnitude being multiplied or passed along, not small values being folded into large ones. This is the overwhelming majority of the arithmetic and the memory traffic, which is why the technique is worth doing at all despite the exclusions. ## The failure signature If the master copy is dropped and the optimizer steps on 16-bit weights directly, the run does not crash. It trains for a while and then flattens, with a loss curve that looks like a learning-rate problem. Raising the learning rate appears to help briefly — because larger updates clear the rounding threshold again — which sends people down the wrong debugging path. Recognizing that signature is the senior half of this question. ## Interview framing Name the rule first, then the list. Candidates who only recite "keep master weights in fp32" get partial credit; candidates who explain that the update is small relative to the weight, and generalize that to reductions, the loss and normalization statistics, get the question.