Mixed-Precision Training

## What "mixed" actually means A training step is a long chain of operations: cast inputs, multiply by weights, apply nonlinearities, normalize, compute a loss, propagate gradients back through the same chain, then update the weights. **Mixed-precision training** runs most of that chain in a 16-bit floating-point format while deliberately leaving a small set of operations in the usual 32-bit single precision. The word *mixed* is the whole point: it is not "train in 16 bits", it is "train in 16 bits everywhere it is safe, and in 32 bits everywhere it is not". Concretely, a typical arrangement looks like this. The optimizer owns a **master copy of every weight in 32-bit**. At the start of a step, those weights are cast down to 16-bit. The forward pass runs on 16-bit weights and produces 16-bit activations, which are the values stashed for the backward pass. The backward pass produces 16-bit gradients. Those gradients are converted back to 32-bit and applied to the 32-bit master weights, and the cycle repeats. ## Where the speed comes from There are two separate wins, and candidates who name only one usually name the wrong one. **Bytes moved.** A 16-bit value is half the size of a 32-bit value. Every activation written out during the forward pass and read back during the backward pass costs half as much traffic, and so does every weight read. Anything whose cost is dominated by shuffling values rather than by arithmetic gets close to a 2x improvement essentially for free. **Matrix-unit throughput.** Modern training accelerators contain dedicated matrix-multiply units that consume reduced-precision inputs and produce a 32-bit accumulator. These units run the dense multiply-accumulates that dominate a deep network at a large multiple of the chip's plain 32-bit rate. The important detail is that the *inputs* are 16-bit but the *accumulation inside the unit is 32-bit*, so a long dot product does not lose precision the way a naive 16-bit summation would. That is why a big matrix multiply is the single operation that benefits most, and why a model built almost entirely from big matrix multiplies sees the biggest gain. There is a corollary: the win is not uniform. A step whose time is dominated by small, oddly shaped operations, or by work outside the accelerator entirely, will not show the speedup the marketing numbers suggest. Mixed precision speeds up the parts of the step it touches; it cannot speed up anything else. ## Where the memory win is, and where it is not The memory story is more subtle than "everything halves". The **activations** stored for the backward pass are usually the largest single consumer of memory in a deep network, and those genuinely do roughly halve, which is why mixed precision so often lets you raise the batch size. But the **parameter side does not halve**: you now hold a 32-bit master copy *and* a 16-bit working copy of every weight, and the optimizer's own state stays in 32-bit. On a parameter-heavy model with a small activation footprint, the net saving can be close to nothing, and in pathological cases the extra 16-bit copy makes it slightly worse. So the honest claim is: activation memory roughly halves; total memory usually falls, but by less than half, and occasionally not at all. ## Why the accuracy usually holds The reason mixed precision is not simply "a worse model, faster" is that the risky operations are excluded from the reduced format. Anything that adds a very small number to a very large one, or sums thousands of terms, or exponentiates something that could overflow, is kept in 32-bit. Everything else — the bulk multiply-accumulates and the activations flowing between layers — tolerates 16 bits, because those values sit comfortably inside the format's range and a little rounding noise per operation is not what determines whether the model converges. When it does go wrong, the failure is usually a *range* failure rather than a *precision* failure: a value becomes too small to represent and turns into zero, or too large and turns into infinity. That is exactly the problem that loss scaling exists to solve for the narrower-range 16-bit format, and the reason the wider-range 16-bit format needs no such trick. ## What it is not Mixed precision is a **training-time** technique. It changes how the arithmetic of a training step is executed; it does not change the model you end up with, which is still a 32-bit set of weights coming out of the master copy. Making the *finished* model smaller or cheaper to serve — pruning it, distilling it, or converting its weights to a low-bit integer format — is a different family of techniques, applied afterwards, with different tradeoffs and different failure modes. Conflating the two is the most common way this question is answered badly. ## Interview framing A strong answer names the two wins separately, notes that the 32-bit master copy is what keeps the update correct, and is honest that the memory saving is concentrated in activations rather than spread evenly. A weak answer says "we cast the model to 16 bits and it runs twice as fast", which is wrong on the mechanism, wrong on the memory, and silent on why accuracy survives.