What does a training loss that spikes and then becomes NaN say about gradients?

## What the shape tells you Three distinct failure shapes are worth separating. A loss that is NaN from the very first step is almost always a bug — a bad target, a division by zero, or a logarithm of zero in the loss expression. A loss that oscillates without diverging is usually a learning rate slightly too high. The one asked about here is the third: many stable steps, then a sudden spike by orders of magnitude, then NaN that never clears. That is the fingerprint of a gradient that grew until one update left the region where the local model of the loss holds. ## The mechanism, step by step 1. The backpropagated gradient at some layer becomes very large, because the product of per-layer factors along the path to the loss has a gain above one and depth compounds it. 2. The optimizer takes a step proportional to that gradient, so the weights move a huge distance in one update. 3. The next forward pass computes pre-activations from those enormous weights. Beyond the range of the floating-point format they overflow to infinity. 4. Infinity then meets an operation that is undefined on it — infinity minus infinity, zero times infinity, infinity divided by infinity — and the result is NaN. 5. NaN propagates backward into the gradients and is written into the weights. The feedback is what makes it sudden: larger weights make larger Jacobians, which make larger gradients, which make larger steps. A run can look completely healthy and then blow up in three or four steps. ## Why it never recovers NaN is an absorbing state. Any arithmetic involving NaN yields NaN, so once even a few weights hold NaN, every subsequent output is NaN, every loss is NaN, and every gradient is NaN. Waiting for the next batch does nothing; neither does lowering the learning rate afterwards, because there is no longer any finite signal to descend on. Recovery means restarting from a checkpoint saved *before* the spike — and restarting from a checkpoint written after it just reloads NaN weights. ## What to check before blaming exploding gradients - **Gradient norm history.** If the norm was already trending upward over the preceding steps, exploding gradients is the strong hypothesis. If it was flat and then a single batch produced a huge value, suspect the data. - **Reproducibility per batch.** Rerun the offending batch on the pre-spike weights. If that batch alone produces the blow-up, it is a data or label problem — an out-of-range target, an unnormalised feature, a mislabelled sample. - **Undefined operations in the loss.** A cross-entropy that takes the logarithm of a predicted probability which has underflowed to exactly zero yields negative infinity in one term; a division by a variance estimate that reached zero does the same. These give NaN without any gradient ever being large. - **Input sanity.** Infinities or missing values in the inputs produce NaN outputs directly, and the training curve will typically break at the step where that sample first appears, without a spike beforehand. ## What to do about it The direct levers are a smaller learning rate, which shortens every step, and bounding the size of the update so no single step can be catastrophic — gradient clipping is the standard containment for exactly this failure. Both of these treat the symptom: the deeper question is why the per-layer gain sits above one in the first place, which is a property of the weight scale and the layer Jacobians rather than of the optimizer. One practical habit matters more than any of this: keep checkpoints frequent enough that a blow-up costs minutes rather than a day, and log the gradient norm alongside the loss, because the norm usually starts rising well before the loss does anything visible.