NaN loss appears at step 12,001 of a previously healthy run — how do you find the cause?
answer
- reproduce before you theorise
- save the batch and the weights
- inf and NaN mean different things
- walk forward to the first non-finite value
- the loss is the last to show it
basics
~20 sMake it reproducible first: checkpoint the batch, the weights and the random state from the last good step and replay them. Then bisect forward — inputs, per-layer activations, loss, gradients — to the first value that is not finite.
solid answer
~50 sFirst I make it reproducible rather than guess. I save the offending batch, the weights and optimizer state from step 12,000, and the random state, then replay that single step offline so I can iterate in seconds. Then I bisect forward: are the inputs and targets already non-finite, then the per-layer activation maximums, then the quantities entering the loss, then the loss, then the gradients walking back. The first non-finite tensor is the crime scene; everything after it is contamination, because any arithmetic touching a NaN returns NaN. The distinction between `inf` and `NaN` narrows it: `inf` points at overflow, a divide by zero or a log of zero, while `NaN` means `0/0`, `inf - inf` or `0 * inf`. Finally I confirm by intervention — remove that batch, or feed only that batch — instead of changing three things at once.
go deeper
Be ready to say what NaN and infinity are and that any arithmetic touching a NaN returns NaN. Knowing that a NaN loss means something upstream already broke, and that you should look at inputs first, is enough at this level.
An interviewer expects you to name which operations produce non-finite values — zero over zero, infinity minus infinity, log of zero, overflow — and to walk the forward pass stage by stage to the first one instead of guessing at hyperparameters.
Show the discipline of capturing the batch, the weights and the random state before forming a theory, and of confirming a hypothesis by a single intervention rather than by changing several things at once.
Own the argument for what a training platform should capture automatically at the first non-finite step, and for when a rare NaN deserves days of investigation versus a documented guard and a monitored counter.
## Two different failures wearing one name Floating point has two classes of non-finite value and they carry different information. `inf` appears when a finite result leaves the format's exponent range (overflow), when a non-zero number is divided by zero, or from the logarithm of zero, which is `-inf`. `NaN` appears when the operation has no defined value at all: `0/0`, `inf - inf`, `0 * inf`, the square root of a negative number, the logarithm of a negative number. NaN is contagious — every arithmetic operation with a NaN operand returns NaN — and it is the only value that compares unequal to itself, which is why `x != x` is the classic finiteness test. This matters for diagnosis because an `inf` usually appears one or more operations *before* the first NaN. A reported `inf` loss steers you toward magnitude: something overflowed, or something took a log of zero. A reported NaN loss means either two bad values met, or a `0/0` happened directly. ## Step 1 — make it reproducible before you theorise A run that was flat and healthy for 12,000 steps and blew up in a single step is almost always a function of what was in that step: that batch's contents, or the exact weights it met. So the first move is to freeze the crime scene. Persist the batch (or the sample ids plus the augmentation seed), the weights and optimizer state from the previous checkpoint, and the random state that produced the dropout masks and augmentations. Then replay that one step in isolation. If it reproduces, you have a loop that runs in seconds instead of hours and you can bisect at leisure. If it does *not* reproduce, that itself is a finding: the failure depends on something you did not capture — a non-deterministic reduction order, an unseeded augmentation, a differently ordered batch — and you should widen to statistics gathered over many steps instead of chasing one. ## Step 2 — bisect forward, not backward from the loss The loss is the last place a NaN shows up and the least informative. Walk the pipeline in execution order and record, per stage, whether every value is finite and what the maximum absolute value is: 1. Raw inputs and targets. A missing sensor reading encoded as a sentinel, a divide during preprocessing, or an already-NaN label is embarrassingly common and costs one minute to rule out. 2. Each layer's output. Track the max absolute activation per layer. A healthy network shows activations of roughly similar scale across depth; a broken one shows them multiplying by a large factor each layer until the forward pass itself overflows to `inf`, with NaN only appearing downstream when that `inf` meets a subtraction or a zero. 3. The quantities entering the loss. 4. The loss. 5. The gradients, walking back from the loss. The first non-finite value in that ordering is the one to explain. Everything after it is contamination. ## Step 3 — the symptom checklist it lands on In practice the first bad value belongs to one of a small family: - **A logarithm or a division applied to something that can be zero.** Hand-chaining a probability and then taking its log is the standard instance; keeping the loss on the unbounded pre-activation scale is the standard cure, and that is a topic of its own. - **A denominator computed from the data.** An average over a mask that is empty for this batch, a normalization by a variance that is exactly zero for a constant channel, an averaged neighbourhood for an element with no neighbours. All of these are `0/0`. - **Forward-pass overflow.** Activations that grow by an order of magnitude per layer reach the format's ceiling; in a reduced-precision format that ceiling is near 6.5e4 rather than around 3.4e38, so overflow arrives many layers earlier for the same model. - **An operation whose value is fine but whose derivative is not**, so the loss prints finite and the gradients are NaN — a square root evaluated at exactly zero is the canonical one. ## Step 4 — confirm by intervention A hypothesis is not a diagnosis. Once you have a candidate, test it: rerun the same step with that batch replaced and confirm it is clean; rerun with only that batch and confirm it fails; guard the one suspect operation and confirm the value becomes finite while the rest of the step is unchanged. Change one thing at a time — a run that stops failing after you simultaneously lowered the learning rate, added three epsilons and shuffled differently has taught you nothing. ## What not to conclude first "Exploding gradients" is a hypothesis, not a default. It is testable and it has a signature: a loss climbing over the preceding steps and a gradient norm growing step over step. A perfectly flat 12,000 steps followed by an instantaneous NaN is a much better fit for a poisoned input or a degenerate denominator in that particular batch. Similarly, resuming from the checkpoint and hoping is not a diagnosis: once a non-finite gradient has been applied, the optimizer's running buffers hold NaN and every subsequent step is NaN regardless of the data.
- The loss prints as inf rather than NaN — what does that change about where you look?`inf` has fewer sources than NaN: overflow of a finite result, a non-zero number divided by zero, or the log of zero. So it steers you toward magnitude and toward a hand-written log or division, not toward `0/0`. It also usually arrives one operation earlier than the first NaN, since NaN typically appears when that `inf` later meets a subtraction, a multiplication by zero, or another `inf`. Track per-layer maximum absolute activations first.
- The forward loss is finite but the gradients are NaN. What do you suspect?An operation whose value is defined at that point but whose derivative is not. A square root at exactly zero is the classic case: the derivative is one over twice the square root, which is infinite at zero. Division by a very small number does the same. The other common cause is a NaN sitting in a branch you thought you had neutralised by multiplying by zero — zero times NaN is still NaN, in both directions.
- You replay the saved batch against the saved weights and it does not reproduce. What now?That means something unrecorded varied: a non-deterministic reduction order, an unseeded augmentation, a dropout mask you did not capture, or a different device. Fix the capture first — seed and persist everything the step consumed. If it still refuses to reproduce, stop chasing the single step and instrument continuously instead: log per-layer activation maxima and gradient norms every step so the next occurrence arrives with evidence attached.
saying these in an interview costs you the question
- Blames the learning rate before reproducing anything
- Assumes every NaN means exploding gradients
- Skips the bad batch and calls the bug fixed
- Watches only the loss, never per-layer statistics
- Thinks resuming the same run clears a NaN already in optimizer state
- Adds epsilons everywhere instead of locating the first bad value