How would you harden a multi-day training loop against an occasional non-finite loss?

## Why the ordering is the whole design A training step is: forward, loss, gradients, update. A finiteness guard placed after the update is worthless, and understanding why is the point of the question. Modern optimizers carry state across steps — a running average of the gradient, a running average of its square, or both. Those buffers are updated from the current gradient. If that gradient contains a NaN, the buffer contains a NaN from then on, and because any arithmetic touching NaN yields NaN, every subsequent update is NaN even on perfectly clean batches. The weights become NaN one step later, and the run produces nothing but NaN until someone notices. So the guard has to sit between computing the gradient and applying it: if the loss is not finite, or the gradient norm is not finite, the update is not applied and the optimizer state is left untouched. This is also why the naive recovery — "resume the run and hope" — never works, and why "restart from the last checkpoint" is only as good as that checkpoint. If checkpoints are written on a fixed cadence, one of them was very likely written after contamination. Validate finiteness of every weight and every optimizer buffer when writing a checkpoint and again when restoring from one; a checkpoint that fails the check should be refused loudly rather than loaded. ## Preserving evidence at the moment of failure The expensive part of a rare NaN is not the lost step; it is that the skip destroys the only reproduction you will ever get. A guard that just continues has converted a diagnosable bug into a permanent mystery. So the skip path should write down enough to replay the step offline: the batch itself, or the sample ids plus the augmentation seed if the batch is large; the step number; a reference to the weights in use (a checkpoint id, or the weights themselves for the first few occurrences); the random state; and the per-layer activation maxima and gradient norms captured at that step. With those, the failing step is reproducible in seconds later, and the bisection work becomes cheap. Capture the first few occurrences in full and then downgrade to a compact record so a persistent failure does not fill the storage. ## The policy layer Skipping is a mitigation, and mitigations rot when they are silent. Make the skip an observable event: - **A counter and a rate.** One skip per hundred thousand steps in a long run is arguably tolerable numerical noise. One per two hundred means something changed — a new data shard, a shifted preprocessing step, activations creeping upward as training progresses — and it is a defect, not weather. Alert on the rate, not on the total. - **A budget with a hard stop.** Define a threshold past which the job aborts rather than continuing to burn accelerator hours producing a model trained on an unknown subset of the data. Silently dropping a growing fraction of batches is also a silent change to the data distribution the model sees, which is a correctness problem and not only an availability one. - **A written statement that the guard is temporary.** The guard buys you the run; it does not close the bug. The ticket stays open with the persisted evidence attached. ## Checkpoint cadence as the other half The cost of a poisoned state is bounded by how much work you lose rolling back. On a multi-day run, checkpoint often enough that a bad rollback costs minutes, and keep more than one generation so you can step back past a contaminated one. Keeping only the newest checkpoint is the classic mistake: the newest is exactly the one most likely to contain the NaN. ## What this does not replace A guard is not a diagnosis and it is not a substitute for making the failure impossible. If the persisted evidence shows an empty mask or a degenerate denominator, the durable fix is upstream, in the loss or in data validation, and the guard should eventually stop firing. If it shows activations climbing steadily across layers or across steps, the answer lives in the model or in the training configuration, and that is a different investigation. The value of the hardening is that it turns a run-ending event into a logged, reproducible one — it should never be the reason nobody looks. ## Judgment calls to be ready to defend How much do you spend? A guard, a counter and an alert are cheap and belong in any long-running loop. Full per-step statistics logging is not free and belongs behind a flag that turns on after the first occurrence. Whether to halt a very expensive run on the first NaN or to skip and continue depends on how reproducible the run is and how close it is to finishing — and that is a decision worth stating explicitly in the runbook rather than leaving to whoever is on call at three in the morning.