A training run OOMs at step 8,000 with no code change — leak or peak, and how do you tell?

## Two failure shapes An out-of-memory error thousands of steps into an unchanged run is one of two things, and they need opposite fixes. 1. **A leak.** Something is retained across iterations, so the live set grows without bound. It was always going to fail; step 8,000 is just where the curve crossed the ceiling. 2. **A peak.** Memory is flat in aggregate, but one particular step demands far more than the typical step. The run was never comfortably inside the budget — you had not yet met the batch that proves it. ## The instrument that separates them Log, per step: allocated memory at the end of the step, the step's **peak** allocated, and the shape of the batch (largest sequence length, or resolution and batch size). Two or three hundred steps of this is enough. - **Straight climbing line, constant increment** → retention. The increment is informative: if it is close to one step's activation footprint, an entire computation graph is being kept per step. - **Flat line with occasional tall spikes** → peak. Correlate the spike heights with the logged batch shapes; they will track the largest sample in the batch. - **Flat line that fails exactly at an epoch boundary** → the evaluation pass, not the training loop. ## The classic retention bug The most common cause has nothing to do with the model. A training loop keeps a running loss or accuracy for logging, and accumulates the loss *tensor* rather than its numeric value. That tensor is still attached to the graph that produced it, and the graph holds references to every intermediate activation saved for the backward pass. Add one per step and after ten thousand steps you are holding ten thousand computation graphs. The signature is unmistakable in the plot: perfectly linear growth, identical increment, and no code change to blame because the bug was there from the first commit and only recently crossed the ceiling as batch size or sequence length crept up. The fix is to accumulate the detached scalar. Related variants: appending per-step predictions or logits to a list "for later analysis"; a cache keyed by batch that is never evicted; a history buffer of model snapshots. All produce the same straight line, and all are found by asking *what does this loop still reference at the end of a step?* ## The peak that hides Activation memory scales with how much data flows through the network, so on variable-length data the **longest sample in a batch** sets the step's peak. Consider a corpus of thirty-minute call transcriptions: almost every utterance is short, and one is enormous. Whichever batch contains it will demand several times the typical step's memory. If the pipeline groups similar lengths together, that one giant sample lands in a batch of its own kind and the peak is even sharper. Worse, that grouping can *hide* the problem: for several epochs the ordering happens to place the outlier where enough memory is free, then a reshuffle — or a different seed, or a resumed run — moves it and the job dies at a step number that means nothing. The tell is that the failing step number is not reproducible across runs, while the failing *batch content* is. Log the batch's maximum length at failure; if it is the corpus maximum, you have your answer. The fix is a budget rather than a count: cap the tokens or pixels per batch, or cap the length of an individual sample, so the worst-case step is bounded by construction instead of by luck. ## The evaluation pass The third suspect explains the most confusing variant: every training step succeeds, and the run dies at the end of the epoch. Two causes dominate. First, gradients are still being tracked during evaluation, so the forward pass saves activations it will never use — evaluation then costs *more* per sample than training did, on top of whatever training state is still resident. Second, evaluation is run at a larger batch because "there is no backward pass", which is true right up until the first cause is also present. Both are one-line fixes, and both are invisible until you look at where in the epoch the failure lands. ## The triage order Ask in this sequence: does memory grow monotonically? If yes, hunt retention — it is the only failure that gets worse with time. If no, does the failing step carry an outlier batch? If yes, bound the batch by a budget. If no, does the failure land at an epoch boundary? If yes, inspect the evaluation path. Only when all three are excluded is this a genuine capacity problem, and only then do you spend the day on the memory levers. One discipline underneath all of it: size the job against the **peak** step, never the average one. A run that survives its typical batch and has never met its largest is not a run that fits.