A training job OOMs the day before a deadline — in what order do you apply the memory levers?

## The principle: order by what it costs, not by what it saves Every memory lever buys room with a different currency. Some cost engineering time, some cost throughput, and a few cost **fidelity** — they change the model you end up with, so the run is no longer comparable to the ones before it. Under deadline pressure the instinct is to reach for whichever lever frees the most bytes. The better instinct is to spend the cheapest currency first and stop as soon as the job fits. ## Step zero: measure the peak Before touching anything, find out how much room you actually need. Instrument the peak step, not the average one, and note the failing request's size against the memory the process already holds. A job that misses by 5% needs one lever; a job that misses by 3x needs a different plan entirely — and knowing which saves you from stacking four changes and never learning which one mattered. This step costs minutes and routinely saves the afternoon. ## The ladder **1. Reduce the micro-batch size.** Instant, no code change, and it attacks the term that dominates most budgets. The cost is that the effective batch has changed, so the learning-rate schedule and any batch-size-dependent behaviour are now off-recipe. Treat this as a diagnostic first: does the job fit at all at a smaller micro-batch? If not, no amount of the later levers is going to rescue it either. **2. Restore the effective batch with gradient accumulation.** This puts the recipe back where it was — same effective batch, same schedule — at the cost of wall-clock time that grows roughly with the number of micro-batches per update, plus a caveat around layers whose statistics are computed per micro-batch. Steps 1 and 2 together are the standard first move, and for a job that misses by a modest margin they are usually the whole answer. **3. Recompute activations instead of storing them.** A large saving on the activation term, paid for with extra forward computation during the backward pass. It is semantics-preserving and needs no change to the optimization recipe, which is why it sits above the precision and sharding levers, but it is a real code change and a real throughput hit, so it comes after the free ones. **4. Reduce numeric precision.** Halving the width of activations and intermediate values is a big saving and is often *faster* rather than slower, which makes it tempting to move earlier. It sits here because it is the first lever with genuine numerical risk: overflow and underflow ranges change, and a run that trains stably at full width can misbehave without the appropriate scaling and accumulation choices. If you have never validated the recipe at reduced precision, doing it for the first time the night before a deadline is how you turn a memory problem into a convergence problem. **5. Shard state across devices.** Splitting optimizer state, gradients and parameters across several devices is mathematically the same computation and can free an enormous amount, but it only exists as an option if you have more than one device, and it adds communication and configuration to a system you are about to run unattended overnight. Powerful, but not a deadline-eve lever unless it is already in place. **Last: the levers that change the result.** Shortening sequences, lowering input resolution, dropping the longest examples, or cutting model width all fit the job by changing the problem. They are legitimate engineering choices — but they belong at the bottom because after using one you can no longer compare the run to yesterday's, and the number you report is a number for a different task. ## Judgment the ladder does not encode The ordering above assumes a single deadline-driven job. Two things change it. If the peak is caused by an outlier batch rather than the model, bounding the batch by a token or pixel budget is cheaper than every lever on the list and should be done instead of them. And if the same team keeps hitting this every month, the right call is not a better ladder but a standing budget: a memory metric logged on every run, a measured peak recorded with each configuration, and a policy that a configuration is only "validated" once it has survived its own worst-case step. ## What a strong answer sounds like The candidate names the ordering, gives the currency each lever spends, and — crucially — says when to stop. Applying every lever because they all help is how a job that needed 10% more room ends up 40% slower, at reduced precision, on a recipe nobody has validated, producing a number that cannot be compared to anything.