When resuming training from a step-50,000 checkpoint, should the learning-rate warmup be replayed?

## The default answer A training schedule is a function of the step counter. If you checkpoint the counter along with the weights and the optimizer state, resuming is simply continuing that function: at step 50,000 the ramp finished long ago, and the resumed run picks up whatever rate the decay schedule specifies for step 50,000. No replay. The most valuable habit here is checkpointing the counter at all. A surprising number of resume bugs come down to a checkpoint that holds weights and optimizer state but no notion of where in the schedule the run was, so the resumed process starts its schedule from zero. ## The bug that actually costs you Replaying just the ramp is a minor inefficiency: a few hundred steps at a reduced rate on a well-trained model, and it costs almost nothing. The expensive mistake is restarting the *entire* schedule. A run that was 80 percent through its budget, sitting at a small decayed rate, is suddenly ramped back up to the full target rate and then re-decayed over another full budget's worth of steps. Well-annealed weights get shaken back out; the loss climbs and takes a long time to come back, if it does. When someone says "the resume made it worse," this is nearly always what happened, and the fix is to persist and restore the step counter rather than to add compensating hacks. ## When a re-ramp is genuinely the right call Ask what the checkpoint contained and what changed across the boundary. **Only weights were restored.** Momentum and the per-coordinate second-moment estimates start from zero again. That returns you to the opening-steps regime: an average over a handful of samples, an unstable denominator in the adaptive update, and a first step whose size is set by the learning rate rather than by the gradient. Worse, the weights are no longer random - they encode a lot of training, and a mis-scaled step now destroys real progress rather than shuffling noise. A short ramp, on the order of a few hundred steps up to the schedule's current rate, is cheap and removes the risk. **Something changed across the resume.** A different batch size, a change in numerical precision, a rebuilt input pipeline that alters the data order or the distribution of sequence lengths, a changed number of workers contributing gradients - each moves the gradient statistics the optimizer's averages were built on. A brief re-ramp lets the averages catch up to the new regime. **Nothing changed and full state was restored.** Continue. There is nothing to warm up; the statistics are exactly as valid as they were one step before the crash. ## Reading the resume The diagnostic is the first few hundred steps after the resume, logged unsmoothed. A loss spike that appears immediately and decays back over a few hundred steps is the signature of missing optimizer state - the run is re-accumulating its averages the hard way. A loss that jumps and stays high, or climbs steadily, points instead at a schedule restarted from zero or at a resumed rate that does not match where the run left off. A loss that continues its previous trend with no discontinuity at all is what a correct resume looks like, and it is worth asserting on: log the rate and the step counter on the first step after every resume, and compare against the last step before the crash. ## The practice this implies Checkpoint the step counter, the optimizer state, and the weights as one unit, and treat a checkpoint missing any of the three as a weights-only checkpoint with the re-ramp policy that implies. Make the resumed rate an explicit, logged value rather than something inferred from a restarted process. And decide the re-ramp rule once - "re-ramp for a few hundred steps whenever optimizer state was not restored or the input configuration changed, otherwise continue" - so nobody has to reason it out at 2 a.m. after a node failure.