How do you set warmup length - a fixed step count or a fraction of the run - when porting a recipe to a far larger dataset?

## The question behind the question Every training recipe expresses warmup in some unit: a step count, a percentage of total steps, or a number of epochs. Nobody thinks about the unit until the recipe moves - to a bigger corpus, a shorter fine-tune, a different batch size - and then the choice of unit silently changes the actual ramp. The interviewer wants to see whether you reason from what warmup is for, or from what the config file happened to say. ## What the ramp is buying, in what currency Warmup exists to carry the run through a transient regime with two components. First, the iterate has to leave the neighbourhood of a random initialization, where the curvature can be sharper than anywhere the target rate was tuned for. Second, the optimizer's running averages - momentum, and for adaptive methods a per-coordinate second moment - have to accumulate enough samples to be stable. Both are measured in **optimizer steps**. Neither has any dependence on how many more batches exist afterward. That single observation settles the unit argument. Steps are the currency in which the ramp's job is denominated, so steps are the unit in which it should be written down. ## What goes wrong with the alternatives **A fraction of total steps.** Convenient, and roughly harmless within one order of magnitude of budget. Take the recipe to a dataset ten times larger, keep "two percent of the run", and the ramp goes from 500 steps to 5,000. That is not a disaster - the extra steps are still training steps, just at a lower rate - but it is 4,500 steps of deliberately reduced progress bought for nothing. Take the same recipe the other way, to a 2,000-step fine-tune, and two percent is 40 steps, which is inside the window where the second-moment estimate is still nonsense. The failure at the short end is real, and it is the one that bites, because short runs are exactly where people reuse a recipe unthinkingly. **A number of epochs.** Worse, because it couples the ramp to two independent quantities. Steps per epoch equal dataset size divided by batch size, so an epoch-based ramp changes when the corpus grows *and* when someone changes the batch size for throughput reasons. A team member doubling the batch to fill a machine has just halved the warmup without touching the warmup config, and the review diff shows nothing suspicious. **A fixed step count.** Ports on the axis that matters. It has one honest weakness: on a very short run, a fixed ramp can become a large share of the budget, so it needs a cap - something like "500 steps, or ten percent of the run, whichever is smaller" is a reasonable compromise that keeps the step-count intent while preventing an absurd result on a tiny fine-tune. ## Sizing the number, not just the unit A defensible starting point is the optimizer's averaging window. With a second-moment decay of 0.999 the effective window is on the order of a thousand steps, and ramps in the hundreds to low thousands are what practitioners converge on for that setting. Then verify on the run itself: log unsmoothed per-step loss and gradient norm over the first couple of thousand steps. If the trace breaks - spikes, plateaus, or diverges - at or just after the step where the ramp ends, the ramp was too short; that coincidence in timing is the signature, and the fix is a longer ramp rather than a lower target rate. If nothing interesting happens anywhere in the opening steps, you may be able to shorten it, though the saving is rarely worth the risk. The asymmetry drives the policy. An over-long ramp costs a small, bounded amount of progress. An under-long ramp costs a dead run, sometimes only on some seeds, discovered hours in. When the downside is that lopsided, err long. ## The organizational call This is where the principal-level answer lives. Pick one unit and make every recipe in the organization use it, because the cost of mixed units is not a bad ramp on one run - it is that nobody can tell by reading two configs whether they are training the same way. Write the ramp in steps, record the derived percentage in the run's logged metadata so comparisons across budgets are readable, and make the opening-steps trace a standard artifact of every launch so an under-warmed run is caught in the first two minutes rather than at the first evaluation. And keep the roles separate: the ramp is sized by the optimizer and the architecture, while the part of the schedule that stretches to fill a bigger budget is the decay that follows it. ## Also on the checklist The ramp shape is a minor knob compared with its length - a linear ramp and a gradual exponential ramp behave similarly, and a short constant-low-rate phase before the ramp is a variation, not a different technique. Spend the argument on length and unit; nobody has ever lost a run to the shape.