Learning-Rate Warmup

## What warmup is Learning-rate warmup is a schedule applied to the very beginning of a training run. Instead of starting at the target learning rate you intend to train at, the run starts at a small value - often zero, sometimes a hundredth of the target - and increases to the target over a fixed number of optimizer steps. The linear form is what people mean by default: for step `t` and ramp length `T`, ``` lr_t = lr_target * t / T for t <= T lr_t = (whatever the decay schedule says) for t > T ``` Some recipes use a gradual exponential or cosine-shaped ramp instead; the shape matters far less than the existence of the ramp and its length. A gradual ramp is not a separate idea from a linear one - both are ways of spending the opening steps at a rate small enough to be safe. ## Why the opening steps are the dangerous ones Three things are true at initialization and stop being true a few hundred steps later. **The weights are random.** A gradient tells you the direction of steepest descent *at the current point*, and it is only trustworthy over a small neighbourhood. At initialization the network computes something meaningless, and the gradients are large in magnitude but say little about where the good region is. Taking a full-size step along such a direction is a large commitment to a low-information direction. **Local curvature can be much sharper than where the rate was tuned.** Stability of a gradient step is a relationship between the step size and the curvature it is stepping across. On a quadratic bowl whose largest curvature (largest Hessian eigenvalue) is `L`, plain gradient descent oscillates and diverges once the rate exceeds `2/L`. Nothing about a target rate chosen from a sweep guarantees that inequality holds at the *initial* point, and deep stacks in particular can start out in a sharp region. Warmup keeps the step small while the trajectory is in that region, and by the time the rate reaches its target the iterate has usually settled somewhere flatter. **The optimizer's running statistics do not exist yet.** Momentum and per-coordinate second-moment estimates are averages over past gradients. In the first handful of steps those averages are computed from a handful of samples, so they are noisy, and an update that divides by such an estimate can be far larger than the nominal rate suggests. ## What the failure actually looks like The observable failure is early and abrupt. Loss rises instead of falling, or jumps by orders of magnitude, and then becomes NaN or a flat constant that never moves again - all within the first tens of steps, long before you would look at a smoothed epoch-level curve. A second, subtler observable is a single spike: a loss spike at roughly step 15 that traces back to three unusually long, unusually high-gradient batches arriving before any useful gradient statistics have accumulated. The run may or may not recover from that spike; behind a ramp, those same batches produce a bump you barely see. The mechanism of the destruction matters. One oversized step can drive activations into a saturating region, blow up the scale of a normalization layer's inputs, or push weights so far that gradients afterwards are either zero or enormous. The optimizer's own state is also poisoned: a huge gradient enters the momentum and second-moment averages and keeps influencing steps for as long as those averages remember it. ## What warmup is not It is not a regularizer. Warmup does not improve generalization by some smoothing effect; it improves final numbers only by way of letting a run survive and by letting you use a larger target rate than you could otherwise. It is not a decay schedule either - the ramp is the front of the run and decay is the rest of it, and they are usually configured together but do very different jobs. It is also not "training on easier data first"; the data stream is unchanged, only the step size moves. It is close to free. A 500-step ramp inside a 200,000-step run costs a quarter of one percent of the budget, and the steps are not wasted - they are ordinary training steps at a smaller rate. ## Relationship to gradient clipping Clipping and warmup are complementary. Clipping bounds the *gradient norm*, which protects against a single pathological batch at any point in the run. Warmup bounds the *step size*, which protects against the whole opening regime, including steps whose gradients are perfectly ordinary but whose direction is uninformative and whose curvature is unforgiving. Many recipes use both, and a run that only needs one of them is not evidence that the other is useless. ## Signs you need it If a run diverges within the first hundred steps at a rate that is stable later; if lowering the target rate fixes divergence but costs final quality; if the architecture is deep, uses post-normalization ordering, or has an untrained embedding table feeding a deep stack - those are all situations where a ramp is the first thing to add, and the cheapest.