How does cosine annealing differ from step decay when both run over a fixed epoch budget?

## Two shapes, same job Both schedules take a peak learning rate and lower it over training. They differ in whether the descent is a staircase or a curve, and in how tightly the shape is bound to the length of the run. ### Step decay Multiply the rate by a constant factor at chosen epochs, and hold it flat in between. The classic form is a tenfold cut at fixed fractions of the budget — in a 90-epoch run, at epochs 30, 60 and 90. Its knobs are the peak rate, the factor, and the cut epochs. The defining property is that the training curve is *readable*. Each cut produces a sharp drop followed by a flatter stretch, because shrinking the step shrinks the region the parameters wander in and the loss immediately settles closer to the bottom of the basin it is already in. That cliff is a measurement, not decoration. If the first cut produces a large drop and the second produces a small one, the run was still travelling at the first and had mostly settled by the second, which tells you the cuts came too early or too late. A schedule you can debug by eye is worth something. A related family is exponential decay, which multiplies by a factor slightly below one every epoch or every step. It is a step schedule with the staircase smoothed out and no cliffs to read; the practical difficulty is that a rate that decays multiplicatively never reaches zero, so the effective end-of-run rate depends jointly on the decay factor and the number of epochs, and re-tuning it after a budget change is fiddly. ### Cosine annealing The rate follows a half-cosine from the peak at step zero down to a floor, usually taken as zero or a small fraction of the peak, at the end of the budget. Written out, the multiplier is `0.5 * (1 + cos(pi * t / T))` where `t` is the step index and `T` the total number of steps. Its knobs are the peak rate and `T` — and `T` is not a free choice, it is the budget. The shape has a useful profile: it stays near the peak for a while, spends most of the middle descending, then flattens out near zero at the end. That gives a long exploratory high-rate phase and a long, gentle settling phase, without any decision about where the boundaries are. In practice it tends to match or beat a carefully tuned staircase, and it removes the two knobs people most often get wrong. ## The budget coupling, which is the real interview point A staircase is defined by absolute epochs; a cosine is defined by a fraction of the run. That makes cosine annealing fragile in exactly one way: it must know the finishing line in advance. Consider a cosine tuned to anneal to zero over 50 epochs, and a project that loses time and can now afford 35. Two options: - **Truncate** — run the same schedule and stop at epoch 35. At that point the rate is still around a third of its peak; the model is under-annealed, its parameters are still wandering in a wide region, and the reported loss is worse than the same compute could have produced. This is the option that looks free and is not. - **Re-fit** — set `T` to 35 epochs and rerun from the start. The curve is a compressed version of the original: the same peak, the same floor, everything squeezed. This is almost always the right call, and it is why a schedule should be written as a function of the budget rather than as a table of epochs. The same logic runs the other way. If the budget grows, a completed cosine cannot simply be extended — the rate is already at its floor, so extra epochs at that rate buy nearly nothing. Restarting the schedule is a separate design with its own name and its own consequences. ## Choosing between them Prefer **cosine** when the budget is fixed and known, when you want the fewest knobs, and when the run is a training run rather than an experiment about the schedule itself. Prefer **step decay** when you are reproducing or comparing against a published recipe that used one, when you want to attribute an improvement to a specific point in training, or when the run is long enough that you want a decision point — a place to look at the curve and decide whether to continue. What you should not do is compare two runs with different schedules and attribute the difference to something else, such as an optimizer or an architecture change. The schedule usually dominates. Any comparison has to hold it fixed, or vary it deliberately as the thing under test.