In the one-cycle schedule, why is the momentum coefficient scheduled opposite to the learning rate?

## The shape The one-cycle policy defines a single cycle across the entire budget rather than a monotone descent. The learning rate rises from a modest value to a peak somewhere in the first portion of the run, then anneals all the way down — importantly, to a rate substantially *below* where the cycle started, not merely back to it. The final low-rate tail is what settles the model. Run alongside it is a second schedule on the momentum coefficient, moving in the opposite direction: it starts high, falls to a lower bound exactly where the rate peaks, and climbs back as the rate anneals. Typical bounds are around 0.95 at the ends and 0.85 in the middle. ## Why the inversion The two knobs are not independent; they multiply into one quantity. With heavy-ball momentum the optimizer keeps a running velocity `v <- m * v + g` and steps by `w <- w - lr * v`. If the gradient were a constant `g`, the velocity would converge to `g / (1 - m)`, so the parameters move by roughly `lr * g / (1 - m)` per step. The **effective step size** is therefore about `lr / (1 - m)`, and the coefficient enters through `1 - m`, which makes it far more sensitive than it looks: - `m = 0.85` gives an amplification of about 6.7 - `m = 0.9` gives 10 - `m = 0.95` gives 20 So moving from 0.95 to 0.85 cuts the effective step by a factor of three — comparable to a large change in the rate itself. Now put the two schedules together. If momentum stayed high while the rate rose to its peak, the two amplifications would stack, and the effective step at the top of the cycle would be several times larger than intended. Runs at that point become unstable or diverge outright. Dropping momentum at the peak is what makes a genuinely large peak rate survivable — the effective step stays roughly in range while the nominal rate does the work. There is a second, softer argument. Momentum averages gradients over a window of roughly `1 / (1 - m)` recent steps. When the rate is large the parameters move a long way per step, so gradients from many steps ago were computed at a very different point and are less relevant; a shorter averaging window is the appropriate response. When the rate is small the parameters barely move, consecutive gradients are near-duplicates that differ mostly by sampling noise, and a longer window is exactly what you want — it averages away noise and pushes steadily through flat regions. High momentum at both low-rate ends and low momentum at the high-rate middle follows directly. ## What the schedule is for The design targets short budgets — the case where the whole run is a handful of epochs and there is no time to explore leisurely. It gets there by spending a long stretch at a rate far higher than a plain decaying schedule would tolerate, which acts as strong regularisation, and then annealing hard to a very small rate so the model actually settles before the budget ends. That deep final anneal is not optional; a one-cycle run stopped before it has completed the tail is worse than the same compute spent on a monotone schedule. ## Practical cautions - **The peak must be chosen, not guessed.** The entire policy rests on running near the edge of stability, so an over-large peak destroys the run rather than merely slowing it. - **It is defined against the budget.** Like any cycle-shaped schedule, the shape is a function of the total number of steps. Change the budget and the whole shape must be re-fitted; there is no meaningful way to truncate it. - **The momentum schedule is not optional decoration.** Running the rate schedule with a fixed high coefficient reintroduces exactly the instability the inversion exists to prevent. - **It transfers imperfectly to optimizers whose per-parameter scaling already normalises the update.** The `lr / (1 - m)` reasoning is cleanest for plain heavy-ball momentum; adaptive methods rescale the update by their own running statistics, so the coefficient governing the first moment does not translate into effective step size quite so directly. The inversion is still commonly applied, but the argument for it is weaker and should be stated as an empirical choice rather than a derivation.