Optimizers and Learning-Rate Schedules

AdaGrad's sum only advances when a parameter actually receives a gradient, so rarely-seen rows keep large steps. RMSProp's average decays on every step, so a long-idle row's denominator collapses and its next update is far larger than intended.

Do not carry the adaptive rate across. Adam's steps are roughly the size of its learning rate whatever the gradient, so momentum using that rate barely moves. Pick a rate one to two orders larger.

Normally no - resume the schedule at step 50,000, ramp already finished. The exception is a resume that restored only weights: with the optimizer's running averages starting from zero again, a short re-ramp is cheap insurance.

The second-moment decay sets how long one batch influences the denominator: 0.999 remembers about a thousand steps, 0.98 about fifty. A lower rate spikes harder on an outlier but forgets it far sooner.

In the common formulation the decay step is multiplied by the current learning rate, so annealing the rate quietly anneals regularization too. Decide that deliberately: either hold the decay-to-step ratio fixed on purpose, or apply a rate-independent decay and accept the different endgame.

Up to a critical batch size, doubling the batch roughly halves the steps to a target loss, so parallelism becomes wall-clock savings. Past it, the step count flattens while compute per step keeps doubling.

You can match the magnitude of the noise but not its structure. Mini-batch noise is shaped by how examples disagree and shrinks as the model fits, while fixed isotropic Gaussian noise points everywhere equally and never fades, so the run keeps wandering and the loss floors out.

Prediction ensembling usually scores highest but multiplies inference cost and memory by five. Collapsing the same checkpoints into one averaged weight vector keeps single-model serving cost and captures part of the gain. Decide from the serving budget, then measure both.