Optimizers and Learning-Rate Schedules

Momentum keeps a running average of past gradients, the velocity, and steps along it rather than along the raw gradient. Consistent directions build up and move faster; components that flip sign each step cancel, so the path stops zig-zagging.

AdaGrad divides the global learning rate by the square root of each parameter's own accumulated squared gradients. Consistently large gradients get small steps, quiet parameters keep large ones, and because the sum only grows, every step shrinks over time.

Both moving averages start at zero, so early on they read too small — the squared-gradient average far more so. Dividing each by one minus its decay rate to the step count removes that startup bias.

Adam keeps two exponential moving averages per parameter: one of the gradient, one of the squared gradient. The update is the first divided by the square root of the second, so each parameter gets its own step size.

Adam gives every parameter its own step size from its squared-gradient history, so training loss drops fast early. That rescaling also changes which solution you reach, and tuned momentum with a decay schedule often ends higher on validation.

Adding an L2 term to the gradient sends that term through the adaptive per-parameter denominator, so each weight ends up with a different effective shrinkage. Decoupled decay subtracts a fixed fraction of the weight itself, shrinking every weight at the same relative rate.

Step decay holds the rate flat and multiplies it by a fixed factor at chosen epochs, leaving visible cliffs in the loss. Cosine annealing slides smoothly from the peak to near zero across the whole budget, whose length fixes the shape.

Multiply the learning rate by the same factor you multiply the batch size by. A run at batch 256 with rate 0.1 moved to batch 8,192 scales both by 32, giving rate 3.2.

Warmup ramps the learning rate from near zero up to its target over the first few hundred to few thousand steps. It prevents the large, poorly informed updates a freshly initialized network takes at full rate, which can spike or destroy the run.

A uniformly sampled mini-batch gradient has the same expected value as the full-batch gradient, so it is unbiased. Only its noise differs: the standard deviation of that noise falls roughly as one over the square root of the batch size.

Late in training the weights bounce inside a noise ball around a minimum rather than sitting in it. An exponential moving average cancels much of that mini-batch noise, so the averaged copy lands nearer the basin's centre and generalizes better.

RMSProp keeps a decaying average of recent squared gradients instead of AdaGrad's ever-growing total. Old gradients fade out, so the denominator can fall again and the step size recovers rather than shrinking toward zero for the whole run.

The mini-batch gradient stops being an unbiased estimate. Each batch holds one class, so its expected gradient is that class's gradient, not the dataset's. That is bias, not noise, and no batch size fixes it.

Square-root scaling, multiplying the learning rate by the square root of the batch-size factor, is the usual starting point with adaptive per-coordinate optimizers, where full linear scaling overshoots. Linear scaling stays the default for momentum SGD.

Heavy-ball measures the gradient at the current parameters. Nesterov measures it at the point the momentum part of the step alone would reach, so an impending overshoot is braked one iteration earlier. Same cost, same velocity recursion.

Epsilon puts a floor under Adam's denominator, capping how large a step a tiny second moment can produce. Raise it when gradients have become genuinely small and the optimizer is amplifying noise into full-size, jittery updates.

Challenge the tuning asymmetry first: an optimizer given a rate search cannot be compared with one whose rate was inherited. Then demand schedule parity, per-optimizer regularization tuning, equal epoch budgets, and multiple seeds.

The coefficient does not carry over. With the L2 term inside the gradient, the adaptive denominator rescales the penalty before it lands, so the same number produces very different shrinkage once decay is decoupled — in practice the decoupled value usually has to be much larger.

A reduce-on-plateau rule fires whenever its monitored metric is noisy enough to sit flat past the patience window by chance. Because the rule only ever reduces, that cut is permanent even if the metric improves next epoch.

A restart raises the step back to the peak while keeping the current weights. Those large steps push the parameters out of the narrow low-loss region they had settled into, so the loss rises before the next anneal brings it down.

Two effects are being confused. A four-fold batch at the same epoch count means four times fewer updates, and an unscaled learning rate leaves each update the same size, so the run underfits. Scale the rate, then judge.

Adam divides by a running estimate of each coordinate's squared gradient, and in the opening steps that estimate comes from a handful of samples. Its noise, not its average, is the problem: some coordinates get steps far larger than intended.

On a plateau the averaged gradient is nearly zero, so a deterministic step is nearly zero and the run creeps. Individual examples still disagree, so a small batch keeps producing non-zero steps that explore directions the average cancels out, and one of them leads downhill.

The momentum coefficient and the learning rate are coupled. Going from 0.9 to 0.99 multiplies the sustained step by ten and stretches the gradient-averaging window to about a hundred steps, so an untouched learning rate is now far too large.

The two runs sit in unrelated regions of a non-convex loss surface, with hidden units learned in different orders and different arrangements. Coordinate-wise averaging mixes unrelated features, and the midpoint has high loss. Averaging only works along one trajectory.

The averaged weight vector never produced any activations during training, so the running mean and variance stored by normalization layers belong to different weights. One extra forward-only pass over training data recomputes them; skipping it can crater accuracy.

The split tracks architecture, not modality. Attention stacks mix parameter groups with wildly different gradient scales and rarely updated embeddings, which one global rate handles badly. Normalized convnets are better conditioned, so tuned momentum wins the last point.

Warmup's job is finished after a certain number of optimizer steps, not after a share of the data, so a fixed step count is the defensible unit. A percentage silently stretches tenfold on a tenfold dataset.

Momentum multiplies the effective step, which scales as the rate over one minus the coefficient. Lowering the coefficient at the peak rate keeps that effective step stable, and raising it again as the rate anneals keeps the low-rate tail moving.