Why can Adam reach a lower training loss faster than SGD with momentum yet finish with worse validation accuracy?

## The two update rules, in one line each SGD with momentum keeps one running average of the gradient and takes a step proportional to it: the same scalar learning rate multiplies every parameter's gradient. A direction with a tiny gradient gets a tiny step; a direction with a large gradient gets a large one. The raw magnitude of the gradient is preserved in the step. Adam keeps two running averages per parameter: one of the gradient and one of the squared gradient. The step is the first average divided by the square root of the second. Because numerator and denominator both scale with the size of that parameter's gradients, the ratio is roughly order one, and the actual step size is roughly the learning rate regardless of whether the gradient was 1e-6 or 1e2. Adam therefore throws away most of the gradient's magnitude information and keeps mainly its direction and its consistency. ## Why that makes early progress fast A fresh deep network has wildly different gradient scales across its layers and parameter groups. Under one global learning rate, that rate must be small enough for the loudest parameters not to diverge, which leaves the quiet ones barely moving for many epochs. Adam removes that coupling: every parameter gets a step of comparable size in units of the learning rate, so the whole network starts making progress on step one. This is why an Adam run at a sensible default rate almost always beats an untuned momentum run on the first few epochs of training loss, and why it is so forgiving of a learning rate that was never really tuned. ## Why fast is not the same as good The crucial point, and the one candidates most often miss, is that this is not an optimization-speed story with a fixed endpoint. Both runs are minimizing a non-convex objective with many near-equivalent minimizers of the training loss. The update rule decides which one you drift into. Adam's rescaling systematically enlarges the steps taken in directions where gradients are small and rare, so the final weight vector differs in both direction and scale from what a momentum run produces. On several standard image-classification setups, published comparisons found that adaptive methods reached training loss as low as, or lower than, momentum while landing at a slightly worse held-out number. That is the signature of a generalization difference, not an optimization failure: equal or better training loss, worse validation. The mechanism behind that difference is still argued about in the literature, and you should say so rather than assert a settled cause. The defensible interview position is empirical: on some architectures and datasets the tuned momentum run ends higher, the gap is small (about a point, sometimes less), and it only shows up at the end of a full schedule. ## What actually shrinks the gap Most reported gaps are smaller than they look once the comparison is done properly: - **A decay schedule.** A momentum run held at a constant rate will lose. Momentum needs its rate decayed toward the end of the run; that final decay phase is where most of its validation advantage appears, which is precisely why an epoch-five comparison inverts the ranking. - **A real rate search for both.** Momentum's best rate is often one to two orders of magnitude larger than Adam's, so inheriting one optimizer's rate for the other guarantees a meaningless result. - **Long enough training.** The ordering can flip after the schedule's decay. A comparison stopped before both schedules complete measures the wrong thing. - **Regularization tuned per optimizer.** Weight decay strength interacts with the step-size rule, so a value copied across optimizers is a confound rather than a control. ## How to talk about it If someone tells you "Adam converged faster," the correct response is a question, not agreement: faster to what, measured on which split, against a momentum run tuned by whom? A convergence-speed claim about training loss and an accuracy claim about held-out data are two different claims, and only the second one decides what ships. State the phenomenon plainly, note that it is architecture- and budget-dependent, and refuse to convert a training curve into a model-quality verdict.