A teammate says Adam beat SGD with momentum after tuning only Adam's rate. What do you challenge?

## Why the claim is empty as stated An optimizer comparison is a controlled experiment with one intended variable: the update rule. The report described here varied two things — the update rule and how carefully each arm's hyperparameters were chosen — and attributes the outcome to the first. That is the single most common defect in optimizer folklore, and it is why the literature is full of contradictory claims about which optimizer wins. The asymmetry has a specific shape here. Adam's step is normalized, so its useful learning rates cluster in a small range and a default often works. Momentum's step is proportional to the raw gradient, so its useful rate depends on the architecture, the loss scale, the batch size and the normalization layers, and it is typically one to two orders of magnitude larger than Adam's. A momentum run handed a rate borrowed from an adaptive recipe will crawl. Concluding "Adam is better" from that is like timing two runners after tying one of their shoelaces. ## The protocol you should ask for **Equal tuning budget per arm.** Fix a number of trials — say the same count for each optimizer — and let each search its own rate over its own sensible range. The claim being tested is "best achievable Adam versus best achievable momentum," so both arms must actually be searched. (How the search itself is organized is a model-selection question in its own right; here the point is simply that the budgets match.) **Schedule parity.** Both arms need a decay schedule run to completion. A constant-rate momentum run is not the practitioner's configuration, and it loses to almost anything. Report the final numbers, not a mid-run snapshot, because the ordering commonly changes during the decay phase. **Regularization tuned per optimizer, not copied.** Weight decay strength is not comparable across step-size rules: the amount of shrinkage a given coefficient produces depends on how the optimizer scales its updates. Copying a value from one arm to the other silently changes the regularization strength and can produce the entire reported difference. **Everything else held fixed.** Same data pipeline and augmentation, same batch size, same initialization scheme, same number of epochs, same early-stopping rule, same validation split, same metric. If wall-clock rather than epochs is the budget, say so explicitly, and note that per-step cost is nearly identical for the two rules, so epochs and wall-clock rank the arms the same way here. **Seeds and spread.** Deep training runs vary run to run. Three to five seeds per arm, reported as mean and spread, is the minimum for believing a gap of about a point; a single seed each cannot distinguish a real effect from initialization luck. If the spread within an arm is as large as the gap between arms, there is no result to report. **One metric decided in advance.** Choosing after the fact between top-1, top-5, loss and a calibration number is a way to find a winner in any experiment. Name the deciding metric before the runs start. ## What a good answer sounds like Do not turn this into a defence of momentum. The right posture is that you do not yet know which optimizer wins on this task, because the experiment as run could not tell you. Offer the cheap version of the fix: rerun with a small matched rate sweep per optimizer, both with the same decay schedule, three seeds each, and compare the final validation means. That is usually a day of compute and it converts an opinion into a measurement. Also be prepared to say when the answer does not matter. If the gap after a fair comparison is smaller than the seed spread, the honest conclusion is that the optimizers are equivalent on this task, and the choice should then be made on operational grounds — tuning effort, robustness across the other models the team trains, and how much the recipe is likely to be re-tuned when the architecture changes.