Why are speech and translation transformer stacks trained adaptively while vision convnet recipes still ship plain momentum?

## The framing to reject first The question is usually posed as "speech and translation use adaptive optimizers, vision uses momentum," as though the input modality decided it. That framing does not survive contact with the evidence. Image models built on transformer backbones are trained adaptively, and convnet-shaped models on non-image data are trained with momentum. What actually varies is the architecture's gradient conditioning and the maturity of its recipe. Saying this out loud is most of the answer at a senior or lead level. ## What makes attention stacks want adaptivity Three properties push these models toward per-parameter step sizes: **Heterogeneous parameter groups.** An attention stack mixes projection matrices, feed-forward blocks, normalization gains and biases, and an output layer over a large vocabulary. Their gradient magnitudes differ by orders of magnitude and do so unevenly over training. A single global rate must be small enough for the noisiest group, which leaves the rest under-trained for a long time. **Sparse embedding updates.** Rows of a token embedding table receive gradient only on the steps where that token appears. Rare tokens therefore accumulate very little movement under a rule that scales the step by gradient magnitude. Dividing by each parameter's own accumulated squared gradient gives those rows a step comparable to everyone else's when they do appear. **Early instability.** The first phase of training a deep attention stack is fragile; loss spikes and divergence are common failure modes, and practitioners handle them with a combination of warm-up and a normalized update rule. An optimizer whose step magnitude is bounded near the learning rate rather than proportional to the gradient is markedly more forgiving when a batch produces an outsized gradient. ## What makes convnet recipes stay with momentum **Normalization removes much of the scale mismatch.** In a convnet with normalization layers, scaling a layer's weights up or down largely does not change its output, because the normalization rescales the activations anyway. The per-layer scale differences that adaptivity exists to fix are therefore already handled structurally, so one global rate is a far less damaging simplification than it is in an unnormalized heterogeneous stack. **The binding constraint is generalization, not optimization.** Standard image-classification recipes train for a long budget with heavy augmentation. Reaching a low training loss is not the difficulty; the difficulty is the held-out number. That is precisely the regime in which tuned momentum's small final advantage shows up, so the extra tuning effort has something to buy. **Recipe maturity.** Those recipes have been tuned collectively for years: the rate, the momentum coefficient, the decay schedule and the augmentation are all co-adapted. A well-worn recipe with a known-good rate removes the main practical cost of momentum, which is having to find that rate yourself. ## The judgment call this is really testing At lead level the question behind the question is how you would set a default for a team. The tradeoff is explicit: - Adaptive optimizers are robust to being under-tuned. Across a portfolio of models, each of which will be re-architected repeatedly, that robustness is worth real engineer-months and it prevents the failure mode where a model quietly underperforms because nobody swept its rate. - Momentum can end higher on a well-understood architecture with a long budget and a stable recipe, at the cost of a rate search whenever anything changes. A defensible policy is therefore asymmetric: default new and exploratory work to an adaptive rule, so that nothing is bottlenecked on tuning, and reserve momentum for the small number of production models whose architecture is frozen, whose budget is long, and where a point of accuracy is worth a recurring tuning cost. Say which models fall in each bucket, and say what evidence would move a model from one to the other — that is the part interviewers are listening for. ## What not to claim Do not assert that adaptive methods generalize worse as a law. The measured gaps are small, architecture-dependent, and sensitive to how carefully each arm was tuned, and plenty of state-of-the-art image models are trained adaptively. The honest position is that the choice is an empirical, budget-dependent one, with a well-known tendency in one specific regime rather than a universal ranking.