Why would you raise the epsilon in Adam's denominator from a tiny default to a much larger value?

## What epsilon actually does Adam's per-parameter step is ``` step = lr * m_hat / (sqrt(v_hat) + eps) ``` The stated reason for `eps` is to avoid dividing by zero when a parameter has seen no gradient. That reason is real but minor. The operational reason is that `eps` sets a **floor on the denominator**, and therefore a **ceiling on the step**: no matter how small the second moment becomes, the step cannot exceed `lr * |m_hat| / eps`. With `lr = 1e-3` and `eps = 1e-8` that ceiling is `1e5 * |m_hat|` — effectively no ceiling at all. With `eps = 1e-4` it is `10 * |m_hat|` — a real constraint. ## The regime where it matters Think of a parameter's root second moment `sqrt(v_hat)` as its recent typical gradient magnitude, and compare it with `eps`. - **`sqrt(v_hat) >> eps`.** The epsilon is irrelevant; the update is the normalized ratio, close to plus or minus one for a consistent gradient. This is the regime most parameters live in for most of training. - **`sqrt(v_hat) ~ eps`.** The epsilon starts to shrink the step, damping this parameter relative to the others. - **`sqrt(v_hat) << eps`.** The update degenerates to `lr * m_hat / eps`, which is simply a momentum step with an effective learning rate of `lr / eps`. Adaptivity is switched off for that parameter and it moves in proportion to its smoothed gradient again. So epsilon is not a numerical nicety; it is the knob that decides *at what gradient scale Adam stops normalizing*. ## The failure it fixes Consider a multivariate time-series forecaster in the late phase of training. The model has captured the systematic structure, and what is left in the gradient is mostly minibatch noise around zero — genuinely tiny in magnitude for many parameters. Adam's normalization does not know the difference between a small consistent signal and small noise: it divides by the same root-mean-square either way. Weights start jittering at close to a full learning rate driven by nothing, validation loss becomes ragged, and each evaluation lands somewhere slightly different. Raising `eps` from around `1e-8` to around `1e-4` puts a floor under the denominator that those noise-scale gradients cannot get under. Parameters whose gradients are still meaningfully large are essentially unaffected — for them `sqrt(v_hat)` still dominates the sum — while the noise-scale parameters quiet down. The run settles. ## Why this is not the same as lowering the learning rate This is the discriminating question, and it is worth having the answer ready. Lowering `lr` scales **every** parameter's step by the same factor, including the ones making real progress. Raising `eps` shrinks **only** the steps of parameters whose second moment has fallen near or below the new epsilon. It is a selective damping keyed on gradient scale, not a global slowdown. If the diagnosis is "the whole model is overshooting", lower the rate; if the diagnosis is "parameters whose gradients have gone to noise are still moving at full speed", raise epsilon. ## The costs and the caveats **Epsilon is not scale-free.** It is compared against a gradient magnitude, so a good value depends on the scale of your gradients: the loss reduction used, the output head, the batch size, the parametrisation. A value carried over from a different model is a guess, not a default. If you rescale the loss, the sensible epsilon rescales with it — this is exactly the point where Adam's otherwise clean invariance to loss scaling breaks. **You lose real adaptivity.** The whole reason to use Adam is that a parameter with a small but consistent gradient still travels. Raise epsilon far enough and you have re-created the problem you were avoiding: layers with genuinely small gradients crawl again. This is why the change is a targeted response to an observed pathology, not a default worth raising "just in case". **It is a single global knob for a per-parameter problem.** Every parameter shares the same epsilon, but only some of them are in the noise regime. Anything you gain by damping the noisy ones you also apply to any legitimately small-gradient parameter. **Placement varies between formulations.** Some write the denominator as `sqrt(v_hat) + eps` and others as `sqrt(v_hat + eps)`; the two put epsilon on different scales, so a value tuned under one formulation does not carry over to the other. When you inherit an epsilon from someone else's recipe, check which form it was tuned against. The short version for an interview: epsilon caps the largest step a vanishing second moment can produce, and you raise it when tiny gradients are being amplified into meaningless full-size updates — accepting, in exchange, that genuinely small gradients will now move more slowly too.