skip to content

Adam Moment Estimates

2 roadmaps4 questions

Adam holds exponential moving averages of the gradient and of its square, then corrects both for starting at zero. Interviewers probe the bias correction because it only bites in the first steps.

on this pageshow

explore

questions

4

Why does Adam divide its moment estimates by one minus the decay rate raised to the step count?

level: middleimportance: must knowfreq 64%

answer

  1. both averages start from zero
  2. an EMA needs time to fill up
  3. the denominator fills slower than the numerator
  4. divide by one minus decay to the t
  5. so early steps come out too large

basics

~20 s

Both moving averages start at zero, so early on they read too small — the squared-gradient average far more so. Dividing each by one minus its decay rate to the step count removes that startup bias.

solid answer

~50 s

Adam initialises both moment estimates to zero, so at step `t` the averages are pulled toward zero by factors of `1 - b1^t` and `1 - b2^t`. Dividing by those factors makes the estimates unbiased under a roughly stationary gradient. The subtle part is the *net* effect on the step: with the usual decay rates the second moment is biased far more severely than the first — at step one it is about a thousand times too small, so its square root is about thirty times too small — and since it sits in the denominator, the uncorrected step comes out about three times **too large**, not too small. That mis-scaling peaks near six or seven times the intended step after a dozen steps and is still about 25 percent oversized a thousand steps in. Both correction factors approach one, so it is a transient that disappears once the averages fill up.

code

python · 17 lines
python
import math

g, lr, b1, b2, eps = 0.3, 0.001, 0.9, 0.999, 1e-8
m = v = 0.0
for t in (1, 2, 3):
    m = b1 * m + (1 - b1) * g          # first moment: average of the gradient
    v = b2 * v + (1 - b2) * g * g      # second moment: average of the squared gradient
    raw = m / (math.sqrt(v) + eps)
    m_hat, v_hat = m / (1 - b1 ** t), v / (1 - b2 ** t)
    fixed = m_hat / (math.sqrt(v_hat) + eps)
    print(t, "v =", round(v, 8), "true g^2 =", g * g,
          "| raw step", round(lr * raw, 6),
          "| corrected step", round(lr * fixed, 6))

# 1 v = 9e-05      true g^2 = 0.09 | raw step 0.003162 | corrected step 0.001
# 2 v = 0.00017991 true g^2 = 0.09 | raw step 0.00425  | corrected step 0.001
# 3 v = 0.00026973 true g^2 = 0.09 | raw step 0.00495  | corrected step 0.001

go deeper

for a junior

Recall that both of Adam's averages start at zero and are therefore too small at first, and that dividing by one minus the decay rate raised to the step number cancels that startup effect.

for a middle

Derive the expectation of the moving average and show why the factor is one minus the decay to the step count. Be able to say which of the two estimates is more badly biased and why the square root in the denominator matters.

for a senior

Show you know the net effect on the step size and how long it lasts, and connect it to real incidents: short fine-tuning runs that live entirely inside the transient, or a resumed checkpoint whose step counter was not restored.

for a principal

Frame this as a case where an implementation detail changes effective step size by several times during the phase that decides whether a run is stable. Be ready to argue where such transients belong: fixed in the optimizer, or absorbed by how the run's opening is managed.

## Where the bias comes from Adam's two estimates are exponential moving averages started from zero: ``` m_t = b1 * m_(t-1) + (1 - b1) * g_t, m_0 = 0 v_t = b2 * v_(t-1) + (1 - b2) * g_t^2, v_0 = 0 ``` Unroll the first one for a gradient whose mean is `g`: ``` m_1 = 0.1 * g m_2 = 0.9 * 0.1 * g + 0.1 * g = 0.19 * g ... E[m_t] = (1 - b1^t) * g ``` The same argument gives `E[v_t] = (1 - b2^t) * E[g^2]`. The zero initialisation is not a neutral choice; it is a phantom observation of zero that the average has to dilute away, and the dilution factor is exactly `1 - b^t`. Adam therefore divides it out: ``` m_hat = m_t / (1 - b1^t) v_hat = v_t / (1 - b2^t) ``` Those are the quantities the update actually uses. ## The direction of the error, which is the part people get wrong It is tempting to say "both estimates start too small, so without correction the early steps are tiny." That is wrong for Adam, because the two biases do not cancel — one of them is in a denominator, and the second moment fills up far more slowly than the first. With `b1 = 0.9` and `b2 = 0.999` at step one: `m_1 = 0.1 * g` and `v_1 = 0.001 * g^2`. The second-moment estimate is a thousand times smaller than the true mean square, so its square root is about 31.6 times too small. The uncorrected ratio is therefore ``` 0.1 * g / (0.0316 * |g|) = 3.16 * sign(g) ``` — a first step roughly **three times larger** than intended, not smaller. In general the uncorrected step is off by the factor `(1 - b1^t) / sqrt(1 - b2^t)`. Evaluated with the usual rates that factor is 3.2 at step 1, rises to about 6.6 around step 12, is still 3.2 at step 100, 1.26 at step 1000, and only reaches one after several thousand steps. So the uncorrected optimizer would take steps several times too large through the entire early phase, precisely when the parameters are least able to absorb them. With the corrections applied, `m_hat = g` and `v_hat = g^2` at step one, so the first update is exactly `lr * sign(g)` — the intended scale from the very first step. ## Why the second moment is the slow one The fill-up time of an EMA is roughly `1 / (1 - b)` steps. For the first moment that is 10 steps; for the second, 1000. This asymmetry is deliberate — a magnitude estimate wants a long, low-variance window while a direction estimate wants to react fast — but it means the two biases live on completely different timescales, and the correction is the only thing that reconciles them. A useful thought experiment: if you set `b1 = b2 = b`, the uncorrected ratio is off by `(1 - b^t) / sqrt(1 - b^t) = sqrt(1 - b^t)`, which is **less** than one. Equal decay rates would make uncorrected steps too *small* rather than too large. The sign of the error is not a fixed property of "starting from zero"; it depends on which estimate fills up faster. ## Practical consequences **It is a transient, not a permanent term.** Both factors approach one, so after the second moment's window has filled, bias correction is doing nothing. Anyone who describes it as an ongoing part of the update rule has misread it. **It matters most in short runs and in fine-tuning.** A run of a few hundred steps spends its entire life inside the uncorrected mis-scaling. A run of a hundred thousand steps would eventually recover on its own — but only after taking thousands of oversized steps first. **The step counter is part of the optimizer state.** If a run is resumed with the moment estimates loaded but the step counter reset to zero, the correction is applied to estimates that are already unbiased: `m` gets multiplied by 10 and `sqrt(v)` by about 31.6, so the effective step becomes roughly three times too *small* for the first few hundred steps, then recovers. Losing the step counter across a checkpoint is a real and quietly damaging bug. **It does not fix anything else.** Bias correction has nothing to do with exploding gradients, with the choice of learning rate, or with the shape of the loss surface. It corrects one specific artifact of initialising two averages at zero. If you can state the direction of the error and the two timescales involved, you have said everything an interviewer wants on this.

  • If both decay rates were equal, would bias correction still be needed?
    Yes, and the error would flip direction. With a common decay `b`, the uncorrected ratio is off by `(1 - b^t) / sqrt(1 - b^t) = sqrt(1 - b^t)`, which is below one — steps would be too small early rather than too large. The corrections do not cancel because one estimate enters under a square root.
  • How long does the uncorrected mis-scaling actually last with a second-moment decay of 0.999?
    Far longer than people expect. The factor `1 - 0.999^t` only reaches about 0.63 at 1000 steps, so the uncorrected step is still roughly 25 percent oversized there, and about 7 percent oversized at 2000 steps. It is not a handful of steps; it is the first few thousand.
  • What breaks if a run is resumed with the moment estimates restored but the step counter reset?
    The correction is then applied to estimates that were already unbiased, inflating the first moment by ten and the root second moment by about thirty, so the effective step is roughly three times too small for hundreds of steps before recovering. Treat the step counter as part of the optimizer state, not as a logging convenience.
  • Does bias correction change the optimizer's behaviour late in training?
    No. Both factors converge to one, so the corrected and uncorrected updates become identical once the averages have filled. It is purely a startup fix, which is why its impact is concentrated in short runs, fine-tuning runs, and the opening of any long run.

saying these in an interview costs you the question

  • Says the uncorrected early steps would be too small
  • Thinks bias correction is applied for the whole run
  • Claims it prevents exploding gradients or divergence
  • Says the correction affects only the first moment
  • Believes the correction can be dropped with no effect on short runs

context

resolving links…

open as a page

What two running averages does Adam maintain, and how does its update rule combine them?

level: middleimportance: must knowfreq 82%

basics

~20 s

Adam keeps two exponential moving averages per parameter: one of the gradient, one of the squared gradient. The update is the first divided by the square root of the second, so each parameter gets its own step size.

open as a page

Why would you raise the epsilon in Adam's denominator from a tiny default to a much larger value?

level: seniorimportance: should knowfreq 41%

basics

~20 s

Epsilon puts a floor under Adam's denominator, capping how large a step a tiny second moment can produce. Raise it when gradients have become genuinely small and the optimizer is amplifying noise into full-size, jittery updates.

open as a page

How should Adam's second-moment decay rate be set when rare batches carry enormous gradients?

level: principalimportance: nice to knowfreq 26%

basics

~10 s

The second-moment decay sets how long one batch influences the denominator: 0.999 remembers about a thousand steps, 0.98 about fifty. A lower rate spikes harder on an outlier but forgets it far sooner.

open as a page