What two running averages does Adam maintain, and how does its update rule combine them?

## The state Adam carries Plain gradient descent stores nothing between steps: it moves each weight by `lr * g`, where `g` is the current gradient of the loss with respect to that weight. Adam stores two extra numbers **per parameter**, updated every step from that same gradient: ``` m_t = b1 * m_(t-1) + (1 - b1) * g_t # first moment v_t = b2 * v_(t-1) + (1 - b2) * g_t * g_t # second moment ``` Both are exponential moving averages (EMAs). `b1` and `b2` are decay rates in [0, 1); the widely used values are `b1 = 0.9` and `b2 = 0.999`. An EMA with decay `b` has an effective memory of roughly `1 / (1 - b)` steps: about 10 steps for the first moment, about 1000 for the second. Both are initialised to zero, which is why the bias-correction terms `m_hat = m_t / (1 - b1^t)` and `v_hat = v_t / (1 - b2^t)` exist. The update itself is: ``` w <- w - lr * m_hat / (sqrt(v_hat) + eps) ``` ## What each moment means The **first moment** is a smoothed estimate of the gradient direction — momentum, written as an average rather than a sum. Averaging over roughly ten steps cancels a good part of the minibatch noise, so the direction the optimizer actually follows is steadier than any single batch's gradient. The **second moment** is the average of the *squared* gradient. Two precision points matter here. First, it is an **uncentered** second moment: it estimates `E[g^2]`, the mean square, not `Var[g]`. It equals the variance only in the special case where the mean gradient is zero. Calling it "the variance" is the single most common slip in interviews. Second, `sqrt(v)` therefore has the units and scale of a gradient magnitude — it is roughly the recent root-mean-square size of that parameter's gradient. ## Why the ratio is the whole idea Because the numerator has the scale of a gradient and the denominator has the scale of a gradient magnitude, the ratio `m_hat / sqrt(v_hat)` is close to dimensionless, and its size is bounded near one in the common cases. Consider two parameters: - **A consistent gradient.** Suppose one parameter's gradient is `+0.0001` on nearly every batch. Then `m_hat` settles at about `0.0001` and `sqrt(v_hat)` at about `0.0001`, so the ratio is about `+1` and the parameter moves by about the full learning rate — even though its gradient is a hundred times smaller than some other layer's. Plain gradient descent would have crawled here. - **An alternating gradient.** Suppose another parameter's gradient is `+0.5, -0.5, +0.5, ...`. The first moment averages toward zero, while the second moment averages `0.25` and its square root stays at `0.5`. The ratio collapses toward zero and the parameter barely moves. Adam damps itself exactly where the gradient signal is inconsistent. So the update behaves like a smoothed sign of the gradient scaled by the learning rate, modulated by how consistent that gradient has been. This is what people mean by "per-parameter learning rates": Adam does not tune anything, it just divides by a running measure of each parameter's own gradient scale. ## Consequences worth naming **Scale invariance.** Multiply the loss by 1000 and every gradient is multiplied by 1000; `m` scales by 1000, `sqrt(v)` scales by 1000, and the ratio is unchanged. The update is invariant to a rescaling of the objective (until `eps` becomes comparable to `sqrt(v_hat)`, which breaks the invariance at the small end). Plain gradient descent has no such property — rescaling the loss is equivalent to rescaling the learning rate. **Still one learning rate to tune.** The normalization removes the need to pick a different step size per layer; it does not remove the need to pick `lr`. "Adam has no learning rate" is wrong, and the global rate remains the highest-leverage hyperparameter. **Two extra tensors of optimizer state.** Every parameter now carries `m` and `v` alongside the weight and its gradient. **The decay rates are separate on purpose.** The first moment must react quickly to a genuine change of direction, so its memory is short. The second moment estimates a magnitude and wants a low-variance estimate, so its memory is long. Setting them equal is not a simplification — it changes both the smoothing behaviour and the way the bias corrections interact. A good one-line summary for an interviewer: Adam takes a momentum step, then divides it by the recent root-mean-square size of that parameter's gradient, so every parameter travels at a comparable pace regardless of how large its gradients happen to be.