How does Nesterov's look-ahead gradient differ from heavy-ball momentum?

## Same state, different evaluation point Write heavy-ball momentum first, in the undampened convention: ``` g = grad(w) v <- beta * v + g w <- w - lr * v ``` The step has two parts: a momentum part `-lr * beta * v` inherited from history, and a correction `-lr * g` from the fresh gradient. Crucially the correction is measured at `w`, before the momentum part is applied. Nesterov's accelerated gradient changes exactly one thing — where the gradient is taken: ``` w_ahead = w - lr * beta * v # where momentum alone would land you g = grad(w_ahead) v <- beta * v + g w <- w - lr * v ``` So Nesterov looks at the slope at the destination of the inertial part of the step, not at the departure point. In implementations the same iteration is often rewritten so the stored parameters are the look-ahead ones, making the change a one-line reshuffle rather than a second gradient evaluation; either way the cost is the same one gradient per step. ## What the look-ahead buys: braking Put both methods on the same two-parameter surface, a narrow valley, and start them off-centre with a velocity already pointing down towards the floor. **Heavy-ball, step by step.** The velocity carries the iterate towards the bottom. At the step where it will cross the minimum, the gradient it uses was measured on the near side, so it still points downhill and *adds* to the velocity. The iterate lands past the bottom. Only on the next step does the gradient reverse and start pushing back — one iteration late. That lateness is precisely the oscillation you see around a minimum at high momentum. **Nesterov, same step.** Before committing, the method evaluates the gradient at the look-ahead point, which is already past the bottom. That gradient points back up the far wall, so the correction subtracts from the momentum part in the very same iteration. The iterate still overshoots, but by less, and the ringing decays faster. A compact way to say it: heavy-ball reacts to where it is, Nesterov reacts to where it is going. The difference is a first-order anticipation of the curvature change — it is not second-order information, and it does not estimate curvature; it just samples the slope one momentum-step downstream. ## Theory, stated carefully For smooth convex objectives, Nesterov's method attains an `O(1/k^2)` convergence rate after `k` steps, matching the lower bound for first-order methods on that class, whereas plain gradient descent attains `O(1/k)`. That is the sense in which it is called *accelerated*. Heavy-ball also achieves an accelerated rate on quadratic objectives with well-chosen constants, but there are smooth, strongly convex functions on which heavy-ball with fixed constants fails to converge, so its guarantee does not extend to the whole class. This asymmetry in guarantees is the honest reason to prefer Nesterov when you want a theoretical justification; it is not a claim that it always trains better. ## What it looks like in practice On stochastic mini-batch training the picture is muddier, for a reason worth stating: the look-ahead gradient is itself a noisy estimate from one mini-batch, so the anticipation is only as reliable as the batch. Empirically the Nesterov variant behaves like a slightly better-damped version of heavy-ball — similar or marginally better final loss, and noticeably steadier behaviour when the momentum coefficient is pushed high, which is where the overshoot it corrects is largest. At small coefficients the two are nearly indistinguishable, because the momentum part of the step is short and the look-ahead point sits almost on top of the current point. ## Interview traps The common wrong answer is that Nesterov uses two gradients per step, or that it uses a second-order or curvature term. Neither is true: one gradient per step, first-order only. The second wrong answer is that it changes the velocity recursion; the recursion is the same, only the evaluation point moves. The third is treating the look-ahead as a prediction of the *next* full step including its own correction — it is only the momentum part that is looked through, since the correction is exactly what is being computed. A good short summary to have ready: same memory, same cost, one gradient measured one inertial step downstream, and the payoff is earlier braking and less ringing at high momentum.