In gradient descent, what does the update rule x = x - eta * gradient actually do at each step?
answer
- one small correction per iteration
- the sign in front matters
- gradient sets direction and part of the length
- eta only scales, never steers
- steps shrink as the slope flattens
basics
~20 sEach iteration moves every parameter a short distance opposite its own partial derivative: new value = old value minus eta times the gradient. The learning rate eta scales how far you move; the loop repeats until the gradient is near zero.
solid answer
~50 sGradient descent is a loop. At the current point you evaluate the gradient of the objective — the vector of partial derivatives — then take one step: `theta_new = theta_old - eta * grad f(theta_old)`. The minus sign is what makes it descent: you move against the direction in which the objective increases. The gradient contributes both the per-coordinate direction (its signs) and part of the step length (its magnitude); eta, the learning rate, is a scalar that scales that length and nothing else. All coordinates must be updated simultaneously from the same old gradient, not one at a time using partially updated values. Steps shrink on their own as you approach a flat point, because the gradient magnitude shrinks there. You stop when the gradient norm, or the change in loss or parameters, drops below a tolerance, or when you run out of iteration budget.
go deeper
Be able to write the update from memory, state that the minus sign is what makes it descent, and hand-execute two iterations on a simple quadratic without a calculator.
Explain what fixes the step length, why the steps shrink near a minimum with a fixed eta, and why the whole gradient must be computed before any coordinate is overwritten.
Show you can debug the loop: a rising loss means a sign error, an exploding loss means the step is too long, and a stalled loss can mean either a tiny gradient or a badly chosen stopping rule.
Own the framing that a single shared scalar eta is a strong modelling assumption about how comparable the parameters are, and that it forces a tuning burden onto whoever operates the training loop.
## The setting You have a differentiable objective `f(theta)` — a loss you want as small as possible — and a parameter vector `theta` with one entry per unknown. The gradient `grad f(theta)` is the vector whose j-th entry is the partial derivative `df/dtheta_j` evaluated at the current `theta`. Gradient descent is the simplest possible use of that vector: repeat ``` theta <- theta - eta * grad f(theta) ``` until some stopping rule fires. Per coordinate this reads `theta_j <- theta_j - eta * df/dtheta_j`. ## What each piece contributes **The minus sign.** The gradient points the way the objective goes up; subtracting it moves you the other way. Getting this sign wrong turns the algorithm into gradient ascent and the loss climbs instead of falls — the single most common implementation bug, and the reason a loss curve that rises smoothly from step one is almost always a sign error, not a bad learning rate. **The gradient magnitude.** The step length is `eta * ||grad f||`. Where the surface is steep the gradient is large and the step is long; where it flattens out the gradient is small and the step is short. That is a useful automatic property: with a fixed eta the iterates naturally decelerate as they approach a stationary point, since the gradient tends to zero there. **The learning rate eta.** A single positive scalar. It scales the step and does nothing else — it cannot change which way you go, and it is shared by every coordinate, so the relative movement of the parameters is set entirely by their partial derivatives. A coordinate whose partial derivative is ten times another's moves ten times as far in the same iteration. Note also that eta is not scale-free: if you multiply the loss by 10, every gradient multiplies by 10, and the same eta now takes steps ten times as long. ## A worked one-dimensional example Take `f(x) = (x - 3)^2`, so `f'(x) = 2 * (x - 3)`, with the minimum at `x = 3`. Start at `x = 0` with `eta = 0.1`: - `f'(0) = -6`, so `x <- 0 - 0.1 * (-6) = 0.6` - `f'(0.6) = -4.8`, so `x <- 0.6 + 0.48 = 1.08` - `f'(1.08) = -3.84`, so `x <- 1.08 + 0.384 = 1.464` The distance to 3 goes 3, 2.4, 1.92, 1.536 — it shrinks by the same factor 0.8 each iteration, which is `1 - 2 * eta`. The iterate approaches 3 geometrically and never overshoots it, and the individual steps (0.6, 0.48, 0.384) get shorter on their own because the derivative is shrinking, not because eta changed. ## Simultaneous updates In more than one dimension there is a subtlety that trips people up in code: every coordinate must be updated using the gradient evaluated at the same starting point. Compute the whole gradient vector first, then apply the whole update. If you overwrite `theta_1`, recompute the derivative, then update `theta_2`, you are running a different algorithm (a coordinate-wise scheme), and its behaviour and guarantees are not the ones you reasoned about. ## What the method needs to work The objective must be differentiable at the points you visit, so that a gradient exists. It must be bounded below, otherwise there is nothing to converge to. And eta must be small enough for the curvature of the surface — too large a step overshoots and the iterates grow instead of shrinking. If the objective is convex, a converged point is a global minimum; otherwise it is only a point where the gradient vanishes. ## Stopping There is no exact arrival. Common criteria, usually combined: the gradient norm falls below a tolerance; the change in the objective between iterations falls below a tolerance; the change in the parameters falls below a tolerance; or a maximum iteration count is reached. Reporting which rule fired matters — hitting the iteration cap means you stopped, not that you converged. ## Why it is worth knowing cold Every first-order training procedure you will meet is this one line plus modifications. If you can state the update, explain the role of each symbol, and hand-execute two iterations on a quadratic, the rest of the optimization material has a foundation to sit on.
- What happens to the step length as the iterate approaches a flat minimum?It shrinks on its own. The step is eta times the gradient magnitude, and the gradient tends to zero at a stationary point, so with a fixed eta the moves get progressively shorter and the iterates decelerate. That is why a constant learning rate can still converge smoothly on a well-behaved convex surface, and also why progress can look stalled near the end even though the method is still working.
- Do all parameters share the same learning rate in plain gradient descent?Yes. A single scalar eta multiplies the whole gradient vector, so each coordinate moves by eta times its own partial derivative. Coordinates with larger partials move further in the same iteration. The consequence is that eta must be chosen for the most sensitive coordinate, which can leave the others crawling.
- How do you decide when to stop iterating?Use an explicit criterion rather than a fixed feel: gradient norm below a tolerance, relative change in the objective below a tolerance, relative change in the parameters below a tolerance, or an iteration cap. Combine at least one convergence test with a cap, and always report which one fired — hitting the cap is a timeout, not convergence.
saying these in an interview costs you the question
- Says the update adds the gradient instead of subtracting it
- Thinks the learning rate changes the direction of the step
- Believes each single step jumps straight to the minimum
- Updates coordinates one at a time from partly-updated values
- Confuses the learning rate with the gradient magnitude