In Newton's method for minimization, what does multiplying the gradient by the inverse Hessian achieve?
answer
- curvature, not just slope
- minimizes the local second-order model
- one exact step on a true quadratic
- long steps flat, short steps steep
- needs a positive definite Hessian
basics
~10 sThe inverse Hessian rescales the gradient by local curvature, giving long steps in flat directions and short ones where the surface curves sharply. The step lands on the minimizer of the local quadratic model.
solid answer
~50 sThe Newton step for minimizing a smooth function is `delta = -H^-1 g`, where g is the gradient and H the Hessian of second derivatives at the current point. Fit a second-order Taylor model around the current point and that step jumps straight to the model's minimizer, so on a genuinely quadratic objective Newton lands on the exact solution in a single step from anywhere. The inverse Hessian acts as a curvature-aware rescaling: directions in which the surface is nearly flat get large steps, sharply curved directions get small ones. It also makes the method invariant to a linear rechoice of coordinates, so no per-direction scaling has to be tuned by hand. The catch is that `H` must be positive definite at the current point — otherwise `-H^-1 g` need not even point downhill, and near a saddle it can steer you toward the stationary point rather than away from it.
go deeper
Recall the step as minus the inverse Hessian times the gradient, and that the Hessian is the matrix of second derivatives. Knowing it uses curvature where a first-order method uses slope alone is enough here.
Explain the derivation from the second-order Taylor model, the flat-versus-curved rescaling of each direction, and why a true quadratic is solved in one step.
Show the safeguards you would actually apply: checking or enforcing positive definiteness, damping the full step, and recognising a run that is converging to a saddle rather than a minimum.
Frame the decision of when second-order information earns its cost on a given problem shape, and what you give up in robustness and engineering complexity to buy the fast local rate.
## The quadratic model Take a twice-differentiable objective `f` and expand it around the current point `x`: ``` f(x + d) ~= f(x) + g^T d + (1/2) d^T H d ``` where `g = grad f(x)` is the gradient (a vector of first partial derivatives) and `H` is the Hessian (the matrix of second partial derivatives, symmetric for smooth f). This is a quadratic bowl in the displacement `d`. Minimising it exactly means setting its own gradient to zero: ``` g + H d = 0 => d = -H^-1 g ``` That displacement is the Newton step. So Newton's method is nothing more than: build the best local quadratic picture of the surface, jump to the bottom of that picture, rebuild, repeat. ## What the inverse Hessian actually does The gradient alone tells you the direction of steepest increase, but it carries the *units* of the parameters. If one parameter is measured in millimetres and another in kilometres, the raw gradient is dominated by whichever coordinate happens to be scaled small. The Hessian records how fast the slope itself changes in each direction; inverting it converts "slope" into "distance to the bottom". Diagonalise H and the picture is clear. Along an eigenvector with a large eigenvalue the surface is sharply curved — a narrow valley — and the corresponding component of the gradient is divided by that large number, producing a short cautious step. Along an eigenvector with a small eigenvalue the surface is nearly flat, and dividing by the small number produces a long stride across the plateau. Steepest-descent-style methods do the opposite: they take their biggest raw steps exactly where the surface is steepest and curving hardest. A structural consequence is **affine invariance**. If you reparameterise the problem by any invertible linear map, the Newton iterates map over exactly. The method does not care about the units or the relative scaling of your parameters, which is why it has no per-direction scale factor to tune. ## One step on a quadratic If `f` really is a quadratic with positive definite `H`, the second-order Taylor model is not an approximation but the function itself, so one Newton step from any starting point lands on the exact minimizer. On a non-quadratic objective this is what buys the fast asymptotic rate: near a strict local minimum with a Lipschitz-continuous Hessian, the iterates converge quadratically, and the error squares each step once you are close. ## Where it breaks **Indefinite or singular Hessian.** The step `-H^-1 g` is a descent direction (that is, `g^T d < 0`) only when `H` is positive definite. At a saddle point the Hessian has both positive and negative eigenvalues, and the raw Newton step happily converges to the saddle, because a saddle is a stationary point of the quadratic model just as a minimum is. Newton's method finds stationary points; it does not distinguish minima from maxima or saddles on its own. Standard fixes are to add a multiple of the identity to `H` until it is positive definite, to modify the negative eigenvalues, or to restrict the step to a trust region where the quadratic model is believed. **Steps that are too ambitious.** Even with a positive definite Hessian, the full step can overshoot when the quadratic model is a poor fit far from the current point. Practical implementations use a damped step — take a fraction of the Newton direction chosen by a line search that requires sufficient decrease — and only switch to full steps near the solution, where they recover the quadratic rate. **Cost.** Forming `H` needs on the order of d^2 second derivatives for d parameters, and solving the linear system `H d = -g` costs on the order of d^3 arithmetic. For anything beyond a few thousand parameters that is the dominant obstacle, and it is why quasi-Newton methods that approximate the inverse Hessian from gradient differences exist at all. ## How to answer in an interview Say what the step is, say it is the exact minimizer of the local quadratic model, explain the curvature rescaling in one sentence, and then volunteer the two conditions — positive definite Hessian, and a model that is trustworthy over the step length. Candidates who only recite the formula and stop are giving the junior answer.
- What goes wrong when the Hessian is indefinite at the current iterate?The step `-H^-1 g` is no longer guaranteed to point downhill, and near a saddle it can converge to the saddle itself, because a saddle is a stationary point of the quadratic model just like a minimum. Standard remedies add a multiple of the identity to the Hessian until it is positive definite, or restrict the move to a trust region where the quadratic model is trusted.
- How many Newton steps does a strictly convex quadratic objective need, and why?One, from any starting point. The second-order Taylor model of a quadratic is the function itself, so setting the model's gradient to zero solves the real problem exactly. Everything slower than that on a real objective comes from the model being only an approximation away from the current point.
- Why is Newton's method described as invariant to a linear reparameterisation of the variables?If you replace x by Ax for any invertible A, the gradient and Hessian transform so that the step maps over exactly, producing the same sequence of points in the original space. Practically this means the method has no per-direction scale factor to tune, and rescaling your features does not change its trajectory.
The gradient tells you which way is downhill; the inverse Hessian tells you how far the hill keeps going that way before it bottoms out.
saying these in an interview costs you the question
- Says the Newton step is always a descent direction
- Confuses the Hessian with the outer product of gradients
- Believes Newton's method distinguishes minima from saddles
- Claims Newton's method needs a hand-tuned per-direction scale
- Forgets the Hessian must be recomputed at every iterate