In Newton's method for minimization, what does multiplying the gradient by the inverse Hessian achieve?

## The quadratic model Take a twice-differentiable objective `f` and expand it around the current point `x`: ``` f(x + d) ~= f(x) + g^T d + (1/2) d^T H d ``` where `g = grad f(x)` is the gradient (a vector of first partial derivatives) and `H` is the Hessian (the matrix of second partial derivatives, symmetric for smooth f). This is a quadratic bowl in the displacement `d`. Minimising it exactly means setting its own gradient to zero: ``` g + H d = 0 => d = -H^-1 g ``` That displacement is the Newton step. So Newton's method is nothing more than: build the best local quadratic picture of the surface, jump to the bottom of that picture, rebuild, repeat. ## What the inverse Hessian actually does The gradient alone tells you the direction of steepest increase, but it carries the *units* of the parameters. If one parameter is measured in millimetres and another in kilometres, the raw gradient is dominated by whichever coordinate happens to be scaled small. The Hessian records how fast the slope itself changes in each direction; inverting it converts "slope" into "distance to the bottom". Diagonalise H and the picture is clear. Along an eigenvector with a large eigenvalue the surface is sharply curved — a narrow valley — and the corresponding component of the gradient is divided by that large number, producing a short cautious step. Along an eigenvector with a small eigenvalue the surface is nearly flat, and dividing by the small number produces a long stride across the plateau. Steepest-descent-style methods do the opposite: they take their biggest raw steps exactly where the surface is steepest and curving hardest. A structural consequence is **affine invariance**. If you reparameterise the problem by any invertible linear map, the Newton iterates map over exactly. The method does not care about the units or the relative scaling of your parameters, which is why it has no per-direction scale factor to tune. ## One step on a quadratic If `f` really is a quadratic with positive definite `H`, the second-order Taylor model is not an approximation but the function itself, so one Newton step from any starting point lands on the exact minimizer. On a non-quadratic objective this is what buys the fast asymptotic rate: near a strict local minimum with a Lipschitz-continuous Hessian, the iterates converge quadratically, and the error squares each step once you are close. ## Where it breaks **Indefinite or singular Hessian.** The step `-H^-1 g` is a descent direction (that is, `g^T d < 0`) only when `H` is positive definite. At a saddle point the Hessian has both positive and negative eigenvalues, and the raw Newton step happily converges to the saddle, because a saddle is a stationary point of the quadratic model just as a minimum is. Newton's method finds stationary points; it does not distinguish minima from maxima or saddles on its own. Standard fixes are to add a multiple of the identity to `H` until it is positive definite, to modify the negative eigenvalues, or to restrict the step to a trust region where the quadratic model is believed. **Steps that are too ambitious.** Even with a positive definite Hessian, the full step can overshoot when the quadratic model is a poor fit far from the current point. Practical implementations use a damped step — take a fraction of the Newton direction chosen by a line search that requires sufficient decrease — and only switch to full steps near the solution, where they recover the quadratic rate. **Cost.** Forming `H` needs on the order of d^2 second derivatives for d parameters, and solving the linear system `H d = -g` costs on the order of d^3 arithmetic. For anything beyond a few thousand parameters that is the dominant obstacle, and it is why quasi-Newton methods that approximate the inverse Hessian from gradient differences exist at all. ## How to answer in an interview Say what the step is, say it is the exact minimizer of the local quadratic model, explain the curvature rescaling in one sentence, and then volunteer the two conditions — positive definite Hessian, and a model that is trustworthy over the step length. Candidates who only recite the formula and stop are giving the junior answer.