Calculus & Optimization Basics

At a smooth interior local minimum the gradient must vanish, so a zero gradient is necessary. But maxima, saddle points and flat inflection points also have a zero gradient, so vanishing slope on its own certifies nothing.

The derivative of x^2 is 2x. The difference quotient ((x+h)^2 - x^2)/h expands to (2xh + h^2)/h, which simplifies to 2x + h, and letting h shrink to 0 leaves 2x.

Each iteration moves every parameter a short distance opposite its own partial derivative: new value = old value minus eta times the gradient. The learning rate eta scales how far you move; the loop repeats until the gradient is near zero.

Differentiate one variable at a time, holding the other fixed: df/dx = 2xy and df/dy = x^2. At (2, 3) those evaluate to 12 and 4, so the gradient there is the vector (12, 4).

The Jacobian is m by n: one row per output component, one column per input variable. Entry (i, j) is the partial derivative of output component f_i with respect to input x_j, evaluated at a specific point.

A function is convex when every chord lies on or above its graph: f(t*x + (1-t)y) <= tf(x) + (1-t)*f(y). Squaring passes everywhere. Cubing fails between x = -2 and 0, where f(-1) = -1 sits above the chord value -4.

The gradient of x^2 - y^2 vanishes at the origin, but the surface curves up along the x-axis and down along the y-axis. A point that is a minimum one way and a maximum another is a saddle.

At 0 the difference quotient |h|/h equals +1 for positive h and -1 for negative h, so the two one-sided limits disagree and no single tangent slope exists. The function is still continuous there.

On f(x) = x^2 the update multiplies x by (1 - 2 * eta) each step, so it converges exactly when eta is between 0 and 1. eta = 0.5 lands on the minimum in one step, eta between 0.5 and 1 oscillates while shrinking, eta = 1 cycles forever, and eta above 1 diverges.

The rate of change in a unit direction equals the gradient's length times the cosine of the angle between them. Cosine peaks at 1 when the direction aligns with the gradient, so that bearing rises fastest.

The Hessian is the square matrix of all second partial derivatives of a scalar function, with entry (i, j) equal to d2f/dxi dxj. It is symmetric wherever those second partials are continuous, by Clairaut's theorem.

Form L = xy - lambda(x + y - 10) and set every partial derivative to zero. The solution is x = y = 5 with lambda = 5, so the largest achievable product is 25.

The inverse Hessian rescales the gradient by local curvature, giving long steps in flat directions and short ones where the surface curves sharply. The step lands on the minimizer of the local quadratic model.

Set the residual r = Wx - y, so L = r transpose r. Entry W_ij affects only r_i, and does so through x_j, so the gradient is 2 (Wx - y) x transpose, a matrix with the same shape as W.

For minimizing f subject to g(x) <= 0 and h(x) = 0, the KKT conditions require stationarity of the Lagrangian, primal feasibility, nonnegative multipliers on the inequalities, and complementary slackness - every inequality's multiplier times its constraint value is zero.

Chain the rates: dV/dt = (dV/dr) times (dr/dt). For a sphere V = (4/3) pi r^3, so dV/dr = 4 pi r^2 and dV/dt = 8 pi r^2 cubic centimetres per second when the radius grows at 2 cm/s.

log(1 + x) is approximately x - x^2/2 near x = 0. The dropped remainder starts at the cubic term x^3/3, so the error shrinks roughly eightfold each time x is halved.

Newton-Raphson replaces the function by its tangent line at the current guess, so each new error is roughly proportional to the square of the previous one. Near the root the number of correct digits roughly doubles per step.

Each factor must consume the output of the stage before it, so shapes conform only outermost-first: (m by q)(q by p)(p by n) yields the m by n total derivative. Matrix products are not commutative, so no other order works.

On an open convex domain, a twice-differentiable function is convex exactly when its Hessian is positive semidefinite at every point of that domain. In one variable this reduces to f''(x) >= 0 everywhere. A positive definite Hessian everywhere implies strict convexity.

At a stationary point c: positive f''(c) means a strict local minimum, negative means a strict local maximum, and f''(c) = 0 is inconclusive, since minima, maxima and inflections all produce it.

No. The implication runs one way only: differentiability forces continuity, never the reverse. The function x times sin(1/x), with value 0 at the origin, is continuous at 0 yet its difference quotient oscillates forever.

The quotient rule turns 1/(1 + e^-x) into e^-x/(1 + e^-x)^2. That expression splits into 1/(1 + e^-x) times e^-x/(1 + e^-x), and the second factor is exactly 1 minus the first, giving s times (1 - s).

The steepest direction limits the step size while the flattest direction needs progress. A step small enough to stay stable along the curvature-100 axis moves the curvature-1 axis by only about one percent per iteration, so the path bounces across the narrow valley and creeps along it.

The gradient of x^2 + 3xy is (2x + 3y, 3x), which is (8, 3) at (1, 2). Combining it with the unit direction gives (8 + 3)/sqrt(2) = 11/sqrt(2), about 7.78 per unit distance.

Along a level curve the function is constant, so the rate of change in the tangent direction is zero. With a nonzero gradient, that combination vanishes only if the two are at right angles.

A Lagrange multiplier is a shadow price: the rate at which the best achievable objective value changes when the constraint is relaxed by one unit. Multiplier 5 means one more unit buys about 5 more objective units.

For a convex f and weights t_i >= 0 summing to 1, f(sum t_i x_i) <= sum t_i f(x_i): the function of the weighted average never exceeds the weighted average of the function values. For a concave f the inequality reverses.

f(x) = x^4 - 4x^2 has minima at x = -sqrt(2) and x = +sqrt(2) with a local maximum at x = 0 between them. A downhill search never climbs that barrier, so the starting sign fixes the answer.