Calculus & Optimization Basics

The subdifferential of |x| at 0 is the closed interval from -1 to 1: every slope of magnitude at most 1 gives a line supporting the V from below. Zero lies inside, which certifies 0 as the minimiser.

A full-batch step uses the exact gradient, so with a small enough step size the loss decreases every iteration. A stochastic step uses a sampled gradient that is correct only on average, so individual steps can go uphill and the iterates settle into a noise ball rather than a point.

Only near the base point, where the loss is smooth and curvature changes slowly. The model's error grows like the cube of the displacement, so it is exact only when the loss is itself quadratic.

IRLS runs Newton's method on the log-likelihood. Each iteration computes fitted probabilities, forms weights p(1-p), and solves a weighted least-squares problem on a working response, repeating until the coefficients stop moving.

L-BFGS never forms an inverse Hessian. It stores only m recent pairs of parameter and gradient differences and rebuilds the step from them, cutting memory from about d squared entries to m times d.

Use a hard constraint when any violation is unacceptable and a feasible solution is guaranteed to exist. Use a penalty when the requirement is a tradeoff you are willing to price. Constraints give guarantees; penalties keep the problem solvable.

The determinant is r. The Jacobian is [[cos theta, minus r sin theta], [sin theta, r cos theta]], whose determinant is r cos^2 theta plus r sin^2 theta, which equals r. That factor is the map's local area scaling.

Take logs first: ln y = x ln x for y = x^x, which differentiates to y'/y = ln x + 1, giving y' = x^x times (ln x + 1). Base and exponent both vary, so neither standard rule applies.

Compute the scalar v^T H v for a unit vector v. That number is the second derivative of the function along the line in direction v: positive bends upward, near zero is locally flat.

For a sum of squared residuals the exact Hessian is J^T J plus a term weighted by the residuals themselves. Gauss-Newton drops that second term, so it needs only first derivatives and always gets a positive semidefinite curvature matrix.

Convexity is defined by the chord inequality, which never mentions derivatives. Each linear piece satisfies that inequality, and taking the maximum of two bounds preserves it, so the max is convex. The corner where the two lines cross affects smoothness only.

At a true stationary point the gradient is exactly zero. On a plateau it is merely small and still consistently signed, so the objective keeps creeping down and a long enough step leaves the flat region.

The gradient of the linear form w^T x with respect to x is w. The gradient of the quadratic form x^T A x is (A + A^T) x, which becomes 2Ax when A is symmetric.

The penalty weight lambda is the KKT multiplier of the constraint ||w|| <= t. For a convex loss the two forms trace the same family of solutions, with a larger budget t matching a smaller lambda.

Iterate. The closed-form route needs a 100,000 by 100,000 cross-product matrix — around 80 GB dense — costing on the order of 10^17 operations to form and 10^15 to factorise, while gradient steps cost one pass over the data each and store only the parameter vector.