Linear Algebra for ML

A nonzero vector v is an eigenvector of A when Av = lambda v. Multiplying by A leaves v on its own line through the origin, only stretching, shrinking or flipping it; the scalar lambda is that scale factor.

A zero determinant means the matrix is singular: no inverse exists. For a 2x2 matrix [[a, b], [c, d]] the determinant is ad - bc, so ad - bc = 0 is the exact test for non-invertibility.

Matrix multiplication needs matching inner dimensions: if A is m x n and B is n x p, then AB is m x p. Entry (i,j) is the dot product of row i of A with column j of B.

Run Gaussian elimination on the augmented matrix. A row that is all zeros on the left but nonzero on the right means no solution. Otherwise, one pivot per unknown means exactly one solution; any free unknown means infinitely many.

Linearly dependent columns means one column equals a weighted combination of the others, adding no new direction. A table holding hours_weekday, hours_weekend and their exact total is the classic case; that matrix is rank deficient.

Every real matrix factors as A = U S V^T. U and V hold orthonormal directions in the output and input spaces, S is diagonal with non-negative singular values in descending order, and the number of nonzero ones equals the rank.

All three measure a vector's size differently. L1 adds the absolute values of the coordinates, L2 is the square root of the sum of squares, and L-infinity is the largest absolute coordinate. For (3, 4) they give 7, 5 and 4.

Solve det(A - lambda I) = 0. For [[2,1],[1,2]] that is (2 - lambda)^2 - 1 = 0, giving lambda = 3 and 1. Substituting each back into (A - lambda I)v = 0 yields eigenvectors (1,1) and (1,-1).

A matrix product is a composition of linear maps, and composition depends on order. In AB the right factor acts first, so rotating then stretching is a different transformation from stretching then rotating. Only special pairs commute.

Choose the x that makes the residual b - Ax as short as possible instead of demanding equality. That least-squares x satisfies A transpose A x = A transpose b, a small square system that always has a solution.

Rank-nullity says rank plus nullity equals the number of columns. A 5x3 matrix of rank 2 therefore has nullity 3 - 2 = 1: the vectors it sends to zero form a line through the origin in three-dimensional space.

The Eckart-Young theorem guarantees it: keeping the k largest singular values and their vectors minimizes the approximation error over all matrices of rank at most k, in both Frobenius and spectral norm. The discarded singular values fix the error exactly.

The L1 unit ball is a diamond with corners on the coordinate axes, so a constrained solution tends to land on a corner where some coordinates are exactly zero. The round L2 ball has no corners, so it only shrinks.

Because the basis vectors are mutually perpendicular and of unit length, dotting a vector with one basis vector cancels every other term and leaves that coordinate directly. No system of equations has to be solved.

The projection of b onto a is (a . b / a . a) times a: scale a by the dot product of a and b over a with itself. The residual, b minus that projection, is perpendicular to a.

They are the principal axes of the data's elliptical cloud: orthogonal directions along which the data spreads independently. Each eigenvalue is the variance measured along its own eigenvector, and the eigenvalues sum to the total variance, the matrix trace.

Its characteristic equation (1 - lambda)^2 = 0 repeats lambda = 1, but the eigenspace is only the line through (1,0). One independent eigenvector instead of two means no eigenbasis, so no invertible P exists: the matrix is defective.

Because the determinant is a volume scaling factor, and applying B and then A scales volume by det(B) and then by det(A). Scaling factors multiply, so det(AB) = det(A)det(B).

The determinant is the signed area scaling factor of the map: the unit square is sent to a shape whose area is the absolute value of the determinant. A negative sign means orientation was flipped.

The trace is the sum of a square matrix's diagonal entries, and both tr(AB) and tr(BA) expand to the same double sum over every pair of entries. The order of summation changes, so the totals match.

Ax is a linear combination of the columns of A, weighted by the entries of x. Applying A to the basis vector with a single 1 in slot j returns column j, so the columns are where the axes go.

Transposing swaps each matrix's row and column counts, so the factors must reverse for the shapes to line up. Entry (i,j) of (AB)^T is entry (j,i) of AB, which is row i of B^T dotted with column j of A^T.

Factoring A into lower and upper triangular factors costs about a third of what forming the inverse costs and is more accurate. It is also reusable: each new right-hand side then needs only two cheap triangular solves.

The singular values of A are the square roots of the eigenvalues of A^T A, which are always real and non-negative. The eigenvectors of A^T A are the right singular vectors, the columns of V.

The Frobenius norm is the square root of the sum of a matrix's squared entries. It is the L2 norm of the matrix flattened into one vector, and it is the usual measure of a weight or residual matrix's size.

For a symmetric matrix, positive definite means every eigenvalue is strictly greater than zero; positive semidefinite means every eigenvalue is greater than or equal to zero. A zero eigenvalue is exactly what separates them, and it makes the matrix singular.

The matrix is ill-conditioned, not the solver buggy. Its condition number - how much a relative change in b is magnified in x - is large, so noise in the last digits swings the answer.

No. A correlation of 0.999 is not an exact linear identity, so the two columns stay independent and the matrix is full rank. It sits a hair from rank deficiency, so arithmetic on it behaves almost as badly.

A 3x3 matrix whose columns span only a plane has rank 2 and a one-dimensional nullspace. If b lies in that plane, solutions form a line: one particular solution plus any multiple of the nullspace vector. Otherwise none exist.