Linear Algebra for ML

Build A^+ = V S^+ U^T from the SVD by inverting each nonzero singular value and leaving zeros as zeros. Then x = A^+ b minimizes ||Ax - b|| and, among the infinitely many minimizers, is the one with smallest norm.

Whenever features are correlated and a point breaks the correlation. Mahalanobis distance divides out the covariance, so a point that is unremarkable on each feature alone but sits off the joint ridge gets a large distance. Euclidean distance sees nothing unusual.

Coordinates are the weights that rebuild a vector from a chosen basis, so they belong to the basis, not to the vector. Rotate the basis and the arrow is unchanged while the list of weights describing it changes.

Squaring grows faster than adding. For points 0, 1 and 2 on a line, squared distance gives 4 end to end but 1 plus 1 for the legs, so a detour beats the direct route. It is a divergence, not a metric.

Each step subtracts projections from a vector, and for nearly parallel inputs that subtraction cancels two almost identical vectors. The tiny survivor is mostly rounding error, and normalising it magnifies that error, so the resulting directions are not truly perpendicular.

Start from a random nonzero vector, repeatedly multiply by the matrix, and renormalize each step. Components along smaller eigenvalues shrink relative to the largest one, so the vector converges to the dominant eigenvector at rate |lambda2 / lambda1| per iteration.

Because the determinant depends on scale: for an n x n matrix, det(cA) = c^n det(A). Shrinking every entry drives the determinant toward zero without making the matrix any harder to invert.

Associativity means (AB)C and A(BC) give identical results, so the grouping is yours to choose. An (m x n) by (n x p) product costs about mnp multiply-adds, so pick the grouping whose intermediate is smallest.

Forming the cross-product matrix squares the condition number, so you lose twice as many digits and can turn a full-rank problem into a numerically singular one. QR instead factors the original matrix with length-preserving transformations.

Treat k as a budgeted tradeoff, not a formula. Let the flattening of the singular-value spectrum bracket a range, then choose inside it by held-out reconstruction error and the downstream metric, subject to storage and serving limits.