How do eigenvalues distinguish a positive-definite matrix from a merely PSD one?

## Two equivalent definitions For a **symmetric** matrix `A` (definiteness is only defined for symmetric matrices in the usual convention), the quadratic form is the scalar `x^T A x`. Then: - `A` is **positive definite (PD)** if `x^T A x > 0` for all `x != 0`. - `A` is **positive semidefinite (PSD)** if `x^T A x >= 0` for all `x`. The eigenvalue characterisation is equivalent, and it is the one worth reaching for. Diagonalize `A = Q D Q^T` with `Q` orthogonal, and substitute `y = Q^T x`: ``` x^T A x = y^T D y = sum_i lambda_i * y_i^2 ``` A sum of eigenvalues weighted by squares is positive for every nonzero `y` exactly when every `lambda_i > 0`, and nonnegative exactly when every `lambda_i >= 0`. Taking `x` to be a unit eigenvector gives the one-line version: `v^T A v = lambda`, so a negative eigenvalue immediately exhibits a direction where the form goes negative. ## The two examples side by side `A = [[2,-1],[-1,2]]`. Trace 4, determinant `4 - 1 = 3`, so `lambda^2 - 4*lambda + 3 = 0` and the eigenvalues are `3` and `1`. Both strictly positive, so `A` is positive definite. Being PD it is invertible, its inverse is also PD, its determinant is positive, and its quadratic form carves out proper ellipses. `B = [[1,1],[1,1]]`. Trace 2, determinant `1 - 1 = 0`, so the eigenvalues are `2` and `0`. All nonnegative, so `B` is PSD, but not PD. The failure is explicit in the quadratic form: ``` x^T B x = x1^2 + 2*x1*x2 + x2^2 = (x1 + x2)^2 ``` This is never negative, but it vanishes on the whole line `x1 = -x2`, i.e. at `(1,-1)`, which is exactly the eigenvector for `lambda = 0`. So the boundary between PD and PSD is precisely whether some nonzero direction is annihilated. ## Why the zero eigenvalue is a practical event, not a technicality The determinant is the product of the eigenvalues, so a zero eigenvalue forces a zero determinant: the matrix is singular and has **no inverse**. Consequences that show up in real work: - Anything that requires inverting the matrix fails outright, and anything that inverts a *nearly* singular matrix returns huge, noise-dominated values. - A Cholesky factorization, which exists only for positive definite matrices, breaks down; it is in fact the standard cheap test for positive definiteness. - For a covariance matrix, `x^T C x` is the variance of the data projected onto `x`. A zero eigenvalue therefore says some linear combination of the variables has **zero variance** — an exact linear dependency among them. Perfectly redundant features, a set of proportions that sums to one, or a dummy-coded variable that kept every level all produce this. A sample covariance matrix is *always* PSD, because it is built as a sum of outer products and so `x^T C x` is an average of squares. It is positive definite only when no such exact dependency exists **and** there are enough observations: with `d` variables and `n` points, the rank cannot exceed `min(n - 1, d)`, so with fewer observations than variables the matrix is guaranteed singular no matter how clean the data is. ## Tests and traps Things that do **not** establish positive definiteness: - **Positive entries.** `[[1,2],[2,1]]` has all entries positive but eigenvalues `3` and `-1`, so it is indefinite. Conversely a PD matrix may have negative entries, as `[[2,-1],[-1,2]]` shows. - **Positive determinant alone.** `[[-1,0],[0,-1]]` has determinant `+1` yet both eigenvalues are `-1`: negative definite. In two dimensions you need positive determinant *and* a positive diagonal entry. - **Symmetry.** Symmetry is a prerequisite for the question to be well posed, not an answer to it. Things that do work: all eigenvalues positive; all **leading principal minors** positive (Sylvester's criterion); or a successful Cholesky factorization. Positive diagonal entries are *necessary* but not sufficient — necessary because `e_i^T A e_i = A[i][i]` must be positive for a PD matrix. ## Repairing a matrix that should have been PD If a matrix that ought to be PD comes back with a tiny negative eigenvalue from rounding, the common repairs are to clip the negative eigenvalues to zero and rebuild the matrix from its eigendecomposition, or to add a small multiple of the identity. The second works because the eigenvalues of `A + eps*I` are exactly `lambda_i + eps`, which shifts the whole spectrum upward while keeping every eigenvector unchanged. Both are deliberate modelling choices with consequences, not free fixes: they change the matrix, so the size of `eps` needs justifying rather than defaulting.