What do the eigenvectors of a 2x2 covariance matrix represent geometrically?

## The setup A covariance matrix `C` for `d` variables is `d x d`, with the variance of variable `i` on the diagonal and the covariance of variables `i` and `j` off the diagonal. Two structural facts drive everything below: `C` is **symmetric**, because the covariance of `i` with `j` equals the covariance of `j` with `i`; and `C` is **positive semidefinite**, because no direction can have negative variance. Symmetry lets the spectral theorem apply: `C` has real eigenvalues and an orthonormal basis of eigenvectors, and can be written `C = Q D Q^T` with `Q` orthogonal and `D` diagonal. ## What the eigenvectors mean The key identity is that the variance of the data projected onto any unit direction `u` is the quadratic form ``` Var(u) = u^T C u ``` If `u` is a unit eigenvector with `Cu = lambda u`, then `u^T C u = u^T (lambda u) = lambda (u^T u) = lambda`. So **each eigenvalue is literally the variance of the data along its own eigenvector**. Maximizing `u^T C u` over unit vectors picks out the top eigenvector, with the largest eigenvalue as the maximum; minimizing picks the bottom one. The eigenvectors are the directions of extremal spread, and because `C` is symmetric they are mutually orthogonal. Drawing the cloud of a two-variable dataset as an ellipse (a contour of constant density for roughly elliptical data), the eigenvectors point along the ellipse's axes and the semi-axis lengths are proportional to `sqrt(lambda)` — square roots, because eigenvalues are in squared units of the data while the picture is drawn in the data's own units. ## A worked example Let ``` C = [[5, 3], [3, 5]] ``` Each variable has variance 5 and they covary positively. Using `lambda^2 - trace*lambda + det = 0` with `trace = 10` and `det = 25 - 9 = 16`: ``` lambda^2 - 10*lambda + 16 = (lambda - 8)(lambda - 2) -> lambda = 8, 2 ``` For `lambda = 8`: `C - 8I = [[-3,3],[3,-3]]` gives `v1 = v2`, so the top eigenvector is `(1,1)/sqrt(2)`. For `lambda = 2`: `C - 2I = [[3,3],[3,3]]` gives `v1 = -v2`, so the second is `(1,-1)/sqrt(2)`, orthogonal to the first as symmetry promises. Interpretation: the cloud is a tilted ellipse whose long axis runs at 45 degrees. Variance along that diagonal is 8; variance across it is 2. The ellipse is `sqrt(8/2) = 2` times longer than it is wide. Because both variables happen to have equal variance, the axes land exactly on the diagonals; unequal variances tilt them off 45 degrees. ## Trace, determinant and total variance Two invariants make the eigen-view easy to sanity check: - **Trace.** `8 + 2 = 10`, and the diagonal of `C` sums to `5 + 5 = 10`. Total variance is conserved: rotating to the eigenbasis moves variance between axes but never changes the total. This is why the eigenvalues are often quoted as a share of the trace. - **Determinant.** `8 * 2 = 16 = 25 - 9`. The determinant, sometimes called the generalized variance, is proportional to the squared area of the ellipse. A determinant near zero means a nearly flat, degenerate cloud. ## Decorrelation Write the eigen-decomposition as `C = Q D Q^T`. If the data (already centred) is rotated by `Q^T`, the covariance of the rotated coordinates is `Q^T C Q = D`, which is diagonal. So expressing the data in the eigenbasis makes the new coordinates **uncorrelated**, with variances equal to the eigenvalues. That is the whole geometric content of the eigendecomposition of a covariance matrix: there is always a rotation of the axes that removes all linear correlation, and the eigenvectors name that rotation. ## Reading edge cases - **Equal eigenvalues.** The ellipse is a circle, spread is the same in every direction, and the eigenvectors are not unique — any orthonormal pair works. Do not over-interpret which pair the computation returned. - **One eigenvalue near zero.** The cloud is nearly flat: some linear combination of the variables barely varies, which is the signature of near-collinearity among the variables. - **Sign.** An eigenvector and its negative describe the same axis. An arrow pointing 'up the diagonal' or 'down the diagonal' is the same finding, so never build logic on the returned sign. - **Scale dependence.** Eigenvectors of a covariance matrix are not invariant to rescaling the variables. Change one variable from metres to millimetres and its variance grows by a factor of a million, dragging the top axis toward it. Whenever variables have incomparable units, standardize first and eigendecompose the correlation matrix instead; the two decompositions generally disagree, and which one is appropriate is a modelling decision, not a numerical one.