PCA and Explained Variance

## The setup Start with a table of `n` rows and `p` numeric columns. PCA rewrites those `p` columns as `p` new columns - the principal components - which are ordered so that the first carries as much of the data's variance as any single direction can, the second as much of what is left as any direction orthogonal to the first can, and so on. Nothing is thrown away until you decide to keep only the first few. ## Step zero: centring PCA is defined on **mean-centred** data. For each column you subtract that column's mean, so every column now has mean zero and the cloud of points is positioned around the origin. This is not cosmetic. Variance is defined as spread *about the mean*; the optimisation PCA performs on centred data maximises the average squared projection, and on centred data the average squared projection **is** the variance. Skip the centring and the same optimisation maximises the raw second moment instead, which is variance plus the squared offset of the cloud from the origin. With columns that sit far from zero - incomes, temperatures in kelvin, test scores out of 800 - that offset dominates, so the "first component" points roughly at the mean vector, i.e. at where the data *is*, not at the direction in which it *varies*. Every later component is then forced to be orthogonal to that useless direction, and the whole decomposition is wrong. Whether you should additionally rescale the columns before running PCA is a separate preprocessing decision with its own trade-offs. ## The objective Consider any unit-length direction vector `v` in the `p`-dimensional space. Projecting a centred row `x` onto it gives a single number, the score `s = x . v` (the dot product). Do this for all `n` rows and you get a one-dimensional summary of the data with some variance. PCA asks: **over all unit-length `v`, which one makes that variance largest?** The answer is the first principal component, and the resulting variance is that component's variance. The second component solves the identical problem with one extra constraint: `v` must be orthogonal (at right angles) to the first component. The third is orthogonal to the first two, and so on. Orthogonality is what makes PCA a *rotation* of the data rather than an arbitrary re-encoding, and it is why the component scores are mutually uncorrelated across the sample - the classic reason PCA is reached for when the original columns are badly collinear. ## The equivalent reconstruction view There is a second description of the same objective that is worth having ready. Take only the first `k` components and use them to describe each row: you get an approximate version of the row, `x` reconstructed as the column means plus a weighted sum of the `k` component directions. The squared distance between the true row and this reconstruction is the reconstruction error. The subspace that **minimises** total squared reconstruction error over the whole dataset is exactly the subspace spanned by the first `k` components. "Keep the most variance" and "lose the least when you project" are two readings of one optimisation, because on centred data total variance splits cleanly into the part you keep and the part you discard. ## Loadings and scores Two different objects get muddled constantly. The **loadings** are the coefficients that define a component - one weight per original column, describing how the component is built. The **scores** are the numbers you get after projecting the rows - one value per row per component, and these are the new columns you feed downstream. A candidate who says "PCA picks the most important features" has confused this: a component is a dense weighted mixture of *all* `p` columns, and a column with a near-zero loading has not been "unselected", it simply contributes little to that particular direction. ## Variance accounting and sign Because PCA only rotates centred data, the sum of the variances of all `p` components equals the sum of the variances of the original `p` columns. Total variance is conserved and merely redistributed, concentrated into the leading directions. That conservation is what makes the per-component share of the total - the explained-variance ratio - a meaningful quantity. One practical wrinkle: the sign of a component is arbitrary. A direction `v` and its negation `-v` span the same axis and give the same variance, so different runs or implementations may return either. Loadings and scores flip together, so the geometry and any downstream model are unchanged - but do not read meaning into a sign, and do not compare signs across two separately fitted decompositions. ## Assumptions to state out loud PCA is linear: it can only find flat subspaces, so structure that curves through the space is not captured. It is variance-based, and variance is a scale-dependent quantity, so the relative units of your columns change the answer. And it is **unsupervised**: it never sees a target, so it ranks directions by how much the inputs move, not by how useful they are for predicting anything.