Is a linear autoencoder trained with squared error equivalent to PCA?

## Setting up the comparison Consider a 50-dimensional customer-behaviour vector — spend rates, session counts, recency measures — and an autoencoder with a single hidden layer of 10 units, **no activation function anywhere**, trained with squared reconstruction error on mean-centred data. The encoder is a matrix `W_e` of shape 10-by-50, the decoder a matrix `W_d` of shape 50-by-10, and the reconstruction is `x_hat = W_d * W_e * x`. The composite map `W_d * W_e` is a 50-by-50 matrix of rank at most 10. So the model class is exactly: all rank-10 linear maps from the input space to itself. The training objective is: minimise the average squared distance between `x` and its image under that map. ## Why the subspace matches The best rank-k linear approximation to a data matrix under squared error is the truncated singular value decomposition — this is the classical low-rank approximation result, and the projection it defines onto the leading k directions of variation is exactly what k-component principal component analysis computes on mean-centred data. Since the linear autoencoder is optimising over precisely that model class with precisely that criterion, its global optimum achieves the same reconstruction error and its code layer spans the same k-dimensional subspace. That is the part of the equivalence that is real, and it is the part worth stating first in an interview: **same subspace, same reconstruction error, same objective**. ## Why the axes do not match Now the part candidates usually miss. Suppose `(W_e, W_d)` is a global optimum. Pick any invertible k-by-k matrix `A` and form `(A * W_e, W_d * A_inverse)`. The composite map is unchanged, so the loss is unchanged, so this is also a global optimum. The set of optima is therefore a continuum, and the individual code directions are identified only up to an arbitrary invertible mixing. Three concrete consequences follow. The learned encoder rows are generally **not orthogonal** to each other and not unit-norm — nothing in the loss asks for that. The code units carry **no ordering**: there is no 'first' direction capturing the most variance, because a mixing can spread that variance across all ten units. And two training runs from different random initialisations will converge to **different axes** spanning the same subspace, which is why a colleague who reruns your training and finds unit 3 means something else is not observing a bug. By contrast, a direct principal-component decomposition gives orthonormal directions, ordered by the variance each captures, computed in closed form, identical on every rerun up to sign. ## The optimisation landscape The classical analysis of the squared-error linear autoencoder shows that its loss surface, while non-convex in the parameters because of the product `W_d * W_e`, has no *suboptimal local minima*: every critical point that is not a global minimum is a saddle point corresponding to projecting onto a non-leading set of directions. That is a pleasant theoretical property and a useful piece of intuition about why non-convexity is not automatically the same as bad optimisation — but it is a statement about this linear model, not a licence to assume the same of deep nonlinear networks. ## What you actually gain from nonlinearity If a linear autoencoder can only ever land on a linear subspace, the interesting question is what changes when you insert activation functions. A nonlinear encoder can map onto a curved surface in input space rather than a flat one. Data that lies near a curved low-dimensional surface — a sheet folded through the ambient space — needs many linear directions to be covered by a flat subspace but few code units to be traced along the surface. A 10-unit nonlinear autoencoder can therefore reconstruct such data far better than any 10-dimensional linear projection, and this gap is the entire reason to train an autoencoder rather than compute a decomposition. The practical corollary: if you train a nonlinear autoencoder and it barely beats a same-width linear baseline, your data is close to linear in the relevant sense, and the extra machinery is buying nothing. That baseline comparison is cheap and worth running. ## Answering the question crisply A good short answer has four beats. The model class is rank-k linear maps under squared error, so the optimum spans the leading principal subspace. The axes within it are free up to an invertible mixing, so they are neither orthonormal nor ordered nor reproducible. A closed-form decomposition gives you all of those properties for free and costs one pass rather than a training run. And the reason to reach for an autoencoder at all is the nonlinear case, where no linear subspace of the same width can compete.