In a Gaussian mixture, what do spherical, diagonal and full covariance each buy you?

## What the covariance controls Each Gaussian component in a mixture has a mean, which places it, and a covariance, which sets its **shape**: how wide it is along each direction and whether the directions are correlated. Drawing the component as an ellipse of constant density makes this concrete — the covariance chooses the ellipse's axis lengths and its orientation. Restricting the covariance restricts the family of shapes the component can take. The standard ladder, from most restricted to least: **Spherical** — the covariance is a single variance times the identity. The ellipse is a circle (a ball in higher dimensions). Every feature has the same spread and no feature is correlated with another. Cost: **1** free number per component. **Diagonal** — one variance per feature, zero off-diagonal terms. The ellipse can be stretched independently along each coordinate axis, but its axes must line up with the feature axes; it cannot tilt. Cost: **d** numbers per component. **Full** — the complete symmetric covariance matrix. The ellipse can have any axis lengths and any orientation, so the component can represent correlated features. Cost: **d(d+1)/2** numbers per component (the diagonal plus one triangle of off-diagonals). Some implementations also offer a *tied* variant, where all components share one covariance matrix; that trades per-component flexibility for a single, much better-estimated shape. ## The discriminating example Picture two clusters in two dimensions, both elongated cigars, both tilted at roughly 45 degrees, lying side by side. Feature 1 and feature 2 are strongly correlated within each cluster. - **Full covariance**: two components suffice. Each fits one tilted ellipse straight onto one cigar. The off-diagonal term is what encodes the tilt. - **Diagonal covariance**: no component can tilt. The best a single diagonal component can do with a 45-degree cigar is an axis-aligned blob that covers the cigar's bounding box, which also covers a lot of empty space and bleeds into the neighbouring cluster. EM typically responds by splitting each cigar into a chain of several small axis-aligned components — the fit improves, but you now have six components where two would do, and the component count no longer means what you wanted it to mean. - **Spherical covariance**: worse again, since it cannot even stretch; it needs still more, still rounder blobs. That failure is diagnosable. If a mixture keeps wanting more components than the domain suggests, and the fitted components come out in strings along a diagonal direction, the covariance restriction is the culprit, not the component count. ## Why you would ever restrict, then Because parameter count is not free. With `K` components in `d` dimensions, the full-covariance model has `K * d(d+1)/2` covariance parameters, and covariance estimation is data-hungry: to estimate a `d x d` covariance stably you want substantially more than `d` effective points, and to have it be non-singular you need at least `d + 1` points in general position contributing to that component. At `d = 50`, full covariance means 1275 numbers **per component**. If a component only owns 300 points, that estimate is wildly noisy; if it owns 40, the matrix is singular and the fit degenerates outright. So the choice is a bias-variance decision made on parameters: - **Full** = low bias in shape, high variance in the estimate. Good when `d` is small and you have plenty of data per component. - **Diagonal** = biased (it cannot represent correlation) but the estimate is stable, and it scales linearly in `d`. This is the workhorse in higher dimensions. - **Spherical** = strongly biased toward round, equal-width clusters, but extremely cheap and hard to break. Reasonable when features are on a common scale and clusters really are isotropic. A practical middle path: rotate or decorrelate the feature space first, so that the correlation the diagonal model cannot represent has largely been removed before fitting; a diagonal mixture in a decorrelated space is far less biased than a diagonal mixture in the raw space. ## Scaling matters more as you restrict Spherical assumes one width for all features, so it is only sensible when features share a scale — a mixture of amounts in dollars and counts in single digits will be dominated by the dollar axis. Diagonal tolerates differing scales, since it estimates a per-feature variance. Full covariance is the least sensitive to relative scaling, because it can adapt shape freely. If you standardise features before fitting, spherical and diagonal both become far more defensible. ## How to choose in practice Fit the candidates and compare them on a penalised likelihood or on held-out log-likelihood. The parameter count is exactly what the penalty is built from, so a full-covariance model has to earn its extra flexibility with a substantially better fit before it wins. Do not choose by eye on a two-dimensional projection: a projection can make correlated clusters look axis-aligned and mislead you toward diagonal. And check for degeneracy — if the full-covariance fit produces a component with a near-singular covariance, that is not a better fit, it is a collapse, and it is a reason to step down the ladder or add a regulariser.