How do BIC and AIC choose the number of components for a Gaussian mixture?

## Why a mixture lets you do this at all A Gaussian mixture is a probabilistic model of the data, so a fit comes with a number that a purely geometric partition does not have: the **log-likelihood** of the observed data under the fitted model. That opens the door to standard model-selection machinery. You cannot simply pick the `K` with the highest log-likelihood — adding components can only ever fit better, so the likelihood rises monotonically with `K` and the maximum is always at the largest `K` you tried. Information criteria fix that by charging for parameters. ## The two criteria For a fitted model with maximised log-likelihood `L`, `p` free parameters and `n` observations: ``` AIC = -2 * L + 2 * p BIC = -2 * L + p * log(n) ``` Both are on a "lower is better" scale, because the `-2L` term turns a higher likelihood into a lower score. The **only structural difference** is the price per parameter: a flat 2 for AIC, `log(n)` for BIC. Since `log(n) > 2` whenever `n > e^2`, roughly 8, BIC charges more than AIC on any real dataset, and the gap widens with sample size — at `n = 10,000`, BIC pays about 9.2 per parameter against AIC's 2. **BIC therefore selects the same or fewer components than AIC**, essentially always. ## Counting the parameters correctly Getting `p` right matters, because the penalty is built entirely from it. For `K` components in `d` dimensions: - Mixing weights: `K - 1` free, not `K`, since they must sum to 1. - Means: `K * d`. - Covariances: `K * d(d+1)/2` for full covariance, `K * d` for diagonal, `K` for spherical. So for `K = 3`, `d = 2`, full covariance: `2 + 6 + 9 = 17`. Note that the covariance form changes `p` dramatically, which means the criteria let you compare covariance forms and component counts **in the same sweep** — a diagonal mixture with 5 components and a full-covariance mixture with 3 are directly comparable if you count parameters honestly for each. ## How the sweep is actually run Take a bid-price distribution from an ad auction and fit `K = 1` through `K = 10`. For each `K`: 1. Run EM from several different initialisations, because EM converges only to a local optimum. Keep the run with the highest converged log-likelihood. 2. Record that log-likelihood, the parameter count and the criterion values. 3. Discard any run with a collapsed component — a degenerate fit has an inflated log-likelihood and will win the sweep for the wrong reason. Then plot both curves against `K`. You are looking for a minimum. Typical outcome: BIC bottoms at, say, 3 and rises after; AIC bottoms at 5 or 6 and stays flatter. That divergence is expected and is not a sign that one of them is broken. ## Reading the curves honestly **A clear minimum.** Take it, then sanity-check the fitted components against the domain. Also look at the *shape*: if `K = 3` and `K = 4` are within a couple of BIC points of each other, the data is not distinguishing them and you should choose on interpretability or downstream cost, not on the second decimal place. **A curve that keeps falling to the edge of the sweep.** This is the common real-world result and it usually means the true density is not a small number of Gaussians. The mixture is using components as basis functions to approximate a skewed or heavy-tailed shape, and each extra bump genuinely improves the fit. Two responses: transform the feature so it is closer to Gaussian (a log transform on prices often collapses ten components into two), or accept that you are density-estimating rather than segmenting, in which case the component count is a smoothing knob and not a count of populations. **A noisy curve with several local dips.** Almost always an initialisation problem. Increase the number of restarts per `K` and re-run before interpreting anything. ## Which one to prefer They optimise different targets, and the honest answer names both: - **BIC** is derived as an approximation to the model evidence and is *consistent*: if the data really were generated by a finite Gaussian mixture, BIC selects the true component count with probability approaching 1 as `n` grows. Choose BIC when the component count is the deliverable — when someone will act on "there are three segments". - **AIC** approximates expected out-of-sample predictive loss. It is not consistent and tends to over-select, but the extra components often help if you only care about how well the model predicts. Choose AIC when the mixture is a density estimator feeding something downstream. - **Held-out log-likelihood** is the third option and often the most convincing: split the data, fit on one part, evaluate the log-likelihood of the other. It makes no asymptotic assumptions and it is the argument stakeholders find easiest to accept. ## Caveats worth voicing Both criteria rest on regularity conditions that mixture models technically violate — component labels are exchangeable and the parameter space has boundaries where a component vanishes — so treat them as well-tested heuristics rather than theorems in this setting. Both assume the same `n` and the same data across every fit, so you cannot compare a criterion computed on one feature set against one computed on another. And neither knows anything about your business: a component count that is statistically justified and operationally unusable is a bad answer. The criterion narrows the field; a domain constraint on how many segments the organisation can actually act on closes it.