Why can a Gaussian mixture's log-likelihood run to infinity during EM, and how do you stop it?

## The singularity, precisely For a Gaussian mixture the likelihood of the data is a product over points of `sum_k pi_k * N(x_i | mu_k, Sigma_k)`. Now do the following thought experiment. Pick any single data point `x*`. Set component 1's mean exactly to `x*` and shrink its covariance toward the zero matrix. The Gaussian density has the determinant of the covariance in the denominator under a square root. As the covariance shrinks, the density evaluated *at its own centre* grows without bound. So the term for `x*` in the product contains `pi_1 * N(x* | x*, Sigma_1)`, which diverges. Every other point still contributes a finite positive amount from the remaining components. The product therefore diverges, and the log-likelihood goes to `+infinity`. This is not a numerical artefact. It is a genuine property of the objective: **the maximum-likelihood problem for a Gaussian mixture with unconstrained covariances has no global maximum**. The supremum is infinite and it is attained only in a degenerate limit. Any "solution" you actually use is a well-behaved *local* optimum, which is a slightly uncomfortable but standard state of affairs. A single point is the cleanest illustration, but the practical trigger is usually a small clump: three near-duplicate rows — the same customer recorded three times, a sensor stuck on one reading, a default value repeated — let a component shrink onto them with a covariance that is singular in every direction where the rows agree exactly. ## Why EM in particular falls in EM has a monotone-likelihood guarantee, which sounds protective and is not. It guarantees the objective never decreases; a run marching toward a singularity is *satisfying* that guarantee perfectly. Nothing in the algorithm knows the difference between climbing toward a good fit and climbing into a spike. The mechanics are self-reinforcing. Once a component's covariance is small, its density at nearby rows becomes enormous, so the E step gives it responsibility ~1 for those rows and ~0 for everything else. Its effective count `N_k` collapses to about three. The M step then computes the covariance as the responsibility-weighted scatter of three points about their own mean — which, in `d` dimensions with `d >= 3`, is singular or nearly so. The next E step sharpens further. The loop tightens each sweep. ## Recognising it The symptoms are specific enough to alert on: - A component whose mixing weight is a tiny fraction of a percent, or whose effective count `N_k` is smaller than the dimension. - A covariance determinant approaching zero, or a condition number exploding; equivalently a near-zero eigenvalue. - A log-likelihood that rises smoothly for many sweeps and then leaps by orders of magnitude. - Refits from different initialisations that disagree wildly, with the "best" log-likelihood belonging to an obviously useless partition. - A numerical failure when inverting the covariance — often the first thing you actually see. Be suspicious of a fit that scores far better than every other run. On this objective, a dramatically higher log-likelihood is more often a collapse than a discovery. ## The fixes **Covariance regularisation (the default).** Add a small positive constant to the diagonal of every covariance in the M step. This puts a floor under every eigenvalue, so no direction can shrink to zero, the determinant stays bounded away from zero, and the singularity is removed from the parameter space entirely. The constant should be small relative to the feature variances — which is another argument for standardising features first, since it makes one floor value meaningful across all of them. This is equivalent to a mild prior on the covariance, and it changes the objective slightly: you are maximising a penalised likelihood that actually has a maximum. **Restrict the covariance form.** Diagonal or spherical covariances are much harder to collapse, because a spherical component has a single variance to drive to zero and needs the clump to be degenerate in *every* direction at once. This costs modelling flexibility, so it is a tradeoff rather than a free win. **Fix the data.** Deduplicate exact and near-exact rows, and look hard at constant or near-constant features — a column that is the same value for 90 percent of rows invites a collapse along that axis. Dropping a zero-variance feature removes a whole direction in which any component can degenerate. **Detect and restart.** Watch each component's effective count and covariance determinant during the run; when one crosses a threshold, reinitialise that component at a random point or drop it and continue. Cruder than regularisation but effective, and it keeps you from silently shipping a collapsed fit. **Reduce the component count.** Collapses become common when you ask for more components than the data supports. If the fit is stable at 4 components and collapses at 9, that is information about the component count, not just a numerical nuisance. **Go Bayesian in spirit.** Placing a prior on the covariance and maximising the posterior rather than the likelihood is the principled version of the ridge fix: the prior vanishes as the covariance shrinks, so the degenerate solution is no longer optimal. ## The interview point The answer that lands states three things: the likelihood is genuinely *unbounded above* (not merely ill-conditioned), EM's monotone guarantee offers no protection because a collapse increases the likelihood, and the standard fix is a floor on the covariance. Candidates who describe this as an overfitting problem or a convergence-speed problem have not understood the shape of the objective.