When can you ignore the marginal likelihood in Bayes' rule, and when must you actually compute it?

## What the quantity is Write Bayes rule for a parameter `theta` under a model `M`: `p(theta | data, M) = p(theta | M) * p(data | theta, M) / p(data | M)` The denominator `p(data | M)` is the marginal likelihood, usually called the evidence. It is defined by integrating the parameter out: `p(data | M) = integral of p(theta | M) * p(data | theta, M) over theta` Two readings are worth holding at once. Mechanically it is the normalising constant that makes the posterior integrate to 1. Interpretively it is the **prior predictive probability of the data**: before any data arrived, the model plus its prior implied a distribution over datasets, and this is the value that distribution places on the dataset you actually got. ## When it is safe to ignore Inside one model the evidence is a single number, identical at every parameter value. Everything that depends only on relative posterior values is untouched by it: - The **shape** of the posterior — where the peak is, how wide the bulk is, whether there are multiple modes. - **Ratios** of posterior density at two parameter values, where the constant cancels. - Any **point summary or interval** derived from the normalised posterior, since normalising restores the constant anyway. - **Numerical normalisation on a grid**: evaluate prior times likelihood at each grid point and divide by the total; that total, times the grid spacing, *is* an approximation of the evidence. You did not avoid it so much as obtain it as a by-product. This is why the working form `posterior is proportional to prior times likelihood` is enough for the overwhelming majority of single-model analysis. ## When you must compute it The evidence becomes the object of interest as soon as the comparison is between models rather than between parameter values. For two models `M1` and `M2`, the Bayes factor is `BF = p(data | M1) / p(data | M2)` Both terms are evidences. Nothing cancels: the number that was a shared constant inside one model is now the whole calculation. If you also carry prior probabilities over the models themselves, posterior model odds are prior odds times the Bayes factor, so again the evidences are required. ## Why the evidence penalises complexity Because the evidence averages the likelihood over the prior rather than maximising it, a model that spreads its prior over a large parameter space pays for it. Suppose a flexible model can fit the data superbly at one narrow region of parameter space but fits badly across the rest of the region its prior covers. The maximum likelihood is high, but the average is dragged down by all the parameter values that predicted poorly. A tighter model whose prior concentrates on decent parameter values can win the comparison even with a lower peak fit. This is often called an automatic Occam effect: it arrives from the arithmetic rather than from an added penalty term. ## The prior-sensitivity trap The same mechanism produces the classic gotcha. Take a prior and widen it substantially — spread the same total mass over a much wider range. Once you have plenty of data, the posterior barely notices, because the likelihood concentrates the mass anyway and the prior is nearly flat over the region that matters. The evidence, on the other hand, drops roughly in proportion to the widening, because most of the prior mass now sits over parameter values that explain the data poorly and drags the average down. So two analysts with essentially the same posterior can report materially different Bayes factors. The practical rule: a diffuse prior that is harmless for estimation is *not* harmless for model comparison, and any Bayes factor should be reported with the priors that produced it and a sensitivity check. ## Why it is hard The evidence is an integral over the whole parameter space, weighted by the prior. In one or two dimensions a grid handles it. As parameters multiply, the region where prior times likelihood is appreciable becomes a vanishing fraction of the space being integrated, so naive numerical integration wastes essentially all of its effort. The difficulty of this integral, rather than any conceptual subtlety, is the reason model comparison is treated as a harder task than estimation. ## What to say in an interview The crisp answer has three beats: it is a constant within a model so it drops out of everything relative; it is the entire quantity in a model comparison because a Bayes factor is a ratio of evidences; and it is an average over the prior, which makes it both a built-in complexity penalty and unusually prior-sensitive.