Under a true null hypothesis, what distribution do p-values follow?

## The claim Run a test on data generated with **no effect whatsoever**, record the p-value, and repeat 10,000 times. Plot the histogram. It is **flat**: roughly the same count in the bin from 0.00 to 0.05 as in the bin from 0.90 to 0.95. That flatness is the uniform distribution, and it is not an empirical accident — it follows from the definition of the p-value. ## Why it is uniform Let `T` be the test statistic and `F` the cumulative distribution function of `T` **under the null**. For a one-sided test in the upper tail, the p-value is ``` p = 1 - F(T) ``` The **probability-integral transform** says that if `T` is a continuous random variable with CDF `F`, then `F(T)` is uniformly distributed on the interval from 0 to 1. Since `1 - U` is uniform whenever `U` is, `p` is uniform too. The same argument works for two-sided tests, where the p-value is the total tail mass beyond the observed statistic in both directions. The crucial condition is that `F` must be the *correct* null distribution of `T`. Uniformity is a property of a well-specified test, not a universal law about p-values. ## Why it matters **It is what makes a threshold mean anything.** If null p-values are uniform, then `P(p <= 0.05 | H0) = 0.05` exactly. The rate at which a correct test produces small p-values on pure noise is known in advance and equals the threshold. Without uniformity, a threshold has no calibrated long-run meaning at all. **It gives you a diagnostic.** Uniformity is testable. Simulate under the null — permute labels, or generate synthetic data with the effect switched off — run the full analysis pipeline end to end, and look at the histogram. This catches an enormous class of bugs and assumption failures: - A histogram **skewed toward zero** means the test fires too readily on noise. Typical causes: dependence between observations treated as independent (clustered users, repeated measurements, time-correlated data), a variance estimate that is too small, or a leak between the units being compared. - A histogram **skewed toward one** means the test is conservative: it rarely produces small p-values even when it should. Discrete statistics with few possible outcomes do this naturally, as do some overly cautious variance estimators. - A **spike just below 0.05**, in a collection of *reported* results rather than simulated ones, is a fingerprint of selection: analyses were adjusted, subsets chosen, or results published preferentially until the number crossed the conventional line. Under a true null the density near 0.049 and near 0.051 is identical, so a discontinuity at the threshold cannot be produced by the test itself. ## The caveats a good answer includes **Discrete statistics.** With a discrete test statistic — small counts, exact tests on tables — the p-value can only take finitely many values, so it cannot be exactly uniform. The distribution is stochastically larger than uniform, meaning the test is conservative: the actual rate of p-values below 0.05 is at most 0.05, often noticeably less. **Composite nulls and approximations.** When the null does not pin down every parameter, or when the null distribution is an asymptotic approximation valid only for large samples, uniformity holds only approximately. In small samples the approximation can be poor in exactly the tail you care about. **Violated assumptions.** This is the practically important one. If observations are correlated but the test assumes independence, the true null distribution of the statistic is wider than the assumed one, the assumed CDF is wrong, and null p-values pile up near zero. The test then produces "significant" results on data containing no effect at all — and no amount of care in interpreting a single p-value will reveal it. Only the distribution of many null p-values will. ## How to say it in an interview State the result (uniform on 0 to 1 under a true null with a continuous statistic), name the mechanism (probability-integral transform: pushing a continuous variable through its own CDF gives a uniform), give the consequence (`P(p <= 0.05) = 0.05`, which is what makes a threshold calibrated), and then offer the diagnostic use — simulating under the null and inspecting the histogram is one of the most effective ways to validate an analysis pipeline before it is trusted.