If you run 20 independent hypothesis tests at alpha = 0.05, how likely is at least one false positive?

## The quantity being asked for A hypothesis test is set up so that, when the null hypothesis is true, the probability of wrongly rejecting it is at most alpha — conventionally 0.05. That is a per-test guarantee. It says nothing about what happens when you run the same procedure many times over. Suppose all 20 nulls are true (nothing real is going on anywhere) and the 20 tests are statistically independent. For a single test: - P(false positive) = 0.05 - P(no false positive) = 0.95 Because the tests are independent, the probability that *none* of the 20 produces a false positive is the product: ``` P(no false positive anywhere) = 0.95^20 = 0.3585 ``` So the probability of at least one false positive is the complement: ``` P(at least one) = 1 - 0.95^20 = 0.6415 ``` Roughly 64 percent. Running twenty null tests at the 5 percent level, you are more likely than not to end up with something that looks significant. ## Family-wise error rate The number just computed has a name: the **family-wise error rate (FWER)**, defined as the probability of making one or more type I errors across a specified family of tests. Contrast it with the per-comparison error rate, which is just alpha for each test taken alone. The whole multiple-comparisons problem is the gap between those two numbers: alpha controls the second, and the reader of your report almost always cares about the first. The general formula, for m independent tests all under true nulls, is: ``` FWER = 1 - (1 - alpha)^m ``` A few values at alpha = 0.05: m = 5 gives 23 percent, m = 10 gives 40 percent, m = 20 gives 64 percent, m = 100 gives 99.4 percent. The growth is fast at first and then saturates. By a hundred tests, at least one spurious hit is essentially guaranteed. ## Why the naive additive answer is wrong A common wrong answer is 20 x 0.05 = 1.00, i.e. certainty. Adding the alphas is the union bound, and the union bound is an upper bound, not an equality — it double-counts the scenarios where two or more tests fire at once. It happens to be a *useful* bound (the Bonferroni correction is built on exactly this inequality), but as an estimate of the actual FWER it overshoots, and it becomes absurd past m = 20 where it exceeds 1. The other common wrong answer is 5 percent — the belief that alpha somehow covers the whole analysis. It does not. Alpha is attached to one decision rule applied once. ## What independence buys, and what happens without it The 0.95^20 step needs independence. Real batches of tests are usually correlated: twenty metrics on the same users, twenty features that move together, twenty endpoints measured on the same patients. Positive correlation makes the tests fire together, which *reduces* the FWER relative to the independent case — in the extreme where all twenty tests are perfect duplicates of each other, the FWER is just 5 percent, because there is really only one test. So 64 percent is the independent-case figure, and correlated families sit somewhere between alpha and that number. What never happens is the FWER staying at 5 percent for genuinely distinct tests. Also note the conditioning: the formula assumes all 20 nulls are true. If only 5 of the 20 nulls are true and the other 15 effects are real, then only those 5 can produce a type I error, and the family-wise error rate is `1 - 0.95^5 = 0.23`. False positives can only come from true nulls. ## Why this matters in practice The scenario is not artificial. A dashboard with forty metrics, a screen of a thousand candidate features, a questionnaire with thirty items, a panel of gene-expression measurements — each is a family of tests being read as if each significant result stood alone. The canonical cartoon version is the lab that tests twenty colours of jelly bean against acne, finds that green beans are significant at p < 0.05, and puts *green jelly beans linked to acne* on the front page. Nineteen null results never appear. The response is to control error at the family level rather than the test level: shrink the per-test threshold so the family-wise rate lands back at 5 percent, or switch to a different, more permissive promise about the *proportion* of your discoveries that are false. Which of those you choose depends on how many tests you are running and how expensive a single false claim is. In an interview, the expected answer is the arithmetic (`1 - 0.95^20`, about 64 percent), the name of the quantity (family-wise error rate), and the recognition that alpha is a per-test guarantee that says nothing about a batch.