Why run one ANOVA across five groups instead of all 10 pairwise t-tests?

## The arithmetic of error inflation Alpha is a per-test promise: run one test when the null is true, and you have a 5% chance of calling it significant anyway. That promise says nothing about a *collection* of tests. With k groups the number of distinct pairs is `k * (k-1) / 2`. For k = 5 that is 10 pairs. Suppose every null is true — all five groups are identical — and you test each pair at alpha = 0.05. Treating the tests as independent: ``` P(no false positive) = 0.95^10 = 0.5987 P(at least one false positive) = 1 - 0.5987 = 0.4013 ``` About **40%**. You believed you were running at 5% risk and you were actually closer to a coin flip. And the growth is fast: three groups give 3 pairs and about 14%; six groups give 15 pairs and about 54%. ## The independence caveat, and why it does not rescue you The 10 pairwise comparisons among five groups are **not** independent — each group appears in four of them, so the tests share data and their results are positively correlated. The exact probability of at least one false positive is therefore somewhat lower than 0.40. Mentioning this is a good sign in an interview, as long as you land the conclusion correctly: the correlation dampens the inflation, it does not remove it. The realised error rate is still multiples of the nominal 5%. ## What ANOVA does instead A one-way ANOVA asks a single question — "are all five means equal?" — and produces a single p-value evaluated at a single alpha. Whatever risk you set is the risk you take. That is the headline reason, but there is a second, less-quoted one: **A pooled error estimate.** Every pairwise test estimates the noise from only the two groups involved. ANOVA estimates it once from all five, so the denominator of the statistic rests on `N-k` degrees of freedom rather than the much smaller count a single pair provides. A better-estimated denominator means a tighter reference distribution and more power to detect a real effect — assuming the equal-variance assumption is reasonable, which is precisely what licenses the pooling. ## "Haven't you just moved the problem?" A sharp interviewer will push: if the ANOVA is significant, you still want to know which groups differ, and now you are back to pairwise comparisons. True — and that is why the pairwise stage after a significant omnibus test uses a **post-hoc procedure** whose critical values are built for the whole family of comparisons at once, rather than a stack of ordinary two-sample tests. Tukey's HSD is the standard choice for "all pairs" after a one-way ANOVA. The two-stage structure is deliberate: one gate, then a controlled set of follow-ups. ## When pairwise comparisons are actually the right call The omnibus test is not always the question you care about. If the study was designed around a small set of **pre-planned comparisons** — each of three treatments against a single control, three comparisons rather than six — then testing those directly, with an appropriate adjustment for the three, is more powerful and more honest than sweeping across all pairs. Plan the comparisons before seeing the data; choosing them afterwards, based on which sample means look furthest apart, silently reintroduces exactly the selection effect the whole discipline exists to prevent. ## The sentence to have ready "Ten pairwise tests at 5% each carry about a 40% chance of at least one false positive, so the error rate I quote stops being the error rate I have. One ANOVA answers the omnibus question at the alpha I chose, pools a better variance estimate across all groups, and if it fires I follow it with a post-hoc procedure that keeps the pairwise comparisons controlled as a family."