How do you decide which set of tests counts as one family needing multiplicity correction?

## Why the question has no formula answer Every multiplicity procedure takes m as an input. Bonferroni divides by m, Holm's denominators count down from m, the false-discovery staircase is `(i/m)q`. None of them tells you what m is. That choice is entirely yours, it changes every conclusion, and it is not checkable from the data. This is the part of multiplicity that is a leadership problem rather than a computation. The orienting principle: **the family is the set of tests over which you are asserting an error guarantee**. If you are going to say *this result is significant* and expect a reader to act on it, the guarantee must cover every test that could have produced that sentence. If a reader would be equally happy with any of fifteen endpoints coming back positive, then all fifteen are in the family, because any of them could have generated the headline. ## Both failure modes are real **Too narrow.** The family is drawn around the tests that happened to look good, or redrawn after the fact so the correction is survivable. Correcting three tests when twenty were run gives a guarantee that means nothing — the arithmetic is right and the claim is still false. The tell is that m was determined after the results were known. **Too broad.** The family is every test anyone ran on the dataset, or every test in the quarter. Now m is in the hundreds, per-test thresholds are microscopic, and nothing is ever significant. This is often presented as rigour, but a procedure that can never detect anything is not conservative — it has simply converted all of its type I error risk into type II error risk and stopped being informative. Deciding to over-correct is a decision to miss real effects, and it should be defended as such rather than assumed to be free. ## The structure that actually works The convention that survives contact with real studies is a **hierarchy of claims**, fixed before analysis: - **Primary.** One, occasionally two, tests that the study exists to answer. These carry the confirmatory claim and get the full error budget — a single primary endpoint is tested at 0.05, not at 0.05 divided by the number of things you also measured. - **Secondary, confirmatory.** A named, ordered set that may also support claims. These are corrected as a family, or tested in a fixed sequence where each is only examined if the previous one succeeded, which spends no extra alpha. - **Exploratory.** Everything else. Reported without correction *and* explicitly labelled exploratory, generating hypotheses rather than conclusions. The label is the honest alternative to both silence and false confirmation. This works because it makes the family small where the stakes are high and admits that the rest is not a claim at all. It also removes the incentive to game m: the boundary was set when nobody knew which way the results would fall. ## Questions worth asking when drawing the boundary - **What single sentence will appear in the summary?** Everything that could have produced that sentence belongs in the family. - **Who bears the cost of a false positive?** A false claim that triggers a costly programme, a regulatory filing, or a public statement argues for family-wise control on a small primary family. A false lead on a shortlist that goes to a cheap validation step argues for a false-discovery criterion on a large family. - **What is the cost of a miss?** A screening study that returns nothing has failed. A confirmatory study that returns a false positive has failed worse. The asymmetry decides the criterion. - **Are these tests answering one question or many?** Fifteen ways of measuring the same underlying effect is closer to one claim than fifteen. Fifteen unrelated endpoints is genuinely fifteen. - **Can I write m down before looking?** If not, the correction will not mean what it says. ## Making the decision auditable Whatever you choose, the analysis plan should carry four things: the family membership, its size m, the criterion (family-wise or false-discovery), and the level. Publish those alongside the results. A reader can then disagree with your boundary and re-derive their own conclusion, which is the whole point — the choice is a judgment, and judgments are defensible when they are visible and made in advance. An adjusted p-value with no stated family is uninterpretable, because the same number means different things depending on what m was. At organisational scale, the useful move is to make this a standing convention rather than a per-analyst decision: a house rule about how many primary endpoints an experiment may declare, which criterion applies to screening work versus decision work, and a requirement that the plan is written before data is examined. That converts a repeated argument into policy, and it is what a lead is expected to own.