In an A/B test, what is the difference between a pre-registered segment cut and a post-hoc one?

## Same numbers, different evidential status A segment cut is just a filter on the experiment's users — mobile only, Brazil only, users with more than 90 days of tenure. Nothing about the arithmetic changes based on when you decided to apply it. What changes is the **error rate you are entitled to quote**. A significance test at alpha = 0.05 promises that, when the treatment truly does nothing, it will declare a difference at most 5% of the time. That promise holds for a test you committed to before looking. It does not hold for a test you chose *because* its numbers looked interesting, because the choosing step already used the data. ## Pre-registered (confirmatory) cuts A pre-registered cut is one that appears in the experiment's design document before the first user is bucketed. A well-formed one carries four things: - **The cut itself** — mobile versus desktop, stated exactly, including how the boundary is defined. - **A reason** — a mechanism you believe in advance. "The treatment changes a layout that only mobile users see" is a reason; "mobile is a segment we have" is not. - **A sample plan** — enough traffic inside the segment to detect an effect worth acting on. Splitting an experiment powered for the whole population usually leaves each half underpowered, so this often means running longer or targeting the experiment at the segment. - **A decision rule** — what launch or no-launch action the segment read licenses, written down before it can be argued about. Because the cut was fixed in advance, its result is a genuine test. If you pre-register several, they still form a family and the threshold has to reflect that, but each one is at least an honest test rather than a selected one. ## Post-hoc (exploratory) cuts A post-hoc cut is anything decided after the data are visible: the flat result you sliced afterwards, the dimension you added because a colleague asked, the tenure band you moved from 30 days to 14 days because the split looked better there. These are not illegitimate — most useful heterogeneity is first noticed this way. The error is in reporting them with the vocabulary of a confirmatory test. Three things go wrong when a post-hoc cut is treated as a finding: 1. **The stated error rate is fiction.** Twenty cuts on a null effect give roughly one significant result by chance, so "p = 0.04" from a hunt carries far less evidence than the same number from a planned test. 2. **The effect size is inflated.** The cut surfaced because its estimate was large; selecting on largeness biases the estimate away from zero, so a replication comes back smaller. 3. **The story writes itself.** Any segment can be rationalised after the fact — a mechanism invented to explain a result is not evidence for it, because you would have invented a different mechanism for whichever cell happened to win. ## How to handle both in one readout A readout that stays honest usually looks like this. The primary all-up metric with its interval is the decision. Below it, a short block of pre-registered segment reads, each labelled as planned. Below that, an explicitly labelled exploratory block that states how many cuts were examined and presents the interesting ones as candidates for a follow-up rather than as results. Nothing in the exploratory block moves a launch decision on its own. The practical discipline that makes this work is small: agree the segment list at design review, and let the analysis tool record which cuts were actually pulled. A cut count in the readout costs nothing and changes how a reader weighs everything under it. ## The junior-level version If you remember one line: **planned cuts test a hypothesis, discovered cuts generate one**. Both belong in the readout; only the first belongs in the decision.