Why does slicing a flat A/B test result into many segments so often produce a false winner?

## The setup An A/B test randomises users between a control and a treatment and compares one primary metric. Suppose the all-up read comes back flat: the estimated lift is near zero and its confidence interval comfortably contains zero. A natural next move is to ask *for whom* it might have worked, and to cut the same data by device, country, tenure, acquisition channel, plan tier and so on. That move is where most invented findings on an experimentation team are born. ## Each cut is another test A significance test at level alpha = 0.05 is calibrated so that, **when there is truly no effect**, it wrongly declares a difference 5% of the time. A p-value is the probability of seeing a difference at least as large as the one observed **if the null hypothesis of no true difference were true**. That 5% is a per-test guarantee. It says nothing about a family of tests. Run `k` cuts on a truly flat experiment and, if the cuts were independent, the probability that at least one comes back significant is `1 - (1 - 0.05)^k`: - k = 5 -> about 23% - k = 10 -> about 40% - k = 20 -> about 64% - k = 30 -> about 79% The expected number of false wins is simply `k x 0.05`: one in twenty cuts, one and a half in thirty. Real segment cuts are not independent — Android overlaps with mobile, Brazil overlaps with Latin America, tenure bands are nested — so the exact family-wise error rate is somewhere below these numbers when the cuts are positively correlated. The arithmetic is an approximation, but the direction is not in doubt: the more cells you look at, the closer to certain a spurious winner becomes. A device x country x tenure grid of thirty cells is a machine for manufacturing one. ## Small cells make it worse in two ways **Wider intervals.** The standard error of a mean scales like `s / sqrt(n)`. A cell holding 1/30 of the sample has an interval roughly `sqrt(30)`, about 5.5 times, wider than the all-up read. Segment estimates are therefore dominated by noise even when the underlying effect is real. **Inflated winners.** Because the cell is noisy, only a *large* observed effect clears the significance threshold. Conditioning on having cleared it selects the draws that happened to land high. So the segments that look like wins carry estimates biased away from zero — often by a factor, not a few percent. This is the winner's-curse or exaggeration problem: even when a real effect exists in that segment, the number you would quote from the discovering test is an overstatement, and a replication will come back smaller. Sign errors are possible too: a cell can be significant in the wrong direction. ## What honest practice looks like 1. **Decide the cuts before the data.** A segment named in the design doc — say, a mobile-versus-desktop split written down before launch because the treatment changes a mobile layout — is a confirmatory test with a real error rate, provided you also planned the sample and the reporting rule. Everything decided after seeing results is exploratory. 2. **Count and disclose the cuts.** The readout should say how many segment views were examined, not just the one that paid off. A p-value of 0.03 out of thirty cuts is unremarkable; the same p-value on a single pre-planned cut means something. 3. **Adjust the bar for multiplicity.** When several segment readouts are treated as decisions, the significance threshold has to account for the family of tests rather than each test alone; the specific adjustment procedures are a topic of their own. 4. **Look at the whole grid, not the winner.** If the treatment really helps heavy users, you expect a gradient across tenure bands and a consistent sign across neighbouring cells, not one lone cell surrounded by noise. 5. **Require replication before shipping.** Treat the segment result as a hypothesis and run a fresh experiment powered for that segment specifically. Effects that survive a confirmatory test are worth acting on; the ones that evaporate cost you only the follow-up. ## The sentence to say in the interview "The flat all-up read is the result. The segment win is a hypothesis whose false-positive rate I cannot state without knowing how many cuts I ran, and whose effect size is inflated by the selection that surfaced it." That framing shows you understand both the multiplicity arithmetic and the estimation bias, which is what separates a candidate who has read about p-hacking from one who has actually had to say no to a product manager holding a slide.