If you stop an A/B test at the first significant look, what happens to the measured effect size?

## Two separate damages Most discussions of optional stopping stop at the false-positive rate — the chance of declaring a winner when the arms are identical. That is real, but there is a second, quieter damage that survives even when the effect **is** real: the number you report is too big. This matters commercially. A false positive gets caught eventually when the metric does not move. An inflated true positive never gets caught — the feature did help, just by a third of what the readout promised, and the forecast built on it silently misses. ## The selection mechanism Declaring significance means the observed difference cleared roughly a fixed number of standard errors: `observed difference > c * SE`. The standard error shrinks as the sample grows, so early in a test that bar is a **large absolute effect**. Late in the test, the same bar is a small one. Now consider what it means to stop at the first crossing. You are not sampling the estimate at a moment chosen independently of the data — you are sampling it at the moment it was most extreme, and specifically at a moment when it exceeded a high bar. Conditioning on "the estimate was big enough to cross" selects sample paths where the noise pushed in your favour. The average of those selected estimates sits above the true effect. This is a selection effect, not a computational error: the arithmetic on the day was correct, and the number is still too large. The severity scales in an intuitive way: - **Earlier stops are worse.** The bar in absolute terms is higher, so more of the crossing has to be supplied by luck. - **Small true effects are worse.** If the truth is a 0.2% lift and the early bar is a 2% lift, essentially all of the observed 2% is noise. - **Low-powered tests are worse.** When the design had little chance of detecting the true effect at all, the crossings that do occur are dominated by favourable noise. In the extreme — a genuinely tiny effect detected by a badly under-powered early look — the selected estimate can even carry the **wrong sign**, because the only paths that cross are extreme excursions and some of those come from the opposite direction of the truth. ## Why teams misread the follow-up When a stopped-early win is re-measured later and comes back smaller, the usual explanation offered is that the effect wore off, or that the launch was diluted, or that the second test was underpowered. Sometimes true. But the mechanical prediction of optional stopping is exactly this: **the first, selected estimate was inflated, and the second, unselected one is closer to the truth.** A candidate who can say "before I reach for behavioural explanations, I would ask whether the first estimate was taken at a stopping time chosen by the data" is demonstrating the judgment being probed here. The sharpest illustration is a stream where the estimate is watched over time: a test that would have been declared a solid win on day 3 and, from the very same accumulating data, read as a loss on day 10. Nothing about the world changed between those readings. What changed is that the day-3 reading was taken at a moment selected for being impressive, and the day-10 one was not. ## Practical consequences **Treat an early-stopped estimate as an upper bound.** If a decision has already been made on it, that is water under the bridge, but do not let the magnitude enter a forecast or a roadmap projection unqualified. **Re-measure before you plan around it.** A follow-up test at a fixed horizon, or a holdback measured after launch, gives an estimate that was not selected for being large. Expect it to come in lower, and say so in advance so the shrinkage does not read as a failure of the follow-up. **Audit the portfolio, not the test.** The strongest evidence that a team has an optional-stopping problem is aggregate: sum the claimed lifts from a year of shipped wins and compare against the actual movement in the top-line metric. A large gap that cannot be explained by interaction or seasonality is the signature of systematically inflated estimates. **Do not try to fix it by shrinking arbitrarily.** Halving every early-stopped estimate because it feels inflated is not a correction; the bias depends on how early you stopped, the true effect and the noise, none of which you know. The honest options are to re-measure, or to have designed for early stopping in the first place so that the estimate has a defined correction available. ## What an interviewer is listening for They want you to separate the two harms — calibration of the decision and bias of the estimate — and to notice that the second one is the expensive one at a company that ships a lot of experiments. A candidate who only recites "peeking inflates false positives" has half the picture; the other half is why the wins that *were* real still did not add up.