An A/B test reads +0.4% lift, 95% CI [-2.0%, +2.8%], planned MDE 2% — is that flat?

## The readout - Point estimate: **+0.4%** relative lift on the primary metric. - 95% confidence interval: **[-2.0%, +2.8%]**. - Planned minimum detectable effect: **2%** — the smallest lift the team decided in advance was worth acting on, and the size the test was powered to detect. Zero sits inside the interval, so the result is not significant at the 5% level. The tempting shorthand is "the feature did nothing". That shorthand is wrong here, and the reason is visible without any further computation. ## Flat versus inconclusive These are two different findings that share a p-value. **Conclusively flat** means the experiment measured the lift precisely enough to rule out anything you would have acted on. Concretely: the entire interval lies inside the band of effects you consider not worth a decision — upper bound below the MDE, lower bound above whatever regression you would treat as harm. An interval of `[-0.3%, +0.5%]` against a 2% MDE is flat. It says: whatever this feature does, it is smaller than the smallest thing we care about. **Inconclusive** means the interval spans the decision boundary. Here the upper end, +2.8%, is 40% larger than the MDE. If the true lift were exactly the 2% the roadmap was built on, this readout would be an entirely unremarkable outcome. The lower end, -2.0%, is a mirror-image problem: a loss of the same magnitude as the win being chased is equally unexcluded. The experiment has not distinguished the three worlds — good, nothing, bad — that would each lead somewhere different. The general rule, worth being able to state cleanly: **compare the interval's endpoints to the thresholds that change the decision, not to zero.** Zero is rarely the boundary anyone acts on. ## Why the precision missed the plan A test powered for a 2% MDE should return an interval roughly of that half-width. This one has a half-width of about 2.4 percentage points of relative lift — wider than planned. That gap is itself a finding, and it has a short list of causes worth checking before any decision: - **Short sample.** The test was stopped early, or the traffic forecast that fed the power calculation was optimistic. Check actual exposures against the planned number. - **Variance underestimated.** The power calculation assumed a metric variance from historical data; if the metric is skewed or driven by a heavy tail (revenue per user is the usual offender), realised variance runs higher and every interval widens. - **Dilution of the audience.** If a large share of assigned users never reached the changed surface, they contribute noise and no signal, and the measured effect shrinks toward zero while the interval stays wide. - **Metric or unit mismatch.** Randomising by user but analysing by session, or the reverse, distorts the standard error. - **Imbalance or instrumentation loss.** A split that did not come out as designed, or logging that dropped events in one arm, both degrade precision. ## What to do **Extend, if extending is honest.** More exposures shrink the interval. This is legitimate only if the extension was planned as an option with an appropriate stopping rule, or the test is simply restarted with the correct sample size — peeking repeatedly and stopping the moment an interval clears zero inflates the wrong-decision rate. **Reduce variance instead of buying traffic.** Where the surface cannot deliver more users, precision can sometimes be recovered by analysing a less noisy primary metric, trimming or capping a heavy tail by a pre-declared rule, or moving to a triggered population if the trigger is properly logged. **Or accept that this question will not be answered by this test.** Low-traffic surfaces often cannot support a 2% MDE at all. Then the honest output is: the experiment was uninformative, and the decision falls back to cost, strategy, and qualitative evidence. That is a defensible call. What is not defensible is laundering it as "we tested it and it did nothing". ## The reporting trap Inconclusive tests get written up as null results because null results are tidy. Six months later somebody cites "we tested that, it was flat" as a reason not to revisit the idea, and the organisation has acquired a false belief from an experiment that established nothing. Record inconclusive readouts with the interval attached and the achieved precision stated, so the next reader can see what the test could and could not have detected.