How should you report an A/B test with p = 0.3 without claiming 'no difference'?

## What p = 0.3 actually claims The p-value is the probability, computed assuming the true difference between the arms is exactly zero, of seeing a test statistic at least as extreme as the one observed. At 0.3, a result like this one is entirely ordinary under a no-difference world, so there is no basis for rejecting that hypothesis. The logical step that fails is the next one. Failing to find evidence against a hypothesis is not evidence for it. The test asked "are these data surprising if the effect is zero?" and got "no". It never asked "is the effect zero?", and the same data would have been unsurprising under many non-zero effects too. ## The reporting sentence that does work Write the estimate, the interval, and what the interval rules out: > Treatment showed **+0.6%** relative lift, 95% CI **-1.1% to +2.3%**. The data are compatible with anything from a 1.1% regression to a 2.3% gain; effects outside that range are not well supported. Everything a reader needs is present: the direction and size of what was seen, the precision, and the bound. Compare the two shorthands people reach for instead: - **"No difference"** — asserts a fact the test cannot establish, and quietly converts an absent finding into a permanent one. - **"Not significant"** — technically correct, decision-useless. It cannot distinguish a tight interval around zero from one that spans the whole roadmap. ## When you may say the effect is not meaningful There is a legitimate version of the no-effect claim, and it depends entirely on the interval, not the p-value. Decide in advance the band of effects too small to act on. If the whole interval falls inside that band, you can report that the experiment found no effect worth acting on, and that statement is supported. Two readouts make the difference concrete, both at p = 0.3: - `+0.1%, 95% CI [-0.4%, +0.6%]` against a band of plus or minus 1% — a real finding. Anything large enough to matter has been excluded. - `+1.5%, 95% CI [-4%, +7%]` against the same band — nothing has been established. A 7% win and a 4% loss are both fully compatible with these data. The p-values are the same and the conclusions are opposite. This is the single most useful thing to be able to explain about a non-significant result. ## Practical habits - **Lead with the interval, not the p-value.** If a template forces a p-value, put it after the interval. - **State the band you are judging against.** "No effect worth acting on" is meaningless until "worth acting on" has a number, and that number should come from the test plan rather than from the readout. - **Never write the point estimate alone.** "+0.6%" with no interval reads as a small win and will be repeated as one. - **Do not shop for a subgroup.** Slicing a null result until something clears significance manufactures findings; that whole practice belongs to a different discipline of pre-registration and correction, not to the readout of this metric. - **Record achieved precision in the log.** Six months on, the only defence against "we tested that and it did nothing" is a written interval showing what the test could have detected. ## Why this matters beyond pedantry Organisations accumulate beliefs from experiment write-ups. A null result written as "no difference" becomes institutional knowledge that the idea does not work, and the idea does not get retried — even when the original test was too small to have detected a large effect. Writing the interval instead costs one extra clause and keeps the record honest about what was and was not learned.