A two-sample test returns p = 0.20. Does that show there is no effect?

## The asymmetry at the heart of testing A hypothesis test is deliberately one-directional. It sets up a null — say, no difference between two groups — computes how surprising the observed data would be if that null held, and reports that surprise as a p-value. When the p-value is small, the data are hard to reconcile with the null and you have an argument against it. When the p-value is large, all you have learned is that the data are **not surprising** under the null. That is a much weaker statement than it sounds, because the same data are usually also not surprising under a whole neighbourhood of *other* hypotheses. A difference of 0 is compatible with the data; so, very often, is a difference of 2%, or 5%, or whatever the interval reaches. The test only ever asked about one of those values, so it cannot single it out as the truth. The slogan is: **absence of evidence is not evidence of absence.** p = 0.20 is an absence of evidence against the null. It is not evidence for it. ## What to do instead: read the interval The practical fix is to stop reasoning from the p-value alone and look at the **interval estimate** around the observed difference — the range of values for the true difference that the data are consistent with. Three shapes, three very different conclusions: **1. Wide interval spanning meaningful values in both directions.** For example, the difference in conversion could plausibly be anywhere from -3 points to +4 points. The p-value is large, but the study is simply **uninformative**: it has not ruled out an important improvement or an important harm. The honest report is "inconclusive" and the next question is what it would take to get a usable answer. **2. Narrow interval sitting entirely inside the range nobody cares about.** For example, the difference is somewhere between -0.1 and +0.1 points, and the team agreed beforehand that anything under 0.5 points is irrelevant. Here you *can* make a positive claim — practical equivalence — but note where it came from: the narrowness of the interval relative to a pre-agreed threshold of importance, not the size of the p-value. This is the only route to a defensible "no meaningful difference" statement. **3. Narrow interval sitting slightly off zero but inside the irrelevant range.** Same conclusion as (2). Whether the interval happens to include exactly zero is not the interesting fact; whether it includes anything worth acting on is. Notice that shapes (1) and (2) can both produce p = 0.20. The p-value alone cannot distinguish "we learned nothing" from "we learned the effect is negligible", and those two conclusions lead to completely different decisions. ## Language discipline How a large p-value is written down determines how it is used downstream. Compare: - **"There is no difference between the variants."** A claim about the world the data do not support. - **"The test was not significant, so we are keeping the current version."** Better, but it hides whether the study was informative. - **"The estimated difference was +0.4 points, with the data consistent with anything from -1.2 to +2.0 points. The study cannot distinguish a meaningful gain from a meaningful loss, so we are keeping the current version and will need a larger sample to decide."** This is what a senior analyst writes: estimate, uncertainty, and an explicit statement of what remains unresolved. ## Where this bites in practice - **Safety and regression checks.** "We tested for a latency regression and it wasn't significant" is often used to close a ticket. If the interval reaches +80 ms, the check has not demonstrated safety at all. - **Killing initiatives.** A promising change is abandoned on a large p-value from a small trial, when the interval was consistent with a substantial win. - **Claiming parity.** Two suppliers, two models, two flows declared "the same" because a test did not reject. Equivalence needs to be argued from an interval and a pre-agreed threshold, and it needs to be planned before the data arrive, not decided afterwards. ## The interview answer Say no, and say why: the test is asymmetric and can only ever provide evidence against the null. Then immediately go to the interval, distinguish "uninformative" from "precisely estimated as negligible", and note that a genuine claim of no meaningful difference requires stating in advance what magnitude would count as meaningful. That progression — refuse the fallacy, then supply the constructive alternative — is what separates a senior answer from a recited slogan.