A KS test on 1,000,000 sessions returns p below 1e-16 for a shift nobody would notice - how do you decide whether to act?

## Why the p-value stopped being informative A p-value answers one question: how surprising is data this extreme if the two distributions were identical? At a million observations per group, that question has almost no practical content, because two real-world populations are essentially never *identical*. Different weeks, different traffic mixes, different device populations - some difference always exists. With enough data the test finds it. The arithmetic is explicit. The large-sample 5% rejection rule for the two-sample maximum-gap statistic is ``` D > 1.36 * sqrt( (n + m) / (n * m) ) ``` With `n = m = 1,000,000` that threshold is about 0.0019. A gap of two-tenths of a percentage point in cumulative proportion - nothing anyone would design a change around - is enough to reject. Push to ten million and the threshold falls by another factor of about three. **Statistical significance is a statement about sample size at least as much as about the world.** ## Switch to the magnitude The statistic you already computed is the effect size. `D` lives on 0 to 1 and reads in plain language: the largest difference, at any cutoff, between the fraction of sessions in group A at or below that cutoff and the fraction in group B. `D = 0.003` means that at the worst point, the two builds disagree by three sessions in a thousand. That sentence is auditable by a product owner in a way that 'p < 1e-16' is not. Three things to put in the report: 1. **`D` with the value at which it occurred.** 'The biggest divergence is 0.003, at around 240 ms.' 2. **The two empirical CDFs overlaid.** With a million points the curves are smooth, and a reader can see instantly whether they are essentially the same line or genuinely separated. 3. **Differences at the decision-relevant cutoffs.** If the SLO is 'under 500 ms', the number that matters is the change in the fraction of sessions above 500 ms, not the maximum gap wherever it happens to fall. ## Pre-commit to a materiality threshold The defensible way to avoid arguing about a result after seeing it is to fix the bar before. Write down, in advance: what magnitude of change would alter the decision we are about to make? That number comes from the economics, not from statistics - the cost of rolling back, the revenue attached to a percentage point, the risk tolerance of the SLO. Once that threshold exists, the analysis becomes a comparison of two numbers rather than a debate about a p-value. You can go further and invert the test into an **equivalence** framing: instead of asking whether any difference exists, ask whether the data lets you rule out a difference larger than the threshold. That flips the burden of proof to where the decision actually needs it - a large p-value from a test of exact equality is not evidence of sameness, whereas an equivalence conclusion is. ## The move that looks clever and is not Someone will suggest sub-sampling to a few thousand rows until the p-value 'behaves'. Refuse it. That does not make the difference smaller, it makes the estimate of the difference noisier. You have discarded information in order to lose the ability to detect the very thing you were measuring, and the p-value you produce is a function of an arbitrary sub-sample size you chose. If the p-value is embarrassing, the fix is to stop using the p-value as the decision rule, not to degrade the data until it agrees with you. The mirror-image error is silent multiplicity: running the test across dozens of metrics on huge tables and reporting whichever ones came back significant. At this sample size, essentially all of them will. ## What to tell stakeholders The useful sentence has this shape: *'The distributions are statistically distinguishable - at this volume they always will be - and the largest difference anywhere is 0.3 percentage points, well under the 3 points we agreed would matter. Recommendation: no action.'* It reports the result honestly, defuses the tiny p-value, and connects to a pre-agreed bar. The converse case matters just as much. If `D` is 0.14 with a gap concentrated beyond the 95th percentile, the same discipline says act, and act on the tail - even though the p-value looks no different from the trivial case. ## The organisational lesson At large scale, hypothesis tests of exact equality stop being decision procedures and become sample-size detectors. A team that keeps them as its default alerting rule will drown in true-but-irrelevant findings and gradually learn to ignore all of them, including the important ones. The durable fix is cultural: **every comparison ships with an effect size and a pre-agreed threshold for what counts as material**, and significance is at best a gate, never the verdict.