How do you tell a real subgroup effect in an A/B test from post-hoc story fitting?

## The problem with stories Human beings are extremely good at explaining any pattern after seeing it. Told that a checkout change worked best for long-tenure users on tablets in one market, a competent product person will produce a plausible mechanism within a minute — and would have produced an equally plausible one for the opposite cell. That is why a narrative attached to a discovered segment is worth almost nothing as evidence. The question is what evidence *does* discriminate. ## Four discriminating tests ### 1. Was the mechanism stated before the data? A mechanism argued in the design doc — "this only changes the mobile layout, so we expect the effect to concentrate on mobile" — is a prediction, and predictions can fail. A mechanism written after the readout is a description, and descriptions cannot fail. Ask when the sentence was written. Timestamped design docs make this checkable rather than a matter of memory. ### 2. Test the interaction, not the two halves The most common technical error in subgroup claims is comparing significance across segments: "it was significant for new devices, p = 0.02, and not significant for old ones, p = 0.11, so the effect is device-dependent". That reasoning is invalid. **The difference between significant and non-significant is not itself statistically significant.** Two estimates can sit either side of a threshold while their difference has an interval comfortably containing zero — especially when the segments have different sample sizes, which alone shifts p-values without any change in the underlying effect. The correct claim is about the *difference of effects*, so the correct test is on that difference — the interaction between the treatment indicator and the segment indicator. Estimate the difference in lifts with its own interval and judge that. Interaction tests are also inherently less powerful than tests of a main effect: detecting an interaction of a given size typically needs substantially more data than detecting a main effect of the same size. That is why so many subgroup claims are underpowered even when they are correctly specified. ### 3. Look for coherence across the grid A real heterogeneity usually leaves a pattern. If the effect grows with tenure, tenure bands should show a gradient rather than one band lit up and its neighbours flat. If the effect is about screen size, phones and tablets should point the same way. A single isolated cell surrounded by cells with the opposite sign is the signature of noise, because noise has no reason to respect adjacency and a real mechanism usually does. Ordered segments give you this check for free; unordered ones (country, channel) give you less, which is another reason to be more sceptical there. ### 4. Replicate before you act The decisive test is a fresh experiment: declare the segment in advance, power the run for that segment specifically rather than for the whole population, and pre-commit to the decision rule. Two things typically happen. Genuine effects replicate but come back smaller, because the discovering estimate was selected for being extreme and the replication is not. Manufactured effects vanish. Both outcomes are informative and both are cheap relative to shipping a targeted change on a phantom. ## Practical guardrails for a readout - Report the number of cuts examined alongside any segment claim, so a reader can weigh it. - Quote intervals on the *difference* between segments, not two separate lifts side by side. - Keep exploratory findings in a clearly labelled section that cannot, by policy, carry a launch decision. - Write the mechanism down at design time even when you are unsure — a wrong prediction that you honour is more valuable than a right story invented later. - Prefer a small list of pre-declared, well-powered cuts to a large grid, since power, not curiosity, is the binding constraint. ## Answering this in an interview The answer that lands names the interaction-test point explicitly, because most candidates do not. Say the sentence — significant here and not there is not evidence of a difference — and then give the replication requirement as the operational rule. Mention that you expect a real effect to shrink on replication rather than hold at its discovered size; that single detail signals you have actually watched subgroup claims survive contact with a second experiment.