How do you find the root cause of a sample ratio mismatch in a running A/B test?

## The mental model: counts as a funnel Every user in an experiment passes through stages, and users can be lost at each one: 1. **Assigned** — the platform decides which arm this user belongs to. 2. **Exposed** — the user actually receives the experience and an event records it. 3. **Analysed** — the user survives whatever cleaning the analysis pipeline applies. A sample ratio mismatch is a claim about the ratio at whichever stage you measured. The diagnostic move is to compute the ratio **at every stage** and find the first one where it breaks. That single step turns an open-ended hunt into a bounded one. ## Stage 1: skewed at assignment If the counts are already off at the moment of bucketing, nothing downstream is to blame. This is the rarest and most serious case, and it belongs to the assignment service itself. Confirm it on raw assignment records that no filter has touched, because measuring "assignment" through a table that has already been joined or deduplicated will mislead you. ## Stage 2: balanced at assignment, skewed at exposure This is where most real mismatches live, and the mechanisms are physical rather than statistical. **The extra redirect.** The treatment variant is served behind a redirect — an additional network round trip before the experiment page renders. Every hop loses users: slow connections time out, impatient users close the tab, some clients mishandle the hop. The users who survive the redirect are systematically faster and more patient than the control arm's, so the arm is both smaller and different. The tell is that the shortfall concentrates in mobile traffic and slow-network regions, and that it appears immediately at launch rather than drifting in. **The one-arm crash.** The treatment code path crashes on a subset of clients — an old OS version, an unusual screen size, a device without some capability. Those users never fire the exposure event, so they vanish from the treatment count entirely. The tell is razor-sharp segmentation: the ratio is fine everywhere except one OS version band, where treatment is missing almost completely. **Asymmetric event timing.** If the exposure event fires at a different point in the render path in each arm — early in control, after an extra component mounts in treatment — then anything that interrupts loading costs treatment more. This is a design bug in the instrumentation, not in the feature. ## Stage 3: balanced at exposure, skewed in analysis Any filter applied **after** assignment can create a mismatch if its hit rate correlates with the arm. Bot and crawler filtering is the standard example: automated traffic is assigned like any other traffic, and if the treatment experience changes what a crawler can reach or how it behaves, the filter removes a different number from each arm. Deduplication, session stitching, outlier trimming and inner joins against another table all behave the same way. The defensive rule is that filters should use only information determined before assignment; a filter that depends on post-assignment behaviour is a mismatch generator. ## Slicing: the second axis Once the stage is known, cut the counts by dimension: - **Platform / OS version / browser / app version** — isolates crashes and rendering defects. - **Country and network type** — isolates latency-driven losses such as the redirect. - **Day and hour** — a ratio that is clean for five days and breaks on the sixth points at a deploy, a config change, or an incident. Overlay release timestamps. - **New versus returning traffic** — an imbalance confined to one of these hints at caching or an identity-resolution problem. A mismatch that is uniform across every slice is unusual and points at the assignment or configuration layer; a mismatch concentrated in one slice hands you the bug. ## Confirming the fix A cause is only confirmed when it explains the **magnitude and the shape**, not just the direction. If a crash on one OS version accounts for 200 missing users but 1,500 are missing, keep looking — partial explanations are how teams talk themselves into re-reading a broken experiment. After the fix, rerun the experiment; do not patch the historical data and reinterpret it. ## What a weak answer looks like "It is probably randomness, let us restart it" — no diagnosis, so the same defect recurs. "The assignment must be buggy" — jumping to the least likely stage. Or reporting the metrics because the cause was found and seemed small: knowing the mechanism does not undo the selective loss it caused.