What does an A/A test validate before you launch a real experiment?

## The idea An A/A test runs the entire experiment machinery with the treatment removed. Users are hashed into buckets, buckets are mapped to two arms, exposures are logged, metrics are computed, and the comparison is run - but both arms see the identical current product. Under a correct platform there is nothing to find, so anything the A/A reports is telling you about the platform rather than about the product. That is the whole point: it turns a silent system into an observable one. ## What it can catch **A broken split.** The assignment mechanism may not be producing what you configured. Symptoms include arm memberships that do not match the intended ratio, or two arms that differ systematically in who is in them - one arm carrying older accounts, one region, or one client version. The first of those is an imbalance in the split itself and is treated as a blocking defect diagnosed on its own terms; the second is what a comparison of pre-existing user characteristics across the arms surfaces. **A broken metric pipeline.** This is the finding that pays for A/A testing. Between two identical experiences, a 15% or 20% gap in an event count cannot be sampling variation at experiment scale - it is instrumentation. The classic case is **double counting**: one arm's events are emitted by both a legacy and a new logging path, so its counts are inflated. Other variants are a join that silently drops sessions for one arm, a deduplication rule applied asymmetrically, or a metric defined over a table that is populated for one code path only. None of these are visible in a real A/B test, because there they masquerade as a treatment effect. **A mis-calibrated analysis.** Even with a clean split and clean data, the statistical procedure can be wrong for the data it is given. If the variance is understated, p-values come out too small and the platform flags differences that do not exist far more often than its stated error rate. You do not see this in a single run; you see it across many A/A runs, where the share that flag any given metric should sit near your alpha and the p-values should be scattered across the whole range rather than piling up near zero. That is the acceptance criterion for the platform. ## What it cannot catch An A/A test says nothing about anything that only happens when the treatment is on: a slower code path in the new variant, an error that fires only on the new screen, a metric that is only emitted by the new UI. It also cannot certify the split as unbiased in general. Like any test, it only detects imbalances that are large relative to its noise, so a small systematic bias passes unnoticed. Treat a single clean A/A as ruling out gross breakage, not as a certificate. ## How teams actually run them Three patterns are common, and they are not exclusive: - **Pre-launch gate.** Run an A/A on the exact split the upcoming experiment will use, particularly when the metric or the population is new. - **Continuous background A/A.** Keep a permanent null experiment running so the platform's false-positive behaviour is measured continuously rather than remembered from a one-off exercise. - **Retrospective A/A.** Re-analyse historical data by splitting a past period with a fresh salt and running the analysis as if it were an experiment. This is cheap because it spends no live traffic, but it only exercises the metric and analysis layers, not the live assignment path. ## Reading a flagged A/A The triage question is always *where does the asymmetry first appear*. Compare the number of assigned users per arm; if that already deviates from the configured ratio, the problem is upstream in assignment and nothing downstream can be trusted. If assignment counts match but a downstream count is inflated, follow a single user through both tallies and find where one of their events is counted twice or dropped. Only when both of those are clean does `expected noise` become a reasonable explanation - and then the size of the discrepancy and how extreme the p-value is decide whether it is credible as noise. ## Interview framing A strong answer names all three failure classes, insists that the biggest practical yield is instrumentation rather than randomisation, and volunteers the limitation: an A/A is underpowered against small biases, so its real value comes from running it repeatedly and checking calibration, not from one green tick.