An A/A run flags 1 of 20 metrics at alpha 0.05 - do you block the launch?

## Why one flag is unremarkable A significance threshold is a deliberate false-positive rate. Setting alpha to 0.05 means that when there is truly no difference, the procedure will still declare one about 5% of the time. Run that procedure over 20 metrics on data where, by construction, no treatment exists, and the arithmetic is immediate: - Expected number of flagged metrics: `20 x 0.05 = 1`. - Probability that at least one flags, if the metrics were independent: `1 - 0.95^20 = 0.64`. So a healthy platform running an A/A across twenty metrics will show a flag on roughly two runs in three. Reporting zero flags every time would be the surprising outcome - it would suggest the analysis is over-conservative rather than that the split is unusually clean. In practice the metrics are correlated with one another, which pulls the at-least-one probability somewhat below 64%, but the qualitative conclusion is unchanged. This is why `one metric out of twenty came back significant` is, on its own, information-free. The number of tests you performed is part of the result and has to be reported with it. ## What actually distinguishes noise from a defect **How extreme, not merely whether.** Chance produces p-values just under the threshold; it does not produce astronomically small ones. A metric at 0.04 among twenty is exactly what the arithmetic predicts. A metric at one in a million is not reachable by sampling variation on a null comparison, so it points to a defect in assignment or logging. **Absolute size, independent of the p-value.** Two identically treated arms differing by 15% on an event count is an instrumentation problem even if the sample is small enough that the p-value looks unimpressive. Always look at the magnitude in the metric's own units, not only at the verdict. **Reproducibility.** Noise does not repeat on the same metric. Re-run the A/A - or look at the last several A/A runs - and see whether the same metric keeps flagging. A metric that flags three runs in a row is not chance; it has a cause. **Where the flag sits.** There is an important asymmetry between downstream behaviour metrics and the split itself. A difference in a conversion rate between identical arms is a candidate for noise. A mismatch between the number of users assigned to each arm and the ratio you configured is not a metric result at all - it says the assignment mechanism is not doing what you asked, and that is a blocking defect handled on its own terms rather than filed under expected variation. ## The right acceptance criterion A single A/A run is a weak instrument, so do not build the gate on one verdict. The stronger criterion is calibration across many runs: over repeated A/A tests, the share that flag any given metric should sit near the alpha you set, and the p-values should be scattered across the whole range rather than piling up near zero. A platform that flags 15% of the time at a 0.05 threshold has a calibration problem no matter how comfortable any individual run looked. This criterion has the advantage of being falsifiable and of not depending on a judgement call each time a run comes back. It also detects the failure mode a single run cannot: an analysis whose variance is systematically understated, which makes every future experiment more likely to declare a win that is not there. ## Decide the rule before you look The organisational failure here is arguing about the criterion after seeing the result. Write down beforehand: which metrics are in the A/A panel, what magnitude of discrepancy blocks a launch regardless of significance, what pattern across repeat runs blocks a launch, and who makes the call. Otherwise the decision is made by whoever is most invested in launching. ## Answering in an interview Open with the arithmetic - one expected flag, about a 64% chance of at least one - so it is clear you are not talking yourself out of a real problem. Then give the escalation criteria: extremity of the p-value, absolute size of the gap, whether it repeats, and whether the flag is on assignment counts rather than behaviour. Finish with the calibration-across-runs criterion. Candidates who either block on any flag or wave away every flag are showing the same weakness from opposite sides.