In a Bayesian A/B test, what is the expected loss of shipping the variant?

## The quantity Expected loss answers the question a probability of superiority dodges: *if I am wrong, how much does it cost me?* For a decision between control A and variant B, the expected loss of choosing B is ``` E[ max(rate_A - rate_B, 0) ] ``` where the expectation is taken over the posterior for the pair of rates. In words: over the whole posterior, average the amount by which the control beats the variant, treating that amount as zero on every part of the posterior where the variant is genuinely ahead. It is sometimes called risk, or expected regret. With paired posterior draws it is a one-liner: sum `rate_A - rate_B` across the draws where A is larger, and divide by the *total* number of draws. Dividing by the count of losing draws instead is the single most common implementation error, and it produces a much larger number that answers a different question — the conditional loss *given* that you were wrong, rather than the unconditional expected cost of the decision. ## Why it makes a better stopping rule A probability threshold compresses the posterior into one bit of direction weighted by belief. Expected loss keeps the units. Consider two experiments: - **Experiment 1**: the posterior says there is a 30% chance the variant is worse, but in the worst plausible case it is worse by 0.001 percentage points. Expected loss is negligible. Ship it; the downside is rounding error. - **Experiment 2**: the posterior says there is a 3% chance the variant is worse, but if it is worse it is worse by five percentage points. Expected loss is around 0.15 percentage points — potentially a serious hit. A 97% probability of superiority looks like an easy green light and is not. A rule keyed on "probability above 95%" ships the second and blocks the first. A rule keyed on expected loss gets both right, because it prices the tail rather than merely counting it. Expected loss also degrades gracefully as evidence accumulates. Early in a test, both arms carry large expected loss because the posterior is wide. As data accumulates, the expected loss of the better arm falls monotonically toward zero in the typical case, which gives you a natural, interpretable stopping quantity: stop when the cost of a mistake has become small enough to accept. ## Setting the tolerance The threshold has to be in the same units as the metric, which is exactly the property that makes it discussable outside the analytics team. On a checkout funnel with a baseline near 5% conversion, a tolerance of 0.02 percentage points says: we accept a decision rule whose expected cost, if it is wrong, is two hundredths of a point of conversion. Product and finance can argue about that number on its merits, which they cannot do with "95%". Because the tolerance encodes willingness to lose, it should be tighter for changes that are expensive or slow to reverse and looser for a copy tweak that can be rolled back in an afternoon. ## Both directions Expected loss is computed for each candidate decision. The expected loss of keeping A is `E[max(rate_B - rate_A, 0)]` — the upside you forgo by not shipping. Two structural cases follow: - **Both losses small.** The posterior for the difference is concentrated near zero. Neither choice can cost you much; decide on other grounds — simplicity, maintenance cost, strategic direction. - **Both losses large.** The posterior is wide and straddles zero. You do not have enough evidence, and either choice carries real exposure. That two-sided reading is the practical payoff: expected loss makes "we genuinely cannot lose much either way" an explicit, defensible conclusion instead of an awkward silence. ## Asymmetric costs The plain formula treats a unit of loss on one side as a unit of gain on the other. When that is false — a regression damages trust or triggers a support burden out of proportion to the revenue involved — weight the shortfall. Multiplying the negative side by a factor greater than one encodes "a point lost hurts more than a point gained helps" and shifts the rule toward the incumbent. Being able to say that a loss function is a modelling choice, not a fixed formula, is what separates a senior answer from a memorised one. ## Common failures Quoting expected loss without units, so nobody can judge whether it is small. Averaging only over the losing draws. Comparing expected loss across metrics measured on different scales. And shipping on a probability threshold while showing the expected loss on the dashboard as decoration.