How does a region of practical equivalence change a Bayesian ship decision?

## The problem it solves A difference of exactly zero is a measure-zero event; no continuous posterior ever concentrates on a point. So "are the arms the same?" has no useful answer as literally posed. What decision-makers actually mean is "is the difference small enough that I do not care?" — and that question is answerable, but only once someone states how small is small enough. A region of practical equivalence, often abbreviated ROPE, is that statement: an interval of differences declared indistinguishable from no effect for the purpose of this decision. On a checkout funnel it might be plus or minus 0.5% relative lift. Inside that band, the two arms are treated as the same product. ## The decision rule With a posterior for the difference in hand, compare it to the band: - **Posterior entirely inside the band.** Accept practical equivalence. This is the verdict that does not exist without a band, and it is a genuine finding: you now have evidence that whatever difference exists is too small to matter. The decision moves to non-statistical grounds — which arm is cheaper to operate, simpler to maintain, better aligned with where the product is heading. On a migration that costs engineering time, this verdict says do not migrate. - **Posterior entirely outside the band, on one side.** The difference is real and large enough to act on. Act. - **Posterior straddling a boundary.** Undecided. Either gather more evidence or decide on other grounds while acknowledging the ambiguity. Teams differ on whether to compare the *whole* posterior or a 95% credible interval — typically a highest-density interval, the narrowest interval containing 95% of posterior belief. The stricter whole-posterior version almost never fires at realistic sample sizes; the interval version is what gets used. Either way, state which one you mean, because the strictness differs. ## Where the band comes from It must be set before you look at the result, and it must come from the cost side of the business, not from the data. The question to ask the product owner is: below what change would you not bother making this change at all? For a migration that costs two engineer-months and adds a service to operate, the honest answer may be a full percent; for a one-line copy change with no maintenance cost, it may be far tighter. Setting the band after seeing the posterior is the failure mode that voids the whole exercise. A band drawn around whatever the data produced can be made to yield any verdict you want, and it makes the equivalence claim unfalsifiable. The band is also directional in some decisions. For a guardrail metric — latency, error rate, refund rate — you often care only that the variant is not meaningfully worse, so the meaningful boundary is one-sided. ## An example verdict A checkout test runs to a posterior for relative lift that spans roughly minus 0.2% to plus 0.3%, entirely within a pre-declared band of plus or minus 0.5%. Read carefully, this is not a failed experiment. It is a successful one: the variant, whatever else it does, does not move checkout conversion by an amount worth the change. If the variant carries an ongoing cost — a new dependency, more code, a vendor bill — the correct call is to ship neither and retire the variant. Teams that lack the band read the same posterior as "inconclusive, run it longer", and burn traffic pursuing a difference they have already established is too small to want. ## The relationship to loss-based rules A practical-equivalence band and an expected-loss tolerance are two ways of encoding the same underlying fact: that the business has a threshold below which differences are not worth acting on. The band expresses it as an interval on the effect scale and answers "are these the same?"; a loss tolerance expresses it as a cost and answers "how much do I risk by choosing?" They are complementary, and mature experiment readouts show both. ## What interviewers listen for That you set the band in advance and from cost. That "inside the band" is an affirmative conclusion rather than a failure to find something. That you say which interval you are comparing. And that you do not let a band be widened after the fact until a decision appears.