When should two experiments be mutually exclusive rather than overlapping in separate layers?

## The two placements A layered platform offers exactly two ways to relate a pair of experiments. **Different layers (overlapping).** Both experiments randomize the full eligible population independently. Every user gets an arm in each, so all four combinations occur. Both experiments enjoy the whole traffic, and each estimate is unbiased for that experiment's effect averaged over the other's arms. **Same layer (mutually exclusive).** The layer's arm space is carved up between the experiments. A user who falls into experiment A's range is in experiment A and is simply not eligible for experiment B. No user ever sees both treatments, and the combination is never observed. ## When exclusivity is the right call **The combination is incoherent.** This is the dominant reason. Two competing redesigns of the same page cannot both render — one of them wins the DOM and the other silently does nothing, or the page becomes a hybrid nobody signed off on. The same applies to two experiments that each change the same copy string, the same pricing rule, or the same ranking decision. Here exclusivity is not a statistical nicety; it is a correctness requirement. **You need a clean read on each variant.** When a strong interaction is expected — two changes both aimed at the same behaviour, pushing on the same friction — the overlapping design still gives you unbiased main effects, but each is an average over the other's arms, and stakeholders may find that hard to act on. If the decision you are about to make is "which of these two designs do we ship", an exclusive group gives you two clean contrasts against a common baseline. **Risk containment.** A user who is simultaneously in three aggressive treatments is the user most likely to have a terrible session. Some platforms keep a small set of high-risk experiments mutually exclusive purely to bound how much novelty any single person absorbs at once. ## When exclusivity is the wrong call It is the wrong call when the only argument is discomfort. "I don't want anything else running during my test" is not a reason — independent layers already give the estimate the cleanliness that argument is reaching for, and granting exclusivity on request turns a scarce, shared resource into a queue. It is also wrong when the combination is exactly what you want to learn about: if two changes are expected to ship together, overlapping them lets you observe all four cells and see whether the pair behaves as hoped. ## The price Suppose an exclusive layer holds three experiments. Each gets roughly a third of the traffic. Because the width of a confidence interval shrinks with the square root of the sample size, a third of the traffic makes each interval about 1.7 times wider at a fixed duration — or, equivalently, forces each experiment to run about three times as long to reach the same precision. Multiply that across an organisation and exclusivity-by-default is the single biggest brake on experiment throughput. A **shared control arm** recovers part of that cost. If three variant experiments in one exclusive group are all being compared against the current experience, they do not each need their own control: allocate one control arm and compare all three variants against it. Four arms of 25% become one 25% control plus three 25% variants instead of three separate 20/20 pairs, so each comparison gets more users on both sides. The caveat is that the three comparisons now share a denominator, so their estimates are positively correlated — a control group that happens to run high makes all three variants look worse together, which matters when you rank them against each other. ## Design guidance that survives contact with an org The workable rule is to define exclusivity by **surface ownership**, not by team request. Each surface that can only be rendered one way — the checkout page, the search results template, the pricing rule — gets a layer, and experiments touching it queue in that layer as an exclusive group. Everything that touches a different surface goes into a different layer and overlaps freely. That policy is mechanical, easy to arbitrate, and puts the cost where the genuine conflict is. ## What a strong answer sounds like Name the mechanism (same layer, split traffic, never both), give the incoherent-combination criterion, quantify the cost in duration or interval width, and mention shared controls as the way to claw some of it back.