How do you set an experimentation platform's policy on which tests may overlap?

## Why this is a policy question, not a statistics question The statistics are settled: independent layers give unbiased estimates of each experiment's effect averaged over everything else running. The open question is organisational — how much throughput to trade for how much interpretive comfort, and who gets to make that trade. Left ungoverned, exclusivity becomes a status good: whoever asks loudest gets a private universe, and the platform's capacity quietly collapses. ## Default: overlap The starting position should be that everything overlaps. Layering exists precisely so that traffic is not the binding constraint on how many decisions the organisation can make in a quarter. Every experiment moved into an exclusive group takes a share of a layer rather than the whole population, and because precision improves with the square root of sample size, splitting a layer three ways roughly triples the time each experiment needs for the same interval width. Across a large program, exclusivity-by-default is the difference between hundreds of decisions a year and dozens. ## The exception rule: surface contention A rule has to be mechanical or it becomes a negotiation. The cleanest criterion is **rendering contention**: if two experiments modify the same thing and only one of them can win, they cannot overlap. That covers competing page redesigns, two changes to the same copy string, two rules setting the same price, two rerankers on the same result list. Give each such surface its own layer; experiments touching it queue there as a mutually exclusive group. This rule has three virtues. It is checkable from the experiment's own configuration, so it can be automated. It puts the cost exactly where a genuine conflict exists. And it removes the argument entirely from the realm of preference — a team cannot request exclusivity, they can only declare which surface they touch. ## Reclaiming traffic inside exclusive groups When several variants of one surface compete, a **shared control arm** cuts the cost substantially: instead of each variant carrying its own control, one control serves all comparisons, so more of the group's traffic goes to the things being tested. The tradeoff to state up front is that the comparisons become positively correlated through the common control, so head-to-head ranking of the variants is less reliable than each variant-versus-control contrast is on its own. For a ship-or-not decision that is a good bargain; for a rank-these-five decision, less so. ## Monitor rather than pre-restrict The instinct to review every launch for possible interactions with everything else running does not scale and does not pay. Interactions between unrelated surfaces are usually small, the four-cell contrast that would detect them is inherently imprecise, and a review board that must clear every start becomes both a bottleneck and a thing teams route around. The better posture is retrospective: periodically scan the highest-traffic concurrent pairs for non-additivity, investigate the small number that look real, and feed genuine findings back as new surface definitions. That converts a permanent tax into an occasional, evidence-driven cost. ## Make the reporting convention explicit The most valuable thing a leader can do here is stop the recurring anxiety that concurrent experiments are corrupting results. Write down, once, that a reported effect is the effect **averaged over the experiment landscape in which it ran**, that this is unbiased for that quantity, and that it is typically the decision-relevant number because the product ships into a world that also contains other changes. Pair it with the honest caveat: for two changes aimed at the same user behaviour, the average may hide a real interaction, and that is when a deliberate 2x2 or an exclusive rerun is worth the traffic. ## Where to spend the scarce judgment A few calls genuinely deserve leadership attention rather than a rule. How many layers to define at all — too few and everything contends, too many and the surface taxonomy stops matching how the product is built. Whether to bound how many concurrent treatments a single user can absorb, trading a little throughput for a less erratic experience for the unlucky. And how to handle a strategically important launch whose read must be unimpeachable, where buying exclusivity is a reasonable one-off purchase rather than a precedent. ## What a strong answer conveys That exclusivity is a resource allocation problem with a mechanical rule attached, that the default must be overlap or throughput dies, and that the organisation is better served by an explicit reporting convention plus retrospective monitoring than by a pre-launch review board.