What do experiment layers do in an A/B platform running many tests at once?

## The problem layers solve An organisation that runs one experiment at a time is limited to a few dozen decisions a year. The obvious fix — cut the traffic into slices and give each experiment its own slice — is worse than it looks. Every experiment now runs on a fraction of the users, so every experiment needs proportionally longer to reach the sample size it needs, and the total number of decisions per unit of traffic does not improve at all. Traffic is the scarce resource, and slicing does not create more of it. ## What a layer is A **layer** (sometimes called a domain or a universe) is one complete, independent randomization of the entire eligible population. A platform defines several layers, and **every eligible user is assigned an arm in every layer**. If there are three layers, a user simultaneously holds three assignments: perhaps treatment in the ranking layer, control in the pricing layer, and treatment in the onboarding layer. The assignments are drawn independently of one another, which is why a single user can be exposed to several experiments at once without the experiments becoming entangled. The key property is **statistical independence between layers**. Knowing that a user is in the treatment arm of layer 1 tells you nothing about which arm they hold in layer 2 — the conditional distribution of the layer-2 arm is the same as its unconditional distribution. ## Why independence makes each test readable Suppose I am running a ranking experiment and, at the same time, a pricing experiment is running in a different layer. My treatment group contains some users who are also in the pricing treatment and some who are in the pricing control — and because the layers are independent, the proportion is the same in my control group. Whatever the pricing change does to the metric, it does it to roughly the same share of users on each side of *my* comparison. It therefore contributes equally to both of my group means and cancels when I take the difference. My estimate stays unbiased for the effect of my change, averaged over the world in which the pricing test is running. That last clause matters. A layered platform does not estimate the effect of my change in a pristine, experiment-free world; it estimates the effect **averaged over the mix of other experiments running at the time**. Usually that is exactly the quantity a shipping decision needs, because the future world will also contain other changes. ## What layers do not do Independence is about *balance*, not *isolation*. Two orthogonal experiments can still genuinely interact: if a treatment that hides shipping cost and a treatment that enlarges the checkout button both push on the same behaviour, the combined effect need not be the sum of the two individual effects. Layering guarantees that the interaction is spread evenly across arms, so each main effect is still estimated without bias — it does not guarantee the interaction is zero, and it does not report it to you unless you go looking. The estimate you get is the average effect over the other test's arms. Layering also does not prevent a **broken combined experience**. If two experiments both rewrite the same page region, a user assigned to both may see something neither team designed and nobody ever reviewed. Independence is no comfort there — the fix is to place those two experiments in the same layer so they are mutually exclusive. ## The small cost of concurrency Running a concurrent experiment in another layer does not reduce your sample size — you still analyse all the traffic. But if the other treatment genuinely moves the metric, that movement becomes extra variation among users *within* each of your arms. The metric's variance goes up a little, so your standard error and confidence interval widen a little. This is a precision cost, not a bias, and in practice it is usually small relative to the natural user-to-user variation in metrics like revenue per user. ## What an interviewer wants to hear A good answer names three things: layers are independent randomizations over the *same* full traffic rather than disjoint slices of it; independence buys balance, so each test's estimate is unbiased even with dozens of concurrent tests; and same-layer placement is the deliberate escape hatch for experiments that must not co-occur.