Why does each experiment layer hash users with its own independent seed?

## How a layered assignment is computed A layered platform assigns arms deterministically from an identifier. For a given layer it forms a string from the randomization unit's id together with a value that is specific to that layer — commonly called the layer's seed, salt, or key — hashes it, and maps the resulting number onto the arm ranges. A 50/50 split, for instance, takes the low half of the hash range as control and the high half as treatment. Determinism is what makes the assignment reproducible: the platform never has to store a per-user decision, and any service can recompute the same answer. But determinism also means the seed is the *only* thing that distinguishes one layer's shuffle from another's. ## What the seed actually buys A cryptographic-quality hash has the avalanche property: changing any input bit changes roughly half the output bits, in a way that is unpredictable from the original output. Consequently, hashing `user123:layerA` and `user123:layerB` produces two numbers with no usable relationship. Across the population, the pair of arm assignments behaves like two independent draws. That independence is the entire mechanism behind overlapping experiments. It gives you, for every experiment, a treatment group and a control group that look alike with respect to **every other running experiment** — same proportion in each of the other layers' arms, in expectation. Any effect from those other experiments therefore lands equally on both of my sides and cancels from my difference of means. ## The failure mode: a shared seed Suppose two layers are configured with the same seed by copy-paste. Now `hash(user:seed)` is the same number in both layers, so a user in the low half of the range is in control for *both* experiments and a user in the high half is in treatment for *both*. The cross-tabulation of the two layers' arms has two full cells and two empty ones. Everyone who saw experiment A's treatment also saw experiment B's treatment. The consequence is not a small bias — it is total non-identifiability. The measured lift for experiment A is the combined lift of A and B together, and so is the measured lift for B. You cannot attribute the movement to either change, and if the two changes push in opposite directions you may see both experiments read flat while both are individually doing something. No statistical adjustment recovers the separate effects, because the data contain no users who received one treatment without the other. A subtler version of the same defect: layers seeded by a short counter concatenated in a weak way, or a non-avalanching hash (a simple modulus of a numeric id, a checksum), can leave partial correlation between layers. The assignments are then neither identical nor independent, and the other experiment is *unevenly* distributed across your arms — a genuine confound that quietly biases your estimate. ## Diagnosing it The direct test is a joint frequency check. Take a period's exposure logs, cross-tabulate the arm held in layer 1 against the arm held in layer 2, and compare each observed cell count with the product of the two marginal proportions times the total. Under independence, a 50/50 layer crossed with a 50/50 layer should put about a quarter of users in each of the four cells. A diagonal concentration means a shared or correlated seed. A milder skew — say 30/20/20/30 — signals partial correlation and is just as much a bug. It is worth stressing that checking each layer *alone* proves nothing here. Both layers can be perfectly balanced on their own marginals while being completely dependent on each other; the marginals of a diagonal table are exactly 50/50. Independence is a property of the joint distribution, so it must be checked jointly. ## Related seed hygiene The seed should also be stable for the life of the experiment, because changing it mid-flight reshuffles the population and mixes two different randomizations into one analysis. And a seed that has been used before deserves care: rotating it when a layer is reused prevents the previous experiment's arm boundaries from lining up with the new one's. ## What an interviewer is listening for That the candidate can say *why* a shared seed is fatal rather than merely undesirable — the two treatments become perfectly collinear and neither is identifiable — and that they know the check lives in the joint table, not in each layer's own balance.