Randomization & Assignment

Layers let many experiments share the same traffic at the same time. Each layer randomizes users independently, so a user's arm in one layer says nothing about their arm in another, and concurrent tests do not confound one another.

A staged ramp limits blast radius: the first small stage exposes few users, so crashes, latency spikes or a collapsing metric are caught cheaply. Later, larger stages exist to measure the effect once the change has been shown to be safe.

Concatenate the user ID with an experiment-specific salt, hash it to an integer, take it modulo the bucket count, and map bucket ranges to arms - 0-49 control, 50-99 treatment. Deterministic, so nothing is stored.

Randomize by session only when the change leaves no trace between visits and the outcome is finished within one visit — a layout tweak judged by same-visit conversion. If the treatment can be learned, remembered or accumulated, randomize by user.

Most likely the winner's curse. The 1% estimate was extremely noisy, and the feature was promoted because that noisy number looked big. Selecting on a large observed effect inflates it, so the precise later read of +2% is the better estimate, not a decay.

The test counts page-views as independent, but page-views from one person share that person's single assignment. The effective sample size is the number of users, so the standard error is understated and false positives run far above the nominal rate.

Per-request randomization re-rolls the assignment on every page load, so one person sees both variants. That breaks the experience and mixes the arms, so the comparison is between two blends rather than treatment versus control.

An A/A test splits traffic normally but serves both arms the identical experience. It validates the plumbing - a split that produces comparable groups, a metric pipeline that agrees on identically treated users, and an analysis that does not flag phantom differences.

Independent seeds make assignment in one layer statistically independent of every other layer. Reusing one seed sends the same users to the same arm position in both layers, perfectly confounding the two experiments.

Make two experiments mutually exclusive when their treatments cannot sensibly coexist for one user, such as competing redesigns of the same page. Everything else should overlap, because exclusivity costs traffic and time.

You never randomize a person, only an identifier. A device cookie splits one person across a phone and a laptop into two units that can land in opposite arms, and signing in mid-visit can switch the arm entirely.

No. Scanning 20 metrics at a 0.05 threshold when nothing truly differs gives one flag on average, and at least one flag about 64% of the time. Investigate the flagged metric, but a single borderline result is not evidence of a broken split.

Those users carry a residual effect from the old treatment - a quarter of exposure changed their habits and their composition. Concentrating them in one arm makes the arms differ before the test starts. Re-salt to spread them.

Ask first whether the pattern is real: the four-cell contrast measuring an interaction carries about twice a main effect's standard error, so flips are usually noise. Then rule out correlated assignment and eligibility differences.

Only when every stage estimated the same thing: unchanged treatment, unchanged eligible population, unchanged metric. Even then, pool by combining per-stage effect estimates, never by summing raw counts across stages with different traffic splits.

Every shopper in a store receives the same assignment and the same local conditions, so their outcomes move together. The independent replicates are the stores, and adding shoppers inside a store adds far less information than adding another store.

Sometimes, and rarely per feature. A holdback buys continued measurement after the experiment ends, but 1% is insensitive and every holdback forks the product. A single company-level holdback covering many launches usually beats one per feature.

Yes, by default. Widen who is eligible while leaving already-assigned users exactly where they are. Re-drawing assignment mid-ramp flips some users between arms, and their later behaviour still carries what they already saw, which contaminates both arms.

Weigh one clean never-treated reference population against three costs: every experiment now draws from 90% of traffic, every shipped feature must keep its old code path alive, and held-out users get a stale product.

Default to overlapping everything and grant exclusivity per shared surface, never per team request. Monitor for interactions retrospectively rather than gating launches, and state openly that reported effects average over the concurrent landscape.

Make the most stable identity the default, record the chosen unit as a first-class property of every experiment, derive the analysis grain from it automatically, and require review before any team randomizes finer than a person.