A/B Testing & Experiment Design

Sticky assignment gives each user one consistent experience instead of a flickering mix. If the variant is re-drawn per visit, returning users receive both treatments, which blends the two arms and shrinks the measured difference toward zero.

Layers let many experiments share the same traffic at the same time. Each layer randomizes users independently, so a user's arm in one layer says nothing about their arm in another, and concurrent tests do not confound one another.

A staged ramp limits blast radius: the first small stage exposes few users, so crashes, latency spikes or a collapsing metric are caught cheaply. Later, larger stages exist to measure the effect once the change has been shown to be safe.

Concatenate the user ID with an experiment-specific salt, hash it to an integer, take it modulo the bucket count, and map bucket ranges to arms - 0-49 control, 50-99 treatment. Deterministic, so nothing is stored.

Randomize by session only when the change leaves no trace between visits and the outcome is finished within one visit — a layout tweak judged by same-visit conversion. If the treatment can be learned, remembered or accumulated, randomize by user.

User behaviour differs by day of week, so a partial week samples an unrepresentative mix. Whole multiples of seven days weight every weekday equally, so the measured lift describes a typical week, not whichever days happened to be included.

The minimum detectable effect is the smallest true difference between variants a test is designed to catch. Given the sample size, significance level and target power, a true effect smaller than the MDE will usually go undetected.

It never says whether 5% means 5 percentage points, 3% to 8%, or a 5% relative lift, 3% to 3.15%. Those targets differ by hundreds of times in required traffic, so sizing cannot start.

A sample count and a calendar window are different requirements. Three days covers only three weekdays and over-represents your most frequent visitors, so the sample is large but not representative. Read the result when both are met.

Decide before launch. The cleanest option is usually to shift the window so it sits entirely before or after the holiday. If the test must span it, keep whole weeks and treat the result as an estimate for holiday traffic.

A count metric sums events, such as total orders. A rate metric divides a count by the units that were exposed, such as orders per visitor. A share metric divides one count by a related total, such as the percentage of orders placed on mobile.

A guardrail is a metric the experiment must not damage — latency, crash-free sessions, unsubscribes — and a big enough degradation can veto the launch. A driver metric only explains why the primary metric moved; it never vetoes.

Almost all the variance in revenue per user comes from a handful of very large spenders, so the interval around the treatment-control difference stays wide. Ordinary experiment sizes then cannot resolve realistic revenue effects.

The Overall Evaluation Criterion is the single metric, agreed before launch, that an experiment's ship-or-not decision is defined against. It expresses what the team means by the product getting better, so the result cannot be reinterpreted afterwards.

A DAU/MAU of 0.2 means the typical monthly active user opens the product on about 6 days out of 30. It measures return frequency, not audience size, and averages over very different user types.

A p-value only says how compatible the data are with zero effect. A confidence interval reports the lift itself with its precision, so you can compare the plausible range against the minimum detectable effect and the launch bar.

Every segment cut is another hypothesis test. At a 5% false-positive rate, about twenty cuts of a genuinely flat experiment throw up roughly one significant segment by chance alone, so the winner you find is usually noise.

Dilution is the shrinking of a measured effect when most assigned users never encounter the change. At a 3% trigger rate an effect among exposed users appears about thirty times smaller in the all-up read, while the noise stays.

Intent-to-treat counts every assigned user and answers what shipping does to the whole population. A triggered analysis keeps only users who met the trigger condition in both arms, answering what the change does to those who encounter it.

No — it is inconclusive, not flat. The interval reaches +2.8%, above the 2% minimum detectable effect the test was planned around, so a lift worth shipping has not been ruled out, and neither has a 2% loss.

Both are temporary reactions to the fact that something changed, not to what it changed into. Novelty inflates an early result as users explore something new; primacy, or change aversion, depresses it as users unlearn a habit.

No. The 0.05 threshold is calibrated for a single analysis at the planned sample size, so a day-2 reading is not the test you designed. Run to the agreed horizon unless the plan already allowed an early call.

A sample ratio mismatch is when the observed split of users across experiment arms differs from the planned split by more than chance can explain. It signals a broken assignment or logging pipeline, so the comparison itself is untrustworthy.

Upward. Treated sellers capture bookings that control sellers would otherwise have received, so treatment is pushed up while control is pushed down. The measured lift counts one transfer twice and mixes real growth with redistribution that vanishes at full rollout.

Put every user on their own clock: days since that user first saw the change. Plot the treatment-minus-control gap against exposure age, grouped by first-exposure date. A gap that shrinks with exposure age, repeating per group, indicates decay.

Use adaptive allocation. With twelve options, clicks arriving in seconds and value gone in hours, the cost of serving losers dominates and no precise per-headline lift is needed. A fixed-horizon test would still be running when the headlines are worthless.

CUPED replaces outcome Y with Y - theta*(X - mean X), where X is a pre-experiment covariate. With theta = cov(Y, X) / var(X), variance falls to (1 - rho^2) of the original, rho being the correlation.

An alpha-spending function states in advance how much of the total 0.05 false-positive budget each interim analysis may consume, as a function of how much of the planned data has arrived. The pieces sum to 0.05.

Because allocation depended on outcomes: an arm gets starved precisely after a run of unlucky results, so the few observations it holds are the unlucky ones. Its sample mean understates its true mean, and ordinary confidence intervals under-cover.

Cumulative regret is the total reward given up by not serving the best arm every round: summed over rounds, the best arm's true mean reward minus the mean reward of the arm actually served.