Where does the 16 come from in the per-arm sample rule n = 16 * sigma^2 / delta^2?

## The rule and its derivation The back-of-envelope every experimenter should be able to write from memory is `n_per_arm = 16 * sigma^2 / delta^2` where `sigma^2` is the per-user variance of the metric and `delta` is the absolute effect you want to detect. The 16 is not arbitrary. The general form is `n_per_arm = 2 * (z_alpha/2 + z_beta)^2 * sigma^2 / delta^2` and it decomposes cleanly: - `z_alpha/2 = 1.96` - the standard normal critical value for a **two-sided test at 5% significance**. This is how far the observed difference must sit from zero, in standard errors, before it is called significant. - `z_beta = 0.84` - the standard normal value corresponding to **80% power**. This is the extra distance the true effect must sit beyond the critical value so that four runs in five land past it. - The **factor 2** - the comparison is between two independent arm means. The variance of a difference of two independent means is `sigma^2/n + sigma^2/n = 2 sigma^2/n`, and that 2 propagates straight into the numerator. Arithmetic: `1.96 + 0.84 = 2.80`; `2.80^2 = 7.84`; `2 x 7.84 = 15.68`, rounded up to **16**. The rounding is generous by about 2%, which is the right direction for a planning rule. ## What changes the constant The 16 is only correct for the specific defaults it encodes. Common variants: | Significance (two-sided) | Power | Constant | |---|---|---| | 5% | 80% | ~16 | | 5% | 90% | ~21 | | 1% | 80% | ~24 | | 1% | 90% | ~30 | For 90% power, `z_beta = 1.28`, so `2 x (1.96 + 1.28)^2 = 2 x 10.5 = 21`. Anyone quoting 16 for a 90%-power plan will under-size by about 30%. A one-sided test at 5% replaces 1.96 with 1.64 and lowers the constant to about 12, which is why the sidedness must be written down rather than assumed. ## Using it on a conversion rate For a proportion, the per-user variance is `sigma^2 = p * (1 - p)`. Take the leaf's worked case: a **3% baseline checkout conversion** and a target of a **10% relative lift**. 1. Convert the relative target to absolute: `delta = 0.10 * 0.03 = 0.003`. 2. Variance term at the midpoint between 3.0% and 3.3%: `p_bar = 0.0315`, so `p_bar (1 - p_bar) = 0.0315 x 0.9685 = 0.0305`. 3. Apply the rule: `n = 16 x 0.0305 / 0.003^2 = 0.488 / 0.000009 ~= 54,000` per arm. 4. Two arms: roughly **108,000 visitors** must reach the checkout surface. Notice how brutal the `delta^2` denominator is. Tighten the ambition to a 5% relative lift and `delta` halves to 0.0015, so the requirement quadruples to about 216,000 per arm. Loosen it to a 20% relative lift and it falls to about 13,500 per arm. ## Why memorise it Three reasons interviewers like this question: - **It makes you calculator-free.** You can sanity-check any planning tool's output in your head to within a few percent, which catches the classic input mistakes: relative entered as absolute, total traffic entered where per-arm was meant, variance taken from a different metric. - **It exposes the assumptions.** Someone who knows the 16 is `2 x (1.96 + 0.84)^2` automatically knows the rule is tied to two arms, an equal split, a two-sided test, 5% significance and 80% power. Someone who memorised only the digit "16" will apply it to a four-arm test or a 90%-power plan without blinking. - **It shows the normal approximation.** The formula is a large-sample result. With very small expected counts - a handful of conversions per arm - the normal approximation degrades, and the planning number should be treated as indicative rather than exact. ## Caveats worth stating out loud - **Per arm, not total.** The rule gives the sample for *each* arm; double it for a two-arm total. - **Equal split assumed.** An unbalanced split needs more total traffic for the same sensitivity, because the smaller arm dominates the standard error. - **Variance must match the metric.** For proportions, `p(1-p)` is trivially available; for revenue-like metrics, the variance has to come from historical data, and outliers make it much larger than intuition suggests. - **The unit of analysis must match randomisation.** If you randomise visitors, `sigma^2` must be the variance of the per-visitor value, not the per-session one.