Minimum Detectable Effect

## The idea in one line The **minimum detectable effect (MDE)** is the smallest *true* difference between control and treatment that an experiment is built to detect reliably. "Reliably" is quantified: at the chosen significance level, with the planned sample, a true effect of exactly the MDE would produce a statistically significant result with probability equal to the target power (80% is the usual convention). It is a **design parameter**, decided before a single user is bucketed, and it is the number that turns a vague ambition ("let's see if the new checkout is better") into a concrete traffic and calendar commitment. ## The equation it comes from For a two-arm test with an equal split, comparing a metric with standard deviation `sigma` between arms, the classical per-arm sample size is `n = 2 * (z_alpha/2 + z_beta)^2 * sigma^2 / delta^2` where `delta` is the effect you want to detect, `z_alpha/2` is the critical value for the two-sided significance level, and `z_beta` is the value corresponding to the target power. For a conversion rate `p`, the variance term is `sigma^2 = p * (1 - p)`. Rearranged for `delta`, the same relationship gives the MDE directly: `delta = (z_alpha/2 + z_beta) * sigma * sqrt(2 / n)` That single rearrangement is the whole concept. Sample size and MDE are not two topics; they are the same equation solved for different unknowns. Two consequences follow immediately: - **MDE shrinks only as `1 / sqrt(n)`.** Four times the traffic halves the MDE. Doubling traffic improves it by about 29%, not 50%. - **Sample size grows as `1 / delta^2`.** Insisting on detecting an effect half as large costs four times the users. ## What you must fix before an MDE means anything An MDE quoted on its own is uninterpretable. To state it you need: 1. **Baseline metric level** — the control conversion rate or mean. It sets the variance and it is the reference for any relative statement. 2. **Variance of the metric** — for a proportion it follows from the baseline; for revenue-style metrics it must be measured from history, and heavy tails make it much larger than people guess. 3. **Significance level `alpha`** — commonly 0.05 two-sided. 4. **Target power** — commonly 0.80. 5. **Traffic split and number of arms** — the per-arm sample, not the total, drives sensitivity. 6. **Unit of analysis** — visitors, sessions or accounts. Randomising by visitor but counting sessions inflates the effective variance and quietly makes the real MDE worse than the plan says. 7. **Direction** — one-sided or two-sided, since the critical value differs. ## What the MDE is not - **Not a prediction.** Saying "our MDE is a 5% lift" says nothing about how large the true lift is. It describes the instrument, not the world. - **Not a significance threshold.** A measured difference smaller than the MDE can still be significant on a given run, and a measured difference larger than the MDE can fail to reach significance. The MDE is a statement about long-run detection probability for a fixed *true* effect, not a cutoff applied to the observed number. - **Not a floor on what the product can achieve.** A change with a true 1% lift is still worth having; a test sized for 9% simply cannot see it. - **Not free to shrink.** The only ways down are more traffic, less variance, or accepting weaker guarantees. ## Why interviewers ask it Because it separates people who have planned an experiment from people who have only read one out. The follow-up is almost always numerical ("what happens to sample size if you halve it?") and the expected answer is the quadratic relationship. The second-order signal is whether you know MDE is chosen from business value, not reverse-engineered from whatever traffic happens to exist and then written into the plan as if it were a decision. ## A worked intuition Suppose you plan a test and the arithmetic says the MDE is a 9% relative lift. That means: if the treatment truly moves the metric by 9%, roughly four runs in five would come back significant; if it truly moves it by 3%, most runs would not. Before spending two weeks of traffic, the team should ask whether a change that only delivers 3% is worth knowing about. If it is, the design is wrong, and finding that out beforehand is exactly what the MDE is for.