What is dilution in an A/B test where only 3% of assigned users ever see the change?

## What dilution is Dilution is what happens to an effect size when the population you average over is much larger than the population the change can touch. It is arithmetic, not a bug, and it is the dominant reason narrowly scoped experiments read flat. ## The arithmetic on the absolute scale Let `p` be the trigger rate, the share of assigned users who reach the change. Let `d` be the average absolute effect on triggered users, and assume untriggered users are unaffected, so their effect is exactly zero. Averaging over everyone assigned: `d_allup = p * d + (1 - p) * 0 = p * d` With `p = 0.03` and `d = 1 percentage point of conversion`, the all-up effect is `0.03` percentage points. Nothing was lost or hidden; the population average of a change that only touches 3% of the population simply is small. ## The relative scale needs one more term Relative lift does not scale by `p` alone. If `m_t` is the control-arm metric level among users who trigger and `m_all` is the level across all assigned users, then `relative_lift_allup = relative_lift_triggered * p * (m_t / m_all)` When triggered users are typical, `m_t / m_all` is about 1 and the shortcut of multiplying the relative lift by the trigger rate works: a `+9%` triggered lift at `p = 0.03` becomes roughly `+0.3%` all-up. But users who reach a settings page are frequently heavier, more engaged users whose baseline is well above average, in which case the shortcut understates the impact. The robust habit is to convert to absolute units, scale there, and convert back at the end. ## Why dilution costs power, not validity The diluted estimate is unbiased for the thing it estimates: the effect of shipping this change on the whole assigned population. Untriggered users contribute the same distribution of outcomes to both arms, so they do not push the difference in either direction in expectation. What they do contribute is variance. Sample size for a fixed power scales like `variance / effect^2`. Dilution divides the effect by roughly `1/p` while leaving per-user variance broadly unchanged, so the squared term dominates. Comparing the two designs in units of assigned users, the all-up read needs on the order of `1/p` times more traffic than a correctly logged triggered read to detect the same underlying effect, holding the outcome variance comparable. At a 3% trigger rate that is a factor of about thirty. An experiment sized for a one-week read becomes a seven-month read. ## How this shows up in practice - A team ships a feature that clearly works when you watch people use it, and the readout is a wide interval straddling zero. - The point estimate moves erratically week to week, because the signal is a rounding error relative to the noise of the untriggered majority. - Someone proposes rescuing the result by filtering to exposed users after the fact, which is the right analysis done at the wrong moment and with no control-side record to support it. ## The healthy response Decide the exposed population before launch, instrument the trigger on both arms, and power the experiment on the triggered read while still reporting the all-up number as the business impact. If the trigger rate is genuinely tiny and cannot be instrumented, be honest at design time that the experiment cannot answer the question in the available traffic, rather than running it and reading tea leaves afterwards. ## The number that dilution actually teaches you Dilution is also information, not just an obstacle. A change with a large triggered effect and a 3% trigger rate has a small business impact today. That points at a different investment question: is the next win in making the feature better, or in making more people reach it? The diluted number is the honest input to that decision, and it should never be quietly replaced by the triggered number when someone asks how much the launch was worth.