When choosing an A/B test OEC, how do you trade sensitivity against long-term alignment?

## Two different properties, often confused **Sensitivity** is a statistical property. It asks: if the treatment really changes this metric by an amount worth caring about, will the experiment detect it? For a per-user average metric estimated from `n` users per arm, the standard error of the difference shrinks roughly like `1/sqrt(n)`, and the scale of the noise is the metric's standard deviation across users, `s`. What matters for detecting a *relative* change is the ratio `s / mean` — the coefficient of variation. A metric with a large coefficient of variation needs far more users to resolve the same percentage effect than a metric with a small one. **Alignment** is a substantive property. It asks: if this metric goes up, is the product actually better in the way the business means over months and years? Nothing statistical tells you this. It is a claim about the world that has to be argued, and ideally checked, not assumed. A metric can score high on one and low on the other, and the pair almost always trade off. ## Why they conflict The outcomes the business really wants — a customer still active a year from now, lifetime value, brand trust — are either unobservable inside an experiment window or enormously variable across users. The outcomes that are easy to move and easy to detect — clicks, page views, sessions — are shallow, and a variant can move them by doing something the user does not benefit from. Take the concrete choice between **revenue per user** and **sessions per user** as the criterion for the same experiment. *Revenue per user* is close to the thing the business ultimately cares about. But spending is extremely uneven: most users contribute nothing in a short window and a small minority contribute a lot, which makes the spread across users large relative to the mean. That large coefficient of variation is exactly what suppresses sensitivity, so a genuine 1% improvement may sit comfortably inside the confidence interval at the traffic a single team can get in two weeks. Reading that experiment as *no effect* would be wrong; it is *no detectable effect*, which is a statement about the design, not the product. *Sessions per user* is much quieter — most users have a small number of sessions, the distribution is tighter relative to its mean — so the same experiment resolves a small relative change. The price is alignment: sessions can rise because users find what they need faster and come back, or because the product became confusing and they had to keep returning. The metric does not distinguish those. That is the trade in its cleanest form: the aligned metric cannot see the effect, and the sensitive metric cannot tell you what the effect means. ## The short-horizon problem A related version of the same tension is time. Suppose the outcome the team genuinely cares about is whether a user is still retained twelve months from now. No two-week experiment can observe that. So the criterion becomes a short-term proxy chosen precisely because the true outcome is out of reach: an early-lifecycle behaviour believed to predict long-run retention. This is a defensible move, but it converts an empirical question into an assumption. The proxy is standing in for the real outcome, and the strength of the whole decision now rests on how well it does that. Two things follow: 1. **The substitution should be argued explicitly**, in the design doc, not smuggled in. Write down which long-run outcome the proxy stands for and why you believe the link. 2. **The link should be validated with data where possible.** The strongest form of evidence is historical: take past experiments where both the proxy and the eventual long-run outcome are now observed, and check whether treatment effects on the proxy predicted treatment effects on the outcome — in direction and roughly in magnitude. Correlation between the two metrics *across users* is much weaker evidence, because a metric can be a fine descriptive correlate of good users while being useless at tracking what an intervention does. A proxy that has never been checked against the outcome it replaces is a hypothesis being used as a decision rule. ## Working the trade in practice A usable procedure: - **Start from the most aligned metric you can name**, whether or not it is measurable. That is the target the criterion is approximating. - **Estimate detectability** for each candidate: given historical variance and available traffic, what relative effect could this metric resolve in the planned window? Anything that cannot resolve an effect of the size the change plausibly produces is disqualified as a criterion. - **Choose the most aligned surviving candidate**, and record what you gave up. - **Try to buy back sensitivity before compromising alignment.** Running longer, pooling traffic across surfaces, restricting to an eligible population where the change can actually bite, and variance-reduction techniques all improve detectability without changing what the metric means. Weakening the criterion should be the last move, not the first. - **Re-examine the choice periodically.** A proxy that was validated two years ago on a different product surface is not automatically still a good stand-in. ## What interviewers are listening for The weak answer treats sensitivity as a property of how important a metric is, or asserts that the business metric must be the criterion because the business cares about it, with no account of whether the experiment can see it. The strong answer separates the two axes cleanly, names the mechanism that makes a metric insensitive (spread relative to mean, at the sample size available), and treats any move toward a shallower proxy as a claim that has to be defended with evidence.