What is an Overall Evaluation Criterion (OEC) in an A/B test?

## What the OEC is The Overall Evaluation Criterion (OEC) is the single quantity an online experiment's decision is defined against. Concretely, before any user is assigned to a variant, the team writes down three things: the metric, the direction that counts as an improvement, and the rule that turns the observed result into a decision (for example, ship if the estimated treatment effect on the metric is positive and its confidence interval excludes zero at the pre-declared level). When the experiment ends, the decision is read off that rule. The word *overall* is doing real work. An experiment platform will compute hundreds of metrics; the OEC is the one that stands for the whole judgment of whether the change made the product better. It is the operational definition of *better* for this experiment. ## Why a single criterion, decided in advance Two distinct problems are solved by fixing one metric up front. **The garden of forking paths.** With enough metrics on a dashboard, some will look good and some will look bad by chance alone. If the decision rule is chosen after seeing the numbers, the analyst is effectively selecting the most favourable of many comparisons, and the reported evidence is far weaker than it appears. Declaring the OEC beforehand removes that freedom: the number that decides was named before it was seen. **Decision deadlock.** If three metrics are all called primary and they disagree — one up, one flat, one down — nothing in the design says what to do, so the call falls to whoever argues hardest in the review meeting. A single criterion forces the disagreement to happen earlier, while the experiment is being designed, which is when it is cheap to resolve. Note that these are different objections. The first is statistical; the second is organisational. Both point the same way. ## What makes a metric fit to be the OEC Three properties matter, and they pull against each other. **Measurable in the window.** The metric has to be observable for the users in the experiment during the time the experiment runs. An outcome that only materialises months later cannot be the criterion of a two-week test, however much the business cares about it. **Sensitive.** A metric can only serve as a decision rule if a real change of plausible size actually shows up in it. Sensitivity depends on how noisy the metric is relative to its mean — a metric with a large spread across users needs many more users to detect the same relative change than a quiet one does. A metric that would need ten times the available traffic to detect the effect you care about is not a usable criterion, no matter how meaningful it is. **Aligned.** Moving the metric has to correspond to the product genuinely improving. This is the property that is hardest to verify and easiest to lose, because the metrics that are easy to measure and sensitive are often shallow ones that a variant can move without helping anyone. The practical craft of choosing an OEC is mostly the negotiation between sensitivity and alignment: the most aligned metric is often too slow or too noisy, and the most sensitive metric is often a shallow proxy. ## The rest of the scorecard Choosing one OEC does not mean ignoring everything else. A well-run experiment reads three kinds of metric with three different jobs: - **The OEC** decides. One metric, one direction, declared first. - **Driver or diagnostic metrics** explain. When the OEC moves, these say through which mechanism — which surface, which step, which segment. They never override the decision; they inform the next hypothesis. - **Checks that can veto** protect against unacceptable side effects. These are a separate concern with their own thresholds and their own conventions, and they are the reason an OEC win is not automatically a launch. The hierarchy matters more than the vocabulary. Interviewers listen for whether a candidate can say which metric decides, which metric explains, and which metric objects — and whether they know that only the first is the OEC. ## Common shapes an OEC takes It is often a simple per-user average — an average of some outcome over the randomised users, so that the analysis unit matches the randomisation unit. It can also be a weighted combination of several components rolled into one number, which keeps the single-criterion discipline while acknowledging that success has more than one dimension; the cost is that the weights themselves become a contested design choice. What it must never be is a metric picked after the data came in, or a list of metrics with no rule for what to do when they conflict.