Metrics & Guardrails

A guardrail is a metric the experiment must not damage — latency, crash-free sessions, unsubscribes — and a big enough degradation can veto the launch. A driver metric only explains why the primary metric moved; it never vetoes.

Almost all the variance in revenue per user comes from a handful of very large spenders, so the interval around the treatment-control difference stays wide. Ordinary experiment sizes then cannot resolve realistic revenue effects.

The Overall Evaluation Criterion is the single metric, agreed before launch, that an experiment's ship-or-not decision is defined against. It expresses what the team means by the product getting better, so the result cannot be reinterpreted afterwards.

A DAU/MAU of 0.2 means the typical monthly active user opens the product on about 6 days out of 30. It measures return frequency, not audience size, and averages over very different user types.

Capping replaces values above the 99th percentile with that value, so you estimate mean capped spend, not mean revenue. Variance falls sharply and power rises, but a real effect in the tail is shrunk or erased.

Pick the most aligned metric that can still detect a realistic effect with the traffic you have. Sensitive short-term metrics decide fast but drift from what the business wants; truer long-run metrics are often too noisy or too slow to see.

A per-impression standard error treats every impression as an independent draw, but impressions from one user are correlated and a few heavy users supply most of them. The effective sample size is users, not impressions.

N-day retention counts users active on exactly day N, rolling counts users active on day N or any later day, and bracket counts users active at least once inside a day range. Each rule admits a different set, so the numbers differ.

The treatment almost certainly increased sessions per visitor. Sessions are an outcome, not a fixed denominator, so extra sessions dilute the per-session rate even while more visitors convert. Report the per-visitor number as primary and sessions per visitor beside it.

Confirm the 20ms is real and not an instrumentation artefact, compare it against the pre-agreed latency budget, segment it by device, then ship, fix, or escalate. Never move the threshold to fit the result.

A 28-day window counts anyone with a single qualifying action in the past four weeks, so it reads higher, moves slowly and hides recent declines. A 7-day window reacts faster, is noisier, and is far more exposed to day-of-week effects.

Only users whose eligibility is settled by facts that the treatment cannot change, snapshotted at assignment. Narrowing to a subgroup such as users with a saved payment method is safe if that status was already true at assignment, and unsafe if the treatment can create it.

Track what a revenue win costs users: unsubscribe and churn rate, ad-blocker installs, return-visit frequency, complaints, and page latency. Short-run ad revenue almost always rises, so revenue alone settles nothing.

A non-significant result means the data does not rule out zero difference; it never proves the difference is zero. Claiming no harm needs a stated degradation margin and a guardrail interval sitting entirely inside it.

Winsorising clips extreme values to a threshold but keeps every randomised user, so the arms stay comparable. Trimming deletes users for their own outcome, conditioning on something the treatment could have caused.

Fix one numeric threshold before the experiment reads out, derive it from pre-period data the treatment cannot have touched, and apply that same number to both arms. Report results at neighbouring thresholds as a sensitivity check.

Clicks measure attention captured, not value delivered. A sensational headline earns the click before the user can tell whether the content is worth it, so a variant that misleads scores as a win on the criterion while making the product worse.

The delta method takes a first-order Taylor expansion of the ratio of two per-user means around their expectations, then reads off the variance. The result combines the numerator variance, the denominator variance and, subtracted, their covariance.

Aggregate each user into revenue and order-count totals, then compute uncertainty across users: either the delta method on those two per-user means, or a bootstrap that resamples users and carries all their orders. Never treat orders as independent rows.

A curve flattening at a positive level means part of every cohort becomes a durable core, so the active base compounds as you keep acquiring. A curve decaying to zero means the base is capped by acquisition rate times average lifetime.

Rank funnel steps by expected additional conversions - users lost times a plausible lift times downstream conversion - not by the worst step rate. And confirm the drop is real rather than a tracking gap before investing.

Treat the uncapped 6% as unresolved until you see its confidence interval and how much survives dropping the largest few spenders. The pre-registered capped metric governs the launch; a whale-driven swing calls for a tail-targeted follow-up, not an override.

Put the components on a common scale first, then set weights from an exchange rate the business will defend — how much consumption one subscription is worth. Weights are a stated value judgment, fixed before launch, not a statistical output.

Its denominator is another outcome. Mobile share rises when mobile orders grow, when desktop orders shrink, or when total orders fall — including outcomes nobody wants. Report it beside the underlying per-visitor counts so a reader can see which side moved.

Queries per user has no fixed good direction. More queries can mean users are engaged, or that results were bad and they had to keep reformulating. A criterion whose direction is ambiguous cannot decide a launch on its own.

A cohort table is triangular: recent cohorts have not lived long enough to appear in later columns. Averaging a column therefore describes only the older cohorts that reached that age, blending different acquisition weeks and channels.

Use a bounded proxy when the hypothesis is about whether users buy at all, not how much they spend, and revenue per user cannot be powered at available traffic. A yes/no purchase indicator has variance at most 0.25.

Give hard veto power to a few non-tradeable trust metrics such as crash-free session rate. Everything else gets a written threshold and a named owner who can accept a recorded breach.

Quantify the damage first by re-analysing past experiments with user-level variance and counting how many decisions flip. Then make correct variance the platform default for ratio metrics, and prepare teams for wider intervals and larger sample-size requirements.