How do you decide which guardrail metrics get authority to veto a launch across an organisation?

### The question behind the question Deciding which metrics can veto a launch is not a statistics question — it is a question about which tradeoffs the organisation refuses to make, and about who is allowed to make the ones that remain. The statistics only tell you whether a degradation is real and how large it might be. ### Two tiers **Tier 1 — non-tradeable trust metrics.** A short, stable list applied to every experiment: crash-free session rate, unhandled error rate, availability, a hard latency ceiling, and data-integrity or security invariants. These earn automatic veto power for three reasons. They are about the product working at all rather than about strategy; the harm they measure is hard to reverse, since users who hit a broken experience often do not return to see the fix; and no plausible business win is worth them, which means arguing the exchange rate case by case wastes time and produces inconsistent outcomes. Making them non-negotiable is a *simplification*, not a bureaucratic tax. **Tier 2 — business floors.** Revenue on an engagement-focused test, engagement on a monetisation test, support volume, unsubscribes. These are genuinely tradeable — the whole point of experimentation is finding favourable trades — so a hard veto is the wrong instrument. Each gets a written degradation threshold and a named owner empowered to accept a breach as a recorded tradeoff. The distinction to defend in an interview is *why* the tiers differ: tier 1 encodes a refusal, tier 2 encodes a price. ### Setting the thresholds Three properties make a threshold real: - **Set before results are visible.** A threshold negotiated after the numbers land is not a constraint, it is a rationalisation. If a threshold turns out to be wrong, change it prospectively as a standing policy, not retroactively for the launch that tripped it. - **Detectable at realistic sample sizes.** A threshold tighter than the experiment can resolve is theatre: the guardrail can never be shown to be met, so in practice it gets ignored. Check at design time that a typical experiment produces an interval narrower than the margin. - **Attached to a consequence.** Block, escalate to a named person, or fix-and-rerun. A guardrail with no defined consequence is a chart. ### The failure modes on both sides **Too many hard vetoes.** Every metric someone cares about becomes blocking. Two things follow. Alarm fatigue: breaches become routine, so overrides become routine, and the whole regime loses force — which is worse than having fewer, respected guardrails. And avoidance: teams shrink experiments, shorten runs, or ship outside the experimentation system entirely, moving risk somewhere with no measurement at all. **Too few.** Each launch degrades something slightly, every degradation is individually defensible, and the aggregate is a product that is slower, noisier and more annoying than it was a year ago, with no single decision responsible. This ratchet is the strongest argument for guardrails as *cumulative budgets* rather than per-test checks: track how much each shipped experiment consumed and hold a standing total, so the drift becomes visible and someone owns replenishing it. ### Governance details that separate a strong answer - **Ownership.** Each tier-2 guardrail needs a named owner — usually the team whose outcome it protects, not the team running the test. Self-approval of one's own breaches recreates the problem the guardrail was meant to solve. - **A standing set versus per-test additions.** Keep a small, universal list so it is not re-litigated each launch, and allow test-specific guardrails where a change carries a specific plausible risk. - **Alerting versus deciding.** Not every guardrail needs to block a decision; some are better as production monitors after rollout, especially slow-moving metrics that cannot resolve within a test window. - **Review cadence.** Revisit the list periodically. Guardrails accumulate — every incident adds one, and almost nothing removes one — until the scorecard is unreadable and nobody looks at it. - **Escalation quality.** A good escalation records the size of the degradation, its confidence interval, the affected segments, the budget consumed and the decision rationale. That record is what lets a future team see how much room is left. ### The summary position A small non-negotiable core protects the things the organisation will not sell at any price; everything else has a price, a written threshold, an owner, and an audit trail. The measure of a good regime is not how many launches it blocks but whether the tradeoffs it permits are visible, deliberate and attributable afterwards.