When should a guardrail metric stop a rollout whose primary metric is up?

## Why a single metric is never enough An online experiment optimizes what you point it at. If you point it at one number, you will get that number moved — sometimes by mechanisms you would never have accepted if you had seen them. A model that produces bolder recommendations raises acceptance and raises returns. A support assistant that answers more confidently resolves more tickets on first contact and quietly resolves some of them incorrectly. A retrieval change that improves answer quality doubles token spend. In each case the primary metric is honest and the change is still bad. Guardrail metrics exist to make those costs visible and binding. They are the metrics you commit, in advance, to respect regardless of how the headline reads. ## Primary versus guardrail versus diagnostic Three roles are worth keeping distinct. The **primary metric** is the one the experiment is powered to detect a change in, and the one the ship decision is nominally about. There should be one, or a small pre-agreed composite. Having three primaries means having none, because you will pick the winner after the fact. **Guardrail metrics** are harms with thresholds and directions declared up front. They are not expected to move; the question asked of them is not "did this help" but "did this break something". They are often noisier and less powered than the primary, which is deliberate — you are watching for damage, not estimating an effect precisely. **Diagnostic metrics** explain mechanism. They neither decide nor block; they tell you *why* the primary or a guardrail moved. Confusing diagnostics with guardrails leads to rollouts halted by noise on metrics nobody was going to act on. ## What deserves to be a guardrail Five families cover most real programs. **Downstream cost the primary ignores.** The primary sits at one point in a funnel; harm often lands later. Acceptance now, refund later. Resolution now, repeat contact later. Any metric that captures "the user came back unhappy" belongs here. **Safety, policy and compliance.** Rate of policy-violating outputs, unauthorized disclosures, refusals to comply where compliance is legally required. These are usually zero-tolerance rather than threshold-based: one confirmed violation stops the ramp. **Operational health.** p95 latency, error rate, timeout rate, dependency saturation. LLM changes routinely trade latency for quality, and the trade must be explicit. **Unit economics.** Tokens or cost per request, and cost per successful outcome. A change that improves quality at triple the cost is a business decision, not an automatic ship. **Segment protection.** An aggregate win frequently hides a segment loss — a language, a device class, a customer tier, a high-value cohort. A guardrail on the worst-affected key segment prevents shipping an average improvement that damages the customers you can least afford to lose. ## Setting thresholds honestly A threshold is a policy statement about acceptable harm, so it should be set by whoever owns that harm and written down before data arrives. Two failure modes are common. **Too tight**, and every experiment breaches something by noise, guardrails get overridden routinely, and the mechanism dies of alarm fatigue. Guardrails are usually stated as non-inferiority bounds — "refund rate must not rise more than X" — with the bound chosen so that ordinary week-to-week variation does not trip it. **Too loose**, and a real harm passes because the bound was set where it could never bind. The sanity check is to ask what movement you would actually act on, then set the bound just inside it. Multiplicity is real: watch fifteen guardrails and something will breach by chance. Keep the list short, prefer metrics you would genuinely act on, and treat a breach as a trigger for investigation rather than an automatic verdict — while keeping the default action *stop*. ## Automatic halts versus human decisions Some guardrails should stop the ramp without a meeting: safety violations, error-rate spikes, a hard latency ceiling. These are fast, unambiguous and cheap to be wrong about, so they are wired to automation. Others — refund rate, escalations, cost — are slower to accumulate and need interpretation. The right pattern is that a breach *pauses further ramp* and forces an explicit, recorded decision with a named owner, rather than either auto-killing a possibly-fine change or letting it drift forward while people argue. Note the asymmetry: pausing is cheap and reversible, ramping into a real harm is not. ## The worked case A 50/50 test of a grocery substitution recommender: primary metric is substitution-acceptance rate, guardrail is refund rate, with a non-inferiority bound agreed before launch. Acceptance rises 3%; refunds rise 1.5%, breaching the bound. The correct call is to hold the ramp, because the guardrail is telling you the mechanism of the win — the model is proposing substitutions people accept in the moment and reject on arrival. Shipping would book the acceptance gain and pay for it in refunds, logistics cost and trust. The next step is to slice: is the refund rise concentrated in one product category, in which case the fix is narrow, or is it spread, in which case the model's substitution policy is simply too aggressive. ## What interviewers listen for The expected answer is that guardrails are declared in advance, encode harms outside the primary metric's view, and bind even against a winning headline. Strong candidates add threshold-setting judgment, the multiplicity problem, and the split between automatic halts and human decisions.