How would you gate a release on an automated red-team suite in CI?

## Why a safety gate is not a quality gate Quality evaluation aggregates: you score many cases, take a mean, and ship if it clears a bar, because one weak answer among a thousand is acceptable. Safety does not aggregate that way. One successful cross-tenant data exfiltration is a breach whether the other 999 cases passed or not. So the gate's decision rule is *worst-case over cases*, not *average over cases*, and everything downstream follows from that inversion. ## Structuring the gate **Tier the cases by severity.** Three tiers work well: - **Blocking.** Outcomes that are unacceptable at any rate — data belonging to another user leaving the system, a state-changing or irreversible tool firing on injected instructions, credentials appearing in output. Any new success here fails the build, full stop. - **Budgeted.** Outcomes with real but bounded harm, where a small residual rate is tolerable and the gate compares against the previous release's rate rather than zero. - **Reporting.** Tone, style, policy drift. Recorded, trended, never blocking — putting these in the blocking path is how teams end up disabling the gate entirely. **Make the check deterministic where the goal allows it.** The strongest safety cases have a mechanical success condition: a canary record planted in a second tenant's data whose appearance in output is a decisive leak; an assertion that the transfer tool was never invoked with the attacker's arguments; a check that no outbound URL was constructed containing case data. Deterministic checks are fast, reproducible, and cheap to rerun — which is what lets them sit in a blocking path. Reserve judge models for categories where the harmful outcome is genuinely a judgement about text, and keep those in the reporting tier unless you have measured the judge's agreement with human labels. **Repeat each case.** Because the system is stochastic, a single run of a case is a coin flip against a probabilistic defense. Run each blocking case k times and treat *any* success as a failure. This is the same any-attempt-wins semantics the attacker enjoys, transplanted into CI. It also means your gate's cost scales with k, which is the main reason to keep the blocking set small and severe. ## Splitting fast from thorough Adaptive red-teaming — an attacker, human or automated, iterating against this specific build — is the evidence that actually characterises safety, and it is far too slow and expensive to run per commit. The usual layering: - **Per release candidate:** the fixed blocking set, repeated k times, deterministic checks, minutes not hours. - **Nightly or pre-release:** a broader campaign including adaptive attempts and best-of-N resampling, whose results inform the launch decision rather than blocking a merge. - **Per incident:** whatever happened in production becomes a permanent blocking case within a day. That last rule is the one that compounds. A red-team corpus grown from real incidents is worth more than a much larger corpus of invented prompts, because it encodes failures the system has actually demonstrated it is capable of. ## Handling the awkward parts **A new success appears and you cannot fix it before the deadline.** The gate should force an explicit, recorded decision — an owner, an expiry, and a compensating control such as disabling the affected tool or gating the feature behind approval — rather than a silent override. Overrides that leave no trace are how a gate stops meaning anything within two quarters. **The suite is a target itself.** Anyone with repository access can see exactly which attacks are checked, which makes the corpus a description of your known blind spots. Treat it as sensitive, and never let "passes the corpus" become the definition of safe — that is the static-suite trap in institutional form. **Cost.** Blocking cases run on every candidate, times k, times model cost. Keep the blocking tier lean and severe; push breadth into the nightly campaign. If the gate is slow enough that people route around it, it has failed regardless of how good its cases are. ## What good looks like A release pipeline where a candidate that lets injected content trigger an exfiltration path simply cannot ship; where the gate's failures are legible enough that an engineer sees the trajectory, the tool call and the canary in one place; where every production incident is represented; and where the number the launch decision actually rests on comes from the slower adaptive campaign, not from the fast gate. The fast gate stops regressions. It does not certify safety, and saying so out loud is part of a good answer.