Which cases earn a permanent slot in a guardrail regression suite — the fixed set re-run whenever a content guard is upgraded or the model behind it changes — and which should stay out?

### What the suite actually is A guardrail regression suite is a fixed, version-controlled list of cases. Each case is a stored input plus the verdict you expect back, and a harness executes the whole list on a trigger: a guard version bump, a policy or threshold edit, a system-prompt change, or a change to the model serving responses behind the guard. "The guard" here means whichever component returns allow-or-block — a hosted moderation endpoint such as OpenAI Moderation or Azure AI Content Safety, a classifier you host yourself such as Llama Guard, ShieldGemma, Prompt Guard or Granite Guardian, or a rule framework such as NeMo Guardrails or Guardrails AI. The suite is not a hunting tool. Hunting is what a broad scanner sweep does: generate many novel probes and see what lands. The suite's claim is narrower and far more durable — *a thing that was once wrong is still right after the stack changed underneath it*. Every promotion rule below falls out of that one sentence. ### The promotion test A case earns a permanent slot when two records exist: the guard returned the wrong verdict, and a specific change made it right. Two shapes qualify. First, a payload confirmed to have got past the guard and then closed. Second, an in-policy request the guard wrongly refused and that was then unblocked. Both are stored with the decision expected from now on, so the suite can fail in either direction — an upgrade that re-opens the bypass, and an upgrade that buys its block rate by refusing normal traffic. Filters to apply per candidate: - **Human-confirmed?** A tool hit is not a finding. Promoting an untriaged hit pins whatever the detector happened to say that night, not a defect the product had. - **Is the expected verdict uncontested?** If two reviewers label the same case differently, no future run can be right about it. It alternates red and green forever and teaches the team that red means nothing. - **Is it dominated by another case?** Forty paraphrases of one technique all move together, so the marginal thirty-nine cost inference and detect nothing extra. Keep one deliberate representative, record the cluster size, sample the rest on a slower cadence. - **Does it have a benign twin?** Most guard "improvements" are bought rather than earned: the block rate rises because the classifier got more aggressive. A small floor of plainly in-policy requests, expected to pass, is what makes that visible. - **Will it survive a swap?** A case that only lands because of one endpoint's phrasing quirk stops meaning anything the day the model behind the guard changes. ### What a run costs Each case is at minimum one guard call; a case that exercises the end-to-end path is also a chat completion, so a 2,000-case suite bills roughly 4,000 calls per run. Per run that is small — cents to a few dollars against a hosted moderation tier plus a mid-size chat model. Per month it is that figure multiplied by every trigger, and the trigger count is the part that grows quietly: a guard bump, three policy edits, a nightly cadence and per-PR runs turn one run into fifty without anyone deciding to spend more. Wall-clock binds sooner than money. Two thousand cases at twenty concurrent requests and roughly a second of latency is a few minutes; the same suite against a vendor tier that rate-limits you to a handful of requests per second is the better part of an hour, and an hour-long suite quietly stops running on the trigger and starts running weekly — at which point it no longer catches the upgrade that broke things. The largest line is human: converting one raw hit into a promotable case (confirm the wrong verdict, fix or accept it, write a stable expectation, record provenance) is tens of minutes of review, which is exactly why 300 overnight scanner hits are not 300 candidates. ### Where the number misleads The output of a run is a pass rate, and a pass rate is read as safety. It is not. A green run means only that *these specific known defects have not returned*; it says nothing about behaviour nobody has yet promoted a case for. Two concrete misreadings follow: - **An attack-only suite's pass rate rises when the guard gets worse for users.** Every case expects a block, so a guard that blocks more indiscriminately scores a perfect run while refusing legitimate traffic. The over-blocking regression is invisible by construction, not by accident. - **Case count is not coverage.** Forty paraphrases of one technique inflate the count forty-fold while covering one behaviour. The denominator that matters is distinct techniques or distinct policy categories exercised, not rows in the suite. ### What to check before believing an inherited suite Count how many cases have ever failed since promotion, and which ones. Count expected-allow cases — if that number is zero, the suite cannot fail on over-blocking, whatever its pass rate says. Cluster the inputs by similarity and compare distinct techniques against total cases. Check that each case carries a provenance line. Finally, read the CI configuration rather than the README to learn which events actually fire a run: a suite that only runs nightly is not protecting the guard upgrade that shipped at noon.