Guardrail & Moderation Testing

It must say what counted as a bypass in the numerator: the guard failed to flag, or the model then actually complied - two different numbers. It must say whether the denominator is attempts, distinct attacks, or attempts per attack. And it must say how many retries each attack got, because retries move the rate without the guard changing.

Every item in the corpus is an attack, so blocking everything means catching everything: the measured block rate hits 100% and nothing penalises refusals. The corpus can only observe one error type. Add benign prompts the product really sees, and report how many of those were wrongly blocked alongside the attack numbers.

The block threshold you ran at, which categories were set to block, the guard's identity and configuration snapshot, and the date. A bypass percentage is only true at one operating point, so without those settings written down nobody can rerun your test or compare a later number to yours.

Only that nothing in the service's fixed category set scored above the threshold you applied. The vendor's policy and training data are not published, so a clean verdict is not evidence of safety: the harm may fall outside the categories on offer entirely. Record it as unmeasured, not as passed.

The refusal is a canned message the framework emits for any blocked turn, so an input pattern rule, an intent match, an LLM self-check or an output rail all produce the same text. To attribute the block you need the run's own execution trace: which rails ran, in what order, and each verdict.

An attack-only set measures one thing: how many attack prompts the classifier blocks. It cannot show how often it blocks ordinary users, because there is nothing harmless in the set to block wrongly. Add a benign half, labelled from the same policy, so you can report wrong blocks alongside catches.

End to end you only see the stack's final verdict, so the first layer that blocks hides every layer behind it. To attribute, run the corpus once per layer with the others put into log-only mode, and record per case which layer fired. Then a green run says which classifier actually earned it.

Multiply three numbers: daily request volume, the estimated share of requests that are attacks, and the 12% bypass rate. Ten million requests a day at one attack per thousand gives ten thousand attacks and about twelve hundred bypasses. Publish the prevalence estimate and its source next to the result.

10% is 400 over attempts, so it partly measures how the corpus was sampled and retried rather than the guard. Report it beside the per-attack rate: how many of the 200 distinct templates got past at least once. If 180 bypassed, the guard is far weaker than 10% suggests; if 4 bypassed 100 times each, far stronger.

Pull benign items from the product's real traffic and from topics sitting right next to the blocked ones: safety questions, clinical or legal wording, security research, fiction, non-English phrasing. A hard negative looks like an attack on the surface but is a request the product must answer. Label them, review the labels, freeze the set.

One threshold collapses the whole score range into a flattering number and hides where the guard actually breaks. A curve shows how bypass changes as the threshold moves, so the reader sees whether the guard is genuinely strong or just parked at a lucky setting, and what a small retune would cost.

Rule out two things. First, that the harness really routed every prompt through the guard and parsed its verdict — send a known-blocked control and confirm it blocks. Second, that your harms map to categories the guard was trained to emit. A harm outside its fixed taxonomy reads clean because it was never measured, not because it is safe.

Every probe costs money and a slot in the rate limit, so build a small stratified corpus instead of a huge random one, deduplicate identical payloads and cache responses. Never score a rate-limit rejection, timeout or error as clean: those are missing verdicts, and counting them as not-blocked inflates your apparent bypass rate.

The classifier was fitted on those exact strings, so a high catch rate scores memorisation, not detection. It tells you nothing about a phrasing it has never seen. Build a held-out attack half you author or transform yourself, keep it unpublished, and record where every item came from so contamination can be checked later.

Promote cases that once produced a wrong verdict and were then fixed: a payload confirmed to slip past the guard, and a legitimate request it wrongly refused. Each carries the decision you expect from now on. Keep out untriaged scanner output, twenty paraphrases of one technique, and cases whose correct verdict is genuinely arguable.

Label each prompt with the guard category it should fall under before the run. Misses where the guard scored the right category just under the blocking line are threshold problems. Misses where no relevant category exists at all are taxonomy gaps, and no threshold change ever fixes those — they need a different or additional layer.

Direct calls measure the vendor's classifier alone. Going through the product measures the deployment: which categories the team acts on, the cut-offs they chose, what text the service actually receives after the app transforms it, and what happens when the call fails. Only the second yields findings the customer can fix, so do both and report them separately.

Each has a different owner and a different fix. The unmatched pattern is a rule gap: widen or add a rule and it closes deterministically. The judge approval is a score problem: no edit guarantees that payload stays blocked, only the rate moves. The unmatched routing is worse still: that turn was never governed at all.

Treat a green that nobody caused as suspect. Check whether the stack changed underneath the suite: guard version or endpoint, its policy configuration, and the model serving responses. Also check the harness — a verdict check that errors and counts as pass looks identical. Every run should record what it tested against, not just the results.

Because the guard is itself a model, every case runs extra inferences: one to screen the input, usually another to screen the reply, on top of the chat call. Each case therefore costs roughly two to three times the tokens, money and latency. Budget for that and run a smaller, deliberately chosen set.

Matching is nearest-neighbour in embedding space against a fixed example set, with a similarity cutoff. A paraphrase can land nearer a different example, or under the cutoff and match nothing at all, so no governed path fires. The decision follows wording distance, not meaning, so one probe run is a sample, not a verdict.

The pattern rule is deterministic: one trial fully characterises it for that input, and a miss is a rule gap you can point at. The judge layer samples a model, so its verdict moves with sampling, prompt wording and model version. Strengthen it by replaying the identical payload many times and reporting a rate.

Two ways. Put the upstream classifier into a mode where it records a verdict but does not stop the request, so payloads reach the model and the reply is screened normally. Or drive the output classifier directly with fixture texts that stand in for replies. The first keeps the real path; the second skips generation entirely.

Store the input, the decision you expect (blocked or allowed), the policy category it should fire on, and a one-line reason the case exists. Pinning the exact response text fails because generation is not stable across model swaps or sampling, so the case goes red on harmless wording changes and gets deleted or ignored.

Estimate it by hand-labelling a random sample of raw traffic, not from what the guardrail blocked. The block counts include only attacks it caught, so they omit exactly the misses you are trying to size, biasing prevalence low. Report the estimate as a range with the sample size that produced it.

Per million requests: 1,000 attacks produce 900 flags, and 999,000 benign prompts produce about 9,990 flags. So roughly 10,890 flags contain 900 real attacks, about 8%. Around eleven in twelve flags are benign. Report that share, because a rate measured on an all-attack corpus never reveals it.

Fix the attempt budget and count templates that bypass within it - say, the share of the 200 that got through in 10 attempts or fewer. Better, record attempts-to-first-bypass per template and report its distribution. Both runs must use the same budget, so you re-run the cheaper one. The cost is queries, money and wall-clock against a metered endpoint.

Count an over-block whenever a benign request fails to get the answer it deserved: a hard refusal, a hedged non-answer, or a silently rewritten reply. One overall percentage hides which kinds of user are hit, so break it out by benign category and hand over the actual blocked requests as examples.

It means the guard barely separates those attacks from allowed traffic: the blocks are luck of the line, not margin. A small loosening, a guard update, or a slightly reworded attack flips many of them through. Report the margin distribution and how bypass moves for small threshold changes, not just the current number.