Guardrail & Moderation Testing

Sweep whatever the service does expose instead of a score: each selectable strictness level and each category toggle. Run the same corpus at every setting and report a step curve over settings rather than a smooth score curve. Pin each configuration exactly, and state that the resolution is limited to the settings offered.

Add cases that probe judgement rather than matching: the same harm in many paraphrases, other languages, and unusual formatting; long or multi-turn inputs where only part of the text reaches the guard; and text whose framing addresses the guard's own reading of the content. A deny-list needs none of these because it has no judgement to sway.

Assume the service changed behind the endpoint; you cannot pin a version. Re-run the stored payload alongside a control set of cases that flagged during the original run. If the controls still flag and the payload now does too, the behaviour moved. Report that with timestamps instead of defending the old result.

If the classifier's own verdict becomes the label, it agrees with itself by construction and every error rate collapses to zero. The label must be an independent human judgement against the written policy. That also means benign items should be chosen near the policy line, not far from it, or the pass half proves nothing.

Because the two 90s may cover the same 450 cases. What matters is per-case overlap, not per-layer totals. Record every layer's verdict for every case in one table, then count the cases only one layer caught. That unique-catch column is the layer's marginal contribution; the aggregate rate hides it.

Prevalence is partly chosen by attackers, so it is not a fixed property of your traffic. A targeted attacker sends 100% attacks and experiences the raw conditional rate; a campaign can raise prevalence overnight. Present both the ambient restatement and the targeted case, and state the prevalence assumption on the same page.

Split the corpus. Freeze a versioned baseline slice and report the bypass rate over that fixed set of distinct attacks at a fixed attempt budget - that number alone is compared quarter to quarter. Report newly added templates as a separate first-run number, never blended in. Blending makes the trend move when the corpus changes rather than the guard.

Spend enough on benign traffic to detect an over-block rate the product would actually care about, then give the rest to attacks. Size the benign half from the smallest rate worth acting on, not from what is left over. Defend it by noting a catch-rate-only report cannot tell a good guard from a closed door.

Hand over a function, not a constant: bypass across the threshold range, the operating point it was measured at, and the raw per-attempt scores so any future setting can be re-derived. Fix the operating point before results are seen, and make a threshold change trigger a re-measurement rather than a reinterpretation of your number.

Map every policy category to a guard category first and mark the unmapped ones unmeasurable at this layer. Report two numbers, never one: prompts blocked out of prompts sent, and how many policy categories this instrument can express at all. Unmapped categories go in the report as untested, with an owner, not as passing.

Spend little on proving a vendor classifier imperfect; the customer cannot fix it. Spend the hours on the decisions they own: which categories they act on, the cut-offs, what text is screened, and what happens on error or timeout. The deliverable should recommend configuration, fallbacks and a residual-risk statement, not a vendor swap.

Keep the balanced mix for measurement — you need enough attack items to estimate the block rate at all — but never quote a corpus-level precision as if it were production's. Report the two rates separately, then project the daily wrongly blocked volume using production's real benign volume and its base rate.

Decompose only where a decision hangs on the answer: a stage someone wants to remove, or one that costs latency or money. Group the rest. Budget one end-to-end baseline plus an isolation run per stage you are actually questioning, and hand back unique-catch per stage against its cost, not four separate pass rates.

Cut redundancy before cutting evidence. Collapse near-identical paraphrase clusters to one representative plus an occasional sample, and drop cases whose defect class no longer exists in the product. Never retire on the grounds that a case has never failed: a case that never fails is doing its job, and it is often the only witness to one defect.

It matches for layers that read the same fixed user text, like two input-side classifiers: OR-ing their independent verdicts predicts the stack. It breaks for anything downstream of generation, because the output classifier reads a reply that only exists if the request ran, varies between runs, and changes if an upstream layer rewrote the input.

Report three numbers with their own denominators and refuse the single one. Rules exercised out of rules configured is countable. Routing is a continuous space with no enumerable denominator, so report a fall-through rate over paraphrase families. The judge layer is a pass rate at a fixed repeat count against a named judge model.