You are red-teaming a chat endpoint that sits behind a safety classifier — a guard that is itself a model, such as Llama Guard or ShieldGemma, which reads text and returns a verdict. Why does each test case in that run cost more than testing the chat endpoint alone, and how should that shape the size of your test set?

## What this layer actually is, mechanically A safety classifier is not a deny-list. It is a model. The self-hosted ones — Llama Guard, ShieldGemma, Granite Guardian, WildGuard, Prompt Guard — are small language models (roughly 1B to 8B parameters) that are handed a prompt template listing a fixed set of hazard categories together with the text under review, and asked to emit a short verdict: a safe/unsafe token, usually followed by the codes of the categories it believes apply. The hosted ones — the OpenAI moderation endpoint, Azure AI Content Safety, Perspective API — do the same thing behind an HTTP call and return per-category scores. Everything about the economics of testing one follows from that single fact: producing a verdict is an *inference*, so it consumes tokens or GPU time, it is billed or queued, and it occupies wall-clock time inside the request path. Contrast the alternative your instinct is calibrated on. A regex or deny-list guard matches in microseconds, costs nothing per call, and can be exercised a million times over lunch. Against a pattern filter, test-set size is limited only by how many strings you can think of. Against a model guard, it is limited by budget. ## Write the arithmetic before you write prompts For a set of N distinct prompts, K samples per prompt, with input screening and output screening both enabled: ``` chat inferences = N x K guard inferences = N x K (input screen) + N x K (output screen) total ~ 3 x N x K ``` So 200 prompts at 3 samples is 600 chat calls and about 1,200 guard calls — 1,800 inferences, not 600. Add a multi-turn probe where the guard rescreens the growing conversation on every turn and the guard term grows with turns as well as samples. Add a paraphrase cluster of eight wordings per harm and you have multiplied the whole thing by eight. ## What it costs, concretely On hosted infrastructure the cost is a bill plus a rate limit, and the rate limit usually binds first: two providers, two quotas, and the slower of them sets your wall-clock. On self-hosted infrastructure the guard is not free either — it is GPU residency and throughput contention. A small guard model serving short classifications will do order tens of requests per second on one modern accelerator, which sounds ample until you notice it is sharing that accelerator with the thing you are testing. The third and most underrated cost is engineer time: guard verdicts arrive as text that must be parsed, mapped to your labels and triaged, and triage does not scale the way generation does. ## Where the number misleads **The pack-size illusion.** A downloadable pack of 100,000 harmful prompts is quoted as coverage. At three inferences per case it is 300,000 inferences, which you will not run. What actually happens is that someone samples two per cent of it, and the coverage denominator silently becomes the sample while the report still names the pack. **"It is a small model, so it is cheap."** Cheap per call, expensive per run, and — the part that hurts — *serial*. Input screening sits ahead of the chat call, so it adds directly to time-to-first-token; it does not amortise away under load. A guard that is inexpensive on the invoice can still be the dominant term in the product's p95 latency, and that is a legitimate finding even when every verdict it returns is correct. **Retries and fail-open.** A rate-limited or timed-out guard call that the harness retries is paid for twice, and if the harness records both attempts it is counted twice in the results. Worse, many integrations fail open on timeout: the fastest, cheapest-looking cases in your log may be exactly the ones where the guard never ran, which quietly inflates the clean rate. ## What you would check Reconcile counted guard calls against expected calls — if you sent 600 samples with two-sided screening, you should see about 1,200 guard invocations, and a shortfall is a wiring or fail-open bug, not a security result. Log the guard's tokens and duration separately from the chat call's, so latency and spend can be attributed. Pull the error and retry counts and subtract retried duplicates before computing any rate. Confirm spend against the plan mid-run rather than at the end. And before the real measurement pass, turn off any verdict cache you used while debugging, so you are measuring the guard rather than your own dictionary.