Your guardrail test corpus is half attack items and half benign items, but production traffic is overwhelmingly benign. How do you choose the corpus mix, and how do you report the results so the numbers still say something about production?

**Why balanced, and what balancing costs you.** Attack traffic against a live endpoint is a thin slice of volume — often well under one percent. Sample a corpus in proportion and nearly the whole hand-labelling budget buys ordinary requests, while the attack half stays too small to distinguish a good block rate from a bad one. Oversampling the rare class is the standard and correct answer. The price is that the corpus no longer resembles production in composition, and that mismatch has to be handled in the reporting instead of quietly ignored. **Which quantities survive the reweighting, and which do not.** A rate estimated *within* one class does not depend on how many of the other class you supplied. The share of attack items blocked is computed over attack items only; the share of benign items wrongly blocked is computed over benign items only. Both transfer. Any quantity that mixes the classes — above all "of everything the guard blocked, what fraction were genuinely attacks" — is a function of the mix you chose, and it moves violently with the base rate. Work it through with a guard that blocks 80% of attacks and 3% of benign items: | corpus | attacks | benign | true blocks | wrong blocks | share of blocks that were attacks | |---|---|---|---|---|---| | 50/50 test corpus | 500 | 500 | 400 | 15 | ~96% | | one production day | 50 | 10,000 | 40 | 300 | ~12% | The guard did not change between the rows. Only the denominator under its positives did. Quoting the 96% to a decision maker is the classic error in this leaf, and it is how a threshold nobody would have accepted gets shipped. **How to report so the numbers say something about production.** Per slice, carry: attack-item count and share blocked; benign-item count and share wrongly blocked; the provenance of each half; and the guard version and threshold. Then add a projection section that applies the two within-class rates to real volumes — at 10,000 benign requests and 50 attack attempts a day, a 3% wrong-block rate is about 300 blocked legitimate requests daily and a 20% miss rate is about 10 attacks through. State the assumed attack volume in the open, because it is the softest number in the report and the one a reader should be free to argue with. **The transfer condition people skip.** Within-class rates transfer *only if that class's composition resembles production's*. The benign half is usually built deliberately heavy on near-miss items, precisely so it discriminates — which means its wrong-block rate is a near-miss rate, not a traffic rate, and projecting it directly onto total benign volume overstates the cost, sometimes by an order of magnitude. The fix is to stratify: keep easy-benign and near-miss-benign as separate strata with separate rates, estimate each stratum's real share of traffic, and reweight. If you cannot estimate the shares, say so and report the stratum rates unreweighted rather than presenting a blended figure that belongs to no population. **What it costs.** The corpus size sets both bills. Labelling is the large one — hundreds of items, minutes each, with a double-labelled sample on top — and slicing multiplies it, because a per-language or per-category rate needs enough items *per cell* to be worth quoting; forty items in a slice gives a confidence interval wide enough to swallow most decisions. Inference is the small one for a hosted guard (cents to a few dollars, minutes of wall clock for a full sweep) and a real one for a self-hosted guard on GPU or an in-situ run that pays application-model tokens per item. Decide the slice list before labelling, since it determines how many items you must buy. **Where the numbers mislead.** Beyond the precision trap: a single blended accuracy over a mix you composed is an average over your own sampling decision and means nothing outside it. A rate quoted without its item count invites a percentage resting on three items. Two guard candidates with equal aggregate scores can differ wildly per language or per category, and the aggregate hides it. And a flattering catch rate measured on a contaminated attack half poisons the projection downstream, so provenance belongs in the same report as the volumes. **What to check.** Recompute the projection yourself from counts, not from percentages, and see whether the arithmetic reproduces the headline. Confirm both rates came from one run at one threshold on one pinned guard version. Ask what the benign half is made of and whether its strata were reweighted. Put a confidence interval, or at minimum the item count, on every rate. Finally, make sure the threshold decision is framed as an explicit trade — blocked legitimate users against prevented incidents — and put it in front of whoever owns the product, rather than letting it be settled implicitly by whoever picked the vendor's default.