To restate a guardrail bypass rate at production prevalence you need to know what fraction of live requests are attacks. How do you estimate that, and why is deriving it from the deployed guardrail's own block counts circular?

### Why the estimate has to exist at all Every restatement of a conditional bypass rate into exposure is a multiplication by prevalence — the fraction of live requests that are attacks. The guardrail harness cannot supply it: it ran a corpus that was 100% attacks, so prevalence inside the experiment is 1 by construction. If the factor is invented, the output is invented, and the more confidently it is presented the more damage it does. So prevalence is a separate measurement with its own method, its own cost, and its own failure modes. ### The circularity, stated precisely Let `p` be true prevalence and `b` the fraction of attacks the deployed guardrail blocks. Block counts observe attacks *it caught*: roughly `p × b` of traffic. Attacks it missed are invisible by construction — they left no record distinguishable from benign traffic, which is exactly the population you are trying to size. Two consequences follow. First, the estimate is biased low by precisely the factor under study. Second, and worse, the exposure figure now contains `b` twice in opposite directions: you multiply an estimate proportional to `b` by a bypass rate of `(1 − b)`. Improve the guardrail and measured prevalence *rises*, so the report claims your traffic became more adversarial because you fixed the screen. Nobody reading the slide will unpick that, which is why the rule is absolute rather than a caveat. (The same objection applies, more weakly, to using flag counts instead of block counts: those also include false flags, so they are contaminated from the other side too.) ### Methods, best to worst, with what each costs 1. **Random sample of raw traffic, labelled by humans** against a written definition. Ground truth. Cost: a labeller sustains roughly 100–200 requests an hour, and the definition meeting is not optional — decide in advance whether a curious policy-probing question, a security-research query, or a low-effort "ignore your rules" joke counts, because that single choice moves the number by an order of magnitude. 2. **Stratified sample with recorded weights.** Sample flagged traffic heavily and unflagged traffic thinly, then reweight each stratum to its share of the population. This is the practical route at low prevalence, and it is only valid if the weights are kept and applied. An unweighted stratified sample is a new bias wearing the costume of a cheaper method — it will read far higher than truth, because you deliberately over-sampled the flagged stratum. 3. **An independent second detector with a characterised error profile**, used as a labelling aid with humans adjudicating disagreements. Cheaper, and inherits that detector's blind spots. 4. **An order-of-magnitude figure carried from a comparable product**, explicitly marked in the report as an assumption, not a measurement. ### Where the sampling itself misleads **Sample size at low prevalence.** At one in a thousand, 200 randomly drawn requests contain an expected 0.2 attacks. Finding zero is the most likely outcome and is consistent with anything from 1-in-1,000 to 1-in-100,000. That is why option 2 exists; it is not a shortcut but the only affordable design. **Frame traps.** Sample requests if the bypass rate is per request, sessions if it is per session — mixing units silently changes the answer. Cover the full daily and weekly cycle, because adversarial traffic is burstier than benign and a business-hours sample understates a night-time campaign. Sample *before* any upstream filter (WAF, rate limiter, edge rules): anything dropped earlier is outside your frame and quietly lowers the estimate. **Stale strata.** If the strata were defined by the guardrail's flags and you then change the guardrail, the weights describe a population that no longer exists. Re-derive them, or the reweighting is arithmetic on a fiction. **Labeller disagreement.** Two humans applying the same written rule to ambiguous prompts will differ. Measure that disagreement on a shared subset; if it is large, the definition is the finding, not the prevalence number. ### How to report it Give a range, not a point. State the sample size, the sampling design and its weights, the labelling rule verbatim, and the date. Then show the restated exposure across the whole range. If the conclusion holds at every prevalence in the band, say so explicitly — that is the strongest result available. If it flips, the honest deliverable is that prevalence must be measured properly before anyone can act on the bypass number, and that is a better outcome than a confident figure built on a guess.