In a rail stack, one layer matches a fixed pattern rule while another asks a judge model whether the turn should be allowed. Why is a single pass from the judge-backed layer weaker evidence in a red-team report than a single pass from the pattern rule, and how do you strengthen it?

### Two mechanisms, two kinds of evidence A pattern rule is a fixed predicate: a regular expression or a deterministic action inside a flow, applied to the turn. It partitions inputs. One trial fully characterises it for one input — it matched or it did not, and it will do the same tomorrow, next quarter, and after a model swap, because no model is involved. A judge-backed rail is a model call wearing a rail's clothes. In NeMo Guardrails, `self check input` renders the template registered under `task: self_check_input` in `prompts.yml`, which wraps the user turn in an instruction ending in a yes/no question, sends it to the model named under `models:`, and parses the completion into an allow/block decision. Every ingredient of that pipeline is a variance source: the decoding settings, the exact instruction text (which is versioned in your config, not in the vendor's), where the payload sits relative to the judge's own soft decision boundary, and which checkpoint the provider is currently serving behind a stable model name. **A single pass therefore has two explanations you cannot distinguish: the layer is blind to this payload, or you drew one sample from a near-boundary distribution.** ### Turning an observation into a measurement Replay the byte-identical payload a fixed number of times against a pinned judge model and report the share that passed. Report the fixed rule's result the same way and let the contrast do the work: it will be a clean 0 or 1, and that contrast *is* the finding's shape. Pin what you can see — judge model identity, decoding settings, the prompt template version, the rail config. And separate the variance you measured from the variance you caused: if you regenerate the payload on each attempt, you have measured your own generator's spread, not the rail's. ### What it costs Every replay is a judge call, plus an application generation for each replay that gets through, plus an output-judge call if one is configured. Twenty payloads at n=30 is 600 judge calls and up to 600 more completions downstream — tens of dollars and, at a few seconds each with modest concurrency, an hour or more of wall clock. Then double it: a judge-layer ticket is only verifiable by re-measuring at the *same* n after the fix, so the lifetime cost of one such finding is at least 2n runs. That is the real reason teams verify these with a single replay, and it is exactly the corner that invalidates the verification. ### Where the number misleads The dangerous reading is a small-n point estimate. **1 approval in 30 replays is not "a 3.3% bypass rate"** — at that sample size the plausible range runs from a fraction of a percent to something near a fifth, so a "3.3%" that later reads "9%" may be pure sampling, not regression. Report the count and the n, or an interval; never the bare percentage. The second trap is **false determinism**: setting the judge to temperature 0 removes sampling noise and nothing else. The prompt template, the served checkpoint and batching-level numerical nondeterminism all still move the verdict, and a payload sitting on the boundary is still on the boundary — greedy decoding does not make the boundary hard. The third is scope creep: a judge-rail pass rate is one layer's rate on one payload family on one day, and it is routinely quoted upward as the application's safety number. Keep it out of executive summaries in that form. ### What you would check Before believing the rate, confirm the payload was byte-identical across replays (hash it) and that the trace shows the judge rail actually executing on every one — an intermittent upstream error that skips the rail looks exactly like a pass. Confirm the judge model identity from the run log rather than the config, since a fallback or a routing layer can serve something else. Run a control payload the judge should obviously block, at the same n, to show the rail is alive and to calibrate what a 0% looks like on your harness. Then write the two findings in different voices: for the rule, *"this input class is not covered; adding a rule closes it, verified by one replay"*; for the judge, *"this payload was approved on k of n replays against this judge model; expect drift when the judge changes."* Collapsing both into "guardrail bypassed" hands the owner a task with no verifiable done state.