In an attacker-model jailbreak loop, one scoring model usually both tells the loop when to stop rewriting a prompt and decides which transcripts enter the report. Why does that double duty make its errors more damaging than the same error rate in an offline evaluation, and what would you change?

The useful framing is that one scoring model is being asked to occupy two roles whose cost asymmetries point in opposite directions, and a single component at a single threshold cannot serve both. ## Role one: the loop's control signal Here the label answers *keep refining this thread, or stop?* The orchestrator reads it as control flow. That single fact is what separates this from an offline evaluation, where labels are applied to a fixed set of responses that already exist. Offline, a wrong label corrupts one row and the rest of the dataset is untouched — the error is noise on a measurement. In a loop, the attacker model's next prompt is conditioned on the label, so a wrong label changes *which responses come into existence*. Two runs of the same campaign under two scorer thresholds do not differ by a relabelling of the same data; they explore different regions of prompt space. Errors therefore compound down a thread instead of averaging out across it. In this role: - a **false refusal** is cheap in truth terms and expensive in budget — you keep attacking something already broken, burning attacker, target and scorer calls to the turn cap; - a **false success** is expensive in coverage — the thread stops and the attempts it would have made never exist. In a branching search the same error at a pruning decision removes a whole subtree rather than one item, so the loss is multiplied by the branch factor. Only one of those two is visible afterwards. The wasted turns show up as spend; the deleted subtree shows up as nothing. ## Role two: the report's gate Here the label answers *does this count as a finding?* and the asymmetry flips: | role | false positive costs | false negative costs | |---|---|---| | in-loop signal | lost coverage, unrecoverable | wasted turns, visible in spend | | report gate | triage hours, then credibility | a missed finding you never learn about | Wire one component at one threshold into both and you are forced to pick a compromise that is wrong for at least one job. Tuned for precision, the loop kills promising threads. Tuned for recall, the report ships unverified hits. ## The decoupled design Cheap, deliberately **recall-biased** scorer as the loop's stop signal, so threads survive; stricter, more expensive **precision-biased** scorer as a second stage over the persisted transcripts; human adjudication above that; and every attempt stored with prompt, full response, raw score (not the boolean), attack strategy, turn index and the reason the thread ended. The economics are what make this work. The in-loop scorer runs once per turn per thread, so its unit cost is multiplied by the whole search; a run with a few thousand attempts calls it a few thousand times, in-line with the loop's latency. The second stage runs once per stored attempt, offline, **and never touches the target** — no target tokens, no attacker tokens, no rate-limit backoff, no fresh authorisation to attack a live system. That is why the second stage can afford to be the expensive model with the long rubric, and why it can be re-run for free when the success criterion changes mid-engagement. ## What this costs you, honestly A second scoring pass over every stored attempt, not just the hits. Storage for full transcripts of a run whose non-hits outnumber its hits by one or two orders of magnitude. And a triage queue that is now *longer* than the raw hit list, by design, because the recall-biased signal deliberately admits more. Engineer hours move from re-running campaigns to adjudicating text. That is the trade, and it is the right one whenever the artefact is a report somebody acts on. ## Where the number misleads A decoupled pipeline produces two counts, and they will differ substantially. The in-loop signal's hit count is not a finding count and must never be quoted as one — it is a queue length. The failure mode in practice is a dashboard that reads the loop's counter because that is the number the tool emits, so the report inherits the recall-biased threshold that was chosen for search behaviour and had nothing to do with what counts as a violation. Equally, a low second-stage count is not evidence the run was thorough: it says nothing about threads the signal stopped early or pruned. ## What to check in a pipeline someone else built Is the same scorer object wired into both the termination condition and the reporting path? Is the raw score persisted or only the boolean — if only the boolean, any threshold change requires re-querying the target. Are non-hit transcripts kept at all? And does anything record *why* each thread ended, so that when you audit you can tell a scored stop from budget exhaustion? Without that last field the two most important failure modes look identical in the store.