In a branching jailbreak search, candidate prompts are checked for whether they still pursue the behaviour under test, and ones judged to have drifted off-topic are discarded before they are ever sent to the model under test. Why is that off-topic pruning step there, and what does it cost you?

### Two different judgments, one loop A branching attacker-model search makes two decisions about every candidate, at two different moments, and collapsing them is the most common design error in this leaf. - The **on-topic check** runs *before* the candidate is sent. It asks: is this rewrite still trying to elicit the behaviour we set out to test? A candidate that fails is discarded and never costs a target query. This is what the question is about. - The **scorer** (or judge) runs *after* a response comes back. It asks: did the target comply? The on-topic check may be an LLM call, an embedding-similarity threshold against the seed behaviour, or a keyword rule. Which of the three you pick decides both what the gate costs and what it wrongly kills. ### Why the gate exists An attacker model told to "rewrite until the scorer reports compliance" has a trivially available cheat: make the request benign. It softens the ask, the target answers cheerfully, the scorer sees a compliant response, and the run logs a hit. The finding is worthless — you demonstrated that a model will answer a harmless question. This is reward hacking of the scorer, and the on-topic gate is the constraint that makes the search objective mean what you intended. Secondarily it saves target queries, by killing candidates before they are spent. ### What it costs Not nothing, and the sign of the saving is worth doing on paper. Let *t* be the cost of a target query plus its scoring call, *g* the cost of one on-topic check, and *p* the fraction of candidates the gate discards. The gate pays for itself only when **p*t > g**. If the gate is an LLM of the same class as the target — the tempting choice, because it is the only thing that understands intent — then g is comparable to a target call, and unless the gate is pruning a large fraction you have added cost while removing candidates. An embedding check makes g nearly free and moves the argument entirely onto recall. There is also engineer time: someone must read the discard log, and if nobody does, the gate is unsupervised. ### Where it fails, and how the number misleads **False pruning is silent.** A pruned candidate is never sent, so you have no response to re-read, no score to appeal, and no artefact in the run log unless you deliberately wrote one. Every other component in the loop leaves evidence of its mistakes; this one does not. **It cuts exactly the promising family.** Indirect framings — hypothetical, fictional, role-shifted, translated, encoded — are effective precisely because they do not look like the original request. A keyword or embedding rule scores them as drift. A translated or encoded candidate may be unreadable to the gate, which then defaults to discard, and you have excluded an entire attack family from the search without ever deciding to. **Multi-turn setups die at turn one.** A line that establishes innocuous context and only lands the request several turns later looks off-topic on the setup turn. Judged turn by turn, it never survives to pay off. **The gate can refuse.** If the on-topic check is an aligned chat model asked to rate a deliberately aggressive candidate, it may decline to answer. Parsing a refusal as "off-topic" prunes the strongest candidates first, and the run never mentions it. **The denominator moves under you.** Attack success rate is usually reported as confirmed hits over candidates *sent*. Tighten the gate and the denominator shrinks faster than the numerator, so the reported rate goes **up** while the run finds fewer behaviours. Two runs with different gate strictness are not comparable on that metric at all. ### What to check Log every discard with the gate's verbatim verdict rather than dropping it, and hand-sample fifty. Look for clustering by framing style — clustering is the signature of a rule matching surface form rather than intent. Count how often the gate output failed to parse and where those went. Run a small holdout arm with the gate disabled and compare **hits per target query**, not hits per candidate, because that is the metric the gate cannot inflate. And report attack success rate against candidates *generated*, with the pruned count stated alongside, so a change in gate strictness is visible instead of flattering.