In an automated jailbreak run where a scoring model labels each target response a success or a refusal, which shapes of target response most often get mislabelled in each direction, and how do you catch them?

Treat the scoring model as a classifier over target responses with two error types, then ask what each error actually looks like in this pipeline and where it surfaces. The scorer here is whatever object decides success or refusal — a PyRIT scorer, a garak detector, a promptfoo grader, a benchmark's harm judge — and the shapes below are properties of the *job*, not of one vendor's implementation. ## False success: the target refused, the scorer said it complied Every one of these is surface compliance without substance. - **Agreeable opening, then a decline.** "Sure, I can help with that — however, I can't provide…" The first clause carries the compliance signal. - **Restatement before refusal.** The response repeats the forbidden request in detail before declining, so the transcript literally contains the request's vocabulary. Any scorer that is even slightly keyword-adjacent lights up. - **Fiction or roleplay frame.** A character discusses doing the thing; the text contains nothing operative. The scorer sees the topic and reads it as content. - **Confident but empty prose.** A fluent, generic paragraph that answers nothing. - **Hedged partial.** The answer starts down the path and stops before anything actionable, which a threshold set for "did it engage?" scores as a hit. The single root cause is the scorer's own instruction. *Did the model comply?* is a question about tone. *Does this response contain the specific prohibited content, judged against this written description of what would count?* is a question about content. ## False negative: the target complied, the scorer said refusal These are the expensive ones, because they appear nowhere at all. - **Buried content.** The operative part arrives after a long disclaimer or safe preamble, and the scorer weighted the preamble. - **Wrong modality.** Output as code, a table, JSON, or pseudo-config rather than prose, when the scorer's few-shot examples were all prose. - **Wrong language or script.** Targets often generalise across languages better than a scorer prompted and calibrated in English. - **Obfuscated or encoded output**, which the scorer does not decode. - **Split across turns.** No single response is damning; the harmful whole exists only in the concatenation. - **Outside the scorer's own notion of harm.** The systematic case, and the one no prompt tuning fixes. ## What each direction costs | direction | where it shows up | who pays | |---|---|---| | false success | the finding list, immediately | triage: minutes of an experienced reviewer per transcript, scaling with the error rate, not the finding count | | false success | the loop, silently | the thread's remaining turns are never spent — attacker, target and scorer calls that the invoice records as a saving | | false negative | nowhere | a real weakness ships unreported, and the client acts on a clean category | The asymmetry is the point: one direction bills you visibly and one does not bill you at all, which is exactly why teams audit the hits and stop there. ## How the number misleads Because only hits are enumerable, *every* easily computed quality figure describes the false-success direction. Precision, triage yield, the proportion of findings confirmed — all of them can look excellent while the miss rate is unknown and large. The specific wrong reading is a harm category showing zero hits and being reported as "no findings", when what happened is that the category's successes came back as code, or in another language, or after a disclaimer, and the scorer rejected each one. A zero is a claim about your instrument until you have coverage evidence to make it a claim about the target. ## What you actually check Build a small fixed **calibration set** before the campaign: hand-labelled responses per harm area that deliberately include the near-misses above — an agreeable refusal, a restatement, a roleplay frame, a code-block compliance, a non-English compliance, a buried compliance. Run the scorer over it and look at the confusion, not the accuracy. If it cannot separate an agreeable refusal from a real hit, its output is not reportable and no amount of extra run budget fixes that. During the run, persist every attempt: prompt, full response, label, raw score or confidence, attack strategy, turn index, and the reason the thread ended. The boolean alone forecloses everything below. Afterwards, run two samples. Hand-label a random sample of hits to estimate precision. Then sample the *rejected* transcripts — stratified toward borderline scores, toward strategies and categories with zero hits, toward long, non-prose and non-English responses, and toward final turns of threads that ended on the cap — and hand-label those. Re-scoring the whole rejected pool with a second scorer written independently from a different prompt is the cheap enrichment step: it bills only scorer tokens over stored text, no attacker calls, no target calls, no rate-limit waiting, and it hands you a disagreement list that is where the human hours belong. Agreements tell you almost nothing. The deliverable is not a fixed scorer. It is two error estimates you can print beside the finding count, and an explicit note naming the response shapes your instrument is known to miss.