You inherit a finished automated jailbreak run that reports sixty successful attacks against a chat endpoint, every one labelled by a scoring model. How do you measure that scorer's error rate in both directions before the report goes out?

You are auditing an instrument, not a model. The sixty is a count of labels produced by a scoring model, and your job before the report ships is to attach two error estimates to it — or to state honestly that one of them cannot be attached. ## Step zero: write the success criterion first One paragraph, agreed before anyone opens a transcript, saying what would count as a violation for each harm area in scope. Without it two reviewers labelling the same sixty transcripts will disagree and you will have measured nothing. Have a second person independently label a shared subset and check how often you agree. If the humans disagree materially, the criterion is the defect, not the scorer, and no amount of sampling repairs that. Do not show reviewers the scorer's confidence while they label. It anchors them to the instrument under test. ## Precision: the easy half The labelled hits are an enumerable population, so sampling is trivial. At sixty items, adjudicating all of them is usually the right call for a customer-facing report — at a few minutes per transcript that is a few hours of one experienced person, which is small against the cost of shipping a finding that does not survive the client's own reading. Report a proportion with an interval, not a point estimate. Sixty items adjudicated to, say, forty-one confirmed is roughly 68% precision with a confidence interval wide enough that quoting "68%" bare is overclaiming. Then look at the *structure* of the errors, not only the count. Hits clustered on one attack template, on turn one, or in one harm category usually mean the scorer matched a phrasing pattern rather than content — and that pattern tells you which of the remaining hits to distrust and which future runs to instrument differently. ## Recall: the half the run either supports or forecloses Misses are not on any list, so they can only be found among the attempts the scorer *rejected*. The first thing to establish is therefore what the finished run persisted. If every attempt is stored with prompt, full response, raw score, attack strategy, turn index and thread-termination reason, recall is estimable. If only hits were kept, it is not, for this run, at any price short of re-running. Where the store exists, sample the rejected pool in strata rather than uniformly, because a uniform sample of thousands of clear refusals finds almost nothing per hour: - near-threshold scores first, where the classifier is uncertain; - strategies or harm categories with suspiciously zero hits; - long responses, non-prose responses (code, tables, structured output), and non-English responses, where scorers are systematically weaker than the targets they score; - final turns of threads that ended on the turn cap. Then re-score the entire rejected pool with a second scorer written independently from a different prompt, and hand-adjudicate only the disagreements. This is the cheap enrichment step: it bills scorer tokens over stored text, no attacker calls, no target calls, no rate-limit backoff and no fresh authorisation to touch a live system, and it hands you a list where at least one of the two labels is wrong. Adjudicating agreements spends hours to learn little. ## Where the number misleads — the paragraph to get right A stratified sample deliberately oversamples where misses are likely. If 12% of your stratified rejections turn out to contain real violations, that 12% is **not** the run's miss rate, and 88% is **not** the scorer's recall. It is an enriched estimate that tells you misses exist and where they concentrate. To quote a population figure you must weight the strata back to their true proportions in the rejected pool, and say which of the two you are reporting. Two adjacent denominator swaps to refuse. Hits divided by attempts is an attack-success rate over attempts; it contains no information about labels the scorer got wrong in either direction. And "sixty this quarter versus forty last quarter" describes the target only if the scorer model, prompt and threshold were identical across both — otherwise the delta is a property of your instrument. ## When the store is missing Say so. The report carries a measured precision on the sixty, an explicit statement that the miss rate is unmeasured for this run, and a note on why. That is a defensible sentence; a guessed recall figure is not. Do not spend target budget re-running the campaign to answer a question stored transcripts would have answered for free, unless you have both the budget and a reason beyond tidiness. The real deliverable of an inherited audit is the change to the pipeline: persist every attempt with its raw score and its termination reason, so that every future audit costs transcripts and reviewer hours instead of target calls.