Your team runs PyRIT campaigns against several internal endpoints on a recurring schedule. What should the scorer be allowed to decide on its own, and what has to reach a human before it counts?

## The authority boundary Automate the decisions whose errors are **cheap and self-correcting**; keep a human on the decisions whose errors **travel**. - **Stopping a conversation is cheap** - if the scorer got it wrong you re-run that objective for the price of one conversation. - **Declaring a finding is not cheap**: it moves to another team, becomes a remediation ask, consumes their sprint, and is remembered for a long time when it turns out to have been a scorer artefact. So inside the run the scorer must be autonomous, because nobody is going to adjudicate every turn of every conversation. Crossing the run boundary, its verdict is a *candidate*, and what leaves the team carries the transcript and the name of a person who read it. ## Ranking is the underused middle A recurring program produces more transcripts than anyone will read. Work the arithmetic honestly: 300 objectives a week, several turns each, is thousands of responses; a realistic human budget is an hour or two, which is perhaps forty to sixty transcripts. The realistic ask of a scorer is therefore not "be right on every verdict" but "put the forty conversations most worth reading at the top". That reframing lowers the stakes on individual verdicts and raises the value of persisting **raw scores** rather than only booleans, because a boolean cannot sort. PyRIT's **human-in-the-loop scorer** fits here too: route only the ambiguous band to a person instead of everything or nothing. ## Separate the stopping decision from the recorded one They need not be the same component. A cheap, deliberately conservative rule can drive the loop while a richer scorer, replayed offline over persisted conversations, produces the verdicts and the reading order the report is built from. That keeps expensive judgment off the hot path of every turn and gives you a second, independent opinion on every candidate hit - for the price of scorer calls only, since no target turns are repeated. **Auxiliary scorers** can annotate in-flight when you want the label recorded but not acting. ## Choose the error direction per campaign type - A **broad coverage sweep** across many endpoints can tolerate false negatives: you are deciding where to dig, and a missed hit costs a bill and a later re-run. - A **deep engagement** on one high-value target cannot tolerate false positives, because every premature stop abandons a line of attack you had budget for. Set the stop rule accordingly, and write down which way you biased it and why, because that choice - not the target - is what makes two campaigns' numbers non-comparable. ## Give the scorer an owner and a test Name who owns the scorer configuration, and hold a **frozen, hand-labelled transcript set** drawn from your own runs as its regression suite. Re-run it before any change to the scorer, its rubric or its backing model is allowed into campaigns. The cost is one scorer call per item plus the labelling you already paid for once; the cost of skipping it is that the definition of "success" in your program drifts with a component nobody is watching, and week-over-week movement becomes unattributable. This is the single most common way a red-team metric silently stops meaning anything: the judge model gets upgraded and the trend line is read as a change in the systems under test. ## Where the numbers mislead at program scale Three readings to refuse. 1. First, a trend line across a scorer or rubric change - it measures the scorer. 2. Second, a league table of endpoints scored by different scorers or objectives - the denominators are not the same. 3. Third, and most dangerous, "no findings this week": a blind scorer and a genuinely resistant target produce identical runs, because both exhaust their budgets. A null result must clear the same evidence bar as a positive one. ## What you would check before a report leaves the team - That the scorer still recognises known-compliant responses, using the frozen set - that is the smoke test that makes a null result publishable. - That every hit going outward has a transcript and a reader. - That early-turn successes were reviewed. - And that nothing in the scorer's configuration changed since the last number you published, or that the change is stated next to the delta. ## What I would not automate - Turning a hit into an external report; - comparing endpoints on scorer output alone; - and any claim of "no findings".