Red-Team Reporting & Operations

The first is evidence; the second is a gap. A reader who sees neither statement assumes the area passed. So the report must list untested areas explicitly, in the same place as the results, rather than leaving them out. Absence of a finding is only meaningful where something was actually attempted.

Ask where the behaviour can actually be changed. If the customer's prompt assembly, retrieval data, tool permissions or output filtering could stop it, they own the fix. If only the model's weights or the provider's own safety layer could, the provider owns it, and your report must still give the customer something to apply now.

Because the report travels much further than the evidence should. A full transcript is a validated working attack plus the harmful text itself, readable by everyone the document reaches. Keep the raw prompt and completion in a restricted evidence store, and put a description, the harm class and a pointer in the report.

You also need the model provider's. The client owns the application, but the traffic you generate lands on the provider's infrastructure under their acceptable-use terms, which often restrict deliberate attempts to defeat safety measures. Check the provider's testing policy, and get the client's written sign-off naming the accounts and endpoints you may hit.

Coverage has no meaning without its denominator. It can mean test families executed out of those the harness offers, attack surfaces reached out of the application's entry points, or harm categories exercised out of a taxonomy. Write the fraction with both numerator and denominator named, not a bare percentage.

Because most rows are one weakness repeated: a single attack template retried with small wording changes, and each case re-sent several times. A grouping key is the field combination you collapse rows on, such as technique plus the behaviour elicited, so each report item is one distinct failure a developer fixes once.

Ship a self-contained reproduction: the exact request they must send, the endpoint and generation settings it was sent with, the response you observed, and a plain statement of what makes that response a failure. Add how often it happened out of how many attempts. No harness install, no red-team-only dependency.

Prompt plus reply says nothing about how the reply was produced. Capture the endpoint and the model identifier the provider returned, the decoding settings you sent, the application's system prompt, every earlier turn, and a timestamp with the provider's request id. Without those, a colleague who reproduces nothing cannot tell drift from a fix.

Grouping by template says how the model was pushed; grouping by behaviour says what it did. The template key hides that one technique unlocked several unrelated harms. The behaviour key hides that one harm is reachable by many independent routes, so a fix aimed at one route leaves the rest open.

State up front that the target is sampled, so one attempt proves nothing. Ship the observed count out of total attempts, the sampling settings you used, a script that repeats the request and tallies how many responses met the criterion, and a written verification rule such as: any hit in twenty attempts is still a failure.

It gives the finding a shared address: a reader can look the technique up, see how other reports used it, and find published mitigations. It does not give severity, likelihood or reproduction detail. Those come from your own evidence, narrative and rating. The identifier is an index entry, not a verdict.

Mark it unmapped and say so explicitly. Describe the behaviour in your own words, name the nearest technique and state exactly why it does not fit. Forcing a fit misdescribes the finding, sends the reader to the wrong mitigations, and hides a gap the reference itself may need to cover.

You lose the denominator, the mix of attacks that produced the rate, and the fact that the control is probabilistic rather than on or off. Record the rate, the suite it came from, the number of attempts, the target configuration and the date beside the verdict, and state the threshold that turned the rate into pass or fail.

Engineers need reproduction: the prompt pattern, the target configuration, what the model returned, and the fix. The register needs a risk statement, which control it is evidence about, how bad it is, and an owner and date. The attack narrative usually shrinks to one line, so keep the detail in a linked annex.

Keep it as one finding — the reader's unit is the delivered impact, not the step count. Name a primary technique for that impact and list the enabling steps as supporting techniques inside the same entry, in order. Splitting one chain into three findings inflates the count and destroys the causal sequence a defender needs.

A scanner hit is one prompt attempt its detector marked as failed. Many hits are the same behaviour repeated across prompt variants, probe classes and reruns, and some are detector false positives. The report's unit is a triaged, deduplicated issue with a cause. Count those, and report attempts separately as volume.

Three: the provider swaps the model behind the same endpoint alias, the safety classifier in front of it is upgraded, or the product team edits the system prompt. None changes the URL or the contract, so no deploy alert or failing test fires. Your filed pass and fail results simply stop describing the live system.

Whatever produces findings cheapest wins the budget. That is usually a high-yield, low-severity probe family against an easy surface. The hard work — a slow multi-turn attack on a tool-using agent — yields few items and looks unproductive. The programme drifts toward whichever instrument is noisiest, not toward where the risk is.

Record the whole configuration, not the numbers: the concrete model identifier and any version metadata returned, decoding parameters and seeds, a hash of the system prompt and context, the guardrail versions and thresholds, the exact attack prompts and harness version, the deciding component's version, trials per prompt, and the raw transcripts.

Decide by finding class and consequence, not by a tool's score. Classes whose harm is irreversible or legally exposed block. A finding ships behind a compensating control only if a control already deployed demonstrably stops it. Everything else ships with a named owner accepting it in writing, with a review date.