In an agent red-team finding you attach both a rate — successes over trials for one attack payload — and a saved recording of one successful run that a reader can replay. What does the replay show that the rate cannot, and what does the rate show that the replay cannot?

## Two claims, two artefacts - "This agent can be driven to do X" is an **existence claim**, and one well-evidenced run settles it. - "This agent does X about a fifth of the time under these conditions" is a **frequency claim**, and only repeated trials settle it. Reports carrying only one of the two draw predictable pushback: a bare rate draws "show me", and a bare recording draws "is this a fluke?". The artefacts are not redundant, and neither degrades gracefully into the other. ## What the rate does Successes over completed trials, with the trial count and run configuration attached, is a severity and ordering input. It tells a defender how often an attacker starting from the same position gets the outcome, which drives priority, and it is the baseline you re-measure against after a mitigation. What it cannot do is say what happened. From 6/30 a reader cannot tell whether the agent leaked a customer record, called a destructive tool, or merely emitted disallowed words — nor whether the object that scored those six hits was scoring the right event at all. ## What the replay does A stored, re-drivable recording of one successful episode shows the **causal path**. To be worth attaching it must carry: - the payload and where it was placed relative to what the agent reads; - the full turn sequence with tool names and arguments; - the harness settings that shaped the episode — turn budget, decoding settings and any seed, which object decided the run was a hit and on what criterion; - and the environment snapshot the episode started from plus the diff it produced. The **world-state diff** is what turns a transcript into evidence. An agent's narration is not testimony about its own actions: an agent can write "I did not send the message" in the turn after the send tool returned success, so a miss scored off transcript wording and a miss scored off the outbound queue are different measurements wearing one label. ## What it costs Capture is not free. - Full-fidelity recording of tool arguments and environment snapshots multiplies log volume per episode, which is why teams sample it — and then discover the one successful episode was the one not recorded. The rule that avoids that: sample freely, but always retain the complete record for any episode the scorer marked a hit. - There is engineer time in building the snapshot-and-diff machinery at all, - and review time in scrubbing what the recording caught, because captured tool arguments routinely contain bearer tokens, session cookies, customer records, or the contents of the mailbox the agent read. An unscrubbed replay artefact pasted into a ticket is its own incident. ## Where each one misleads The **replay's failure mode** is the implied promise "run this and watch it break". Against a sampled agent a replay may simply not land, and a reader who tries twice and fails concludes the finding was fabricated. State the reproduction fraction — "replayed 3 of 10 attempts" — because intermittency is the property under measurement, not an embarrassment. The replay also drifts silently: re-driven under a different turn budget, a different tool set, or a mutated environment it is a different experiment, so pin the artefact to the run manifest it came from. The **rate's failure mode** is the opposite. It looks objective while inheriting every assumption of the scorer that produced it; a rate computed by a judge that rewards refusal-shaped wording will report a comfortingly low number on an agent that is quietly acting. A number cannot flag its own scorer, and the replay is the only artefact that lets a sceptic audit one. ## Where the pair pays off: fix verification After a mitigation ships, the recording tells you which exact scenario to re-drive, and the pre-fix rate tells you how many trials the post-fix run needs before "we saw no successes" carries weight. - Without the rate you cannot size the verification; - without the recording you cannot be sure you re-tested the same thing rather than a rephrasing the mitigation happens to catch. And when the pre-fix rate is very low, the pair tells you honestly that re-measurement cannot settle it and the verification has to be structural. ## What to check - That the recording holds a world diff, not only a transcript. - That credentials and customer data are scrubbed before it leaves the harness. - That a manifest is attached so a replay is comparable to the original. - And that the replay's reproduction fraction is stated rather than implied.