Agent Red-Team Harnesses

Split raw from shareable. Keep the unredacted trace in a restricted store with a short retention, and derive a redacted copy by replacing secrets and personal data with stable placeholders. Stable means the same value maps to the same token, so the reader still sees the causal chain.

Compare trial counts first — 5% may be 1 of 20. Then decoding settings and any fixed seed, the turn budget per episode, which object decided a hit and on what threshold, the starting environment each trial began from, and whether aborted or retried episodes were counted as trials.

Two stages. Spend a cheap screening pass of a few trials on all 40 payloads to find which ones ever land, then spend the depth only on those, repeating them enough to quote a rate. Stop early when the interval already answers the decision, and price a trial as a full episode.

A diff only sees the surface you snapshot. Outbound network calls, writes to a third-party service, data read and emitted without changing local state, persisted agent memory and delayed jobs all escape it. Bound it by enumerating every sink the sandbox exposes and recording each one, not just the database.

It is the wrong instrument when you need a ship decision. Its mock tools, data shapes and action space are not yours, so its number describes the fixture. Keep it as a cheap, comparable regression check, and stand up a bespoke world replaying your own tool schemas with your own harm predicates.

Give them the pair, not a scalar. The exchange rate between one blocked injection and one failed user task is a product decision, not a benchmark output. B keeps most of the utility; A buys a lower attacker-action rate by breaking a third of the work. Ask what a residual success actually costs.

Do all iteration on the stand-in, where the surface is mutable and repeats are cheap, and spend live calls only on confirming a shortlist. Decide the confirmation criteria before touching production. Never claim the partner is exploitable, or quote a rate, from stand-in runs alone.

Split by reversibility and by who else sees the effect. Reads and reversible writes inside a scratch tenant run for real. Anything reaching a third party, moving money, or paging a human gets stubbed, and you argue that gap separately. Decide before the run, write the table down, publish it with the findings.

Agree up front how much of the stack you may modify, and write it down. Every layer you replace is a layer the owning team can blame. Prefer the least invasive interception that still proves the claim, ship a matched control run with each finding, and state exactly what the harness changed.

Capture fully by default and cut elsewhere. You can always delete a trace later; you can never reconstruct one you did not record, and a failed rerun is what kills a finding. Trim by truncating giant tool payloads to a hash plus a head, and by shortening retention, not by sampling which attempts you trace.

Tie the trial floor to the claim: an existence finding needs one replayable success with evidence; a quoted rate needs a denominator and an interval in the text. Ban bare percentages. The one-in-fifty still gets filed on impact, marked low-rate and not verifiable by re-measurement at that budget.

Mostly that you only forbade what you thought of. A pass against a hand-written forbidden list is evidence about that list, not about the agent. Invert it: assert the exact set of effects the task permits, and fail the run on any effect outside that set, including ones nobody predicted.

Split by what the run has to answer. Breadth finds which task families the agent will engage with at all, but one attempt per task has no repeatability and tight caps truncate long chains. Depth gives a defensible per-task number with an interval, but says nothing about families you did not run.

Tie tiers to decisions, not to narrative detail. Use the fewest checkpoints that change what someone does: attempted, reached the irreversible action, completed. Every tier needs an objective world-state check and a named arbiter for disputes, or graders drift and cross-team numbers stop comparing.