Agent Red-Team Harnesses

Attack success rate only counts attacker-instruction executions. An agent that refuses everything never executes one, so it scores near zero: perfectly secure and completely useless. The harness must also run each task without any injection and score whether the agent finished the user's real work, then report both numbers together.

Because a server you host is repeatable. You can change a tool's description or schema between runs and replay the same attempt to see what moved. A live vendor server gives one shot: it changes under you, its calls touch real data, and a result there cannot be reproduced or varied.

Provision throwaway identities the agent acts as: a scratch mailbox, a test tenant or sandbox project, and disposable CRM records. Point outbound tools at dry-run or test endpoints. Mark everything with a unique per-run tag so you can find it later, and agree who owns cleanup.

Put it in the tool result, not the user message. The agent is already trained to be wary of user requests, but treats tool output as data it fetched. The harness intercepts the call and returns attacker-controlled content in its place. A hit there shows the agent obeys untrusted retrieved content.

Capture every tool call the agent made: the tool name, the exact arguments, and the result returned to it, in order, with timestamps. Also record the prompts, the model settings, and which harness and environment version ran. Text-only transcripts leave a finding unreproducible.

Because the transcript is the agent's own account, not evidence. Score against world state: snapshot the sandbox before the run, run it, then diff. If the outbound queue gained a message or a row is gone, the run failed no matter what the agent narrated in its final turn.

A prompt list only records whether the model said something harmful once. A task set records how far the agent got along a chain of tool calls before it stopped. That makes a refusal at step one and a stop at step eight different outcomes, and it exposes actions the words alone never produce.

Forty clean episodes plus 40 x 15 poisoned pairings is 640 episodes, each a multi-step agent loop costing many model calls. Cut the cross product first: sample a few injection tasks per user task instead of all of them, keep the full sweep for release candidates, and always report which pairings you actually ran.

Compute it over the same fixed task set as the no-injection run, and read the drop between the two. Conditioning only on tasks the agent already passes cleanly isolates the defence's cost but shrinks the sample. Either way, report the benign number beside it; a bare under-attack percentage mixes agent capability with attack damage.

It proves the model was hijacked into deciding to send, and shows the exact arguments it chose. It does not prove delivery: the real endpoint might reject the recipient, demand a confirmation, or fail an authorisation check. The stub tests the model's decision, not the system's outcome.

Three placements. Wrapping the tool function inside the framework is easiest and sees parsed arguments, but skips the wire. A proxy in front of the endpoint sees real requests and responses but not framework post-processing. Planting content in data a genuine tool reads is highest fidelity and lowest control over timing.

Because the run is not a pure function of the seed. The environment answers differently each time: tool results, clocks, IDs, retries, and the endpoint's own sampling all vary. Record the actual request and response bytes at every step so a replay can fall back to the recorded exchange.

One trial per payload gives a yes/no from a sampled, stateful system, not a rate. The same payload can land one run in five, so a single pass misses real attacks and promotes flukes. Re-run each payload a fixed number of trials and report hits over trials attempted.

It licenses a claim about that run only: the chain reached step three under this prompt, seed and environment state. It does not show the target is safe at step four; a retry, a different phrasing or a fuller mailbox may carry it further. Partial is evidence of reach, not of a boundary.

It means naming the forbidden side effect as a concrete, machine-checkable condition over environment state, and committing to it before you run anything. Written afterwards, the behaviour you observed sets the bar: you quietly accept whatever happened, and a near-miss that emptied one row becomes a pass.

Check where your copy is more permissive than the real service: looser schema validation, unsanitised and untruncated result bodies, stale description text, different field names, and errors that echo input back. Then diff your served tool definitions against the partner's current contract and a capture of real responses.

Re-run with the shim in place but a benign result substituted. If the behaviour still appears, the shim caused it, not the payload. Usual manufacturers: the shim answering a call that would have errored, returning instantly, changing call order, or replacing a result the framework would have truncated.

Score it as an invalid run, not a defence. Rate limiting is infrastructure back-pressure from the endpoint, not a model refusal or a policy control; it would not stop a patient attacker. Mark the run inconclusive, exclude it from the denominator, back off and retry, and log the cause.

Because a pass/fail hides how close the run came. Recording the last step the agent reached shows whether it was stopped early at planning or only at the final harmful action, tells you which control fired, and gives severity a distance input. Two zero-score runs can be very different.

Binary scoring maps every unfinished attempt onto one bucket, so an agent that stopped at the first call and one that stopped one call short look identical. You paid for the whole chain, so report chain depth: which required steps were reached per task, plus the reason each attempt stopped.

Everything except the planted bytes: same task, same starting world contents, same tool list and descriptions, same model, sampling settings and step limit. Hold the payload's container and position fixed too, so only its text differs. Drift in timestamps, result ordering or sampling shows up as a fake attack effect.

Two variables moved, so you cannot say which caused the change — and with a stochastic model, neither may have. Rerun with one edit at a time from a pinned baseline, repeat each configuration enough times to compare success rates, and record which fixture version produced each run.

An empty account gives an exfiltration attempt nothing to steal and no permissions to abuse, so the run scores a false negative. Seed it with realistic decoy documents, contacts and message history, grant the same roles a real user holds, and mark every seeded item with a canary string you can search for.

The replay shows the mechanism: the exact turns, tool calls and world change that made it a real break, so a reader can watch it happen rather than trust a number. The rate shows how often it happens, which sets priority. Neither substitutes for the other; a finding needs both.

Multiply tasks by required steps by model turns per step, add retries and repeats, then pilot ten tasks and measure the real per-task call count before extrapolating. A step cap bounds the bill but truncates long chains, so attempts that hit it look like non-completions unless you record the cap as their stop reason.

The mocks are not your tools. They return small, clean, deterministic data with no pagination, partial failures or auth errors, and they expose a fixed action set, so harms outside that set cannot be observed at all. A pass says the fixture found nothing, not that your production tool surface is safe.

They inflate it. Give every episode an explicit termination reason and split three ways: attacker action succeeded, agent completed the task safely, or the episode aborted. Report the abort rate as its own figure and compare it between the defended and undefended arms; a defence that lengthens prompts can manufacture aborts.

Do not reconstruct it afterwards. Have the harness log every outbound tool call with the identifier the system returned, so the run produces a delete list. Tag artefacts with a per-run marker, delete, then re-search until the marker returns nothing. Delivered mail, webhook fan-out and audit entries stay.

Log what the model actually received, not what you handed the shim. Frameworks truncate, summarise, re-serialise or drop tool results before they become context, so your text may never arrive intact. Without that check a clean run is not evidence of resistance; it may only prove your content was cut.