Containing Side Effects

Containment for an agent red-team run is not a switch in the harness. It is a set of accounts, endpoints and markers you create **before the first prompt goes out**, chosen so that every side effect the agent can produce lands somewhere you own, can find, and can delete. The harness — your own wrapper around the agent's tool loop, or a benchmark runner — only constrains calls that pass through it. The credential the agent carries is what the outside world honours, and that is a different object. ### The four layers you provision **1. Identity.** The agent authenticates as a principal created for this engagement: a service account or a test user in a directory you control, never the operator's own SSO session and never a real employee's account. The roles on that principal are the ceiling on the whole run. "The harness only exposed three tools" is not a boundary — a model can reach further by chaining a tool that itself calls another API, by following a callback, or by writing code that a code-execution tool runs, and every one of those paths carries the same token. **2. Data plane.** A scratch tenant, sandbox project or test mailbox: a container you can drop wholesale afterwards. The test is whether you personally hold the permission to delete it, not whether someone believes it is disposable. **3. Egress.** Decide per outbound tool, in advance, which of three modes it runs in: real execution against your scratch tenant; the vendor's own test or dry-run mode where one exists (test API keys, a mail provider's sandbox that accepts and discards); or a recording stub inside your harness that logs the call and returns a canned result. Write the choice down per tool — that table becomes the caveat section of the report. **4. Markers.** A unique per-run token goes into every free-text field the test can create: mail subjects, record names, file bodies, ticket titles. This is the cheapest item on the list and the one most often skipped, and without it post-run cleanup degrades to searching by timestamp, which collides with real traffic. ### What it costs The expensive resource here is engineer time, not inference. A first scratch tenant for a mail-and-documents assistant is typically half a day to two days of work plus a ticket to whoever owns the identity provider; each additional connected system (tracker, CRM, object store) adds hours and often a paid seat. The model calls in an agent engagement are trivial by comparison — a few hundred to a few thousand calls, single-digit to low-tens of dollars. On a week-long engagement the containment setup, not the tokens, is the budget line, which is precisely why teams talk themselves out of it. ### Where the number misleads Two readings of a contained run go wrong, in opposite directions. The first is a **clean run read as a safe agent**. A freshly provisioned tenant usually has no history, no shared drives and no crown jewels, so an exfiltration attempt has nothing to take and a confused-deputy attempt has no reach to abuse. The run returns zero findings, and the zero describes the *environment*, not the agent. Containment and fidelity pull against each other; any score is a property of the pair. The second is a **success count read as impact**. An attack-success rate computed over a run where most tools were stubbed counts model decisions, not realised effects. Published without the mode table it is uninterpretable and, worse, uncomparable: switch one tool from stub to live next quarter and the same system "gets worse" with no change in its security. ### What you check before the first attack - Read the throwaway principal's effective roles back from the API rather than trusting the provisioning console; defaults frequently grant more than the form suggested. - Confirm the mail path has a genuine sandbox mode. A "test" recipient address on the production relay still performs real DNS, real delivery and a real bounce to a real postmaster. - Enumerate fan-out: webhooks to partner systems, CRM-to-warehouse sync, notification rules that page an on-call human. Those escape the tenant you can delete. - Confirm you can delete the tenant, and that soft-deleted objects can be purged rather than left recoverable. - Agree a named cleanup owner, an abort signal with the system owner, and disclosure of run windows and markers to the blue team so your traffic is attributable. Simulated-environment benchmarks — agentdojo, agentharm and injecagent are of that class — dodge this entirely by shipping the environment with the tests: no provisioning, no residue, and a repeatable number. They also say nothing about your deployment's real connectors, so they complement an engagement rather than replacing one.