In an agent red-team harness you replace the agent's real email-sending tool with a stub that records the call and returns success. An injection attempt now shows the agent calling that stub with attacker-chosen recipients and body. What does that result prove, and what does it not?

A stub is a function you register in the harness under the same tool name and JSON schema the model already sees, which records the call and returns a canned result instead of forwarding it. The model cannot tell the difference: to it, the tool exists and it worked. That property is exactly what makes a stub safe and exactly what makes its evidence partial. ### The evidence chain, link by link A live-fire hit demonstrates six links in sequence: 1. untrusted content reaches the model's context (a retrieved document, a tool result, an email body); 2. the model treats that content as instruction rather than data; 3. the model emits a tool call with attacker-chosen arguments; 4. the tool layer in front of the real API authorises the call; 5. the external system accepts it; 6. an observable effect exists in the world. A stubbed run establishes links 1–3 and asserts nothing whatsoever about 4–6. That is the whole answer, and everything below is what follows from it. ### What links 4–6 might have contained The untested region is usually where the undocumented controls live: the API's own server-side authorisation check on the calling principal, a recipient allowlist or external-domain block, a rate limit, a human confirmation step in the send path, a DLP scan on the outbound body, an anti-abuse heuristic on the sending account. Any one of them could downgrade your critical finding to a defence-in-depth win. Equally, none of them may exist — and a defender who cannot name the control that would have stopped it does not have one. Both possibilities are live until someone exercises the path. ### The second distortion: the missing error branch A stub that always returns success does more than hide the outcome; it changes the run. Agents branch on tool results. A real permission error may make the agent retry with different arguments, escalate to the user, or abandon the task; a synthetic success walks it straight down the happy path. So multi-step traces gathered behind an optimistic stub are unrepresentative in both directions — they can overstate how smoothly an attack completes, and they can hide adaptive behaviour that only appears after a refusal. Returning realistic error shapes for some fraction of calls is a cheap corrective, and the fraction you chose belongs in the write-up. ### What it costs A recording stub costs an hour to write and produces zero residue, which is why it is the default. Buying back links 4–6 is the expensive part, and the options are not equally priced. Reading the tool's server-side authorisation code is free if you can get the repository. Asking the owning team what would have happened costs a meeting and yields a *claim*, which you record as a claim. Executing the path once against a scratch tenant and a recipient you control costs the provisioning already discussed plus one artefact to clean up, and is usually the right answer. Executing against production with the owner watching is the last resort and needs an agreed abort signal. Note the shape: one targeted execution, not a live-fire campaign, is what settles a disputed severity. ### Where the number misleads The reporting failures are symmetric and both are common. **Over-claiming:** "the agent exfiltrated the customer list to an attacker-controlled address" when the mail tool was a stub and the real relay only accepts internal recipients. One engineer who knows that discounts the entire report, including the findings that were sound. **Under-claiming:** dropping the finding because "no real send happened", when a fully hijacked model plus an unknown downstream control is precisely the risk the engagement existed to surface. There is a quieter numeric failure too. An attack-success rate aggregated across a run mixes tools that executed for real with tools that were stubbed, so the single percentage answers no question anyone actually has. Report the rate per instrumentation mode, or do not report it as one number. ### What you check State the instrumentation next to every finding: which tools executed for real, which were stubbed, what each stub returned, and whether error injection was used. Prefer severity phrased as coverage — "model-layer control failed; send path stubbed and not exercised" — over a single ordinal, because the ordinal invites an argument about a fact you did not establish. Before the run, confirm the stub's return shape matches the real API's, including its error shapes; a stub whose success object differs from the real one can itself change the agent's next step and produce behaviour the real system would never show.