An agent red-team run that substitutes tool results through a shim reports a large number of policy violations. Before you write any of them up, how do you establish that the shim did not manufacture them?

A hit produced by an interception shim reads exactly like a real breach. The transcript shows the agent calling a tool, receiving content, and then doing something it should not. Nothing in that artefact distinguishes *the payload persuaded it* from *the harness put the agent in a state it could never have reached*. Because the two are indistinguishable after the fact, the separation has to be designed into the run. ### The control that actually isolates the payload The instinctive control -- run it again with the shim removed -- is the wrong one: it changes the instrumentation and the content at the same time, so a difference cannot be attributed to either. The control you need is **shim present, payload absent**. Hold fixed the task, the seed, which call is intercepted, whether that call succeeds, the injected latency, and the result's shape and approximate length; swap only the adversarial text for benign text. Any violation that still occurs is caused by the instrumentation, not by the injection. ### The structural artefacts, and why each one manufactures hits 1. **Never-failing tools.** The shim answers every call with success. If the real service would have returned a permission error, an empty result or a rate limit, the agent never meets the failure path that stops it in production, and walks on to actions the real environment gates. Everything downstream of that point is a statement about your shim's permissiveness. 2. **Collapsed latency and suppressed retries.** An instant return changes how many steps fit inside a turn budget or a wall-clock timeout, and removes retry loops that would otherwise have consumed the run. Agents behave differently near their step limit. 3. **Reordering and position.** Injecting on the first call rather than where the workflow would naturally return that content puts the payload earlier in the context, with less competing material after it. That is a different experiment with a different answer. 4. **Bypassed result handling.** A pre-baked string that skips the framework's truncation, escaping, schema validation or summarisation reaches the model in a condition no real tool output ever would. The payload gets a delivery quality no attacker could obtain. 5. **Missing side effects.** A short-circuited tool never writes the state a real call would have written, so later steps see a different world for reasons unrelated to your content. ### What the control costs, and why teams skip it The matched control roughly doubles the run: a 200-case suite at five samples and ~8 model turns per sample is on the order of 8,000 model calls per arm, so budget two arms, tens of dollars of inference, and an hour or more of wall clock per full pass -- plus the engineering to make the shim parameterised enough that latency, success and intercept index can be pinned identically across arms. That parameter plumbing is usually a day of work, and it is precisely the day teams skip, which is why so many interception reports cannot be defended when challenged. ### Where the number misleads "142 violations" from an interception run is not a measurement until you can say how many of the 142 survive the benign-content arm. Two specific misreadings follow from omitting it. First, a shim that always succeeds inflates the numerator with cases that are really about a missing permission gate, and those get filed as prompt-injection findings the scaffold team cannot fix. Second, the denominator is usually attempts, not distinct behaviours: the same underlying failure repeated across 40 task variants is one report item, and quoting 40 makes the agent look forty times worse than it is. A count without its control and without deduplication is an opinion with a number attached. ### What you would check Run the matched control and put its outcome next to the hit count in the write-up. For each surviving hit, ask whether it depended on a call succeeding that would really have failed -- if it did, report it as a dependency on that gate rather than as a breach. Reproduce the survivors with the least instrumentation that still shows the behaviour, ideally by planting the content in data a real tool returns, and treat that reproduction as the finding. Finally, keep an explicit list of every property the shim controls -- success, latency, ordering, shape, position, side effects -- and for each one either hold it at a realistic value or state in the report that you changed it.