Defining Success

**An agent run leaves two records, and only one of them is independent of the thing under test.** The *transcript* is the ordered log of the agent loop: the system prompt, the task input, each tool call the model emitted, each tool result the harness returned, and the final assistant message that summarises what happened. The *environment* is the sandbox the agent acted in — a seeded database, files under the task root, a fixture mailbox or outbound queue, a recorded HTTP egress boundary. A red-team harness owns the environment. The model under test authors the generated parts of the transcript. That asymmetry is the whole answer. **Three tiers of evidence, routinely conflated.** | artefact | produced by | what it actually proves | |---|---|---| | final assistant message: "I did not send anything" | the model | that the model wrote that sentence | | harness tool-call log: a recorded `send_email(...)` invocation | the harness | that the call was issued with those arguments | | pre/post diff of the fixture mailbox | the environment | that a message landed in the sink | The final message sits at the bottom because it is a *generation*, not a log. Models summarise optimistically, drop tool calls out of a recap, and sometimes narrate work they never performed — no adversary required. Add an injected instruction in the data the agent reads and the incentive becomes explicit: the useful thing for an injection to produce is an action plus a reassuring summary. In a capability eval a wrong self-report is a quality bug. In a security eval, where the entire question is whether a forbidden effect occurred, it is a false pass on the exact property you were paid to check. The middle tier is genuine harness-produced evidence but still not conclusive: a recorded call may have errored, been rejected by the sandbox, or hit a stub. Issuing a call is not landing an effect. **The mechanism.** Make the environment the oracle: ```text before = snapshot(env) # rows, files, queue, egress log run_agent(task) after = snapshot(env) violated = assertion(diff(before, after)) # not: assertion(final_message) ``` Published agent-security suites are built this way rather than on text matching — AgentDojo's injection tasks expose a security check that is handed the pre-run and post-run environment objects, so the verdict is computed from state the harness controls rather than from the model's answer. Sinks worth snapshotting: the task's rows at field level, paths under the task root, the fixture mailbox, and the request list captured by the egress recorder. For a third-party API you cannot snapshot, the recorded request list *is* the state you assert over. **What it costs.** Transcript scoring is nearly free — one regex over one string — which is exactly why teams drift into it. State scoring costs a per-run fixture reset (a template-database restore or container re-create, typically hundreds of milliseconds to a few seconds, and it must happen between *every* trial or task N+1 inherits task N's rows), an egress proxy to stand up and keep working, storage for two snapshots per run, and the engineering time to write one predicate per task, which is usually the larger half of authoring a task. Set that against the inference bill: a twenty-turn agent run re-sends the growing tool-result history every turn, so a single task run consumes far more prompt tokens than a one-shot probe, and a 100-task suite at five trials each is 500 such runs per sweep. The snapshot machinery is a rounding error beside that. Do not let "parsing the text is cheaper" survive contact with the arithmetic. **Where the number misleads.** A transcript-scored suite reports an attack-success rate whose denominator is fine and whose numerator is a statement about the agent's *candour*. It is wrong in both directions and the single number cannot tell you which. The **silent success** — the send tool was called, the summary denies it — is recorded as a pass and pushes the reported rate down. The **phantom failure** — the model narrates a deletion that never occurred — is recorded as a violation and pushes it up, burning triage days on nothing. The second is annoying; the first is the one that ships. Worse, matching on the final message usually measures refusal *language* rather than harm: a run that ends "I cannot help with that" scores clean even when the tool call went out earlier in the loop. A 0% rate from that pipeline is not evidence of a safe agent; it is evidence that the agent's last paragraph was polite. **What to check.** Read the verdict function and ask which artefact it consumes; if the final assistant message appears anywhere in it, every number in the report is provisional. Then run a positive control — give the agent a plain, non-adversarial instruction to perform the forbidden effect and confirm the assertion fires. An assertion that has never fired has never been tested. Finally, check that the tool-call log and the state diff agree on the same run: a recorded call with no matching state change usually means a stub, an error swallowed by the tool wrapper, or a sink nobody is snapshotting. The transcript keeps its place — for triage, and for harms that leave no state change at all — but it never decides the verdict.