Your agent red-team harness replays a recorded attempt with the same prompt, the same sampling settings and the same seed, and the agent takes a different action. Why is recording the seed not enough to make an agent run replayable, and what does the trace need instead?

**Why the seed argument fails.** A seed constrains the pseudo-random sampling *you* control inside a single generation call. An agent run is not one call; it is a loop over a live system, and a loop amplifies divergence. One differently-worded argument at step two changes the tool result at step two, which changes the messages at step three, and by step six the two runs are not comparable at all. Reproducibility therefore has to be engineered into the trace. It cannot be borrowed from a seed. **Where the entropy actually enters.** Three sources, worth separating because the fixes differ. The **endpoint under test**: a hosted model is a shared service, and identical settings do not guarantee identical decoding — batching, expert routing, kernel and hardware differences, and silent version rolls all move the output. Temperature zero narrows the variance; it does not remove it, and a seed parameter, where the API offers one, is best-effort rather than a contract. The **environment**: tool results carry fresh identifiers, current timestamps, listings whose order is not stable, quota and rate-limit errors, and state left behind by earlier attempts in the same campaign. The **harness**: retries on timeout, truncation at a token cap, and — when the model emits several tool calls at once — a resolution order that follows latency. Any one of these flips a borderline attempt from hit to miss. **What the trace records instead.** Per model call: the message list as sent, the settings and seed, the raw response including the tool-call structures, and the finish reason. Per tool call: the arguments, the result payload, latency, and any error and retry count. Per run: harness and environment revision, and the initial state of anything the tools read. Recording raw requests and responses — not a parsed summary of them — is what buys two distinct replay modes. | mode | what it re-executes | what it proves | what it costs | |---|---|---|---| | recorded | nothing; stored responses are served back in order | the reported transcript is complete and internally consistent — the behaviour happened as written | near zero: no tokens, no side effects | | live | the model and every tool, against the real target | the behaviour still happens now | one full attempt per rerun, plus containment and cleanup | **What it costs.** Live reruns are the expensive half, and the cost is superlinear in loop length because an agent resends the whole growing transcript at every step: a fifteen-step attempt can bill six figures of prompt tokens even though the operator typed one paragraph. Ten reruns is that ten times over, plus ten more sets of side effects to contain and clean up, plus the wall-clock of a loop that waits on real tools. That economics is exactly why teams rerun once, get a hit, and write "reproducible". **How the number misleads.** Two readings go wrong, and the second is the dangerous one. First, a rerun count with no denominator: "reproduced" on the strength of one successful repeat is a coin landing heads twice. If the behaviour fires on three of ten live attempts, the honest finding is "intermittent, 3/10" — still real, still worth fixing, and materially different from "the agent always does this", which is what a reader takes from an unqualified claim. Second, the replay mode gets lost between operator and reader. A recorded replay is green *by construction*: it serves stored responses, so it cannot fail for the reason anyone cares about, and it will pass ten times out of ten forever. Demonstrate it to a stakeholder without naming the mode and they will conclude the issue is live today, which the run never tested. The inverse trap is just as common: after a vendor rolls a model or tightens a filter, a 0/10 live rerun does not mean the finding was wrong — it means the world moved, and the recorded trace is what preserves the original claim. **What I would check.** Confirm the "live" rerun was actually live: count outbound requests, or read the endpoint's own usage counter for that window. A live rerun that consumed no tokens was a recorded one. Deliberately corrupt one stored response and confirm the recorded replay diverges rather than sailing past, which tells you it is genuinely following the trace and not re-deriving anything. Reset or snapshot the environment between reruns so leftover state is not quietly doing the work. Then put the rate and the mode in the finding text itself, not in the operator's memory.