A goal-substitution finding reproduces once in five runs and the step trace is green — how do you adjudicate it?

## Why this is a judgment call and not a technical one A one-in-five reproduction against a probabilistic system arrives at an organisation as a dispute. The person who filed it believes something real. The owner sees a green step-by-step trace, an objective revision written by the agent's own consolidation step in a legitimate session, and a demonstration that failed four times out of five, and reasonably asks what they are supposed to do. Somebody has to decide what the finding is, what it is worth, and who owns it — and could defensibly refuse it. That decision is the answer. ## Separate the two claims The single most useful move is refusing to let one report carry two kinds of evidence. **The outcome claim** is that a particular run ended up pursuing a substituted objective. This is inherently probabilistic: sampling, retrieval ordering, whether the row fell in scope, and whether the consolidation step kept the content all vary. Report it the way an empirical result is reported — successes over trials, with the conditions held fixed — and state plainly that five trials support essentially no confidence interval. Inflating a 1-in-5 into "the agent can be redirected" costs credibility the next time; deflating it into "unreproducible, closing" throws away the part that is certain. **The structural claim** is not probabilistic. Two facts about the deployment either hold or do not: the standing objective is re-read from a record that a step in the run writes, and the per-step verifier dereferences that same record. Both are inspectable, both reproduce every time, and together they entail that a green trace is precisely what a successful substitution looks like. That is the durable finding. It says the assurance the trace was supposed to provide does not cover this class at all — not that it is weak against it. ## What the green trace is worth as evidence Nothing either way, and saying so is half the adjudication. A fully green step-verification trace attests that every action matched the objective record at the time it was proposed. It does not attest to the record's authorship or its correspondence to the request. A run with a substituted goal produces exactly the same trace as a clean one, which means the trace cannot be cited as evidence against the finding. The same holds for the objective record's own writer field: it truthfully names the consolidation step, and that is equally true of every legitimate refinement. ## Bug or design limit Both framings are available and they lead to different places. Filing it as a bug points at the instance — this row, this span, this revision — and invites a fix at the instance: purge the row, add a pattern to whatever screens the retrieved text. That closes the ticket and leaves the property, so the next construction that reaches the same writer succeeds identically. Filing it as a design limit says the deployment has a run-writable premise that its own verification consumes as ground truth, and that this is a consequence of a real architectural choice — usually made for good reasons, because a multi-hour objective cannot be carried in context and is expected to gain detail as work proceeds. The instance then becomes a demonstration attached to the property, not the subject of the report. That is the shape a lead should insist on. ## Ownership and cost asymmetry Be explicit about the asymmetry, because it is what makes routing contentious. Making the standing objective something the run cannot rewrite touches how the agent is built: how long-running work is carried, how the goal is refined, what the loop reads at the top of each step. Tightening whatever screens the incoming text is comparatively cheap. Cheap and relevant are different questions, and the cheap change does not address a property that lives in the state model. Route the finding to whoever owns that state model, name the expensive option honestly, and let them decide with the cost visible — including the legitimate decision to accept the exposure for this deployment. A finding routed to the cheap owner because it will be accepted there is a finding you have closed rather than resolved. ## What a programme should conclude Two things generalise. First, on a probabilistic system, a red team's most defensible product is often a property with a demonstration rather than a reproducible exploit — build the reporting norms around that in advance, or every low-rate finding turns into the same argument. Second, treat controls that dereference run-mutable state as unable to attest to that state, and audit which assurance claims currently rest on them. That audit is a lead's job because it changes what the organisation is allowed to say it knows. ## The trap to avoid Do not accept, and do not offer, "we verify every step against the plan" as the resolution. It is the wrong answer restated as a remedy, and the whole point of the finding is that this particular control confirms the attack rather than catching it.