An agent's call log shows an approved operation with valid arguments — how do you triage a report that it should not have run?

## Triaging a call that was allowed, valid, and wrong You have inherited a report. It says an autonomous procurement agent 'made a call it should not have made'. The attached evidence is a call record: an approved operation, on a real invoice, an amount inside policy, every argument passing validation. Nothing in it is anomalous. Somebody upstream has already suggested closing it as working-as-designed. ### First, name what the artefacts actually prove This is the discipline that unblocks the triage. Take each artefact and state its claim precisely: - **The call record proves which operation ran**, when, under which identity, and that its arguments satisfied the declared rules. That is all it was built to prove. - **A validation pass proves the values were of admissible shape.** Not that they were correct, and not that anyone with authority selected them. - **The absence of an alert proves nothing was of a kind anybody watches.** Kind, rate and sequence were all normal here, because the operation was the one the workflow performs daily. None of those three touches the question in the report, which is about a *value*, not an operation. ### The wrong answer to confront early 'We log every tool call, so we would have seen it.' It is a reasonable-sounding claim and it is the one that closes these tickets prematurely. The log is complete along the axis of *which call*; the dispute is along the axis of *where each value came from*, and that axis is not recorded at all. Completeness on one dimension is not coverage of another. Nothing in an ordinary record diffs argument provenance, and almost nobody is doing that comparison by hand either — which is exactly why the class survives in systems with excellent logging. ### The reconstruction that settles it Provenance is recoverable even when it is not recorded, because the pipeline is deterministic enough to walk. For each argument on the disputed call, ask: what was supposed to determine this value, and what actually did? 1. Take the arguments one at a time and identify the intended source — internal state such as the matched purchase order, or the ingested document. 2. Fetch the internal record and compare. If the beneficiary was meant to be fixed by the purchase order and instead corresponds to a field on the submitted invoice, the comparison is the finding. 3. Check whether that mapping is stable across other runs. If every invoice contributes that field, the exposure is structural rather than an incident. At the end of this you can state the claim in one sentence with evidence behind it: this argument was determined by content an outside party authored, and the workflow's owners believed it was determined by an internal record. ### Against whose expectation was the call wrong? This is the adjudication, and it is where triagers disagree honestly. Two positions are available: - **A design limit.** The workflow was specified to take that field from the document. The system did what it was written to do, and the report is a disagreement with the specification. - **A vulnerability.** The specification was written without anyone realising that the field's author is an outside party, and every downstream consumer — reviewers, approvers, the finance team reading the record — reads the value as though the organisation chose it. The honest verdict is usually that both are true, and that the second is the reportable half. A specification that grants an outside party the choice of what a payment lands on is a trust decision; if nobody knew they were making it, the fact that it was implemented faithfully does not settle anything. Write the finding against the trust assumption rather than against the code, and say which stakeholders were relying on the assumption. ### Reproduction, and what a single success proves Expect this to reproduce unevenly. Extraction and matching may involve a model, submissions vary, and the same document can land differently on different runs. A construction that lands one time in five is not a flaky test to be dismissed; it is a class that works whenever the mapping fires, on a queue that processes every submission. But be precise about the claim: a successful reproduction demonstrates that this construction worked once, against this deployment, in this configuration. It is not a rate, and generalising from it to 'this always works' or 'this is unexploitable' are both unsupported. ### What to hand back A good triage output states: the operation was approved and in character; the arguments were valid; one argument's value originated in content authored outside the organisation; here is the internal record it should have matched and did not; here is how often the mapping fires; and here is the question the existing artefacts cannot answer, which is who chose the value. That last sentence is the substance of the report, and it is worth writing down explicitly, because the natural instinct of everyone reading a clean log is to conclude there was nothing to find.