What must a multi-agent trace record for failure analysis to be possible later?

## Why tracing is the substrate, not a nice-to-have In a single-model call, the request and response are the whole story. In a multi-agent system, the interesting behaviour lives *between* the agents: what the orchestrator decided to delegate, what context each subagent was given, what it returned, and what the next agent made of it. None of that is visible from the final output. And because the run is nondeterministic — sampling, retrieval results, tool responses and timing all vary — you cannot recover it by executing the task again. Every downstream evaluation activity depends on the trace existing: per-agent scoring on real inputs, first-divergence analysis, counterfactual replay, cost accounting, and mining production runs into eval cases. ## The structure: one task, nested spans The shape that works is hierarchical. A single **trace id** identifies the whole task, from the user's request to the final answer. Under it sits a span for the orchestrator's run, and under that a span per agent turn, with tool calls as child spans of the turn that issued them. Nesting matters because it preserves *who invoked whom* — a flat log of agent turns loses the delegation structure, which is precisely what you need when attributing a failure across a graph of agents. Parallel subagents produce sibling spans under the same parent, each with its own start and end, which is how you later distinguish a genuinely concurrent branch from a sequential one. ## What each span must carry - **Verbatim input context.** The actual prompt and retrieved material the agent saw, not a summary of it. A summary of the input cannot answer "did this agent have the information it needed?" - **Verbatim output.** Including the reasoning surface if the provider exposes one, and the final artifact. - **The handoff payload.** The structured object or message passed to the next agent, recorded at the boundary. This is the single most valuable field for multi-agent attribution and the one most often omitted, because teams log each agent's output but not what was actually forwarded after any post-processing. - **Tool calls.** Name, arguments, result or error, and duration. - **Model identity and version, plus the sampling settings.** Without it you cannot tell whether last week's regression came from a model change or a prompt change. - **Token counts and latency**, per span, so cost and time decompose by agent rather than arriving as one bill. - **Status and termination reason** — completed, budget exhausted, error, cancelled. ## Standardise the schema Bespoke trace formats rot: they are tied to one framework, they break when you change orchestrators, and every tool you want to point at them needs an adapter. OpenTelemetry's GenAI semantic conventions give a vendor-neutral schema covering agent, workflow, tool and model spans with token and latency metrics, and by 2026 it is the common denominator across observability vendors and agent frameworks. Emitting to it means your traces outlive your framework choice and can be read by generic tooling. ## Replay is the payoff A complete trace lets you replay a run like a flight recorder: step through the agent turns in order and watch where the state went wrong. It also enables the two techniques that make attribution rigorous. **Substitution replay** re-executes downstream agents with one step's output replaced by a corrected value, confirming or refuting an attribution hypothesis. **Input-faithful scoring** grades each agent on the input it actually received, which is the only honest per-agent number. Neither is possible without verbatim payloads. Traces are also the best source of new evaluation cases: the failures you find in production are, by construction, the distribution your fixtures were missing. ## The costs you must plan for **Volume.** A multi-agent run carries far more text than a chat turn, and full-fidelity traces of every production run get expensive quickly. The usual policy is to keep everything for failing or flagged runs and sample successful ones, with a short hot-retention window and longer cold storage for the sampled set. **Sensitive data.** Verbatim context contains whatever the user and your retrieval layer put into it — personal data, documents, credentials pasted by mistake. Redact at write time rather than at read time, and treat trace stores as a data-protection surface with real access control, since they aggregate more user content in one place than the application database does. **Instrumentation drift.** If each agent is instrumented by hand, coverage decays as the system changes and you discover the gap exactly when you need the trace. Instrument at the orchestration layer, so every agent turn is captured by construction rather than by a developer remembering. ## The failure mode to recognise The common story is a team with dashboards full of aggregate metrics — success rate, average latency, tokens per run — and no way to answer "what happened in run 4f2c?". Aggregates tell you something is wrong; only a trace tells you what. If the first question in your incident review cannot be answered from the record, the tracing is inadequate no matter how good the dashboards look.