A trip-planning agent recommended a fully booked hotel — how do you localize the fault in its trace?

## Why status is useless here The defining property of LLM failures is that they are successful. No exception was thrown, every HTTP call returned 200, the tool answered, the model responded. So the first instinct from classical debugging — filter for errors — finds nothing, and people conclude the trace is not helpful. It is; you just have to read it differently. Instead of asking 'what failed', you ask 'where did the correct information stop being present'. ## The two-way split Every wrong answer of this kind reduces to one of two cases: 1. **The fact never reached the model.** It was not retrieved, it was retrieved and then filtered out, the tool was called with wrong arguments, the tool returned a stale cached answer, or the result was truncated before it was serialised into context. 2. **The fact reached the model and was not used.** It was in the assembled input and the model still recommended the booked hotel — an instruction-following, context-position or truncation problem. These have completely different fixes, and confusing them is the most expensive mistake in LLM debugging: teams rewrite prompts for weeks over what was a reranker cutoff. The trace decides the split in one look, provided the spans record shapes and inputs. ## The walk, span by span **Root run span.** Read the attributes, not the status: session id, turn index, prompt and app version, entry point. If the failure started last Tuesday, the version attribute is what tells you it coincided with a release. **Retrieval span.** The single most informative attribute is the count pair: candidates returned versus candidates that survived into the prompt. Eight hotels in, two out is a complete explanation on its own if the available hotel is among the six dropped. Also check the query text — an agent that searched for the wrong city was never going to be saved by a better reranker. **Tool span.** Look at the arguments the model produced, not just the result. Availability checked for the wrong date range, the wrong occupancy, or a stale cached response are all common, and all look like a healthy span. Check timings too: a suspiciously fast tool call often means a cache hit on data that has since changed. **Model-call span.** If you captured the assembled input for this trace, this is where the question is settled. Was the availability information in the prompt? If yes, and the model contradicted it, you are in case 2. Check `finish_reasons` — output stopped for length means the answer was cut mid-thought. Check how large the input was and where the fact sat within it; a fact buried in the middle of a very large context is materially less likely to be used, and that is a context-construction problem rather than a model defect. **Missing spans.** An absent tool span has two very different causes, and separating them is the first move: either the model never emitted the tool call at all — visible in the preceding model-call span's output, which will contain no tool-use request — or the tool ran somewhere the trace did not reach and its span was never attached. The first is a model/prompt problem; the second is an instrumentation gap that will keep costing you. ## Making the walk fast On a trace with 300 spans, browsing is not a strategy. Three things make it quick. Put outcome attributes on the root — the feature, the verdict if you have one, the number of agent iterations — so you can find suspect traces before opening them. Record the count-in/count-out shape on every retrieval and tool span, since shapes are cheap to store and often sufficient without payloads. And keep the span taxonomy consistent, so you can filter a trace to just its retrieval spans, or just its model calls, instead of reading the tree linearly. ## What you say in an interview The weak answer is 'I would look at the trace'. The strong answer names the hypothesis structure — did the fact reach the model, yes or no — says which span answers it, and admits what the trace cannot tell you: if you did not capture the assembled input for this trace, you cannot resolve the split, and your next move is to reproduce the turn with capture enabled or to look at a sampled sibling trace on the same path. That admission is the difference between someone who has debugged a production LLM system and someone who has read about it.