In an LLM rollout, what can shadow traffic measure and what can it never measure?

## What shadow evaluation is Shadow evaluation — also called dark launching or mirroring — routes a copy of real production requests to a candidate system that runs alongside the live one. The candidate's response is captured and scored; it is discarded from the user's point of view. The user is served entirely by the incumbent and cannot tell anything else ran. The mirrored share is a knob. Mirroring everything doubles inference cost, so teams typically sample: mirror 5% of live requests, or mirror everything for a subset of segments you care about. Mirroring is usually asynchronous so the shadow call cannot add latency to or fail the user's request — a shadow path that can break production defeats its own purpose. ## The gap it closes The central weakness of offline evaluation is that inputs are a sample somebody chose. Shadow evaluation removes that weakness completely: the inputs *are* production, in the proportions production actually has them, including the seasonal spike, the malformed client payload, the query in a language nobody put in the eval set, and the long tail that individually looks rare but collectively is most of your volume. That unlocks several measurements a harness cannot make honestly: - **Operational behaviour under real load** — p50/p95 latency, timeout rate, error rate and throughput at production concurrency rather than in a sequential batch run. - **Real cost per request** on the true input-length distribution, which is what your bill will look like. - **Failure rates that are only visible at volume** — schema violations, refusals, truncation, tool-call errors that occur in one request in ten thousand and therefore never appear in a thousand-case eval set. - **Paired comparison** — because both systems saw the *same* request, you can diff outputs one to one. Where they agree, nothing changed; where they disagree, you have a small, high-value review queue rather than a whole corpus to grade. That last property is the underrated one. A/B tests compare distributions across different users; shadow compares two answers to the same question. For detecting *changed behaviour* — as opposed to measuring *value* — the paired design is far more sensitive per unit of traffic. ## The gap it cannot close No user ever sees a shadow output, so every metric that lives downstream of the user is unavailable: purchases, task completion, escalation, retry, session length, revenue. Shadow tells you what the candidate *says*; it cannot tell you what anyone *does* about it. Second, systems with side effects cannot be shadowed naively. If the candidate is an agent that calls tools, mirroring it means it will really send the email, really issue the refund, really write to the database — twice. Shadowing anything stateful requires stubbing or sandboxing the write path, and the moment you stub it, the shadow run stops being fully faithful to production. Third, shadowing cannot capture feedback effects. In a search or recommendation system, what the model returns changes what users click, which changes the training and ranking data, which changes future results. A shadow system sits outside that loop by construction. Fourth, scoring is still your problem. Shadow gives you real inputs; it does not give you labels. You still need programmatic checks, a judge model, or human review to say whether a shadow output was good — and at production volume you are sampling those, not scoring everything. ## Where it sits in the release path The natural sequence is offline gate, then shadow, then canary, then split test. Offline catches known regressions cheaply. Shadow proves the candidate behaves sanely on the real request distribution and is operationally viable — right latency, right cost, no unexpected refusal or malformed-output rate — all without a single user exposed. Only then does a canary put real users behind it, and only then does a randomized split answer whether it is actually better. Skipping shadow is common and usually fine for a prompt tweak. It earns its cost when the change is structural — a new model, a new retrieval stack, a rewritten pipeline — where the questions "does this survive real traffic" and "what does it cost at our actual input lengths" are exactly the ones a frozen set cannot answer and an A/B test answers too expensively. ## A worked shape For a grocery search-query rewriter: mirror 5% of live queries into the candidate rewriter, run both rewritten and incumbent query through the same retrieval, log both result sets. Score automatically for empty-result rate, latency delta and out-of-catalogue terms; send the disagreeing pairs to a judge and a human sample. If the candidate produces empty results twice as often on seasonal terms, you learn that before any shopper sees it. If it looks clean, you still do not know it sells more — that requires a split test where shoppers can actually add things to the cart. ## What interviewers listen for The expected answer names the exact tradeoff: shadow buys the production input distribution and operational truth at zero user risk, and pays by learning nothing about user response. Bonus credit for raising side effects on stateful systems, the cost of doubled inference, and the paired-comparison advantage over a split test.