How do you turn a production failure trace into a permanent regression test case?

## Why the loop exists Every production incident in an LLM system is an expensive discovery: a case nobody imagined, found by real users. The waste is letting that discovery evaporate once the fix ships. The trace-to-eval loop is the discipline of converting each live failure into a permanent, automatically-checked case, so that a prompt tweak, model upgrade or retrieval change six months later cannot silently reintroduce it. It is the mechanism that makes a quality programme cumulative rather than reactive. ## Step 1: capture enough to replay A replayable case needs everything the system saw, not just what the user typed: - the full sequence of user turns, in order - the system instructions and prompt version in force - whatever was retrieved or injected as context - **every tool call and the exact response it received** - the model and configuration identifiers The tool responses are the part teams most often skip and most often regret. If the case calls a live service at replay time, it will drift: the order that was undelivered in July is closed by September, and the test becomes flaky or quietly stops exercising the failure. Freeze the responses with the case so it is hermetic. ## Step 2: sanitise before it becomes an artefact A production trace contains real people. Once promoted into a repository it will be copied, read in CI logs and possibly sent to a third-party model. Redact or pseudonymise names, addresses, account and payment identifiers, and free text that could identify someone — while preserving the structural features the failure depended on. If the bug only reproduces when an account number has an unusual format, keep the format and change the digits. Record provenance (source trace id, date, incident) so the case can be traced back internally even after the content is scrubbed. ## Step 3: assert the property, not the transcript This is where the loop most often goes wrong. Free-text output is not stable across model versions, so an exact-match assertion against a golden reply will fail on the next upgrade for reasons unrelated to the bug, and the team will delete it. Instead, name the defect and check for it directly. Useful assertion shapes: - **Must-not** — the reply does not contain an unauthorised commitment, does not disclose another customer's data, does not invent a policy. - **Must** — the reply cites at least one retrieved document, or offers the escalation path, or refuses. - **Structural** — the response validates against the required schema; the tool was called with the correct arguments. - **Programmatic verification** — where the task has a checkable end state, assert that state rather than the prose. The test is: would this assertion have caught the original failure, and would it stay quiet on a correct answer worded differently? If the second half fails, the assertion is too tight. ## Step 4: verify the case has teeth Run the new case against the build that produced the failure. It must fail. This one step separates a real regression case from decoration: an assertion that passes on the broken build is testing something other than the bug. Then run it against the fix; it must pass. Only then does it join the suite. ## Step 5: tag, batch and prune Cases arrive in bursts. After a bad week you might promote around forty traces at once. Without structure that becomes an undifferentiated pile: - **Tag by failure mode** — unauthorised commitment, retrieval miss, tool-argument error, loop, premature termination. Tags let you report which failure families are covered and collapse near-duplicates. - **Deduplicate.** Twenty traces of the same failure mode add runtime and no information; keep the two or three that differ meaningfully in trigger. - **Budget the suite.** Cases cost inference time and money on every run. Track total runtime and cost, and prune cases whose failure mode is now covered by a cheaper deterministic check. - **Handle nondeterminism.** A single pass can pass by luck. For cases where the failure was intermittent, run repeats and require consistent success rather than one green result. ## What closes the loop The loop is only closed when the case runs automatically on every candidate change and its failure blocks the change. A suite that is run manually before big releases is a checklist, not a regression barrier. Wire it into the same gate that guards the prompt, the retrieval configuration and the model version — those are the three things most likely to resurrect an old failure. ## The payoff Six months in, the suite is a written record of every way this system has actually broken in front of users — far more predictive of real risk than any generic benchmark, because every case is drawn from your traffic, your tools and your policies. That is the argument for spending an hour promoting a trace on the day of the incident, when the context is still fresh and someone still remembers exactly what went wrong.