Offline vs Online Evaluation

## The two halves of an eval program An LLM feature has no single correct output, so "is it good?" cannot be answered by a unit test. Practitioners split the question into two measurement regimes that run at different times, on different data, and answer genuinely different questions. **Offline evaluation** happens before exposure. You take a fixed collection of inputs — a *frozen dataset*, meaning it does not change between runs — feed each one through the candidate prompt, model or pipeline, and score the outputs. Scoring can be programmatic (does the JSON parse, does the SQL run, does the answer contain the required entity), reference-based (compare against a stored expected answer), or a judge model applying a written rubric. The output is a number per slice: accuracy on refunds, faithfulness on long documents, format-validity overall. **Online evaluation** happens on live production traffic. Its forms escalate in exposure and in what they can tell you: mirroring real requests to a candidate whose output is scored but never shown; splitting users between a control and a treatment arm and comparing product metrics; exposing a tiny slice first and ramping it if nothing burns. The scores here are not rubric grades but outcomes — did the shopper buy, did the ticket get resolved without escalation, did the agent finish the task. ## What offline is good at Offline evaluation is **repeatable, cheap, and safe**. The same dataset scored twice gives roughly the same number (modulo sampling nondeterminism), so you can compare a candidate against a baseline directly. No user sees a bad output. A run costs tokens and minutes, not customers, so you can afford it on every change and iterate ten times a day. It is also the only way to test rare and dangerous cases *on purpose*. Live traffic will not conveniently contain the prompt-injection attempt, the malformed invoice, or the multilingual edge case at the moment you need to test them; a curated set holds them permanently. And it lets you slice: overall quality can rise while one important segment collapses, and a frozen set with segment labels shows that immediately. ## What offline can never tell you An offline set is a **sample someone chose**, scored by a **proxy for value**. Both assumptions can be wrong. The inputs may not match today's traffic. A set assembled last quarter carries last quarter's request mix; live traffic drifts with season, campaigns, new user segments and new failure modes. The score is then honest about a population you no longer serve. The metric may not proxy what users reward. A rewriter can produce more relevant, better-formed queries by every rubric a judge applies, and still make people buy less because the results feel unfamiliar. Offline metrics measure output properties; businesses are paid in user behaviour, and the mapping between the two is an assumption, not a measurement. And offline cannot capture **response**: users react to outputs, learn the system, change how they phrase requests, sometimes stop trusting it. A frozen harness has no user in it to react. ## What online is good at, and what it costs Online evaluation measures the thing you actually care about, on the population you actually have, including effects nobody thought to encode in a rubric. It is the decision of record. Its costs are real. Exposure means someone gets the worse variant. Statistical power means waiting — small effects on noisy business metrics take days or weeks of traffic to resolve. Attribution is hard: outcomes are influenced by pricing, seasonality, other teams' experiments. Turnaround is measured in days, so you cannot iterate on prompt wording this way. And you cannot A/B-test something catastrophic; that is what the offline gate is for. ## How they compose in practice The standard shape is a funnel. Every change is scored offline against the frozen set; a regression on a tracked slice stops it there. Changes that pass are exposed narrowly — first as scored shadow runs on mirrored requests, then to a small canary slice, then to a randomized split with a pre-declared primary metric and a set of harm metrics that can stop the rollout. Whatever the online experiment discovers is fed back as new offline cases, so the frozen set slowly learns the failure modes production found. The two are not competing methods and neither substitutes for the other. Offline is a **fast, cheap filter with weak external validity**; online is a **slow, expensive measurement with strong external validity**. Confusing the roles produces both classic failures: shipping on an offline number and being surprised, or A/B-testing every prompt tweak and shipping nothing. ## What interviewers listen for A weak answer describes offline as "testing" and online as "monitoring". A strong answer states the epistemic difference — chosen sample and proxy metric versus real population and real outcome — and names the concrete online mechanisms (shadow, split test, canary, harm metrics) with the exposure and latency each one costs you.