Why can't a LangSmith online evaluator score live traffic against a reference answer?

## The asymmetry between online and offline An offline experiment evaluates a dataset. A dataset example has two halves: the inputs, and the expected outputs someone curated. That second half is what makes exact match, semantic similarity to a gold answer, or reference-based recall computable at all. A production run has no second half. It has what the user sent, what the system did, and what came out. The reference is the one thing production never provides, and no configuration flag conjures it. This is why the same evaluator can be perfectly valid in an experiment and worthless attached to an automation rule — the metric did not change, the availability of its input did. ## The failure is quiet, which is the danger If a reference-requiring evaluator simply crashed on every production run, this would be a five-minute lesson. Often it does not. The judge prompt has a slot for the reference, the slot is empty or filled with something meaningless, and a language model asked to compare an answer against nothing will still return a number in the requested format. You get a full chart. It trends. People make decisions from it. The score is noise wearing the costume of a metric, and the only tell is that it does not move when quality obviously does. So the practical advice is not only "choose the right metric" but "before trusting any online series, look at a handful of the individual judge outputs and confirm the judge was actually given what it needed". ## What is legitimately computable online Everything whose evidence is on the run: - **Grounding against retrieved context.** The retrieved documents are on the trace, so a judge can ask whether each claim in the answer is supported by them. This needs no gold answer at all. - **Relevance to the question asked.** The input is on the run, so "does this response actually address it" is answerable. - **Format and schema compliance.** Deterministic, cheap, no model needed — did it return valid JSON, did it include the required disclaimer, is the citation list non-empty. - **Policy and safety checks.** Did it disclose what it must, did it refuse an out-of-scope request, did it leak an internal identifier. - **Tool-call validity.** Were the arguments well-formed, did the tool error, did the agent loop. - **Self-consistency and structural signals.** Length anomalies, repetition, an unusual number of retries. Notice that most of these are checks on *behaviour* rather than on *truth*. That is the honest limit of online evaluation: it catches the system misbehaving, not the system being confidently wrong in a way that looks perfectly well-formed. Accepting that limit is what separates a realistic online setup from a fantasy one. ## Manufacturing ground truth after the fact There are two ways truth arrives late. **A human writes it.** Route interesting runs into an annotation queue. The reviewer scores it and, crucially, writes a correction — the answer that should have been given. Now that run has a reference. Promote it into a dataset, where its inputs become the example's inputs and the correction becomes the expected output. Reference-based metrics now work on it, offline, forever. **The world reveals it.** Many applications produce a delayed outcome that is better than any label: the user accepted the drafted email unchanged, the support ticket did not reopen, the extracted invoice total matched what accounting later booked, the suggested code was merged. These are ground truth, they cost nothing to obtain, and they arrive minutes to days after the run. Because feedback is attached by run id with no time limit, a downstream job can write them back onto the original run under their own feedback key whenever they become known. Designing for that second channel is one of the highest-leverage things you can do in a production LLM system, and it is invisible to anyone who thinks of evaluation as something that happens at request time. ## The loop this implies Online reference-free evaluation detects that something is off and selects candidates. Human review or delayed outcomes supply labels for a subset. Those labelled runs become dataset examples. Offline experiments over that dataset — where references exist — are what you actually gate a change on. Then the change ships and online evaluation watches it in the wild. Each half does the job the other cannot: online has all the traffic and none of the truth; offline has all the truth and none of the traffic. ## What a weak answer sounds like "You just configure the evaluator with the expected output." Ask where that expected output comes from for a request that arrived four seconds ago, and the answer dissolves. The second weak answer is "use a stronger judge model" — a better judge with no reference is still a judge with no reference.