How do you set up a Langfuse LLM-as-judge evaluator to score production traces?

## Where the evaluator lives Unlike a metric you call in code, a Langfuse LLM-as-judge evaluator is a piece of **project configuration**. It watches traces as they arrive and writes scores back onto them. Nothing changes in the application; the same traces you were already sending become the evaluator's input. That is the whole appeal, and also the source of its failure modes, because the evaluator can only see what your instrumentation happened to record. ## The four decisions **Template.** A template is the judge prompt: an instruction, one or more `{{variables}}` to be filled from the trace, and an output contract (a score plus a short reason). Langfuse ships a managed library of templates for common dimensions, and you can copy one and edit it or write your own. Templates are versioned in the project, so a judge prompt change is a visible event rather than a silent drift. **Variable mapping.** Each variable must be bound to something concrete on the trace or on a chosen observation type: the trace input, the trace output, a field inside metadata, the output of a particular kind of observation. This is the configuration step people get wrong. If your RAG traces put retrieved chunks in a retriever observation but the mapping points at trace metadata, the judge receives an empty context and confidently scores hallucination on nothing. After creating an evaluator, always inspect the first few resulting scores and read the judge's input on one trace. **Judge model.** The judge needs credentials for a model provider configured in the project. Choose it deliberately: judging is usually a cheaper task than generation, but too weak a judge produces noise, and using the exact model under test invites the model to like its own style. The judge's cost is separate from your application's inference bill and is easy to forget when you turn the evaluator on across all traffic. **Target and sampling.** An evaluator is pointed at a filtered slice of traces — an environment, a tag, a trace name, a time window — and given a sampling rate. Langfuse can also run an evaluator over historical traces, not just new ones, which is how you backfill a new dimension over last month's traffic. Sampling is the cost dial: at a few thousand traces an hour, one judge call each is a bill you did not have yesterday. A few percent of traffic is usually enough to move a trend line, and you can always raise it for a suspicious cohort. ## Reading the result Each run produces a score whose source is `EVAL`, with the judge's explanation in the comment. Two things follow. First, aggregate views over that score name give you the trend, sliced by any trace attribute you recorded — user, release, prompt version. Second, the individual bad scores are clickable: you land on the trace, see the prompt, the retrieved context and the model call, and read the judge's stated reason next to them. That round trip from a metric to the offending execution is what makes an online judge worth configuring at all. ## The limits An evaluator on live traffic can only judge what needs no ground truth — groundedness against the retrieved context, relevance to the question, tone, refusal. Anything that compares against a correct answer has nothing to compare with on production traffic, and belongs to an offline dataset run where the expected output exists. Whether the resulting score is *valid* — how the judge is biased, whether its numbers track human labels — is a separate discipline from configuring it, and the configuration screen will happily produce a well-formed number for a badly conceived judge. If you need judging logic Langfuse's templates cannot express, the escape hatch is to compute the score yourself in your own code or job and write it with the score API; a self-computed score is indistinguishable to every dashboard except for its source field.