skip to content

LLM evaluation & observability

2 roadmaps121 questionsupdated

Two related problems: scoring whether an LLM's output is good, and seeing what a live LLM or agent application actually did. Interviewers ask because "it worked in the playground" is not something you can ship, and both halves turn up in every production LLM postmortem.

on this pageshow

guide

overview

~1 min

LLM evaluation and observability covers two problems that most production LLM features run into: deciding whether a model's output is good enough, and reconstructing what a live LLM or agent application actually did on a particular request. Interviewers ask because a prompt that looked fine on a few hand-typed inputs says little about the next ten thousand, and nondeterministic output breaks the habit of asserting an exact value in a test. They listen for what you are measuring, where the expected answer comes from, what does the judging, and what a trace must hold to explain a bad answer later. The hub is organised around six tools, each a lens on one part of the problem. [DeepEval](/topics/ai-deepeval) and [Ragas](/topics/ai-ragas) are evaluation libraries: DeepEval frames evaluation as pytest-style test cases with thresholds you can run in CI, while Ragas concentrates on retrieval-augmented generation and separates retrieval quality from generation quality. [LangSmith](/topics/ai-langsmith) and [Langfuse](/topics/ai-langfuse) are platforms that join the two halves, tracing runs, turning traffic into datasets, scoring against them and monitoring production; Langfuse is open source, can be self-hosted, and includes versioned prompt management. [Helicone](/topics/ai-helicone) and [OpenLLMetry](/topics/ai-openllmetry) are about capture: Helicone records calls mainly by sitting in the request path as a proxy, and OpenLLMetry emits OpenTelemetry spans into a tracing backend you already run. Start with an evaluation library, because the words it introduces (test case, metric, reference, judge model, threshold) are reused by every other section. Move to a tracing platform next, where that vocabulary meets production traffic, and finish with the capture layers. Junior questions are mostly wiring; senior and principal ones ask why a suite is flaky, what a proxy outage takes down, or how an organisation should attribute spend. Tool syntax changes quickly; the reasoning underneath carries between vendors.

primer

A few ideas carry the whole subject; most questions below apply them to one tool. - **"Good" has to be decomposed before it can be measured.** An answer can be relevant to the question and still unsupported by its sources, or correct and still in the wrong format. Each is a separate metric because each fails, and is fixed, somewhere different; a blended score hides which one moved. - **Where the reference comes from decides where a metric can run.** Some metrics compare the output with an expected answer; others judge it only against the input and the retrieved context. Curated datasets can carry expected answers and live traffic cannot, so reference-based metrics belong offline. - **An LLM judge is a measuring instrument with its own error.** A model grading a model makes evaluation affordable at scale, but the judge is nondeterministic, billed per call and biased. Treat its scores as noisy readings: pin the judge, fix the rubric, check it against human labels, leave margin around thresholds. - **Retrieval-augmented systems fail in two places.** A wrong answer can come from retrieving the wrong context or from generating badly on the right context. Metrics that score retrieval and generation separately turn "the RAG is bad" into a statement about one stage. - **The trace is the unit of observability.** Each user request or agent execution is recorded as a tree of timed steps, with model calls carrying the model, parameters, token usage and cost. Without nesting, a multi-step agent is a pile of unrelated calls. - **Evaluation and tracing form one loop.** Production traces surface failures, the interesting ones become dataset examples, changes are compared on that dataset before release, and sampled live traffic is scored and annotated after it. Interviewers expect the loop, not one tool in it. - **Where you capture decides what you see and what you risk.** In-process instrumentation, a proxy or an OpenTelemetry exporter differ in visibility, integration effort and failure modes. And because traces hold user input and retrieved documents, what to capture and where it is stored are design decisions.

Test case
One record to evaluate: the input, the output your application produced, and optionally the expected answer and the context that was available.
Reference answer
A human-written or approved expected output that reference-based metrics compare against; present on curated datasets, absent on live traffic.
Reference-free metric
A metric that scores an output using only the input, the output and retrieved context, so it can run on production traces.
LLM-as-judge
Using a language model, prompted with a rubric, to score another model's output; scalable and cheap per case, but noisy and biased.
Faithfulness
Whether every claim in an answer is supported by the context it was given; the usual measure of hallucination in retrieval-augmented systems.
Context precision and recall
Retrieval-side metrics: how much of the retrieved context is useful, and how much of the needed information was retrieved at all.
Synthetic test set
Evaluation examples generated by a model from your own documents, used to bootstrap coverage before real traffic or labels exist.
Trace
The record of a single request or agent execution from start to finish, structured as a tree of nested, timed steps.
Span
One timed step inside a trace, with a parent, inputs, outputs and attributes; a model call is a span with model and token fields.
Dataset
A stored, versioned collection of examples, often promoted from production traces, that changes are evaluated against.
Experiment
One run of an application version over a dataset, with every output scored, so two versions can be compared on identical inputs.
Online evaluation
Scoring a sampled share of live production traffic after the fact, usually with reference-free metrics and an LLM judge.
Semantic conventions
Agreed OpenTelemetry attribute names for a domain, so any tracing backend can read model, provider and token data without custom parsing.

The six sections sit at three layers, and most answers draw on more than one. **Evaluation libraries compute scores.** DeepEval and Ragas take records shaped like test cases and return metric scores with reasons. They do not keep production history; you run them in a notebook, a scheduled job or a CI gate. DeepEval leans toward the software-testing workflow; Ragas toward diagnosing retrieval pipelines. Both can generate a synthetic test set from your documents, a way to start before real labelled data exists. **Platforms keep the history and close the loop.** LangSmith and Langfuse store traces, datasets, experiments and scores in one place. Shared storage is what lets a trace become a dataset example, an experiment be compared with the last, and an evaluator or annotator score a live run. The choice between them in an interview often turns on hosting and data control rather than on features. **Capture layers feed the other two.** Helicone and OpenLLMetry decide how call data is collected rather than whether an answer was good. Used as a proxy, Helicone sits between your code and the provider, so it can also cache, rate-limit and retry. OpenLLMetry instruments the client libraries in-process and writes standard GenAI attributes, so model spans join the traces your services already send. **Offline and online evaluation share metrics but not inputs.** Offline runs score a fixed dataset that can carry references, before a change ships. Online runs score sampled traffic that cannot, after it ships. Human annotation is the bridge, turning judged production runs into the references the next offline run needs. **Cost runs through every layer.** Judge calls are billed like any model call, token and cost attributes live on spans, and tags attached at capture are what let spend be sliced by feature or user later.

  1. Built-in Metrics →

    Start with what a metric is, which fields of a test case it reads, and how a threshold turns a score into a pass or fail.

  2. Metric Classes →

    Learn to split retrieval quality from generation quality and to tell which metrics need a reference answer and which do not.

  3. Traces & Observations →

    The trace and observation model shows how one request becomes a tree of timed steps carrying model, token and cost data.

  4. Datasets & Experiments →

    Datasets and experiments connect the two halves: production examples become a fixed benchmark that versions are compared on.

  5. Online Eval & Annotation →

    Online evaluation and annotation cover what happens after release: sampling live traffic, judge cost, and human feedback.

  6. GenAI Semantic Conventions →

    Finish with standard GenAI span attributes, the answer to how LLM telemetry fits a tracing stack a team already runs.

  • Quoting a single quality score without saying what it measures, which reference it compares against, or which model did the judging.

  • Proposing a reference-based metric for live traffic, where no expected answer exists; production scoring needs reference-free signals or later human labels.

  • Treating a judge model's score as ground truth: an unpinned judge, a rubric that changes between runs, and no check against human labels.

  • Setting per-case thresholds inside the score's own noise, so a CI evaluation suite fails at random; measure variance before deciding what counts as a failure.

  • Blaming the prompt or the model for a bad RAG answer without first checking whether retrieval returned the right context.

  • Recording only flat model calls, so a multi-step agent run cannot be read or timed as one operation and the failing step stays hidden.

  • Putting a proxy in the request path with no way to bypass it, so an observability vendor's outage becomes your feature's outage.

  • Capturing full prompts and completions by default in a regulated product without a decision on redaction, retention and where traces are stored.

  • Scoring every production request with multi-call judge metrics, then discovering the evaluation bill rivals the application's own model spend.

The same choices recur across sections; naming the one you are making is the core of a senior answer. - **Offline versus online evaluation.** Offline runs on curated data can use references and gate a release, but only cover what someone thought to include. Online scoring sees real inputs, but mostly with reference-free metrics and only on a sample. - **Judge model versus human versus deterministic check.** Exact match, schema validation and keyword checks are cheap and stable but narrow. Human labels are trusted but slow and scarce. An LLM judge sits between the two: broad and scalable, noisy and billed per call. - **Proxy versus in-process instrumentation.** A proxy needs almost no code change and can add gateway features, at the price of a third party in the request path. SDK or OpenTelemetry instrumentation keeps the path direct but needs code and dependency upkeep. - **Per-case gates versus aggregate gates.** Failing a build on any single case catches regressions early and amplifies noise; gating on an average is stable and can hide a broken edge case.

A few shapes recur across the tools under different names. - **Failure to regression test.** A bad production trace is promoted into a dataset, the fix is measured against that dataset, and the example stays as a guard against the same failure returning. - **Break it down, then judge the pieces.** Many judge metrics split an output or a context into smaller units and score each, which makes the result more stable and the explanation more specific than one holistic verdict. - **Pin and record.** The judge model, the rubric, the prompt version and the dataset version are stored next to every score, so a moved number can be traced to a changed input rather than instrument drift. - **Filter, sample, then score.** Expensive evaluation of live traffic narrows to the runs that matter, takes a share of those, and only then spends judge calls. - **Tag at the source.** User, feature, environment and prompt version are attached when the call is captured, because cost and quality can only be sliced later along dimensions recorded at the time. - **Fail open around observability.** Logging and tracing paths are given timeouts, fallbacks or asynchronous delivery so that losing telemetry never means losing the feature.

explore

report an issue with this guide →

questions

121 · 6 sections

In DeepEval, what is the difference between a Golden and an LLMTestCase?

level: juniorimportance: must knowfreq 72%
basics
~20 s

A Golden holds only the static half of a test — input, expected_output, context — with no actual_output. An LLMTestCase adds the actual_output your application produced on this run, and that is what metrics score.

open as a page

What does evaluation_params control in a DeepEval GEval metric?

level: juniorimportance: must knowfreq 65%
basics
~20 s

evaluation_params is the list of LLMTestCaseParams members — INPUT, ACTUAL_OUTPUT, EXPECTED_OUTPUT, RETRIEVAL_CONTEXT and so on — naming which fields of the LLMTestCase the judge model is shown. Fields you leave out are invisible to the judge.

open as a page

In DeepEval, how do you score one LLMTestCase with a built-in metric?

level: juniorimportance: must knowfreq 78%
basics
~10 s

Build an LLMTestCase with at least input and actual_output, construct a metric such as AnswerRelevancyMetric(threshold=0.7), then call metric.measure(test_case). DeepEval then fills metric.score, metric.reason and metric.is_successful() for that one case.

open as a page

In DeepEval, what does assert_test() do inside a pytest test function?

level: juniorimportance: must knowfreq 70%
basics
~10 s

assert_test(test_case=..., metrics=[...]) scores the test case with every metric and raises AssertionError if any metric lands below its threshold. That turns an LLM evaluation into an ordinary pytest failure.

open as a page

What does DeepEval's Synthesizer.generate_goldens_from_docs actually do?

level: middleimportance: must knowfreq 62%
basics
~20 s

It reads the documents you point it at, splits and groups them into contexts, then asks an LLM to write synthetic inputs grounded in each context, rewrites them to be harder, and returns Golden objects — optionally with a generated expected_output.

open as a page

In Ragas, how do you run evaluate() over an EvaluationDataset of samples?

level: juniorimportance: must knowfreq 70%
basics
~20 s

Ragas scores a whole dataset in one call. Put each record into a SingleTurnSample, wrap the list in an EvaluationDataset, then call evaluate(dataset=..., metrics=[...], llm=...). It returns an EvaluationResult carrying per-metric averages and a to_pandas() view.

open as a page

Which Ragas calls turn your own documents into a synthetic testset?

level: juniorimportance: must knowfreq 55%
basics
~10 s

Two calls. TestsetGenerator.from_langchain(llm, embedding_model) builds the generator from a LangChain model pair, then generator.generate_with_langchain_docs(docs, testset_size=N) returns a Testset of N synthetic samples with questions, reference contexts and reference answers.

open as a page

In Ragas, what do RunConfig's max_workers, timeout and max_retries control?

level: middleimportance: must knowfreq 58%
basics
~20 s

RunConfig tunes how a Ragas run executes: max_workers caps how many metric calls run concurrently (default 16), timeout bounds a single call (default 180 seconds), and max_retries is how many times a failed call is retried with backoff (default 10). Pass it as evaluate(run_config=RunConfig(...)).

open as a page

In Ragas, how do you give metrics an evaluator LLM and embedding model?

level: middleimportance: must knowfreq 78%
basics
~20 s

Wrap a LangChain chat model in LangchainLLMWrapper and a LangChain embeddings object in LangchainEmbeddingsWrapper, then pass them to evaluate() as llm= and embeddings=, or set them on individual metric objects. Ragas ships no model of its own.

open as a page

Which SingleTurnSample fields does each Ragas metric class require?

level: middleimportance: must knowfreq 68%
basics
~20 s

Each Ragas metric reads only the SingleTurnSample fields it declares. Faithfulness uses user_input, response and retrieved_contexts; LLMContextRecall and FactualCorrectness also need a human-written reference; ResponseRelevancy needs an embedding model in addition to a judge LLM.

open as a page

How do you create a LangSmith dataset and add examples with the Python SDK?

level: juniorimportance: must knowfreq 62%
basics
~10 s

Instantiate langsmith.Client, call client.create_dataset(dataset_name=...) to get a Dataset, then client.create_examples(dataset_id=dataset.id, examples=[...]) where each example is a dict with an inputs dict and, usually, a reference outputs dict.

open as a page

What signature and return value does a custom LangSmith evaluator function need?

level: juniorimportance: must knowfreq 78%
basics
~20 s

A LangSmith evaluator is an ordinary Python function that declares any subset of the arguments inputs, outputs and reference_outputs, and returns a dict such as {"key": "exact_match", "score": 1}. You pass it in the evaluators list of evaluate().

open as a page

How do you attach an end-user thumbs-down to a LangSmith run from your app?

level: juniorimportance: must knowfreq 52%
basics
~20 s

Capture the traced call's run id, then call Client.create_feedback with that run id, a feedback key such as "user_score", and a score. The key names the metric, the score is the number, and comment or correction can carry the user's words or the fixed answer.

open as a page

How do you enable LangSmith tracing for a plain Python function with @traceable?

level: juniorimportance: must knowfreq 78%
basics
~10 s

Set LANGSMITH_TRACING=true and LANGSMITH_API_KEY in the environment, then decorate the function with @traceable from the langsmith package. Each call is logged as a run in the LANGSMITH_PROJECT project, nested under any traced caller.

open as a page

In LangSmith's evaluate(), what must the target function accept and return?

level: middleimportance: must knowfreq 74%
basics
~20 s

The target takes one argument — a single example's inputs dict — and returns a dict of its outputs. LangSmith calls it once per example in the dataset and records each call as a run inside one named experiment.

open as a page

In Langfuse, what does compile() do to a fetched prompt, and what placeholder syntax does it use?

level: juniorimportance: must knowfreq 60%
basics
~20 s

compile() substitutes values into a Langfuse prompt's double-curly-brace placeholders, such as {{question}}. A text prompt compiles to a plain string; a chat prompt compiles to a list of role/content message dicts, ready to pass straight to a model SDK.

open as a page

In Langfuse, how do prompt versions and labels decide what get_prompt() returns?

level: juniorimportance: must knowfreq 72%
basics
~20 s

Every save of a Langfuse prompt creates a new immutable numbered version. Labels such as production and latest are movable pointers to one of those versions. langfuse.get_prompt("name") resolves the version labelled production; pass label= or version= to target a different one.

open as a page

In Langfuse, how do you attach a score to a trace or an observation?

level: juniorimportance: must knowfreq 72%
basics
~20 s

A Langfuse score is a named value hung off a trace or off one observation inside it. From inside an active observation call score_current_span() or score_current_trace(); from anywhere else call score_trace(trace_id=...) with a trace id you stored earlier.

open as a page

How does Langfuse's @observe decorator trace a Python function?

level: juniorimportance: must knowfreq 70%
basics
~20 s

@observe wraps a Python function so each call becomes one Langfuse observation, named after the function, timed, with the arguments captured as input and the return value as output, nested automatically under whatever observation is already active.

open as a page

In Langfuse, what is a trace and what are the observations inside it?

level: juniorimportance: must knowfreq 74%
basics
~20 s

A Langfuse trace is one end-to-end unit of work, typically one request or agent run. Observations are the timed steps inside it: nested and typed, with GENERATION recording an LLM call and SPAN any other step.

open as a page

How do you turn on Helicone's response cache and control how long entries live?

level: juniorimportance: must knowfreq 60%
basics
~10 s

Send Helicone-Cache-Enabled: true on the request, and set the lifetime with a standard Cache-Control: max-age=<seconds> header. Caching only works through the Helicone proxy, because something has to answer in place of the provider.

open as a page

How do you route OpenAI traffic through Helicone's proxy, and which header authenticates it?

level: juniorimportance: must knowfreq 72%
basics
~10 s

Point the OpenAI client's base URL at Helicone's proxy host, https://oai.helicone.ai/v1, and send a Helicone-Auth header holding "Bearer" plus your Helicone API key. Your provider key still travels in the usual Authorization header.

open as a page

When would you choose Helicone's async logging over its proxy integration?

level: middleimportance: must knowfreq 62%
basics
~20 s

Async logging keeps the provider call direct and ships the log separately, so Helicone adds no latency and cannot take your feature down. Choose it when the request path must stay untouched; choose the proxy when you want gateway behaviour, not just logs.

open as a page

What breaks when Helicone's proxy is slow or unreachable, and how do you limit the blast radius?

level: seniorimportance: must knowfreq 58%
basics
~20 s

With the base URL pointed at the proxy, every model call goes through it, so a degraded proxy degrades the feature itself — calls slow down, hang or fail. Limit the damage by making the base URL a flippable runtime setting, bounding timeouts, and moving latency-critical paths to async logging.

open as a page

In Helicone, what does the Helicone-Cache-Seed header change about cache hits?

level: middleimportance: should knowfreq 42%
basics
~20 s

Helicone-Cache-Seed partitions the cache into namespaces. Two identical requests sent with different seed values never share an entry, so you can isolate caches per user or tenant, and changing the seed instantly invalidates everything cached under the old one.

open as a page

What does Traceloop.init() do so existing OpenAI calls emit spans in OpenLLMetry?

level: juniorimportance: must knowfreq 74%
basics
~20 s

Traceloop.init() starts OpenLLMetry's tracing pipeline and monkey-patches the LLM client libraries already installed in the process — OpenAI, Anthropic, LangChain, LlamaIndex, vector-store clients — so their calls emit spans. Your existing call sites stay unchanged.

open as a page

Which OpenTelemetry GenAI span attributes carry the model name and token counts?

level: juniorimportance: must knowfreq 68%
basics
~10 s

gen_ai.request.model records the model you asked for, gen_ai.response.model the one that answered, and gen_ai.usage.input_tokens plus gen_ai.usage.output_tokens the token counts. Standard names let any backend chart cost without reading your code.

open as a page

Why add OpenLLMetry's @workflow and @task decorators when init already traces model calls?

level: middleimportance: must knowfreq 66%
basics
~20 s

Auto-instrumentation only sees individual library calls. The decorators create named parent spans for your own multi-step logic, so a retrieval plus three model calls appears as one named operation you can time, compare and filter on rather than four unrelated spans.

open as a page

What does setting TRACELOOP_TRACE_CONTENT=false change in OpenLLMetry spans?

level: middleimportance: must knowfreq 58%
basics
~20 s

It stops OpenLLMetry recording prompt and completion message text on spans. Metadata still flows: provider, model, token counts, parameters, latency and errors. Content capture is on by default, so this is the switch a regulated team flips.

open as a page

When should you pass disable_batch=True to OpenLLMetry's Traceloop.init(), and what does it cost?

level: middleimportance: should knowfreq 48%
basics
~20 s

Pass disable_batch=True in short-lived processes — serverless handlers, CLI scripts, notebooks, tests — where the process can end before buffered spans are sent. It exports each span as it finishes, which is safe but adds export work on the calling path, so leave it off in long-running services.

open as a page