DeepEval

evaluation_params is the list of LLMTestCaseParams members — INPUT, ACTUAL_OUTPUT, EXPECTED_OUTPUT, RETRIEVAL_CONTEXT and so on — naming which fields of the LLMTestCase the judge model is shown. Fields you leave out are invisible to the judge.

Build an LLMTestCase with at least input and actual_output, construct a metric such as AnswerRelevancyMetric(threshold=0.7), then call metric.measure(test_case). DeepEval then fills metric.score, metric.reason and metric.is_successful() for that one case.

assert_test(test_case=..., metrics=[...]) scores the test case with every metric and raises AssertionError if any metric lands below its threshold. That turns an LLM evaluation into an ordinary pytest failure.

It reads the documents you point it at, splits and groups them into contexts, then asks an LLM to write synthetic inputs grounded in each context, rewrites them to be harder, and returns Golden objects — optionally with a generated expected_output.

criteria is a sentence GEval expands into judging steps with an extra model call each time the metric is built. evaluation_steps hands GEval that list yourself, so every run applies the identical rubric. Pass one or the other, never both.

All metrics need input and actual_output. FaithfulnessMetric and ContextualRelevancyMetric add retrieval_context; ContextualPrecisionMetric and ContextualRecallMetric add both retrieval_context and expected_output; HallucinationMetric needs context. Missing a required field raises an error rather than scoring zero.

deepeval test run invokes pytest with DeepEval's plugin active and its own flags on top: parallel processes, repeats, result caching, a run identifier, and an aggregated end-of-run report that can be pushed to Confident AI. Plain pytest still runs the assertions but produces none of that.

Separate metric errors from genuine low scores with the runner's ignore-errors flag, measure each case's score variance with repeated runs, move thresholds off the noise floor, mark genuinely unstable cases flaky so they warn instead of raising, and gate on an aggregate rather than every case.

Construct ConversationalTestCase with a turns list of Turn objects, each carrying a role of user or assistant and its content. Optional scenario and expected_outcome describe what the conversation was supposed to achieve, and conversational metrics score the exchange as a whole.

Evolutions are rewrite passes that make a freshly generated question harder — adding reasoning steps, forcing several contexts to be combined, adding constraints or hypotheticals. Each pass is another LLM call, and too many push the question away from its source context.

rubric takes a list of Rubric objects, each pairing a score_range with an expected_outcome description. It tells the judge what each band of scores means, so scores land on defined levels instead of on the judge's private sense of what 0.7 is worth.

strict_mode turns the metric into a binary one: the score becomes 1 for a perfect judgement and 0 otherwise, and the threshold is overridden to 1. Partial credit disappears, so anything less than flawless fails.

context holds human-supplied ground-truth background for the input; retrieval_context holds the chunks your retriever actually returned at runtime. HallucinationMetric reads context, while FaithfulnessMetric and the contextual metrics read retrieval_context — swapping them is a common wiring bug.

Bad. HallucinationMetric and ToxicityMetric are lower-is-better: the score measures how much hallucination or toxicity was found, and the case passes when the score is at or below the threshold — the inverse of AnswerRelevancyMetric or FaithfulnessMetric.

Use evaluate() when you want scores back rather than a pass/fail gate. It takes a list of test cases plus metrics, runs them with its own concurrency settings, and returns an EvaluationResult you can inspect — no pytest, no raised assertion.

Parametrize the test over the dataset's goldens — @pytest.mark.parametrize("golden", dataset.goldens) — and build one LLMTestCase inside the test body. Each golden then becomes its own pytest node, so failures are isolated and the cases can be distributed across processes.

Generate from the chunks your production retriever really returns, weight evolutions toward reasoning and multi-context, use StylingConfig so inputs read like real user messages, set a strong critic model in FiltrationConfig, and sample-read the output before it becomes the standard.

Subclass BaseMetric and implement measure(test_case), the async a_measure, and is_successful(), setting self.score and self.success inside measure (plus self.reason if you want an explanation). Expose a name property so the metric is labelled in reports.

Use DAGMetric when the judgement is a sequence of decisions rather than one holistic score — a decision tree of judgement nodes ending in verdicts. It makes the reasoning path explicit and auditable, where GEval collapses everything into one opaque judged number.

More than one per metric. Each built-in metric runs its own chain of judge prompts — extracting statements, judging them, and with include_reason=True writing an explanation — so cost scales as cases times metrics times several calls, not as one call per case.

AnswerRelevancyMetric, FaithfulnessMetric, ContextualRelevancyMetric, ToxicityMetric and BiasMetric all run without a reference. ContextualPrecisionMetric and ContextualRecallMetric need expected_output, and HallucinationMetric needs human-supplied context, so none of those three can score raw traffic.

It is a tradeoff between reproducibility and collaboration. Files in the repo are pinned by commit and reviewable in a pull request; a hosted dataset pulled with dataset.pull(alias=...) lets domain experts curate it, but the suite can change under CI without any code change.

Decide what the merge path must prove, then run only that: a small marked subset per pull request with parallel processes and the result cache on, the full sweep on a schedule, cheaper judges for breadth and expensive ones for the gate, and hyperparameters logged so runs stay comparable.