Running Evaluations

## What a Ragas run actually is Ragas does not score "a RAG pipeline". It scores **records that a pipeline already produced**. You run your application first, capture what it did for each question, and then hand that captured data to Ragas. That separation is why the API has three distinct objects: a sample (one record), a dataset (a homogeneous collection of records), and a run (`evaluate`) that returns a result object. ## The sample `SingleTurnSample` is a typed container for one question-and-answer interaction. Its fields map onto what RAG metrics need: - `user_input` — the question that was asked. - `retrieved_contexts` — the list of context strings your retriever returned. - `response` — what your generator produced. - `reference` — the ground-truth answer, present only when you have one. - `reference_contexts` — the contexts that *should* have been retrieved, for recall-style metrics. Not every field is needed for every metric; each metric declares which fields it reads, and a dataset missing a required field fails validation rather than silently scoring badly. For conversations there is `MultiTurnSample`, whose `user_input` is a list of message objects instead of a string; a dataset must be all one sample type, and only multi-turn-capable metrics accept it. ## The dataset `EvaluationDataset(samples=[sample1, sample2, ...])` is the direct constructor. When your data is already a list of dictionaries — the usual shape after pulling rows out of a warehouse, a JSONL file, or a dataframe converted with `to_dict("records")` — `EvaluationDataset.from_list(records)` builds the samples for you, and the dictionary keys must match the sample field names. The dataset is deliberately dumb: it holds records and knows its sample type. It contains no scoring logic and no configuration. ## The run `evaluate(dataset=ds, metrics=[...], llm=evaluator_llm)` is the single entry point. Two things about it surprise people the first time: **It is not a loop.** Internally Ragas builds one job per (sample, metric) pair and executes them concurrently through an executor, with a progress bar. A hundred samples and three metrics is three hundred jobs, not three hundred sequential HTTP round trips. Concurrency, per-call timeouts and retries are configured through a `RunConfig` passed to `evaluate`. **It needs a judge model.** Almost every metric in Ragas is LLM-based: it prompts a model to extract claims, check them against context, or produce a verdict. You supply that evaluator model either once on `evaluate(llm=...)` or individually on each metric object. Forgetting it is the most common first-run error. Some metrics also need an embedding model. ## The result `evaluate` returns an `EvaluationResult`. Printing it renders a dictionary of one averaged number per metric — the headline. `result.to_pandas()` returns a dataframe with one row per sample: the sample's own fields as columns, plus a column per metric holding that sample's individual score. That per-row view is where the useful information is; a single mean tells you a number moved but never which query broke. ## Why running it this way matters Calling `single_turn_ascore` on one sample at a time in a Python loop technically works and is how you debug a single case, but it gives up the executor, the shared run configuration, the progress reporting, the aggregate result object, and the dataframe. For anything beyond a handful of records, the batch call is the API you want. ## The shape of a real workflow Capture production or golden-set records → build `SingleTurnSample`s → wrap in `EvaluationDataset` → choose the metrics that match the fields you have (reference-requiring metrics are unusable on raw production traffic with no ground truth) → wire an evaluator model → `evaluate` → read `to_pandas()`, sort ascending by the metric column, and read the worst rows. The last step is the one candidates skip and interviewers ask about.