A Ragas run finishes but some metric scores are NaN — what happened, and what do you do?

## What a NaN actually means Ragas treats an evaluation as a batch of independent jobs, and it is deliberately tolerant of individual job failure. `evaluate(..., raise_exceptions=False)` is the default: when a scoring job raises after its retry budget is spent, the framework records no score for that sample-and-metric pair and moves on. In the dataframe from `to_pandas()`, that shows up as NaN in the metric column. This is a reasonable default — one poisoned row should not destroy a forty-minute run — but it converts a loud failure into a quiet one, and quiet failures in evaluation are dangerous because the output still looks like a result. ## The realistic causes, roughly in order **Retries exhausted.** The provider was throttling or erroring for long enough that backoff ran out. Symptom: NaNs cluster in time, and often across every metric at once, because the whole executor was hitting the same wall. **Per-call timeout.** A slow judge, or a metric issuing several sequential calls, ran past the deadline. Symptom: NaNs concentrate on the samples with the longest contexts or the most expensive metrics. **Provider-side rejection.** Context-length errors from a sample whose `retrieved_contexts` are enormous, or a safety filter refusing to process the content. Symptom: NaNs correlate with a content property, not with time. **Unparseable judge output.** LLM metrics ask the judge for a structured answer and then parse it. A judge that returns prose, truncates, or emits malformed JSON produces a parse failure. Symptom: a weaker or smaller judge model has far more NaNs than a stronger one for the same dataset — this is the classic reason a cost-saving judge swap goes wrong. ## Why the average lies Aggregation is over the samples that produced a score. If 300 of 1,000 rows failed, the headline is the mean of the 700 that survived — and those 700 are not a random sample. If the failures were context-length rejections, you have quietly excluded exactly the hardest, longest-context queries and made the system look better than it is. This selection effect is the reason NaN counting comes before interpretation, always. ## The diagnostic sequence 1. `df = result.to_pandas()`, then `df.isna().sum()`. Any non-zero count invalidates the headline until explained. 2. Look at *which* rows failed. Are they the longest ones? All from one time window? All on one metric? 3. Re-run a small failing subset with `raise_exceptions=True`. The run now stops on the first real exception and you see the actual error rather than a silent gap. 4. Fix by cause: reduce concurrency for throttling, raise the timeout for a slow judge, truncate or chunk oversized contexts for length rejections, use a stronger or better-constrained judge for parse failures. 5. Re-run and confirm the NaN count is zero before comparing to anything. ## Distinguish this from an upfront validation error A different and louder failure happens *before* any scoring: a metric declares the sample fields it requires, and a dataset that does not carry them fails validation rather than producing NaNs. Handing reference-free production traffic to a metric that requires `reference` is that case — you get an explicit error about the missing field, not silent gaps. Knowing which of the two you are looking at tells you whether the problem is your dataset shape or your run configuration. ## Reporting discipline Any automated report over a Ragas run should carry the scored-row count next to the score, in the same way a survey reports its response rate. "Faithfulness 0.86" and "faithfulness 0.86 over 702 of 1,000 samples" are different claims, and only the second one can be safely compared to last week's run. If a run cannot be repaired in time, reporting it with the completeness figure is honest; reporting the mean alone is not. ## The failure mode to name in an interview The worst version of this is a scheduled evaluation whose judge provider degrades, whose rows quietly stop scoring, and whose dashboard keeps drawing a flat, healthy line computed over a shrinking survivor set. A completeness check is the guard, and it belongs in the same job that computes the score.