How do you validate an LLM judge used to filter synthetic fine-tuning data?

## The filter is a classifier, so measure it like one Running a judge model over generated training rows and keeping the ones it scores highly is a filtering decision, not a scoring exercise. Every row is being classified keep or drop, and the resulting dataset is whatever the classifier let through. If you would not deploy an unmeasured classifier into a production path, you should not deploy one over your training data either — it decides what the model learns. The measurement is the ordinary one. Draw a sample of generated rows — stratified across the score bands, not just the borderline ones — and have a domain expert label each keep or drop under the same written rubric the judge was given. Then compute, at the threshold you intend to use: - **Precision** — of the rows the judge kept, what fraction the expert also would have kept. Low precision means bad rows are entering training. - **Recall** — of the rows the expert would keep, what fraction the judge kept. Low recall means you are discarding good data, and paying for it twice: once in generation cost, once in coverage. A few hundred labelled rows is usually enough to see whether the filter is trustworthy at all; sweep the threshold across that same labelled sample rather than accepting the first cut-off someone picked. ## The two errors are not symmetric **False accepts** are the dangerous ones. A wrong row that survives the filter becomes a training target, and the model learns it as the correct behaviour. In a clinical-adjacent domain, one confidently wrong triage instruction repeated across a few dozen surviving rows is a real harm, not a statistical nuisance. **False rejects** look benign — you lose a row you paid to generate — but they are not random. The judge rejects according to its own preferences, so the rows it drops share properties: they are terser, less conventionally formatted, more hedged or less hedged, further from the register the judge finds "good". The survivors are therefore a biased sample of what you generated, and the fine-tuned model inherits that bias. This is the second-order effect people miss: a filter does not merely subtract quality problems, it reshapes the distribution. Where the filter also removes the harder or more unusual cases, it can undo the coverage work that went into generating them. ## Correlated blind spots When the generator and the judge are the same model, or close relatives, their failure modes overlap: the judge is least likely to notice exactly the mistakes the generator is most likely to make, and it will favour text in its own style. Using a different model family for judging, or a rubric written from the domain expert's failure list rather than from general notions of quality, reduces the overlap. It does not eliminate it, which is why the human-labelled sample remains the ground truth for the filter. ## Prefer checks that do not need a judge Anything mechanically verifiable should be verified mechanically, before the judge sees the row: schema validity, required fields present, the urgency band drawn from the permitted set, no contradiction between the stated band and the recommended action, length within bounds, no leakage of the generation prompt. Programmatic checks are deterministic, free, and never drift. Reserve the judge for the genuinely subjective residue — is the clinical detail specific enough, does the reasoning actually support the conclusion — and write its rubric as concrete criteria rather than "rate the quality". ## Ongoing spot-checks and drift One validation is a snapshot. Keep a continuous sample — a few percent of accepted rows reviewed by the expert — because the thing being filtered changes: new generation prompts, a new seed batch, a new judge model version, a rubric edit. Any of those invalidates the earlier precision and recall figures. Cheap monitoring signals help between full validations: the accept rate per generation batch, the score distribution shape, and the share of accepted rows falling in each coverage cell. ## Cost, and when to skip the filter Judging every generated row costs a model call per row, which for a large synthetic set is a real line item. Cheaper arrangements: run programmatic checks first so the judge only sees rows that could plausibly pass; judge a sample to estimate batch quality and regenerate whole batches that fail rather than filtering row by row; use a smaller judge for an initial pass and a stronger one on the borderline band. Whichever you choose, the validation requirement does not change — you still need to know the precision and recall of the rule you are applying.