Evaluation & Testing

LLM-as-judge means prompting a model with a rubric to score another model's output. The score is a noisy estimate, not truth: the judge has its own biases and blind spots, so it must be validated against human labels before anyone trusts the number.

Offline evaluation scores a fixed, frozen set of saved examples in a harness before release, so it is repeatable and cheap. Online evaluation measures real user traffic after release, where actual behaviour and business outcomes decide whether the change helped.

Sample by strata, not uniformly. Bucket traces by the dimensions you care about — request type, customer segment, known failure mode — then take a quota from each bucket, so rare but costly cases appear in numbers large enough to score.

The recurring four are position bias (order of presentation sways pairwise verdicts), verbosity bias (longer wins), self-preference (a judge rates its own family higher), and formatting halo (bullets and confident tone read as quality). Controls: swap orders, normalise or cap length, judge across families, and strip presentation from the rubric.

Temperature 0 only removes sampling randomness. Serving-side variance (batched floating-point arithmetic, expert routing), silently repointed model aliases, and unpinned fixtures all still move scores, so a suite needs a full pin set and repeated runs rather than one greedy pass.

Those benchmarks score agentic coding and command-line work, and a construction-permit assistant writes no code — the memo ranks models on a construct the product does not use. Replace it with a suite of real permit questions scored on citation correctness and safe abstention.

An aggregate averages cohorts together, so a large healthy segment hides a small broken one. Slicing by who the request came from and by what went wrong exposes concentrated regressions, and lets you gate on the worst slice rather than the mean.

Have humans label a sample against the same rubric, run the judge on that sample, and measure chance-corrected agreement such as Cohen's kappa rather than raw percent agreement. Below roughly moderate agreement the judge is not usable, and the usual cause is an ambiguous rubric, not a weak model.

Three causes dominate: the frozen set no longer matches live traffic, the offline metric proxies something users do not reward, and the harness has no user in it to react. A large offline gain on the wrong population or the wrong proxy buys nothing online.

Measure the suite's run-to-run spread on unchanged code first, then set the fail margin above that band — for example baseline minus two points — and decide on repeated runs with a majority rule so a single unlucky sample cannot block a release.

Run a fixed set of scored eval cases as a build step and compare the aggregate score against the last accepted baseline. The build fails on a score drop beyond an agreed margin, not on any difference in wording.

Saturation. Frontier models score roughly 88-99% on all three, so the surviving gap is mostly ambiguous items and label errors rather than capability. Once scores sit at a benchmark's ceiling it stops discriminating, which is why harder successors replaced it.

Route the disputed items to a third, independent adjudicator, then read the resolved cases together. Most disagreement is a symptom of an underspecified rubric, so the real output is a sharper rubric plus a re-label of the affected category — not just 31 settled labels.

Use pairwise when you need to resolve which of two candidates is better, because models compare far more reliably than they assign absolute numbers. Use pointwise when you need a per-item score that is comparable over time, across releases and by criterion.

Shadow traffic mirrors real requests to a candidate whose output is scored but never shown, so it measures behaviour on the true request distribution at real latency and cost with zero user risk. It can never measure user response, because no user ever sees the output.

Contamination is test data leaking into a model's training corpus, so the score measures recall of seen items instead of the capability. Detect it by comparing performance on freshly authored or perturbed items of equal difficulty: a large drop is the signature.

Almost certainly not. On 60 pass/fail items scoring around 80 percent, the 95 percent interval is roughly plus or minus 10 points, so a 4-point gap is well inside noise. Detecting it needs pairing, graded scores, or several hundred items.

Whenever a pre-declared harm metric breaches its threshold, even if the primary metric improves. Guardrails encode costs the primary metric ignores — refunds, escalations, safety violations, latency, spend — and they are set before the experiment precisely so a good headline number cannot argue them away.

Diagnose the flap first — an ambiguous expected answer, a judge scoring near a rubric boundary, or an unpinned fixture. Fix what is fixable, move the irreducible remainder into an advisory tier that reports but cannot block, and delete cases that stabilise nothing.

Enough to build a shortlist, never enough to close a decision. Reported scores come from the party selling the model, often under undisclosed scaffolding and effort settings, and they omit cost, latency and reliability. Internal suites hold the deciding vote.

On a scheduled cadence plus event triggers. Schedule a review each quarter; trigger immediately when the rules the labels encode change, when the product gains a capability the set never covers, or when the traffic mix shifts. Version the set, never edit it silently.

Treat the judge as a versioned instrument: pin the judge model version and rubric text, record both with every score, and never compare numbers across a change without a bridge. When either changes, re-run a frozen human-labelled set and dual-run old and new judges over an overlap to measure the offset.

A canary controls risk: a tiny exposed slice bounds blast radius while you watch for breakage, and it is read as a safety check. An A/B test controls inference: randomized arms and sufficient sample size let you attribute a measured effect to the change.

Treat cost and wall-clock as gate design constraints. Put a small stratified subset covering every failure mode on pull requests so it finishes in minutes, run the full suite nightly and before release, and measure how many nightly regressions the subset would have caught.