Two reviewers disagree on 31 of 400 eval labels — what is your adjudication protocol?

## Why disagreement is the useful signal When two qualified reviewers label the same eval item differently, one of three things is true: one of them made a mistake, the rubric does not decide the case, or the case is genuinely ambiguous and no rubric could decide it without an arbitrary convention. Only the first is noise. The second is the most common and the most valuable, because it points at a definition your whole evaluation depends on and has never actually written down. Teams that treat disagreement purely as a quality problem in their annotators — retraining people, tightening SLAs — usually leave the underlying ambiguity in place, where it keeps producing inconsistent labels forever. ## The protocol **1. Label blind and independently.** Both reviewers see the same input and the same candidate output, and neither sees the other's label. Critically, reviewers should not know which system version produced the output, and ideally should not see the model's own justification before deciding — reading a fluent rationale anchors a reviewer toward accepting it. Where you can, show outputs in randomized order and strip version identifiers. **2. Pilot on a small batch first.** Run twenty or thirty items through both reviewers before committing to four hundred. The disagreements from that pilot are the cheapest rubric feedback you will ever get, and fixing the rubric before the bulk pass avoids re-labeling the whole set later. **3. Adjudicate the splits.** A third reviewer, again blind to who labeled what, resolves each disputed item and records a short reason. The reason is the artifact that matters; a bare tiebreak label teaches you nothing. **4. Cluster and generalize.** Read the adjudicated items as a set. Do they concentrate in one category, one failure mode, one boundary condition? A cluster means a rule is missing. Write it into the rubric with a worked example on each side of the line, then re-label everything in the affected category — because the agreed labels in that category were agreed for the wrong reason as often as the disputed ones were disputed. **5. Version the rubric with the set.** A label is only meaningful under the rubric that produced it. When the rubric changes, the set version changes, and scores from before and after are not directly comparable until the previous system version is re-scored. ## What to do with irreducible ambiguity Some items resist every rubric: two defensible answers, a request whose intent is genuinely unclear, a case where the correct behaviour depends on information the system was never given. Forcing a label on those items injects noise into every future comparison and, worse, injects it in a way that looks like signal. The cleaner move is a third category — unlabelable, or ambiguous — kept in the set but excluded from the headline pass rate and reported as its own slice. Its size is informative on its own: a growing ambiguous slice means the product's own definition of correct is drifting, which is a design conversation, not a labeling one. Some teams instead label such items with an accepted-set of valid answers, which works when the ambiguity is over form rather than over what the right decision is. ## The label-noise ceiling Human labels are the measuring instrument, and no instrument reports differences smaller than its own error. If independent reviewers agree on roughly 92 percent of items in a category, then two system versions whose true quality differs by two points in that category cannot be distinguished on that set no matter how many items you add, because the disagreement is not random noise that averages out — it is systematic, tied to the same hard cases every time. This is the most common reason a carefully built eval suite fails to reproduce a difference the team is sure exists. The practical consequences: measure the disagreement rate and publish it next to the scores; spend labeling budget on rubric clarity before spending it on volume; and be explicit that any automated grader you later validate against these labels inherits the same ceiling — it can at best reproduce the humans, including their inconsistency. ## Cost discipline Double-labeling everything is expensive. A common compromise is to double-label a random sample — perhaps 15 to 20 percent — plus every item in high-stakes categories, and single-label the rest. That gives an ongoing measurement of reviewer consistency without paying twice for the whole set. Re-measure it after every rubric change, because the number you are quoting was earned under the old definition.