In a content-moderation labeling operation, why is each reported post judged by three annotators instead of one?

## The problem with a single verdict A labeling operation exists to **manufacture** the training label for a moderation model: the platform does not find labels lying in its logs, it pays people to produce them against a written policy. A single annotator's verdict on a reported post is an assertion with nothing attached to it. The record says `remove`, and the record alone cannot distinguish a correct read of an obvious case from a rushed read of a genuinely hard one. Nothing in the operation flags that item as worth a second look, and nothing in the aggregate tells you whether the policy document the annotator worked from is even usable. Redundancy is what turns an opinion into a measurement. ## What the second and third verdicts actually produce They produce two separate outputs, consumed by different people: - **An item-level disagreement flag.** A post on which the three annotators split is, by definition, a post the guideline does not resolve cleanly. That flag is the routing signal into the adjudication path. - **An operation-level agreement trend, per policy category.** Rising disagreement inside one category is the earliest evidence that its wording is ambiguous, that it overlaps another category, or that what the queue is sending has shifted. - **A qualification surface.** Overlap between a new annotator and the established pool is how you tell, in their first days, whether they have absorbed the guideline, before their verdicts silently enter the training set. ## The adjudication path 1. The three verdicts are captured **blind**: no annotator sees another's answer. If the third reviewer can see the first two, you have bought an echo rather than a verdict. 2. Unanimous items pass straight through; the shared verdict becomes the training label. 3. A split item leaves the normal queue for an escalation queue, where a senior reviewer decides and **cites the guideline clause** that decided it. 4. The **adjudicated verdict, not the majority**, becomes the training label. Two annotators being outvoted by the reviewer is the path working as designed. 5. When the adjudicated case is the first of its kind, it is written back into the guideline as a worked example. This is how a policy document accumulates its own case law and stops producing the same split next month. ## Why three rather than two | annotators per item | what you learn | relative cost | where it fits | |---|---|---|---| | one | nothing about the verdict's correctness | 1x | settled categories already covered by known-answer items | | two | that they disagree, never which one is right | 2x | categories with a long history of high agreement | | three | a majority resolves the common case, so only a real split escalates | 3x | new clauses, new annotator cohorts, contested categories | Two annotators detect disagreement but resolve nothing: every split goes to an adjudicator, which is the most expensive seat in the operation. A third verdict lets the routine case settle itself and reserves escalation for items that are genuinely on the boundary. ## Agreement is a health metric, not a proof of correctness High agreement means the pool applies the guideline **consistently**. It does not mean the pool applies it **correctly**: a group trained together can settle on a shared misreading of a clause and agree with each other all the way to a wrong training set. Consistency and correctness are different properties, and only an item whose correct verdict is already known can separate them. Two further cautions: - With three or more annotators the measure has to correct for the agreement you would get by chance, which is what statistics such as Fleiss' kappa and Krippendorff's alpha are for; how any of them reads depends on the category's base rate. - The useful artefact is the **trend per category**, not one number for the whole operation. A single operation-wide score averages a healthy category against a broken one and hides both. ## Spending the redundancy where it pays Every extra verdict is annotation capacity that could have bought coverage of fresh volume instead, so redundancy is allocated, not applied uniformly: - A category at sustained high agreement can drop to fewer annotators per item, with a known-answer rate holding the floor, and the freed capacity moves to a contested category. - A newly edited clause or a newly onboarded cohort goes back to three until the agreement trend recovers. - Disagreement is **never** resolved by discarding the item. The contested posts sit exactly on the decision boundary the model most needs to learn; dropping them leaves a training set made only of the easy cases, and a model that looks excellent until it meets a real one.