Why report Cohen's kappa instead of raw percent agreement between two annotators?

## The problem with percent agreement Suppose three crowd annotators are asked whether a comment is toxic, and you compute pairwise agreement on the items two of them both saw. You get 92%. Is that good? The number alone cannot say, because two annotators who never read the comments and simply guessed at the observed base rate would already agree most of the time when one class dominates. Percent agreement has no fixed zero: its floor moves with the label distribution. ## What kappa does Cohen's kappa rescales agreement so that chance sits at zero: ``` kappa = (p_o - p_e) / (1 - p_e) ``` - `p_o` is **observed agreement** -- the fraction of items the two annotators labelled the same. - `p_e` is **expected agreement** -- what they would reach if each annotator kept their own observed rate of using each label but assigned labels independently of the item. For two classes, `p_e = a_yes*b_yes + (1-a_yes)*(1-b_yes)` where `a_yes` and `b_yes` are the two annotators' marginal rates of the positive label. The numerator is the agreement above chance; the denominator is the agreement above chance that was available. So kappa answers: of the headroom that existed, how much did they use? Kappa is 1 for perfect agreement, 0 when the pair does no better than their own marginals predict, and negative when they systematically disagree more than chance -- which usually means a label was inverted somewhere. ## The prevalence trap This is the single most-asked follow-up. When one class is rare, `p_e` is already close to 1, so `1 - p_e` is tiny and kappa becomes extremely sensitive. A pair can agree on 95% of items and still score kappa near 0.2. That is not a bug -- it is telling you honestly that almost all of the agreement was about the easy majority and very little shared judgement was demonstrated on the cases that matter. The correct response is not to abandon kappa but to report it with the prevalence and the full agreement table, and often to measure agreement on an enriched sample where positives are over-represented. ## Which coefficient for which situation - **Cohen's kappa** -- exactly two annotators, nominal categories, both rating the same items. - **Fleiss' kappa** -- a fixed number of ratings per item, where the raters need not be the same people on every item. This is the crowd-annotation case. - **Weighted kappa** -- ordinal categories, where being one step apart should count as a smaller disagreement than being three steps apart. - **Krippendorff's alpha** -- the most general: any number of raters, missing ratings allowed, and a distance function chosen for nominal, ordinal or interval data. Using Cohen's kappa on three raters, or an unweighted coefficient on an ordinal scale, is a common and visible mistake. ## Using it in practice Agreement is a gate you run *before* you trust a labelled dataset, on a deliberate overlap set: route some fixed share of items -- often 5-10% -- to two or more annotators, and compute agreement only on that overlap. Do it early, on the first batch, while the guidelines can still be changed. Interpretation benchmarks such as 0.41-0.60 'moderate' and 0.61-0.80 'substantial' come from an old convention and are rules of thumb, not thresholds you can defend on their own; what matters is whether the disagreement rate is small relative to the effect you are trying to detect. ## When agreement comes back low The reflex is to blame annotators. Usually the guideline is at fault. Low kappa most often means the label definition is ambiguous at exactly the cases that occur most: the boundary between sarcasm and toxicity, between a complaint and a threat, between 'inactive' and 'churned'. The productive loop is: pull the disagreed items, adjudicate them with a senior annotator, turn each adjudication into a worked example in the guideline, retrain the annotators, and re-measure on a *fresh* overlap set. Re-measuring on the same items you just adjudicated only tells you the adjudication happened. The agreement number is also a planning input. A pair whose kappa is 0.9 barely benefits from a third rater; a pair at 0.4 tells you that single-rater labels are close to worthless and that either the guideline needs work or every item needs multiple ratings with a resolution rule.