What does Hamming loss measure for a news tagger that assigns several topics per article?

## The multilabel setting Single-label multiclass forces one class per sample. Multilabel does not: a news article may be tagged `politics` and `europe` and `energy` at once, and the model outputs an independent yes/no for every topic in the vocabulary. The evaluation question changes shape — a prediction can now be *partly* right, and metrics must decide how to price that. Represent the truth as an N x L table of 0/1 cells (N articles, L candidate topics) and the prediction as another table of the same shape. Every metric in this setting is a way of comparing those two tables. ## Hamming loss ``` Hamming loss = (count of cells where prediction differs from truth) / (N * L) ``` It is the per-cell error rate, and it treats a **false positive** (a tag the model invented) and a **false negative** (a tag it missed) identically — each is one mismatched cell. It runs from 0 (perfect) to 1 (every cell flipped), and lower is better, which is the opposite direction from almost every other metric on a scorecard; mixing it into a table of accuracy-style numbers without a clear label invites misreading. Example: 4 articles, 5 candidate topics, so 20 cells. If the model misses two true tags and invents one, three cells disagree and Hamming loss is 3/20 = 0.15. ## Why partial credit matters The alternative extreme is **subset accuracy** (also called the exact-match ratio): the share of articles whose predicted tag set matches the true set exactly, with no credit for four out of five. Subset accuracy is brutally strict as L grows — with five true tags and a large vocabulary, getting all five and no extras is rare, so the metric can sit near zero across models that differ enormously in usefulness, giving you no gradient to reason with. Hamming loss and subset accuracy bracket the sensible range. Hamming loss says "how many individual decisions did you get wrong"; subset accuracy says "how often was the whole answer right". A reasonable review shows both: a low Hamming loss with a near-zero subset accuracy means the model is roughly right everywhere and exactly right nowhere, which for a tag-suggestion feature may be entirely acceptable and for an automated routing rule may not be. ## The sparsity trap Multilabel vocabularies are usually large and true tag sets small. With L = 200 topics and an average of 3 true tags per article, 98.5% of all cells are legitimately 0. The all-negative tagger — predicts nothing, ever — therefore achieves a Hamming loss of about 0.015 and would top a leaderboard ranked on that number alone. This is the single thing to say out loud when the metric comes up. The defences are straightforward: - **Report label-wise F1 alongside it.** Micro-F1 over label decisions pools TP, FP and FN across all cells and ignores the true negatives that dominate the table, so the empty tagger scores zero. Macro-F1 over the L topics averages per-topic F1 equally and additionally exposes topics the model never predicts. - **Report the empty-prediction baseline explicitly.** Any Hamming loss must be read against "what would predicting nothing score", the way any score should be read against a trivial baseline. - **Track label cardinality** — the average number of tags predicted per article against the average number of true tags. A model predicting 0.4 tags where the truth averages 3.1 is under-tagging, and no single aggregate will say so as clearly. ## Micro-F1 is not accuracy here A reflex carried over from single-label multiclass causes real errors: there, micro-averaged F1 equals accuracy because every sample produces exactly one prediction, so each mistake creates one false positive and one false negative and the pooled totals match. In multilabel that symmetry is gone — the model may output four tags where two are true, so pooled FP and FN differ, micro-precision and micro-recall differ, and micro-F1 is a genuine F1 rather than a disguised accuracy. Likewise `1 - Hamming loss` is a per-cell accuracy, not subset accuracy, and the two can be miles apart. ## Asymmetric costs Hamming loss prices a missed tag and an invented tag the same. That is often wrong for the product: on a news site an invented `obituary` tag is embarrassing while a missed `europe` tag merely reduces reach. When the two cost differently, Hamming loss is the wrong headline — separate the per-cell false-positive and false-negative rates, or move to an F-beta over label decisions that encodes the ratio you actually care about. Hamming loss is best understood as a fast, symmetric, partially-crediting sanity number: useful to watch, unwise to optimise alone.