Why do content moderation classifiers return per-category scores, not one flag?

## What a harm taxonomy is A harm taxonomy is the enumerated set of categories your moderation system recognises: hate, harassment, sexual content, violence, self-harm, and so on, often with subcategories. A classifier scores a piece of text against each category independently and returns a score per category, not a single verdict. The taxonomy is the contract between the classifier and the product: it defines what you can even talk about detecting. ## Why one flag is not enough A single `unsafe: true` flag forces three things you almost never want: 1. **One action for every harm.** Blocking is right for some categories and actively wrong for others. Suppressing a message that scores high on self-harm hides a user who may need help. 2. **One threshold for every harm.** Categories have wildly different base rates and wildly different error costs. A cutoff calibrated for mild profanity is far too loose for credible threats of violence. 3. **One review path.** Operationally you need to know *why* something was flagged in order to route it — to an automated mute, to a general review queue, or to an urgent queue with a tight response time. A boolean throws away all the information needed to make those decisions. ## Designing categories: the response test The useful design rule is that **a category earns its existence when it maps to a distinct response**. Not when it feels conceptually different, and not because a vendor's default list happens to contain it. Apply the test: if category A and category B always produce the same action at the same threshold, merge them. If one broad category is producing two different actions depending on what a reviewer sees inside it, split it. ## A worked example: party chat in a competitive shooter Consider voice-to-text party chat in a competitive online shooter. A flat "harassment" category performs badly here, because the same words carry opposite meanings. "You're trash, get off my team" between friends who queue together every night is banter. The identical string aimed at a random matchmade stranger is abuse. And "this map is trash, whoever designed it should be fired" is neither — it is trash talk about the match. Splitting harassment into **targeted-at-a-player** and **trash-talk-about-the-match** passes the response test, because the responses genuinely differ: the first can lead to a chat mute or a report escalation, while the second is left alone at any score. That split also lets you set the targeted subcategory's threshold low (catch more, tolerate some false positives) and effectively disable enforcement on the other, which a merged category cannot express. Note what the split does *not* solve: relationship context. The classifier sees text, not the fact that these two players are friends. Systems handle that with signals outside the classifier — party membership, prior reports, whether the recipient blocked the sender — combined with the score. Naming that limitation is what separates a thoughtful answer from a naive one. ## Per-category thresholds Once you have categories, each gets its own cutoff. Rough shape in practice: - **Categories where a miss is catastrophic** (child safety, credible threats): low threshold, accept false positives, human review on everything flagged. - **Categories where a false positive damages the product** (mild profanity, edgy humour in an adult context): high threshold, or no automated enforcement at all. - **Categories that need routing rather than blocking** (self-harm): moderate threshold, action is escalation and resource surfacing, not suppression. ## Where taxonomies go wrong - **Categories that never fire.** Dead categories add classifier cost and reviewer confusion. Measure per-category volume and prune. - **Overlapping categories with no precedence rule.** One message scoring high on three categories needs a defined winner, or your action is nondeterministic. - **Adopting a default taxonomy unchanged.** Off-the-shelf category lists are a starting point; the categories that matter to a shooter's chat, a health forum, and a legal-research tool are not the same set. - **Assuming categories are mutually exclusive.** They are not — scores are independent and a message can be high on several at once. ## What interviewers listen for They want the response test articulated, at least one concrete example where two harms demand different actions, and awareness that thresholds are per category rather than global. A candidate who says "we call the moderation check and block if it says unsafe" has described a system that will both over-block and under-protect.