How do you size and operate the human review queue behind a moderation classifier?

## What the queue is for A human review queue exists because classifiers are wrong in both directions and some decisions are too consequential to automate. It handles three distinct workloads, and conflating them is the first mistake: 1. **Enforcement decisions in the uncertain band** — items scoring between the escalate and auto-action cutoffs, where a human decides. 2. **Appeals** — users contesting an automated action. 3. **Measurement sampling** — items pulled deliberately, including ones the classifier allowed, to estimate true error rates. The third is the one teams skip, and skipping it is why so many moderation programmes cannot state their false-negative rate. ## Sizing The arithmetic is straightforward and rarely done. Expected escalations per day = daily items scored x flag rate at the chosen thresholds, summed over categories, plus the measurement sample, plus appeals. For a shooter's party chat that lands around 900 escalations a day. Divide by items-per-reviewer-hour (this varies hugely: a short chat line is seconds, a long conversation with context is minutes) to get reviewer hours, then add headroom. Headroom is not optional. Moderation load is spiky in ways ordinary traffic is not: a game update, a tournament, a coordinated brigading campaign, or a viral incident can multiply volume overnight. Staffing exactly to the mean guarantees a backlog during precisely the events where timely review matters most. Plan for burst capacity — overflow staffing, a temporary threshold raise on low-severity categories to shed load deliberately, or explicit shedding rules that state which categories get dropped first. Note the coupling: threshold choice and queue capacity are one joint decision. Tightening a threshold reduces escalations and raises misses; loosening it does the reverse. A threshold whose flag volume exceeds capacity is not a policy, it is a backlog. ## Priority tiers and response targets A single FIFO queue is a design error, because it makes a self-harm report wait behind a routine profanity call. Tier by the cost of delay: - **Urgent** — self-harm signals, credible threats, child-safety: a tight response target, on the order of 15 minutes, staffed around the clock. - **Standard** — targeted harassment, hate: hours. - **Routine** — low-severity, appeals on minor actions: same-day. Each tier needs its own target, its own staffing, and its own alerting when the target is breached. Report attainment per tier, never as a single blended number, because a blended figure hides exactly the misses that matter. ## Sampling the allowed traffic If reviewers only see flagged items, you can compute precision — of what we flagged, how much was really harmful — but you can never compute recall, because you never look at what you let through. The fix is a random sample of allowed traffic, stratified across the score range, sent to reviewers as if it were an escalation. It is pure cost with no enforcement yield, which is why it gets cut, and it is the only way to detect a classifier that has quietly stopped catching a whole class of abuse after a slang shift or a model change. ## Reviewer wellbeing as a capacity input Content moderation is psychologically demanding work, and fatigue shows up directly as decision quality. Practical measures: cap exposure time on the most distressing categories, rotate reviewers between category pools rather than specialising them into the worst one, provide tooling that truncates or blurs by default with opt-in reveal, and fund mental-health support. Treat throughput assumptions accordingly — a sustainable items-per-hour figure is lower than a peak-hour measurement, and planning against the peak produces both burnout and attrition, which is a capacity problem as much as an ethical one. Guideline quality is the other half. Ambiguous policy is the largest source of inconsistent decisions, so decision guidelines need worked examples and edge cases, and genuinely ambiguous items need an escalation path to a policy owner rather than each reviewer inventing an answer. ## Closing the loop The queue is not a sink; it is your best source of signal. Track per category: - **overturn rate** — how often reviewers disagree with the classifier. A climbing overturn rate on flagged items means the threshold is too loose or the category is mis-specified. - **hit rate in the allowed sample** — rising means recall is degrading. - **time-in-queue distribution per tier**, not just the mean. - **appeal overturn rate** — high means automated enforcement is firing too aggressively. Those numbers feed back into threshold moves, taxonomy splits, and the decision to retrain or replace the classifier. A programme where reviewer decisions never change a threshold has a queue but not a feedback loop. ## What interviewers listen for Capacity derived from arithmetic rather than guessed; tiers with per-tier targets; the insight that flagged-only review is blind to false negatives; burst planning with an explicit shedding rule; reviewer sustainability treated as a capacity constraint; and a named path from reviewer decisions back to threshold and taxonomy changes.