How do you choose decision thresholds for a 50-tag multi-label audio tagging head?

## Why a global 0.5 is arbitrary In a multi-label head each output unit produces a logit `z_i` and a probability `p_i = 1 / (1 + exp(-z_i))`. The cutoff `p_i > 0.5` is exactly `z_i > 0` — it is a property of where the sigmoid crosses its midpoint, not a statement about the tag. Two things make it a bad default in practice. **Base rates differ enormously.** In a 50-tag music-audio tagger, 'acoustic guitar' might be positive in 30% of clips and 'live recording' in 2%. Training with a per-label objective on sparse positives pushes rare tags' biases negative, so their whole score distribution sits low. A rare tag can be perfectly rank-ordered — every true positive above every true negative — and still never cross 0.5. **Error costs differ.** A wrongly applied 'explicit content' tag may be far more damaging than a missed 'reverb-heavy' tag. The threshold is the only knob that trades precision for recall after training, and it should be set per tag by the cost the product pays for each error type. ## The tuning procedure 1. **Hold out a split that thresholds never train on.** Ideally a third split, distinct from the one used for model selection, because thresholds are themselves fitted parameters. 2. **Score it once.** You need the raw per-tag probabilities for every held-out example; the sweep is then pure post-processing and costs nothing per additional candidate threshold. 3. **Sweep each tag independently.** Because the units are independent, the tags do not interact: sweeping candidate cutoffs for tag i changes only tag i's predictions. Evaluate the chosen metric at each candidate and keep the argmax. 4. **Choose the metric deliberately.** Per-tag F1 is the common default. If the product has a hard constraint — 'no more than one false tag in twenty' — tune to the highest recall subject to precision at or above that floor. Optimising micro-averaged F1 across all tags instead produces a different answer, one dominated by the frequent tags; say which you chose and why. 5. **Freeze and version the thresholds with the model.** They are part of the deployed artefact. A model checkpoint without its thresholds is not a deployable system. ## The rare-tag trap The procedure above overfits badly where it has least data. With 12 positive validation clips for a tag, the F1-optimal cutoff is chosen from a step function with 12 rises in it; the winning point is often a knife-edge that generalises poorly and swings wildly between retrains. Defences: - **Require a minimum positive count** (say a few dozen) before trusting a per-tag threshold; below it, fall back to a global default or to a threshold shared across a group of similar tags. - **Prefer smooth choices.** Instead of the single best point, take the centre of the plateau where the metric is near-optimal, which is far more stable than the peak. - **Sanity-check the implied volume.** If a tag's tuned cutoff would fire on 40% of the catalogue when its true base rate is 2%, the threshold is fitting noise regardless of what the validation metric says. ## Alternatives and complements **Top-k instead of thresholds.** For a UI that shows a fixed number of tags, ranking and taking the top k sidesteps thresholds entirely — but it forces exactly k tags onto every clip, including clips that deserve none. A hybrid works well: rank, then require the score to also clear a floor. **Calibration first.** If the scores are pushed towards the extremes, a monotone recalibration per tag (fitted on held-out data) makes the numbers readable as probabilities, which matters if downstream logic consumes probabilities rather than the binary decision. Calibration does not change the ranking, so it does not by itself change achievable precision/recall — it changes where the useful cutoffs sit and makes them interpretable. ## Operating them over time Thresholds are the part of the system most likely to silently rot. Re-tune after **every** retrain: two checkpoints with identical ranking quality can have quite different score distributions, and reusing the old cutoffs can halve a tag's recall overnight. Also watch for input drift — a shift in what gets uploaded moves score distributions without any model change. Monitoring per-tag firing rates against their expected base rates catches both failures faster than watching aggregate metrics, because one tag collapsing is invisible in a micro-average over 50 tags.