How does label noise put a ceiling on the accuracy any model can reach?

## The arithmetic of the ceiling Let a fraction `e` of the recorded labels differ from the truth. Take the best possible model -- one that outputs the true label for every case. On every clean row it matches the record; on every corrupted row it disagrees. Its measured accuracy is therefore `1 - e`. That is the ceiling, and it is set by the data, not the algorithm. This is why the first question to ask when a stakeholder demands 99% accuracy is not about the model, it is: how often do two competent annotators labelling the same case produce the same answer? If two radiologists reading the same chest scan disagree on 8% of cases, no model reading those scans can be scored above roughly 92% against single-reader labels -- and a model that scores 96% is not better than the radiologists, it is agreeing with one reader's idiosyncrasies more often than chance. ## Two shapes of noise, two different harms **Symmetric noise** flips labels at the same rate in both directions. Its effect on the *optimal decision rule* is surprisingly mild: as long as the flip rate is below 50%, the class with the higher true probability still has the higher observed probability, so the ideal boundary is unchanged. What does change is the posterior -- the observed probability becomes `(1-e)*p + e*(1-p)`, which pulls every prediction toward 0.5. So calibration degrades even when the ranking survives, and with finite data the learner burns capacity trying to fit points it cannot possibly explain. **Class-conditional noise** flips at different rates per class -- for example, positives are frequently recorded as negatives because the annotator only marks what they are sure of. Now the harm is bias, not just variance. The observed base rate is wrong, the boundary shifts toward the under-reported class, and precision and recall measured against those labels are systematically off in a direction you can predict but not remove without cleaning. ## The evaluation set is the expensive half Most people's instinct is to clean the training set. That is the less important half. A model can tolerate a fair amount of training noise, especially with enough data and a loss that does not chase outliers. But a noisy *test* set corrupts the measurement, and every decision downstream -- which model ships, whether the retrain helped, whether an A/B result is real -- is made from that measurement. Two failure modes matter: - **Downward bias.** A genuinely better model is punished for disagreeing with wrong labels, so your reported number understates it. - **Rank inversion.** If the noise is class-conditional, a model that shares the annotators' bias scores higher than a model that does not. You ship the worse model and cannot see why production disagrees with the offline number. The standard remedy is a **gold set**: take a modest evaluation sample -- often a few thousand rows -- have it labelled independently by two or three of your best annotators, adjudicate every disagreement, and freeze it. Report headline metrics on the gold set and use the large noisy set only for training and for trend monitoring. ## Finding the suspect rows You can locate probable mislabels without any ground truth. Train a model with cross-validation and collect **out-of-fold** predictions, so every row is scored by a model that never saw it. Then rank rows by how confidently the model contradicts the recorded label -- high loss, or a confident probability on the other class. Methods in the confident-learning family formalise this by estimating the joint distribution between the recorded label and the likely true label from the confidently-predicted counts, which gives per-class flip-rate estimates rather than a single ranked list. A simpler neighbourhood check works too: flag a row whose nearest neighbours in feature space are overwhelmingly the other class. The crucial discipline is what you do with the list. **Do not auto-delete it.** Confidently-contradicted rows are a mixture of genuine mislabels, genuinely hard cases, and legitimate minority patterns the model has not learned yet. Deleting all of them removes exactly the tail you most need, and it flatters your metric because you have quietly deleted the hard test. Send the top of the list for re-annotation, measure what share were actually wrong, and let that rate tell you how much noise remains in the rest. ## Training under noise When re-labelling is not affordable for the whole set, the levers are: choose a loss that does not explode on a single contradicted point (absolute or Huber-style errors for regression rather than squared error, which weights a badly wrong target quadratically); stop before the model drives training error to zero, since the last fraction of fit is where memorising wrong labels happens; and downweight rather than delete the suspect rows so an incorrect suspicion is recoverable. ## What to say in the interview Name the ceiling explicitly, tie it to a measured annotator disagreement rate rather than a guess, insist that the evaluation labels are cleaned first, and be honest that above the ceiling the extra points are the model learning the annotators' errors.