Your booster's training loss reaches zero on 5%-mislabelled data while validation loss rises after round 300 — why?

## Why boosting is specifically vulnerable here Bagging-style averaging fits many models to the same target independently; a mislabelled row contributes one wrong vote and gets outvoted. Boosting does the opposite by design. Each round computes what the current ensemble still gets wrong — the pseudo-residuals — and fits the next tree to exactly that. A row whose label is wrong is a row the ensemble can never explain by learning real structure, so its residual stays large round after round and it keeps presenting itself as the most valuable thing left to fix. The algorithm has no way to distinguish 'hard but real' from 'impossible because the label is wrong'. Both look like a large residual. That is the whole mechanism. Boosting's greatest strength — relentlessly attacking remaining error, which is why it reduces bias where averaging reduces variance — becomes its failure mode when a slice of the remaining error is not error at all but corruption. ## Reading the two curves The pattern in the question is the classic signature: - **Training loss to zero.** Given enough rounds, an additive ensemble of shallow trees can isolate essentially any individual training point: successive trees keep splitting until a mislabelled row sits in a region of its own with the prediction it was told to produce. Zero training loss means the model has memorised its way to the corrupt labels, and it says nothing about generalisation. - **Validation loss turning up after round 300.** Round 300 is roughly where the marginal tree stops finding structure that transfers and starts fitting the corruption. Before it, each round improves both curves; after it, they diverge. That inflection point is the model. The corollary matters more than the diagnosis: **the training curve carries no stopping information**. In a setting with 5% noise it will descend to zero whatever you do. Anyone tuning against it will train forever and ship the worst available model. ## The floor you cannot cross With 5% of labels wrong, there is a lower bound on validation loss that no model can beat: even a perfect predictor of the true relationship disagrees with the corrupted labels on those rows. A team chasing validation loss below that floor will keep adding capacity and keep making the model worse. Recognising that the curve has bottomed out at the noise floor — rather than at some fixable modelling shortfall — is the senior judgment in this scenario, and it is the point at which the effort should move off the model. ## What actually helps 1. **Stop at the validation minimum.** This is the direct fix and it is nearly free: the round count already tells you where the noise-fitting begins. Ship 300-ish rounds, not 3,000. 2. **Lower the learning rate.** At a coarse rate the ensemble crosses the minimum in a few big jumps and you cannot tell where it was; at a small rate the curve is smooth and the turning point is well resolved, so the stopping decision is more accurate. The rate does not make the model immune to noise, but it makes the optimum findable. 3. **Subsample the rows.** If each round fits on 70-80% of the rows, no mislabelled row is present in every round's target, so the ensemble's pursuit of it is intermittent rather than relentless. This measurably softens the turn-up. 4. **Choose a less aggressive loss for regression.** Squared error penalises a residual by its square, so one grossly wrong target dominates the round's fit. Huber loss is quadratic near zero and linear beyond a threshold, and absolute-error loss is linear throughout; both cap how much a single outrageous residual can pull the tree. For classification, log loss is the standard choice and is already gentler than losses that grow exponentially in the margin. 5. **Look at what the last trees are fitting.** The rows with the largest persistent residuals at the end of training are, disproportionately, the corrupt ones. That list is often the most useful artefact of the whole exercise. ## What does not help - **More rounds.** The noise does not average out; boosting is not averaging. Each additional round after the turn makes it worse. - **A larger learning rate 'to get there faster'.** Fewer, coarser steps make the minimum harder to locate, not easier to avoid. - **Judging progress on the training curve.** It will be zero, and it will be zero for a bad model. ## Answering the question in an interview Name the mechanism first — sequential residual-chasing means a permanently-wrong row attracts unbounded attention — then the two curves and what they mean, then the fact that the training curve is not a signal, then the mitigations in order of directness. Finishing with the noise-floor observation is what separates a diagnosis from a plan.