When reporting a regression model's worst error slice, how do you know it is not noise?

## The maximum is not an average When you compute a metric over fifty slices and report the worst one, you are not reporting a measurement — you are reporting the maximum of fifty noisy estimates. Even if every slice has exactly the same true error, the largest of fifty sample estimates will sit above that common truth, and it will sit further above it the more slices you cut. This is the same winner's-curse effect that makes the best-looking arm of a many-armed experiment disappoint on replication. Worst-slice reporting therefore has a built-in pessimistic bias about the model, and the bias grows with your enthusiasm for slicing. ## Small slices are imprecise, in a way you can quantify An error metric on a slice is essentially an average over that slice's rows, so its sampling precision improves with the square root of the row count. Four times the rows halves the interval. That means the interval on a slice of thirty rows is roughly five times wider than on a slice of eight hundred, and a slice metric can differ substantially from the truth purely by which thirty rows landed in the split. When you then rank slices by that metric, the small slices systematically populate both ends of the ranking — the worst list and the best list alike. A worst-slice table sorted by point estimate is, in practice, largely a list of your smallest slices. ## The discipline **Pre-register the slice list.** Decide which dimensions you will always cut by — region, tenure band, product line, channel, prediction magnitude — before looking at any errors. A standing report is a measurement; a search for the cut that makes the model look broken is a story. **Set a minimum slice size, and derive it.** Do not pick a round number. Work backwards from the difference you would act on: choose the row count at which the interval on the slice metric is narrower than that difference. Slices below it are still computed, but reported rolled up, or shown with the interval attached and explicitly marked as indicative. **Attach uncertainty to every slice number.** Resample the slice's own rows with replacement, recompute the metric a few hundred times, and take the spread as the interval. Then compare intervals rather than points; a slice whose interval overlaps the overall error is not yet a finding. **Require replication.** The cheapest and most convincing check is whether the slice looks bad again on a different period, a different split, or the next evaluation cycle. A slice that is genuinely broken repeats; a slice selected by noise usually does not. **Scale the bar with the number of slices.** If you cut fifty ways, demand a bigger gap than if you cut five. You do not need a formal correction to act on this instinct, but you do need to be conscious that each additional cut buys another chance at a false alarm. ## Hard slice or broken model A slice can be honestly, reproducibly worse and still not be a model defect. Some cohorts have intrinsically noisier targets: sellers in their first month have no history to learn from and behave erratically, and no model can predict them as well as it predicts a five-year seller. The test is to compute the slice's own trivial baseline — the best constant prediction using only that slice's rows — and compare the model against it inside the slice. - Model error triple the average, slice baseline also triple: the target is harder there, the model's lift is normal, and the right output is an honest accuracy expectation for that cohort rather than an engineering ticket. - Model error triple the average, slice baseline normal: the model really is failing on that cohort, and that is a defect worth investigating — thin training data, a missing feature, a different regime. This single comparison converts most alarming slice tables into a much shorter list of real problems. ## The report format that survives scrutiny One row per slice with: the slice definition, the row count, the metric, its interval, the slice's own baseline, and the lift over that baseline. Sort by lift shortfall rather than by raw error, and mark which slices met the minimum size. Anyone reading it can then see immediately whether the worst number is a real failure, a hard cohort, or three dozen rows. ## What interviewers listen for That you recognise the maximum-of-many-estimates problem without prompting; that you quantify precision rather than eyeballing it; that you insist on replication before acting; and that you separate the model failing from the world being noisy. Candidates who chase every worst slice burn the team's time on sampling artefacts, and candidates who ignore slice tables entirely miss the real failures inside them.