Per-class AP on a 12-instance defect class swings 8 points on one missed box - how do you report it?

## Why the number is coarse Average precision summarises a precision-recall curve, and recall has denominator equal to the number of ground-truth instances of the class. With 12 instances, recall is confined to the 13 values 0, 1/12, 2/12, ... 1. There is no such thing as a small change in recall for this class - the smallest possible move is 8.3 points. Precision at each of those points is estimated from a similarly small number of detections. A single borderline box that slips below the IoU threshold, or a single annotation the labeller drew a little differently, moves the class's AP by several points. The key framing for an interview: **this is sampling noise, not model variance.** Retraining with a different seed will move it. So will re-splitting the data. So will a slightly different annotation of one image. Reporting a point estimate to two decimal places implies a precision the evaluation set cannot deliver. ## How it contaminates the headline mAP is an unweighted mean over classes. That choice is deliberate - it stops a few common classes from hiding total failure on the rest - but it has a direct consequence here: a class with 12 instances carries exactly the same weight as one with 5,000. If a change flips one box on the rare class, the mean moves by roughly 8/N points for N classes. On a 20-class problem that is a 0.4-point swing in the headline metric from a single box, which is the same order as the improvements teams routinely celebrate. So when someone reports that mAP went up 0.4 after a change, the first question is: did the common classes move at all, or is this one rare class's coin flip? ## What to actually do **Publish the breakdown, always.** Per-class AP next to that class's instance count in the evaluation split, in the same table. A reader who sees `n = 12` calibrates instantly; a reader who sees only the mean cannot. **Attach uncertainty.** Bootstrap over *images* (not over detections - detections within an image are not independent), recomputing the whole matching and AP on each resample, and report an interval. It will be embarrassingly wide for the rare class, and that is the point: the interval is the argument you will use to stop a bad decision later. **Set a decision rule before the experiment.** Define, in advance, the minimum number of instances a class needs before its AP is allowed to gate a release, and the effect size that counts as real for the classes above that bar. Use paired comparisons - evaluate both models on the same images and compare per-image differences - which removes most of the split-to-split variance and is far more sensitive than comparing two independent point estimates. **Fix the measurement, not just the report.** If this defect class matters to the business, 12 instances is not an evaluation set; it is an anecdote. Options in rough order of cost: mine and annotate more instances specifically for evaluation, even if training data stays as it is; collect a targeted slice from production; or, where instances are irreducibly scarce, change what you measure - report a fixed-operating-point recall with an exact confidence interval, or count misses directly and set an absolute limit, which is easier to reason about with tiny counts than an area under a curve. **Consider whether the mean is the right contract at all.** For a defect-detection product, the rare class is usually the expensive one, and hiding it inside an unweighted average across classes is a governance failure regardless of the statistics. A common resolution is a two-part contract: a headline metric for tracking overall progress, plus explicit per-class floors on the classes that carry business risk, each with its own evaluation slice sized so that the floor is actually measurable. ## The trap to name out loud Averaging AP over ten IoU thresholds does *not* fix this. The ten evaluations re-use the same 12 objects, so they are strongly correlated; averaging smooths the curve a little but adds no independent evidence. Sample size is the binding constraint, and only more annotated instances relax it. Similarly, adding more classes to the benchmark dilutes each class's contribution to the mean but does nothing for the rare class's own estimate - it just hides the noise better, which is worse.