A precision-recall curve built on only 60 positives is jagged. How much do you trust its average precision?

## Where the noise comes from A precision-recall curve is built by walking down the ranked scores. Recall only changes when a **positive** is admitted, and it changes by exactly `1 / (number of positives)`. With 60 positives, recall moves in jumps of about 0.017 and the curve has at most 60 vertical resolution points, no matter whether the test set holds 6,000 rows or 6,000,000. The row count buys precision resolution; the positive count buys the curve. On top of the coarse grid, the left end is intrinsically unstable. At recall 0.05, three positives have been caught; precision there is `3 / (number flagged)`, and one negative slipping above one positive in the ranking changes it by a large fraction. Small denominators produce large relative swings. So the region of the curve people care most about — the high-precision top of the ranking — is precisely the region estimated from the fewest observations. ## Average precision is an estimate, not a measurement It helps to name the thing properly. There is a true average precision for this model on the population it will run against; what you computed is an **estimate** of it from one finite sample. Like any estimate it has a standard error, and here that standard error is governed mainly by the 60 positives. The consequence is blunt: quoting `AP = 0.412` implies a precision the data cannot support, and a leaderboard ranking two models 0.412 against 0.389 is very likely ranking noise. A second, subtler effect: with few positives the estimate is not merely noisy but skewed by which positives happen to be easy. If two of your 60 positives are near-duplicates of training rows and both land in the top ten, the left end of the curve — and therefore a disproportionate share of the AP sum — is carried by two rows. ## Quantifying the uncertainty The standard move is resampling. Draw bootstrap resamples of the test rows with replacement — stratified by label so each resample keeps a comparable positive count — recompute AP on each, and read the 2.5th and 97.5th percentiles of the resulting values as a 95% interval. With 60 positives that interval is routinely wide, often plus or minus 0.05 or more, and seeing it is the point: it converts an argument about decimals into an argument about overlap. For comparing two models, resample the **difference**, not the two numbers independently. Both models are scored on the same resampled rows in each iteration, which removes the shared difficulty of the sample and gives a much tighter interval on the delta than comparing two separately-computed intervals. If the interval on the difference straddles zero, you have not shown an improvement. A cheaper sanity check costs nothing: recompute AP with the single highest-scored true positive removed. If the number collapses, the result rests on one row. ## Getting more signal Uncertainty this large is usually a data problem, and the fixes are about positives: - **Extend the evaluation window.** Six months of history instead of one multiplies the positive count directly. Weigh this against drift: an older window may no longer describe the population you will run on. - **Pool repeated cross-validation.** Rather than one held-out split, run repeated stratified splits and either pool all out-of-fold scores into one curve or average AP across folds and report the spread across repeats. Pooling gives one curve built on every positive in the dataset. - **Do not add negatives.** Growing the test set with more negatives makes the sample bigger and the curve no smoother, because recall resolution never improves. It will, however, lower AP if the extra negatives change the prevalence — which is a reason to keep the evaluation set's base rate matched to production. - **Keep the test set fixed** across model iterations so comparisons stay paired, and accept that a fixed set gradually gets overfit by repeated selection against it. ## What to actually report A defensible report for a rare-positive detector is: average precision with a resampling interval, the number of positives and the positive rate of the evaluation set, and the curve itself drawn raw rather than smoothed. Smoothing a 60-positive curve into an elegant line communicates a confidence that does not exist. And when a stakeholder asks whether the new model beat the old one by 0.02, the correct senior answer is that the test set cannot tell — followed by a plan to get more positives.