How do you put a percentile band on a classifier's accuracy using bootstrap replicates?

## The recipe You have a trained classifier and a set of evaluation rows it has never seen - say 500 documents with sentiment labels - and a point accuracy of 0.85. A single number invites over-reading, so put a band on it: 1. Draw 500 rows **with replacement** from the 500 evaluation rows. 2. Recompute accuracy using the predictions already made for those rows. 3. Repeat B times (2,000 is a comfortable default). 4. Sort the B accuracies. The 2.5th and 97.5th percentiles are the endpoints of a 95% **percentile band**. The model is fitted once and never refitted; you are resampling *scored rows*, which makes the whole thing cheap - no retraining, just re-averaging predictions that already exist. ## What the band means It answers: *if I had drawn a different 500 documents from the same population and scored this same fixed model on them, how much would the reported accuracy have wobbled?* That is evaluation-set sampling noise. On 500 rows at 0.85, the standard error of accuracy is about `sqrt(0.85 * 0.15 / 500) = 0.016`, so the band lands near [0.82, 0.88] - about three points either side. If your product decision turns on a two-point difference, the band tells you the evaluation set is too small to support it. Interpretation discipline matters. The band is a frequentist coverage statement about the procedure: bands built this way contain the fixed underlying accuracy about 95% of the time. It is not "a 95% probability that the true accuracy is inside this particular interval". ## What the band does NOT include **Training variability.** The model is frozen. If you retrained on a different sample, the model itself would change - possibly by more than the evaluation noise. To capture that you have to resample the *training* rows, refit the model on each replicate, and score it on the rows that replicate left out. That version measures training instability plus evaluation noise together, but it costs B full refits instead of B cheap averages, and it estimates the error of the *procedure* rather than of one specific trained model. Decide which question you are answering before you pick. **Selection effects.** If the same rows chose the model - picked the threshold, the features, the hyperparameters - then the point accuracy is already optimistic and a band around it is a band around an inflated number. Bands do not repair a contaminated estimate; they only quantify noise around whatever you measured. **Distribution shift.** Every replicate is drawn from your evaluation set, so the band describes noise under the assumption that production data looks like that set. It says nothing about next quarter's documents. ## More replicates do not narrow the band This is the most common misunderstanding. B controls **Monte Carlo** error - how precisely you have located the percentiles of the resampling distribution. Going from 500 to 20,000 replicates makes the endpoints reproducible to more decimal places; it does not change the underlying width, which is governed by the evaluation-set size `n` and the metric's variance. Want a tighter band? Score more rows. A useful rule of thumb: a few hundred replicates suffice for a central estimate, one to two thousand for stable 2.5%/97.5% endpoints, more if you are reading extreme tails. ## Practical wrinkles **Discreteness.** With few evaluation rows, accuracy can only take a handful of values, and the percentile band inherits that lumpiness - it may look artificially narrow or step-shaped. Report the sample size next to the band so the reader can judge. **Rare positives.** If the positive class is scarce, some resamples will contain very few positives, or none. Metrics like recall or AUC become unstable or undefined in those replicates. Either stratify the resample so each replicate keeps the class counts of the original evaluation set, or report explicitly how many replicates were degenerate. Silently dropping them biases the band. **Grouped rows.** If evaluation rows are not independent - several documents per author, repeated measurements per customer - resample the *groups*, not the rows. Row-level resampling on clustered data pretends you have more independent evidence than you do and yields a band that is too narrow. **Any metric works.** The same machinery applies to F1, precision at a fixed threshold, AUC, mean absolute error - anything computable from a set of scored rows. Recompute the metric inside each replicate rather than averaging per-row values, because most useful metrics are not simple row averages. ## How to report it Write it as "accuracy 0.85, 95% percentile band [0.82, 0.88] from 2,000 bootstrap resamples of 500 held-out documents". That single line tells a reader the estimate, the uncertainty, the source of the uncertainty and the sample size behind it - and it makes clear the band is about evaluation noise, not about model retraining.