How do you put a confidence interval on the AUC difference between two models?

## Why this metric needs its own treatment For accuracy or log-loss, each test example contributes one number to each model, so the comparison collapses to per-example differences and the paired machinery is straightforward. AUC does not decompose that way: it is computed from comparisons between positive and negative examples, so a single example participates in many comparisons and there is no well-defined "AUC of example `i`". That rules out the per-example approaches and forces you to resample at the level of examples while recomputing the whole metric. ## The paired bootstrap, step by step 1. Fix the held-out set of `n` examples, with both models' scores already computed for each. 2. Draw `n` examples with replacement to form one resample. 3. Compute `AUC_A` and `AUC_B` **on that same resample** and record `delta = AUC_A - AUC_B`. 4. Repeat for `B` resamples - two thousand is a reasonable default for an interval. 5. Report the 2.5th and 97.5th percentiles of the recorded `delta` values as a 95% interval for the difference. The load-bearing detail is step 3. Because both models are evaluated on the identical resample, every draw carries the same examples for both, and the shared difficulty cancels inside each `delta`. If you instead drew a separate resample per model, the two AUCs would vary independently, the interval on the difference would be much wider, and you would have thrown away the whole benefit of a shared test set. ## Stratifying by class AUC depends on having both positives and negatives present. When positives are rare - a few dozen in a large test set - an unstratified resample can contain very few, making the recomputed AUC unstable or, in the extreme, undefined. Resampling positives and negatives separately, keeping each class count fixed at its observed value, avoids that and is standard practice for imbalanced evaluation sets. Mention this and you signal that you have run the procedure rather than read about it. ## DeLong's method The analytic alternative estimates the variance of each AUC and, crucially, the **covariance** between two AUCs computed on the same data, from which a variance for the difference follows: ``` Var(AUC_A - AUC_B) = Var(AUC_A) + Var(AUC_B) - 2 * Cov(AUC_A, AUC_B) ``` A normal-based interval on the difference then follows immediately. Its appeal is that it is deterministic and cheap - no resampling loop, no Monte Carlo noise - and its covariance term is exactly the paired structure that the naive treatment of two independent AUCs would drop. Its cost is that it leans on a large-sample normal approximation, which is shakier when the test set is small or one class is very thin. The two approaches usually agree closely on reasonably sized sets; the bootstrap is the safer default when they do not. ## Do not compare two individual intervals A common error is to draw an interval around each model's AUC and check whether they overlap. Overlap is neither necessary nor sufficient for the difference to be significant, and it is especially misleading here because the two AUCs are strongly positively correlated - the interval on the *difference* is typically far narrower than the individual intervals suggest. Build the interval on the gap and see whether it excludes zero. ## What the interval covers, and what it does not A bootstrap over test examples quantifies one source of randomness: which examples ended up in the test set. It does not cover training-seed or data-order variance - retrain either model with a different seed and its AUC will move by an amount this interval knows nothing about. It does not cover the effect of having chosen either model by looking at this same test set. And it does not cover any difference between the test distribution and the one the model will meet in the future. When an offline gap is small, seed variance is frequently the larger of the two effects, and a defensible comparison retrains each model a handful of times and reports that spread alongside the interval. ## Reporting Write the interval on the gap, not two numbers with a winner circled: "model A's AUC is 0.008 higher, 95% interval 0.001 to 0.015, from a paired bootstrap over 2,000 resamples of the test examples, stratified by class." That sentence states the effect, its uncertainty, the procedure and the scope, which is exactly what a reviewer needs in order to disagree with you productively.