Two candidate models' ROC curves cross - how do you decide which one to ship?

## What a crossing actually tells you One ROC curve **dominates** another if it lies weakly above it at every false positive rate. Under dominance the choice is easy: the dominating model is at least as good at every possible threshold, and its AUC is necessarily higher. A **crossing** is the explicit statement that dominance does not hold. Model A is above model B up to some false positive rate; past that point model B is above A. Each model is the better model somewhere. This is common in practice, and it usually has a mechanical cause. A model with a very confident, sparse high-score head - a boosted tree ensemble that isolates a small set of near-certain positives - can win decisively at the strict end while flattening out later. A smoother model - a regularised linear score over broad features - can lose the strict end and win the middle, because it never assigns extreme scores it cannot back up. ## Why the AUC comparison is the wrong instrument here AUC integrates true positive rate over false positive rate with **uniform weight**. That means it treats a gain at `FPR = 0.9` as worth exactly as much as the same-sized gain at `FPR = 0.001`. For a fraud queue, a moderation pipeline, or a credit decision, false positive rates above a few percent are unreachable - the alert volume or the customer harm is unacceptable long before you get there. So the total area is partly a summary of behaviour in a region the model will never see. When curves cross, this stops being a philosophical objection and becomes a live risk: the model with the higher AUC can be the one that is worse at every operating point you could ever choose, because it banked its area in the far right of the plot. A single scalar cannot express "better here, worse there", and a crossing is precisely the case where that distinction is the whole decision. ## How to decide instead **1. Fix the operating region before comparing.** Establish, from the product side, the band of false positive rates that is reachable at all. This is not the same as picking the exact threshold - it is bounding the plot. Everything outside that band is irrelevant evidence. **2. Compare inside the region.** Two standard readouts: - **TPR at a fixed FPR** - "at a 1% false positive rate, model A catches 62% of positives and model B catches 54%". This is the most legible comparison to non-specialists and it is exactly a point-wise read of the curve. - **Partial AUC** - the area restricted to a false-positive-rate interval, typically rescaled to run from 0 to 1 over that interval. It preserves AUC's summary character while discarding the unreachable region. The cost is variance: fewer negatives fall inside a narrow low-FPR window, so the estimate is noisier than full AUC and needs a bigger evaluation set to be trusted. **3. Check that the crossing is real.** On a finite evaluation set two curves will jitter around each other; a crossing near the middle of the plot with only a handful of examples separating them may be pure sampling noise. Whether an observed gap between models is statistically meaningful is its own discipline - the point here is that a visible crossing is not automatically a structural fact about the models, and a resampled comparison should be run before a shipping decision rests on it. **4. Check stability, not just the average curve.** Recompute the region-restricted comparison per segment - by geography, device, tenure, traffic source - and across several time windows. A model that wins the operating region on average but loses it on your largest segment, or that changes places from month to month, is a worse ship than the average curve suggests. **5. Consider not choosing.** Crossing curves are a hint that the two models make different errors. Averaging their scores, or averaging their ranks - which is the natural combiner when only ordering matters - frequently produces a curve that beats both, precisely because the disagreement carries information. Validate the combination on held-out data rather than assuming it. **6. Weigh the non-curve factors.** Latency, retraining cost, feature dependencies, explainability obligations and operational familiarity are real inputs. When two models are close inside the operating region, these decide - and pretending the curve settled it is a way of hiding the actual reason. ## The judgment being tested The question is not really about ROC geometry. It is about whether you will let a convenient scalar make a decision that the scalar cannot see. The expected answer refuses the framing of "which AUC is higher", replaces it with "which model is better in the region we can operate in, and is that difference stable and real", and is explicit that the conclusion must be written down - including the fact that the headline AUC pointed the other way, so that the next person to read the dashboard does not silently reverse the call.