A fraud model shows 0.97 ROC-AUC but poor live precision — which sklearn.metrics calls do you reach for?

## Why the two numbers disagree The report is not contradictory. `roc_auc_score` integrates true-positive rate against false-positive rate over all thresholds. The false-positive rate has the negative count in its denominator, so when negatives outnumber positives a thousand to one, thousands of false alarms barely move it. Precision has the *predicted-positive* count in its denominator, so those same false alarms dominate it. A model can rank almost perfectly and still hand a review queue mostly noise, and no operating threshold is even implied by an AUC figure. In `sklearn.metrics` terms: you asked a threshold-free ranking question and got a threshold-free answer. ## The summary to switch to `average_precision_score(y_true, y_score)` summarises the precision-recall curve instead. It takes the same continuous score array — `predict_proba(X)[:, 1]` or `decision_function(X)` — and returns the precision-weighted mean over the recall steps. Its baseline is the positive-class prevalence rather than the fixed 0.5 of ROC-AUC, so it moves when the class balance moves, which is exactly the sensitivity you wanted. A specific caution: do not compute this as `auc(recall, precision)`. The generic `auc` helper applies trapezoidal integration, which linearly interpolates between adjacent curve points. The PR curve advances in discrete steps as samples cross the threshold, and interpolating across those steps is optimistic — scikit-learn's own documentation flags this. `average_precision_score` uses the step-wise summation and is the one to quote. ## Reading precision_recall_curve correctly `precision_recall_curve(y_true, y_score)` returns three arrays: `precision`, `recall`, `thresholds`. The first two have length `n + 1`; `thresholds` has length `n`. The extra point is the degenerate endpoint at recall 0 and precision 1, which corresponds to predicting nothing positive and therefore has no threshold behind it. Any code that zips the three arrays together, or indexes `thresholds` with an index taken from `precision`, is off by one at the tail. The correct alignment is `precision[:-1]` and `recall[:-1]` against `thresholds`. This off-by-one is one of the most reliable senior-level details on this whole topic, precisely because it produces a plausible wrong threshold rather than an exception. `roc_curve` returns `fpr, tpr, thresholds`, all the same length, with a synthetic first threshold above the maximum score — a different convention, so do not carry the habit across. ## Choosing and then measuring at the operating point Once you have the curve, threshold selection is a constrained pick: the largest recall whose precision clears the floor your downstream process can absorb, or the highest precision that still catches enough cases, depending on which side is scarce. Then — and this is the step most often skipped — you must *re-measure at that threshold*. `clf.predict(X)` does not use your threshold; for probabilistic classifiers it takes the argmax of `predict_proba`, which in binary is the fixed 0.5 cut, and for margin classifiers the sign of `decision_function`, which is 0. So every label metric computed from `predict()` describes a threshold you did not choose. The re-measurement is `confusion_matrix(y_true, (scores >= t).astype(int))` and `classification_report(y_true, (scores >= t).astype(int))`. The confusion matrix at the chosen threshold is what actually translates into queue volume: false positives are reviewer hours, false negatives are losses. `precision_score` and `recall_score` on those hard labels then match what operations will see. ## Reporting honestly A good evaluation of this system therefore carries three layers: a ranking summary (`roc_auc_score`) that is comparable across datasets with different prevalence, a prevalence-sensitive summary (`average_precision_score`) that reflects this stream, and the confusion matrix at the deployed threshold that converts to headcount and money. Quoting only the first is what produced the original surprise. `PrecisionRecallDisplay.from_predictions` is a convenient way to put the curve in front of stakeholders alongside those numbers. ## The evaluation-data caveat One more thing to check before blaming the metric: the threshold must be chosen on data whose class balance matches production. A threshold tuned on a downsampled or class-balanced validation set will not deliver the same precision on a live stream where positives are a hundred times rarer, because precision depends on prevalence while the score distribution does not shift with it. Tune the threshold on a held-out sample with realistic prevalence, and re-check it as prevalence drifts.