Evaluation Metrics

No. Conformal coverage rests on exchangeability between the calibration rows and the rows you predict on. A pricing change makes post-change traffic a different population, so the stored quantile is stale and realised coverage can drift below 90%.

On its own the number means nothing. Compare it with the constant forecaster that predicts 0.12 on every row, which scores about 0.367. A log-loss of 0.31 is therefore roughly a 15% improvement on that reference: real, but modest.

Balancing the training data shifted the log-odds by a constant. Undo it: multiply the predicted odds by the ratio of true prior odds to training prior odds. A 0.5 score becomes 0.04 at a 4% base rate.

Almost nothing about production. Accuracy depends on the class mix it was measured on, so a 50/50 test set reports balanced accuracy, not the live figure at a 0.3% defect rate. Recombine the per-class rates at the true base rate.

Cohen's kappa subtracts the agreement expected by chance from the observed agreement and rescales the remainder, so a model that merely mimics the grade distribution scores near zero even when raw agreement looks high.

True positive rate is computed only among positives and false positive rate only among negatives, so randomly discarding negatives leaves both rates unchanged in expectation. ROC-AUC measures ranking quality and is therefore blind to the class mix.

Flagged volume rises sharply. If only the base rate moved, precision at the fixed cut rises while recall holds, so the queue floods rather than degrades. If the traffic itself changed shape, false positives rise and precision can fall.

Picking the cut that first clears 85% in a sweep selects the point where sampling noise flattered you, and precision at a strict cut rests on few items. The estimate is optimistic; aim above the target on untouched data.

R-squared divides the model's squared error by the target's spread on the rows being scored. Narrow that spread and the same absolute errors give a much lower score, so the number is not comparable across splits.

Minimising absolute error drives predictions toward the conditional median, and on a right-skewed target the median sits below the mean. Median-like predictions therefore sum low, even though each individual row looks accurate.

Hamming loss is the fraction of individual label decisions that are wrong across all articles and all candidate topics. Each missed tag and each spurious tag counts once, lower is better, and partly-correct tag sets earn partial credit.

Youden's J, sensitivity plus specificity minus one, weights both error rates equally and ignores class sizes. Maximising F1 ignores true negatives entirely and implies a trade-off that shifts with the operating point rather than a fixed cost ratio.

RMSLE is the root mean squared error of log(1 + value), so it scores the ratio between prediction and actual, not the currency gap. Being twice too high on a $20 order costs what it costs on a $2,000 one.

Beta sets how many times more recall matters than precision: F2 weights recall twice as heavily, F0.5 half as heavily, F1 equally. Derive beta from the cost ratio of a missed case to a false alarm.

Lift measures who is likely to respond, not who responds because you called. Much of the top decile would have bought anyway, so targeting them redistributes credit rather than creating sales. Measuring the campaign's real effect needs a randomised control group.

Not much as a point value. The curve's resolution comes from the positive count, not the row count: with 60 positives each is a 1.7% recall step, so average precision carries sampling variance. Report a resampled interval, not three decimals.

A maximum over many slices is biased: cut finely enough and one always looks terrible by chance. Fix the slice list and minimum size in advance, attach a resampling interval to each slice's error, and require it to repeat on fresh data.

Choose the miscoverage level from the cost of missing the true diagnosis, not from convention. Higher coverage means larger label sets and more escalations, and the workflow must handle sets of any size, including two labels or none.

Choose by the decision the probabilities feed. Log-loss when an overconfident mistake is expensive and you want extremes punished hard; the Brier score when you need a bounded, stakeholder-legible number that no single row can dominate.

Crossing curves mean neither model dominates - each wins over a different range of false positive rates. Compare them where you will actually operate, such as true positive rate at a fixed low false positive rate, not by total area.