Evaluation Metrics

The four cells count true positives, false positives, true negatives and false negatives - one per pairing of predicted and actual label. Accuracy is (TP + TN) divided by all four counts; the error rate is 1 minus accuracy.

Because a model that never predicts 'defective' already scores 99.7% on that data. The majority class supplies almost every row, so accuracy measures performance on good units and says nothing about whether a single defect was ever caught.

With true classes as rows and predicted classes as columns, a class's recall is its diagonal cell divided by its row total, and its precision is that same diagonal cell divided by its column total.

Precision is the share of predicted positives that are truly positive; recall is the share of actual positives the model catches. Loosening the decision rule labels more examples positive, which raises recall and usually lowers precision.

A cumulative gains chart plots the share of all responders captured against the share of the file contacted, after sorting by score. A lift chart shows the same result as a multiple of random targeting, where random equals 1.

A precision-recall curve plots precision against recall as the decision threshold sweeps from strict to permissive. Average precision compresses that curve into one number: the recall-weighted mean of the precisions along it, estimating the area beneath.

An ROC curve plots true positive rate on the y-axis against false positive rate on the x-axis. Each point is one decision threshold, and sweeping the threshold from strictest to loosest traces the curve from (0,0) to (1,1).

More items are labelled positive, so true positives and false positives can only rise, recall can only rise, and precision usually falls. The fitted model, its scores and its ranking of items do not change at all.

Two: the best constant prediction — the target's median for MAE, its mean for squared error — and the rule of thumb the business already runs. Both on the same held-out rows, they turn a bare error number into a decision.

R-squared on the fitting rows is measured on data whose residuals the fit already minimised, so it flatters the model. Held-out R-squared scores rows the model never saw, the only honest estimate of future performance.

RMSE squares every error before averaging, so one huge miss contributes out of all proportion. MAE averages the absolute errors, giving each mistake weight in line with its size. RMSE tracks the worst errors, MAE the typical one.

Split conformal holds out a calibration set the model never trained on, scores every row by its absolute residual, and takes the residual sitting at the 90% rank. Each new prediction then gets that one value added and subtracted.

Log-loss charges minus the log of the probability the model gave to the outcome that actually happened, so a confident miss costs unboundedly much. The Brier score squares the probability error, so no single row can cost more than 1.

Macro averages per-class F1 scores with equal weight, so a rare class counts as much as a common one. Micro pools every class's true positives and errors first, so frequent classes dominate. Weighted averages per-class scores by class support.

Both are right; they use different denominators. With 0.2% fraud, tens of thousands of false positives barely move the false positive rate, but they dominate the flagged set and crush precision. Read average precision against a 0.002 baseline, not 0.5.

Draw one clicker and one non-clicker at random: the model scores the clicker higher 78% of the time, counting ties as half a win. ROC-AUC is a statement about ranking order, not about the numeric size of the scores.

Flag an item when the expected cost of flagging is below the expected cost of not flagging. With false-positive and false-negative costs only, the break-even is cost_FP / (cost_FP + cost_FN), which is 0.5 only for equal costs.

Overall MAE is a row-weighted average, so a small cohort can run three times worse and barely move it: a cohort holding 3% of rows at triple the error lifts overall MAE by only about 6%. Cut the metric by cohort.

A negative held-out R-squared means the model's squared error on those rows exceeds what predicting one number, the split's mean, for every row would give. The model lost to a constant.

A scoring rule is proper when a forecaster gets their best expected score by reporting the probability they genuinely believe, rather than by shading it. Log-loss and the Brier score are strictly proper; thresholded accuracy is not.

Choose isotonic regression when the calibration set is large, a few thousand rows or more, and the distortion is not sigmoid-shaped. Platt scaling fits only a two-parameter logistic curve, so it is the safer choice on small calibration sets.

Balanced accuracy averages the per-class recalls, giving every class equal weight, while plain accuracy weights each class by how many rows it contributes. On skewed data the two diverge sharply, and a constant predictor scores 1/k instead of the majority share.

The harmonic mean is dominated by the smaller of the two values, so F1 stays low unless precision and recall are both decent. Precision 0.90 with recall 0.10 gives F1 0.18, while the arithmetic mean would flatter it at 0.50.

Specificity is measured only among patients who do not have sepsis, so it ignores how rare sepsis is. At a 2% rate, 5% of the huge non-sepsis group yields far more false alarms than true cases, leaving precision near 25%.

KS 38 means that at its best cut-off the scorecard separates 38 percentage points more of the bads than of the goods. Gini 0.52 is a whole-curve summary equal to two times AUC minus one, so AUC is 0.76.

It means the score stops rank-ordering cleanly in that region. Usually it is sampling noise in small deciles; sometimes it is a genuinely unstable model, a shifted population, or a scoring bug. Check decile sizes before you diagnose anything else.

Residual spread grows with the size of the prediction, so error is roughly proportional to the quantity rather than constant. Any absolute-error headline is then mostly a statement about the largest predictions. Report error banded by predicted magnitude.

MAPE divides each error by the actual value, so a 30-second miss on a one-minute wait scores 50% and near-zero waits explode. It is also asymmetric: over-prediction is unbounded, under-prediction capped at 100%.

Conformal prediction guarantees marginal coverage: 90% averaged over the whole population, not inside every subgroup. One global quantile under-covers groups whose errors are larger and over-covers the easy ones. Group-wise calibration or an input-adaptive score is the fix.