Confusion Matrix Measures

## What the matrix is A **confusion matrix** is a small table that cross-tabulates what a classifier predicted against what was actually true, for every row in an evaluation set. For a binary problem - two possible labels, conventionally called *positive* and *negative* - the table has two rows and two columns, giving four counts. | | actually positive | actually negative | |---|---|---| | **predicted positive** | true positive (TP) | false positive (FP) | | **predicted negative** | false negative (FN) | true negative (TN) | The naming rule is worth internalising because candidates fumble it under pressure: the **second word is what the model said**, and the **first word says whether the model was right**. A *false negative* is therefore a row the model called negative and was wrong about - an actual positive that slipped through. A *false positive* is a row the model called positive and was wrong about - a false alarm. Every scored row falls into exactly one cell, so `TP + FP + TN + FN = N`, the size of the evaluation set. That single fact is what makes the matrix the common ancestor of essentially every classification metric: each metric is just a different ratio built from these four numbers. ## Accuracy and error rate **Accuracy** is the share of rows the model got right - the diagonal of the table over the whole table: ``` accuracy = (TP + TN) / (TP + TN + FP + FN) ``` **Error rate** (sometimes called misclassification rate) is the complement, the off-diagonal share: ``` error_rate = (FP + FN) / (TP + TN + FP + FN) = 1 - accuracy ``` They carry identical information; teams differ only in which they prefer to quote. Error rate reads better when you are describing improvement, because relative change is meaningful: dropping from 4% error to 2% error is a halving, while '96% to 98% accuracy' sounds like a two-point nudge. ### A worked example Score 200 rows. 40 are actually positive, 160 actually negative. The model flags 50 rows as positive; 32 of those flags are correct. - TP = 32 (flagged and truly positive) - FP = 50 - 32 = 18 (flagged but actually negative) - FN = 40 - 32 = 8 (truly positive but not flagged) - TN = 160 - 18 = 142 (correctly left alone) Check the total: 32 + 18 + 8 + 142 = 200. Accuracy = (32 + 142) / 200 = 174 / 200 = 0.87. Error rate = (18 + 8) / 200 = 0.13. Being able to reconstruct all four cells from partial information like this - 'the model flagged 50, 32 were right' - is a common whiteboard drill. ## The three properties that decide where accuracy is safe 1. **It weights every row equally.** One misclassified row moves the number by 1/N regardless of which row it was. If a missed case costs a thousand times more than a false alarm, accuracy does not know that. 2. **It pools the two error types.** A model with 18 false positives and 8 false negatives, and a model with 8 false positives and 18 false negatives, score identically. Two very different operational behaviours, one number. 3. **It depends on the class mix of the evaluation set.** Because it sums over rows, the class that supplies most of the rows supplies most of the score. On a set that is 99% negative, accuracy is essentially a report on the negative class with a rounding error attached. Property 3 is what makes accuracy collapse under class imbalance, and it is the reason interviewers almost always follow this question with a low-base-rate scenario. Properties 1 and 2 are why the rest of the confusion-matrix family exists at all: the four cells are kept separate precisely so you can ask which kind of mistake the model is making, not just how many. ## Which label is 'positive' The choice of which class to call positive is a convention, not a property of the data - usually the rarer, more interesting or more actionable class (the defect, the churner, the alert). Swapping the convention swaps TP with TN and FP with FN. Accuracy is unchanged, because the diagonal is the diagonal either way; every asymmetric metric flips. Say the convention out loud when you present a matrix, because a reader who assumes the other one will read your false positives as false negatives. ## Multiclass For k classes the matrix is k-by-k: rows are predictions, columns are truth (or the transpose - label your axes). The diagonal holds correct predictions, and accuracy generalises directly as the sum of the diagonal over the sum of the whole matrix. The off-diagonal cells are the useful part, because they tell you *which* pairs of classes are being confused with each other - information no single scalar preserves.