Comparing Two Models

## What "0.3 accuracy points" actually means Accuracy on a 1,000-example test set can only move in steps of 0.1 percentage points, because one example is `1/1000`. So a 0.3-point gap is not an abstract quantity - it is **three examples**. Whatever the two models did on the other 997, model B ended up with three more correct answers than model A. Saying it that way already reframes the conversation: nobody ships a model on the strength of three examples without checking whether three is more than the test set's ordinary jitter. ## The comparison is paired, not two separate scores Both models were run over the *same* 1,000 examples. That means the two accuracy numbers are not two independent measurements; they share every quirk of the test set. The information about which model is better lives entirely in the examples where the two models **disagree**. Write the four paired outcome counts for the test set: - `a` - both models right - `b` - model B right, model A wrong - `c` - model A right, model B wrong - `d` - both models wrong Model B's correct count is `a + b` and model A's is `a + c`, so the accuracy gap is ``` accuracy_B - accuracy_A = ((a + b) - (a + c)) / n = (b - c) / n ``` The `a` and `d` cells cancel exactly. A 0.3-point gap on `n = 1000` therefore means `b - c = 3`, no matter whether the models disagreed on 5 examples or 300. ## Why a net of three examples cannot be significant McNemar's test asks whether the disagreements split more lopsidedly than fair coin flips would. Its chi-square form, with one degree of freedom, is ``` X^2 = (b - c)^2 / (b + c) ``` The 5% critical value for one degree of freedom is about **3.84**. With `b - c = 3`, the numerator is fixed at 9, and the denominator `b + c` is at least 3 (you cannot have a net of three from fewer than three disagreements). So ``` X^2 <= 9 / 3 = 3 < 3.84 ``` and the best case - `b = 3`, `c = 0`, meaning model A never once wins an example that B loses - still falls short. The exact form makes the same point more directly: under the null, the discordant examples behave like `b + c` fair coin flips, and three flips all landing the same way has two-sided probability `2 * 0.5^3 = 0.25`. Every other split of a net-three win is weaker still. There is no arrangement of a 0.3-point gap on 1,000 examples that clears a conventional threshold. ## How much data would you need? Suppose the two models keep disagreeing on a fraction `r` of examples and keep the same 0.3-point edge. Then `b - c = 0.003n` and `b + c = rn`, so ``` X^2 = (0.003n)^2 / (rn) = 0.000009 * n / r ``` Setting that at 3.84 gives `n = 426,667 * r`. With 5% disagreement that is roughly 21,000 examples; with 10% disagreement, roughly 43,000. A 0.3-point effect is simply a large-sample question, and a 1,000-example test set was never going to answer it. ## What to report instead Report the paired gap with an uncertainty statement rather than a bare winner: "model B is ahead by 3 examples out of 1,000; the paired interval comfortably covers zero; this test set cannot resolve differences below roughly one point." Naming the resolution of your test set is far more useful to the reader than a p-value, and it turns the next conversation into "do we need a bigger evaluation set?" rather than "which model won?". ## Traps that show up in interviews The most common mistake is comparing the two accuracy numbers as if they were independent proportions - that both ignores the pairing and still would not rescue a three-example gap. The second is treating a positive difference as evidence of any size at all, when the sign of a tiny difference is close to a coin flip. The third is quoting the standard error of a *single* accuracy estimate (about 0.95 points at 90% accuracy on 1,000 examples) as the yardstick for the gap; that number is too conservative for a paired comparison, because the paired standard error of the difference is usually much smaller. Use the paired machinery, and then note that even the paired machinery cannot make three examples significant.