Why is comparing two models on one shared held-out set a paired problem, not two proportions?

## One draw of data, two numbers When you evaluate two models on the same held-out set, there is exactly one random thing in the experiment: which examples landed in that set. Both accuracy figures are computed from that single draw. The two-independent-proportions machinery assumes there were two separate draws, one per model, and that assumption is simply false here. The consequence is a variance error. For the difference of two estimates, ``` Var(pA - pB) = Var(pA) + Var(pB) - 2 * Cov(pA, pB) ``` The unpaired formula keeps the first two terms and assumes the third is zero. But models trained on similar data make similar mistakes: an ambiguous, mislabelled or genuinely hard example tends to be missed by both. That shared difficulty is exactly a positive covariance, and it is often the dominant term. ## A concrete size for the error Take 10,000 shared test examples, both models at 90% accuracy, disagreeing on 4% of examples. Fill in the paired table: both right on 88%, both wrong on 8%, and 2% each way in the two disagreement cells. Then ``` Cov = 0.88 - 0.9 * 0.9 = 0.07 Var per example, each model = 0.9 * 0.1 = 0.09 Var of the per-example difference = 0.09 + 0.09 - 2 * 0.07 = 0.04 ``` So the paired standard error of the accuracy gap is `sqrt(0.04 / 10000) = 0.002`, or 0.2 percentage points. The unpaired formula gives `sqrt(2 * 0.09 / 10000) = 0.0042`, or 0.42 points. The correlation between the two per-example outcomes is `0.07 / 0.09 = 0.78` - very high, and entirely typical for two models trained on the same corpus. Notice also that the variance of the per-example difference, 0.04, is exactly the disagreement rate. That is the general identity: for accuracy, the paired variance is driven only by how often the models disagree. Two models that agree on 99% of examples can have a tiny standard error on their gap even on a modest test set. ## Conservative is not the same as safe An unpaired test on paired data does not usually inflate your false-positive rate - overstating the standard error makes it harder, not easier, to declare a difference. It is conservative. But it is the wrong kind of caution: it makes real improvements invisible, so teams conclude "no significant difference" from evaluations that would have resolved the gap easily. Reviewers care about this because the cost lands as silently rejected good work rather than as a visible error. ## What to do instead Work with the **per-example difference** as the unit of analysis: - For accuracy or any 0/1 metric, the differences are `+1`, `0`, `-1`, and the whole comparison collapses to the two disagreement counts - which is what McNemar's test consumes. - For a continuous per-example metric such as log-loss or squared error, take `d_i = loss_A(i) - loss_B(i)` and analyse the mean of the `d_i`, either with a resampling procedure or by reporting its interval directly. - For a metric that does not decompose per example, resample the test examples once and score **both** models on each identical resample, so the pairing survives the resampling. In all three, the rule is the same: one shared random draw, then a difference computed inside it. ## Overlapping intervals prove nothing A related trap: plotting a confidence interval for each model's accuracy and concluding "no difference" because the intervals overlap. Overlap of two individual intervals is neither necessary nor sufficient for the difference to be significant, and it is especially misleading when the two estimates are positively correlated - which is exactly the situation here. Build the interval on the **gap**, and check whether that interval excludes zero. ## When pairing does not help Pairing helps because the correlation is positive. If the two models had negatively correlated per-example outcomes - one succeeding precisely where the other fails - the variance of the difference would be *larger* than the unpaired formula suggests, and ignoring the covariance would be anti-conservative. That case is rare between two versions of the same system but is worth naming, because it shows you understand the covariance term rather than reciting a rule. Finally, if the two models were evaluated on *different* test sets, none of this applies: the pairing is gone, and any gap now confounds model quality with test-set difficulty. That is a design flaw to fix, not a statistic to correct.