How do you compute the standard error of the difference between two independent sample means?

## What is being estimated Comparisons are almost never about one group; they are about a **difference**. The estimate is `mean1 - mean2` and, like any statistic, it has its own sampling distribution and therefore its own standard error, describing how much that difference would move if both samples were redrawn. ## The rule: variances add For independent random variables `A` and `B`: ``` Var(A - B) = Var(A) + Var(B) ``` The minus sign does **not** become a minus on the right. Subtracting a noisy quantity injects noise just as adding one does; uncertainty accumulates in either direction. Applying this to two sample means: ``` Var(mean1 - mean2) = s1^2/n1 + s2^2/n2 SE_diff = sqrt(s1^2/n1 + s2^2/n2) = sqrt(SE1^2 + SE2^2) ``` The recipe is: **square each standard error, add, take the square root.** Standard errors combine in quadrature, like the sides of a right triangle, never by simple addition. ## A worked comparison Two independent schools sit the same test. | school | mean | s | n | SE = s / sqrt(n) | |---|---|---|---|---| | A | 512 | 40 | 100 | 4 | | B | 498 | 30 | 100 | 3 | - Difference in means: `512 - 498 = 14` points. - Standard error of that difference: `sqrt(4^2 + 3^2) = sqrt(25) = 5` points. So the observed 14-point gap is estimated with a wobble of about 5 points per redraw. Note the arithmetic: 5 is smaller than the 7 you would get by adding the standard errors, and larger than either one alone. Both bounds are worth remembering as a sanity check - the combined standard error always sits between `max(SE1, SE2)` and `SE1 + SE2`. ## Why adding standard errors is wrong Adding assumes the two errors always push in the same direction, which is exactly what independence rules out. Sometimes school A's sample runs high and B's runs high too, and the errors partly cancel in the difference. Because cancellation is possible, the honest combined uncertainty is less than the sum. Adding standard errors inflates uncertainty, which sounds conservative but leads to declaring real differences unmeasurable. ## The smaller group dominates Unequal sample sizes have a consequence people repeatedly miss. Suppose the control group has 1,000,000 observations with `s = 40` (`SE = 0.04`) and the treatment group has 400 with `s = 40` (`SE = 2`): ``` SE_diff = sqrt(0.04^2 + 2^2) = sqrt(0.0016 + 4) = 2.0004 ``` The giant group contributes essentially nothing. Precision of the comparison is set by the smaller arm, so pouring more data into the already-large group is wasted effort; the lever is the small arm. This is why balanced allocation is roughly optimal when the two groups have similar variance. ## Unequal variances Nothing above required `s1 = s2`. The formula `sqrt(s1^2/n1 + s2^2/n2)` handles unequal spreads directly and is the safe default. An alternative, the **pooled** standard error, first combines the two samples into a single variance estimate and is only valid when the population variances really are equal; when they are not and the sample sizes differ, pooling can badly misstate the standard error. Using the separate-variance version costs almost nothing when variances happen to be equal, so it is the sensible habit. ## When independence fails: paired data If the same students are measured before and after a programme, or the same users see both designs, the two means are **not** independent. The general identity is ``` Var(A - B) = Var(A) + Var(B) - 2 * Cov(A, B) ``` With positive correlation - a strong student is strong on both occasions - the covariance term is positive and subtracts, so the true standard error of the difference is **smaller** than the independent formula suggests. The clean way to handle it is to stop treating them as two samples: compute each subject's own difference, then take the standard error of those `n` differences as an ordinary mean, `s_d / sqrt(n)`. That automatically captures the covariance. Applying the independent formula to paired data throws away the pairing and leaves real effects looking noisier than they are - a genuine loss of statistical efficiency. ## Sanity checks to carry into an interview - The combined standard error is never smaller than the larger of the two inputs and never larger than their sum. - Halving the standard error of a difference still needs roughly four times the data, in **both** arms. - If one arm is far larger, quote its contribution and show that it is negligible; interviewers like seeing the term dropped for the right reason rather than by feel.