Why is Welch's t-test usually a safer default than Student's pooled t-test?

## Two ways to build the denominator Both two-sample tests have the same numerator, the difference in sample means `xbar1 - xbar2`. They differ in how they estimate the variability of that difference. **Student's pooled t-test** assumes a single common variance `sigma^2` in both groups and estimates it by a degrees-of-freedom-weighted average: ``` s_p^2 = ((n1 - 1) * s1^2 + (n2 - 1) * s2^2) / (n1 + n2 - 2) t = (xbar1 - xbar2) / sqrt(s_p^2 * (1/n1 + 1/n2)) df = n1 + n2 - 2 ``` **Welch's t-test** makes no such assumption and never mixes the two variance estimates: ``` t = (xbar1 - xbar2) / sqrt(s1^2/n1 + s2^2/n2) df = (s1^2/n1 + s2^2/n2)^2 / ( (s1^2/n1)^2/(n1 - 1) + (s2^2/n2)^2/(n2 - 1) ) ``` That second expression is the **Welch-Satterthwaite** approximation. It is not required to be an integer. ## Where pooling goes wrong Pooling weights each group's variance by its degrees of freedom, so the bigger sample dominates. That is the right thing to do *if* both groups genuinely share a variance — you are combining two estimates of the same quantity. If they do not, the pooled number is an average of two different things, and whether it is too big or too small depends on how variance lines up with sample size: - **Larger variance in the smaller group** (the dangerous case). Take one arm with `n1 = 10` and a large variance against another with `n2 = 200` and a small variance. The pooled estimate is pulled toward the low-variance arm because that arm carries 199 of the 208 degrees of freedom. The standard error of the difference is then understated, the t statistic is inflated, and the claimed degrees of freedom of 208 make the critical value smaller still. The test is **anti-conservative**: its true false-positive rate can run well above the nominal 5%. - **Larger variance in the bigger group.** The bias runs the other way and the test becomes conservative — it under-rejects and loses power. - **Equal sample sizes.** Pooling is fairly forgiving of unequal variances when `n1 = n2`, which is why the pooled test survives so long in balanced experiments. The trap is that unbalanced designs are common — a small pilot arm against a large control, a rare segment against everyone else — and unbalanced designs are exactly where pooling misbehaves. ## What Welch's degrees of freedom do Welch's df always lands between `min(n1, n2) - 1` and `n1 + n2 - 2`. Which end it approaches depends on which group contributes more to `s1^2/n1 + s2^2/n2`: - If one group's `s^2/n` term dominates the sum, the effective degrees of freedom collapse toward that group's own `n - 1`. In the 10-versus-200 example, the small noisy arm's term dominates and the df land near 9 or 10, not 208. - If the two terms are comparable and the samples balanced, the df approach `n1 + n2 - 2` and Welch nearly coincides with Student. So Welch does two things at once: it fixes the standard error, and it charges you honestly for the fact that most of your information about the difference came from a small, noisy group. Reporting a fractional value such as 9.4 degrees of freedom is expected, not a bug; the fraction is what the Satterthwaite approximation produces when it matches moments between the true distribution and a t distribution. ## What does defaulting to Welch cost? Very little. When the variances are genuinely equal and the groups balanced, Welch's degrees of freedom sit just below the pooled value and the two p-values differ in the third decimal. Simulation studies of this comparison consistently show Welch holding its nominal error rate across variance ratios where Student's does not, while giving up only a sliver of power in the equal-variance case that Student's assumes. That asymmetry — large protection, tiny cost — is the whole argument for making Welch the default. Several statistical environments have made it the default two-sample test for precisely this reason. ## Related mechanics worth knowing - Welch's test still assumes the two samples are **independent** and that each group's sampling distribution of the mean is approximately normal. It relaxes the equal-variance assumption only. - With `n1 = n2` and equal variances, the Welch and pooled statistics are algebraically the same number; only the degrees of freedom differ slightly. - The confidence interval for the difference follows the same split: `(xbar1 - xbar2) +/- t_crit * sqrt(s1^2/n1 + s2^2/n2)`, with `t_crit` taken at the Welch degrees of freedom. ## How to answer State the assumption that separates them (equal variances), give the formula difference in one line (pool versus keep separate), name the failure case concretely — a small high-variance arm against a large low-variance arm inflates false positives — and close with the practical rule: default to Welch, because when Student's assumption holds the two agree, and when it does not, only one of them is still correct.