Test Families

A two-sample t-test assumes observations are independent within and between groups, that the sampling distribution of each group mean is approximately normal, and, for the pooled version, that the two populations share a common variance.

A chi-square goodness-of-fit test compares observed category counts with the counts a hypothesised distribution predicts. A fair die over 600 rolls predicts 100 per face; the statistic sums (observed minus expected) squared, divided by expected, across the six faces.

A two-sample Kolmogorov-Smirnov test compares the two samples' entire distributions. It builds a step-shaped empirical CDF for each sample and takes the largest vertical gap between them, so a difference in location, spread or shape can all trigger rejection.

Rank-based tests replace each value with its position in sorted order, so a mistyped 10,000 becomes only the largest rank, not a huge number. Its influence is capped at one rank; a mean and a variance have no such cap.

Use a paired t-test when each value in one group is naturally matched to one in the other, such as the same patient measured before and after treatment. It tests the mean of the within-pair differences.

F is the between-group mean square divided by the within-group mean square. With k groups and N observations the numerator has k-1 degrees of freedom and the denominator N-k. F sits near 1 when all group means are equal.

Each expected count is that cell's row total times its column total, divided by the grand total. Degrees of freedom are (rows minus one) times (columns minus one), so a 3x4 table has (3-1)(4-1) = 6.

Pool both samples, rank them, and U counts how often a value from one group beats a value from the other, testing whether P(X > Y) = 1/2. It compares medians only when the two distributions share a shape.

A z-test for a mean is valid only when the population standard deviation is known rather than estimated. If you plug in the sample standard deviation, the extra uncertainty makes the statistic follow a t distribution instead.

The independence assumption breaks. Measurements from one patient are correlated, so 200 rows carry far less information than 200 independent ones: standard errors come out too small, confidence intervals too narrow and p-values far too optimistic.

Each pairwise test spends its own 5% false-positive budget, so 10 of them carry roughly a 40% chance of at least one spurious significant result. A single ANOVA asks one question at one alpha.

Bartlett's test assumes normality and mistakes heavy tails or skew for unequal variance, so it rejects far too often on real data. Levene's test compares absolute deviations from each group's centre and stays reliable when the data are not normal.

An S-shape means both tails are heavier than a normal's: the lowest points sit below the reference line and the highest above it. A curve where both ends bend the same way, one tail far more than the other, means skew.

The common rule wants every expected count at least 5, or more leniently no expected count below 1 with at most a fifth below 5. Below that, collapse categories, gather more data, or run Fisher's exact test.

Fitting the reference curve to the same data pulls it toward the sample, shrinking the maximum gap. Standard tables assume a reference fixed in advance, so p-values come out too large and the test under-rejects.

Only that the groups are not all alike — at least one tends to produce larger values than another. It is an omnibus test on pooled ranks with k-1 degrees of freedom and never says which pair differs.

It takes each pair's difference, drops zeros, ranks the absolute differences from smallest to largest, and sums the ranks belonging to positive differences. Large or small sums indicate the differences are not centred at zero.

Student's two-sample t-test pools the two groups into one variance estimate, which is only valid when their variances are equal. Welch's test keeps them separate and adjusts the degrees of freedom, so it stays accurate under unequal variances.

A significant omnibus F only says at least one mean differs. Follow it with a post-hoc procedure such as Tukey's HSD, which compares all pairs at once while holding the error rate across the whole family of comparisons.

A normality test measures evidence against normality, not the size of the departure. At n = 5,000 it detects deviations far too small to matter; at n = 8 it has almost no power, so passing proves nothing about the population.

On a 2x2 table the Pearson chi-square statistic is algebraically identical to the square of the pooled-variance two-proportion z statistic. A chi-square variable on one degree of freedom is a squared standard normal, so the two-sided tail areas coincide exactly.

A Kolmogorov-Smirnov test is unreliable on heavily tied data: its null distribution assumes continuous measurements with no exact repeats. With five rating values the empirical CDFs jump in blocks and the classical p-value comes out too conservative.

They quietly change the question. A rank test asks how often one group is higher, not by how much, so it can be significant while the total the business banks is unchanged — and it costs about five percent efficiency.

Pool for the hypothesis test, not for the confidence interval. The test assumes the null that both rates are equal, so one combined proportion estimates their shared variance; an interval must allow them to differ.

At a million observations the KS rejection threshold shrinks toward zero, so any real difference becomes significant. Judge the magnitude instead: read the statistic D as an effect size and compare it against a materiality threshold agreed beforehand.

Only the direction: which member of the pair was larger. Magnitudes are discarded, ties are dropped, and the count of positive pairs is compared with a binomial distribution with success probability one half.

Both are symmetric bell curves centred at zero, but t with 4 degrees of freedom has heavier tails and variance 2 rather than 1. Its two-sided 95% cutoff is about 2.78 against the normal's 1.96.

A significant interaction means the effect of one factor depends on the level of the other: the drug's effect changes across doses. The main effects then describe averages that may not hold at any particular drug-dose combination.

One about geometric means, not arithmetic ones. A t-test on logged values compares mean log income between groups, and exponentiating that difference gives a ratio of geometric means - not the difference in average income the business usually asked about.