Why can Shapiro-Wilk reject normality at n = 5,000 yet pass a clearly skewed n = 8 sample?

## What the test is asking Shapiro-Wilk takes a sample and tests the null hypothesis that it was drawn from a normal distribution. The statistic is built from a comparison of the sorted sample against the values expected from a normal distribution - essentially a numerical summary of how well the points would line up on a normal Q-Q plot. A small p-value is evidence against normality; a large one is not evidence for it. ## Why sample size dominates the verdict Every hypothesis test has power: the probability of rejecting the null when it is false. Power increases with sample size and with the size of the true departure. Two facts then collide. First, exact normality never holds for real measurements. Data are bounded below by zero, rounded to two decimals, censored at a cap, or generated by a mixture of processes. There is always some departure, so the null is always false in the strict sense. With enough data, any test of a null that is false by any margin will eventually reject. At n = 5,000 the amount of departure needed to produce p < 0.001 is far smaller than the amount that would change any conclusion you would draw. Second, at n = 8 the test has so little power that even a strongly skewed population usually fails to produce a significant result. Non-significance means the data could not convict, not that the population is innocent. This is the standard asymmetry of null hypothesis testing: rejection is informative, non-rejection is not. Put together, the test's sensitivity moves in exactly the opposite direction from the assumption's importance. For a comparison of means, non-normality of the raw data matters most at small n, where the central limit theorem has not had room to work - and that is where the test is blind. At large n the central limit theorem has largely absorbed the issue - and that is where the test screams. ## The right question: how big, in what direction, and does it matter? Swap the yes/no question for a magnitude question. Look at a normal Q-Q plot to see whether the departure is skew or tail weight, whether it is driven by the bulk or by a few points, and how far the ends stray. Quantify with descriptive measures such as skewness and kurtosis if you want numbers. Then ask what the procedure is sensitive to: - Comparing means at a healthy sample size: moderate skew is absorbed; what remains dangerous is a handful of extreme values, which inflate the sample standard deviation and drain power. - Small samples: the raw shape matters directly, and this is where you should be most cautious - though it is also where you have the least evidence about the shape. - Anything about the tails - a high percentile, a prediction interval, a probability of exceeding a threshold: the tail behaviour is the estimand, so heavy tails are not a technicality but the whole answer. When you genuinely cannot tell whether a departure matters, resample: draw repeatedly from your own data and see whether the interval or error rate you plan to report actually holds. That answers the operational question directly, rather than the philosophical one about whether the population is exactly normal. ## Legitimate uses of the test None of this makes the test useless. It is a cheap screen on a moderate sample. It is defensible as a documented, pre-declared step in a process that requires one. It works as triage across many series at once, where nobody can eyeball every plot. And significance on a moderate sample size, combined with a plot showing a large bend, is a real signal. What it must not be is the automatic gate that decides your analysis, because its verdict tracks n more strongly than it tracks anything you care about. ## Responding when it rejects If a large sample fails a normality test and you wanted to compare means, the usual answer is to proceed, after checking the plot for outliers or a second mode - those distort the mean itself and are a substantive finding, not a technicality. If the sample is small and the plot shows a serious bend, the honest options are to transform the outcome, to use a procedure that does not rely on the shape, or to accept wider uncertainty. And if the outcome is strongly skewed, the deeper question is whether the mean is the right summary at all, which no normality test will ever tell you.