How does lowering the significance level from 0.05 to 0.01 affect Type II errors at a fixed sample size?

## What the significance level actually controls A hypothesis test converts data into a test statistic and rejects the null hypothesis when that statistic falls into a **rejection region**. The significance level `alpha` is what defines the boundary of that region: it is chosen so that, if the null hypothesis were true, the statistic would land in the rejection region only a fraction alpha of the time. So alpha is a knob on the threshold, and moving it moves the boundary. For a two-sided test based on a statistic that is approximately standard normal under the null, `alpha = 0.05` puts the boundary at roughly `|z| = 1.96`, and `alpha = 0.01` pushes it out to roughly `|z| = 2.576`. Nothing about the data changed; only the bar the data must clear. ## Two distributions, one boundary The clearest mental picture has two sampling distributions of the same statistic drawn on the same axis. One is the distribution assuming the null hypothesis is true, centred at zero. The other is the distribution assuming a particular real effect exists, shifted away from zero by an amount that grows with the size of the effect and shrinks with the noise and with the square root of the sample size. The threshold is a vertical line. The area of the null-centred distribution beyond the line is alpha, the false-alarm rate. The area of the shifted distribution that falls *short* of the line is beta, the miss rate. There is one line and two areas, and they move in opposite directions: - Push the line outward (smaller alpha) and the null distribution's tail beyond it shrinks — fewer false alarms — while more of the shifted distribution now sits on the non-rejection side, so beta rises. - Pull the line inward (larger alpha) and you accept more false alarms in exchange for fewer misses. At a fixed sample size, fixed variance and fixed true effect, that is the entire trade-off. The two distributions overlap, and no placement of a single line can avoid cutting into both of them. ## What actually moves both rates down The threshold is not the only thing on the diagram. The *separation* between the two distributions can change, and that is the real lever: - **More observations.** The spread of a sample mean shrinks with the square root of the sample size: `SE = s / sqrt(n)`. Narrower distributions overlap less, so at any given alpha the miss rate falls. - **Less measurement noise.** Reducing the variability in the underlying measurements, through better instruments, cleaner data or blocking out a known nuisance source, has the same narrowing effect as adding data. - **A more efficient design.** Paired or within-subject comparisons remove between-unit variability from the comparison and can shift the alternative distribution further out for the same number of units. - **A larger true effect.** Not usually under your control, but it is why a design that reliably catches a large effect can be helpless against a small one. This is why the honest answer to "we got a null result, should we just use a stricter threshold next time?" is no: strictness is not rigour, it is a re-allocation of risk from one error type to the other. ## A worked intuition Suppose a design gives a statistic that, if the real effect is what you expect, sits on average at `z = 2.2`. At `alpha = 0.05` the bar is 1.96, and a typical outcome clears it. Move the bar to 2.576 and that same typical outcome now falls short: the effect is real, present in the data at its expected magnitude, and the test reports nothing. Nothing was measured worse; the decision rule simply stopped counting that much evidence as enough. ## Where the choice comes from Because moving alpha only re-allocates risk, the choice has to come from outside the mathematics — from what each mistake costs. A setting where a false alarm triggers an expensive, hard-to-reverse commitment argues for a strict threshold; a setting where a miss is the expensive outcome and a false alarm merely triggers a cheap follow-up argues for a loose one. What is never defensible is choosing the threshold after seeing which side of it your result landed on, because the stated false-alarm rate then no longer describes the procedure you actually ran. ## Common confusions to avoid Candidates sometimes claim a stricter alpha makes a significant result "more likely to be true" in a way that costs nothing. It does buy something real — surviving results were held to a higher evidentiary bar — but the cost is paid in silence, in the real effects that never get reported. Others assume beta has a fixed value that alpha cannot touch; in fact beta depends jointly on alpha, the sample size, the variability and the size of the true effect, and quoting it without those four is meaningless.