A protected-group slice of 38 applicants shows a 12-point pass-rate gap. Is the gap real?

## The trap Slice the funnel finely enough and some protected-group slice will always look alarming. The question is whether a 12-point gap on 38 people is a finding or an artefact — and the answer, almost always, is that on that sample size you cannot tell. ## The arithmetic Suppose the slice has 38 applicants of whom 15 pass (39.5%), against 402 applicants elsewhere of whom 208 pass (51.7%). The observed gap is 12.3 points. A **Wilson score interval** is the standard interval for a proportion at small `n` — better behaved than the plain normal approximation, which misbehaves when `n` is small or `p` is near 0 or 1. For `p̂` successes-proportion out of `n`, with `z = 1.96` for 95%: ``` centre = (p + z^2 / (2n)) / (1 + z^2 / n) half = z * sqrt(p*(1-p)/n + z^2/(4n^2)) / (1 + z^2 / n) ``` On the small slice this gives roughly `[0.26, 0.55]` — an interval about 30 points wide. Practically, the true pass rate for that group is consistent with anything from one-in-four to a bit over half. Overlapping intervals are suggestive but not the correct test; the honest quantity is an interval on the **gap itself**: ``` se = sqrt(p1*(1-p1)/n1 + p2*(1-p2)/n2) gap = p2 - p1 -> gap +/- 1.96 * se ``` Here `se ≈ 0.083`, so the 95% interval on the 12.3-point gap runs from about **-4 to +29 points**. Zero is inside it. Nothing has been established. ## Both wrong conclusions **"The gap is real, escalate."** Rebuilding features or changing the model on a result this noisy is chasing sampling variation. Next month's 38 people will produce a different number, the change will look like it worked or backfired for no reason, and the team learns to distrust fairness reporting. **"Not significant, so the model is fair for that group."** Equally wrong, and more common among people who have just learned the first lesson. A non-significant result on an underpowered slice is an absence of evidence. Compute the **minimum detectable effect**: at `n = 38` against a large comparison group, roughly a 16-to-17-point gap is needed before a 95% interval clears zero. A genuine 10-point disparity would slip through most of the time. Saying "we could not detect a gap smaller than about 17 points" is honest; saying "there is no gap" is not. ## Multiplicity Slices multiply. Cut the funnel by four attributes with three or four levels each, cross a couple of them, and you have twenty or more comparisons. At a 5% level, one in twenty flags on chance alone, so a single significant slice out of twenty is exactly what you expect from a model with no disparity at all. Options: prespecify a small set of slices you always report; apply a multiplicity correction such as controlling the false discovery rate; or treat all slice findings as hypothesis-generating and require a fresh sample before acting. What you cannot do is scan twenty slices, report the worst one, and present its nominal p-value as if it were the only test you ran. ## What to actually do with a small slice - **Report the interval next to every rate.** A dashboard of bare point estimates on small slices manufactures alarm and false comfort in equal measure. - **State power up front.** "This slice can detect gaps of about 17 points or more" tells the reader what the number is worth before they read it. - **Pool.** Accumulate over a longer window, or aggregate to a coarser grouping, until the interval is narrow enough to act on. Say how long that will take. - **Watch the direction across periods.** A gap of the same sign in six consecutive underpowered windows is meaningful evidence even though no single window is significant; that is a different and much stronger argument than one flagged slice. - **Do not delete the slice from the report.** A group too small to measure is itself a finding about your data collection, and it is precisely the group least likely to be noticed if things go wrong. ## Talking to non-statisticians The sentence that lands is: "with this many people, the number would bounce by that much even if the model treated everyone identically." Then give the interval and the number of applicants you would need before the number means something. Handing a stakeholder a raw 12-point gap on 38 people with no interval is the failure the interviewer is probing for — not the statistics, the reporting judgment.