Why did the 1936 Literary Digest poll get the election wrong despite millions of returned ballots?

## What happened In 1936 the *Literary Digest* ran what was, by volume, the largest election poll ever attempted. It mailed roughly ten million straw-vote ballots and received back on the order of 2.4 million. It forecast a comfortable victory for the Republican challenger, Alf Landon. Franklin Roosevelt won 46 of the 48 states. Meanwhile a far smaller poll run by George Gallup, with a sample in the tens of thousands, called the result correctly. The episode is the canonical demonstration that **sample quality dominates sample size**. ## Failure one: the sampling frame A **sampling frame** is the list from which you actually draw. It is not the population; it is your operational stand-in for the population. The Digest built its frame from telephone directories, automobile registration records and its own subscriber list. In the middle of the Depression, owning a telephone or a car was a marker of relative affluence, and affluence was correlated with the very thing being measured — vote intention. Large parts of the electorate had **zero probability** of ever being sampled. That is **undercoverage**, also called frame error or coverage error. The formal point matters: any sampling theory that promises unbiasedness assumes every member of the target population has a known, non-zero inclusion probability. When a subgroup has probability zero, no amount of sampling from the frame recovers it, and no confidence interval computed from the sample accounts for the missing group — the interval describes uncertainty about the frame's mean, not the population's. ## Failure two: who mailed the ballot back About one in four recipients returned a ballot. Response is a second, self-administered selection step layered on top of the frame. Whether someone bothers to fill in and post a straw ballot depends on how strongly they feel, and strength of feeling was itself related to which candidate they preferred. The people who responded therefore differed from the people who did not, on the exact variable being estimated. Here non-response is best understood as **a frame-and-weighting problem**: the effective sample is not "people mailed a ballot" but "people mailed a ballot who chose to answer", and unless you can characterise how those two groups differ, you cannot correct the estimate. A high absolute response *count* tells you nothing; the response *rate*, and how it varies across subgroups, is the informative quantity. ## Why the enormous n did not save it Decompose the error of an estimator into bias and variability: `error = (estimate - frame mean) + (frame mean - population mean)` The first term is sampling variability and shrinks roughly as `1 / sqrt(n)`. The second term is the bias induced by frame and response selection, and it does **not** depend on `n` at all. Going from ten thousand to two million responses divides the first term by about fourteen and leaves the second untouched. The practical result is worse than a small poll: the reported margin of error becomes tiny, which manufactures confidence in a wrong number. A biased big sample is more dangerous than a biased small one because it looks authoritative. ## What a better design would have looked like - **Fix the frame first.** Use a frame that reaches the whole target population — in that era, area or household-based sampling rather than lists of consumer goods owners. - **Control who responds, not just who is contacted.** Chase non-respondents, offer alternative response modes, and measure the response rate by subgroup so the shortfall is visible. - **Compare the achieved sample to known population composition.** If your respondents are richer, older or more urban than the population, you can see the coverage problem before the result embarrasses you. - **Report the response rate next to the estimate.** A poll quoting a raw count without a rate is hiding the second selection step. ## The modern version Swap telephone directories for an opt-in web panel, a banner ad recruiting respondents, or an intercept survey on a single page of a site, and the identical two-step failure reappears: a frame that reaches only part of the population, and a response step that filters again by interest. Very large modern samples reproduce the Digest's mistake routinely, and for the same reason — the analyst reports `n` and never reports who could not be reached and who declined. ## What interviewers listen for A strong answer separates the two failures rather than blurring them into "the sample was bad", states that bias does not shrink with `n`, and names the diagnostic that would have caught it: compare the composition of respondents against a trusted external benchmark, and look at the response rate rather than the response count.