How does the file-drawer problem bias a published literature, and can you detect it?

## What the file drawer is Rosenthal's file-drawer metaphor: for every study that appears in print, some number of similar studies sit unwritten in a drawer because they found nothing. If publication depends on the result, then the published literature is not a sample of the studies conducted — it is a sample **selected on the outcome**, which is the classic recipe for a biased estimate. The selection has two distinct consequences, and strong candidates name both. **Too many positives.** If a large number of teams independently test null hypotheses, a fraction will hit significance by chance alone. When only the hits get published, the literature is populated by exactly those chance hits, each looking like a legitimate discovery. **Effects that are too large.** This one matters more and is missed more often. A small, noisy study reaches significance only when the sampling variation happens to push its estimate away from zero. So conditioning on significance conditions on a favourable draw, and the surviving estimate sits in the tail rather than at the truth. A literature assembled from surviving estimates therefore reports effects meaningfully larger than reality, even for effects that are genuinely non-zero. ## Funnel plots The standard visual diagnostic across a set of studies on one question. For each study, plot its effect estimate on one axis against a measure of its precision on the other — larger samples or smaller standard errors sit at the top, small imprecise studies at the bottom. With no bias, the picture is a symmetric inverted funnel. Precise studies cluster tightly around the common effect at the top; imprecise studies scatter widely, but symmetrically, to both sides. That symmetry is the signature of a complete record. Publication bias eats one lower corner. Small studies that found a null or opposite-signed effect never entered the record, so the bottom-left of the funnel is sparse while the bottom-right is full. The plot looks lopsided, and the average effect across studies is dragged upward by the surviving small studies. **Read funnel asymmetry carefully.** Asymmetry is evidence, not proof. Other explanations exist: real heterogeneity where smaller studies genuinely differ (different populations, more intensive interventions), quality differences correlated with size, or simply too few studies for the shape to mean anything. Treat asymmetry as a prompt to ask what is missing, not as a verdict. ## The replication evidence The Reproducibility Project in psychology took roughly a hundred published findings and repeated them with high-powered direct replications. Only around a third to forty percent produced a statistically significant result in the same direction, and the replication effect sizes averaged roughly half the originals. Comparable exercises in other empirical fields have landed in similar territory. What that number does and does not mean matters. It does **not** mean two-thirds of the studies were fraudulent, or that any single failed replication proves the original wrong — replications have their own sampling variation and can differ in population or procedure. What it does show is a literature whose reported effects are systematically larger than what careful repetition finds, which is precisely the fingerprint of selection on significance combined with flexible analysis. ## Why it compounds with analytic flexibility Selective publication and selective analysis multiply. An analyst with many defensible analysis choices can usually find one that reaches significance; the journal then selects among those already-selected results. Each filter passes through the estimates that noise flattered most. The reader sees the end of a long selection chain and has no way to reconstruct its length from the paper. ## Fixes that actually work - **Preregistration.** A time-stamped, public analysis plan filed before the data are seen. It fixes the hypothesis and the analysis, and it leaves a record even if the paper never appears. - **Registered reports.** The stronger version, and the one that attacks the file drawer directly: the journal reviews the introduction and method *before* data collection and grants in-principle acceptance. Publication then depends on the question and the design, not the result — so a null cannot be filed away. - **Study registries and results databases.** Making the existence of a study public at launch means a missing write-up is visible as a gap rather than invisible. - **Publishing nulls, and valuing them.** A well-powered null is genuinely informative about the world, and treating it as a non-result is what fills the drawer in the first place. - **Reading the literature as a selected sample.** Practically: assume published effects are upper bounds, weight large pre-registered studies far above small surprising ones, and be sceptical of a striking effect that exists in one small study. ## In an interview Expect this in research-adjacent or data-science roles as "why should I not just trust the published effect size?" The two-part answer — the record is selected on significance, so both the count of positives and the magnitude of surviving effects are inflated — plus one detection method and one structural fix is a complete answer.