What is HARKing, and why does it invalidate the p-value it is reported with?

## The definition HARKing stands for **Hypothesizing After the Results are Known**, a term introduced by Norbert Kerr in 1998. It describes a specific move in scientific writing: after seeing which analysis produced a significant result, the author writes the paper's hypothesis section so that this result appears to have been predicted from the start. It is not lying about the numbers. The data are real, the test was computed correctly, the p-value is arithmetically right for the comparison shown. What has been rewritten is the *order of events* — and the order of events is exactly the information a reader needs to interpret the number. ## The canonical shape A study collects an outcome and eight ways to slice the sample: age bands, gender, region, tenure, device, channel, first-time versus returning, weekday versus weekend. The overall effect is null. One slice — returning users on mobile, say — shows p = 0.02. The HARKed write-up says: *"We hypothesised that the intervention would be most effective for returning mobile users, because habit strength moderates response to interface changes."* Then it reports p = 0.02. The theory is plausible, the citation is real, and the sentence would never have been written if that slice had come out null. ## Why the p-value is no longer interpretable A p-value is a statement about a procedure: if the null is true, a procedure that names one comparison in advance and tests it at alpha = 0.05 produces a false alarm 5% of the time. A procedure that scans eight comparisons and reports the best is a different procedure with a much larger false-alarm rate. HARKing does not change which procedure was followed — the search happened either way. It changes which procedure the *reader believes* was followed. And because interpretation of a p-value is entirely relative to the procedure, the reader's inference is wrong even though the arithmetic is right. The number in the paper is correct for a test that was never pre-specified. A useful way to say this in an interview: p-hacking is what you do to the analysis; HARKing is what you do to the write-up. They usually travel together, and HARKing is the half that makes the other half undetectable. ## The near-neighbours HARKing gets confused with - **Reporting an unexpected finding.** Entirely legitimate — and it is legitimate precisely because it is labelled. "We did not predict this; the effect appeared only among returning mobile users and should be treated as exploratory" is honest science. The same sentence with the prediction moved to the introduction is HARKing. - **Generating hypotheses from data.** Also legitimate, and the normal engine of research. The problem starts when the hypothesis born from a dataset is then tested — and reported as confirmed — on that same dataset. - **Tidying the narrative for readability.** Papers are written after the work, so some retrospective ordering is unavoidable. The line is whether the reordering hides how many comparisons were in play. Cutting a paragraph is editing; promoting the winning subgroup to a prediction is HARKing. ## Why it is so common The incentives are brutally aligned against disclosure. A paper with a clean predicted effect reads as competent; a paper reporting "we looked at eight subgroups and one was significant, which may be noise" reads as a failure, even though it is more informative. Kerr's original point was partly sociological: HARKing survives because the literature rewards tidy stories, not because anyone decided to deceive. ## How to detect and prevent it Detection, from the outside, is mostly about asking what is missing: - Does the hypothesis section name a subgroup or moderator suspiciously specific to the significant result? - Are all the measured outcomes and slices reported, including null ones? Silence about a slice that was obviously collected is the tell. - Does the theory feel reverse-engineered — plausible for the finding, but equally plausible for its opposite? Prevention is structural. A time-stamped analysis plan written before the outcome data is inspected fixes the order of events in a way that cannot be quietly rewritten. Reporting standards that require every collected measure and every planned comparison to be listed remove the option of silent omission. And separating a report into a confirmatory section (pre-specified, p-values meaningful) and an exploratory section (labelled, treated as hypothesis-generating) lets a genuine surprise be published without dressing it as a prediction. ## The correct move when you find a surprise Say it plainly: this was not predicted, here is the full set of comparisons examined, here is the one that stood out, and here is the pre-specified test we will run on new data to find out whether it is real. That report is more useful than the HARKed version, and it is the version that survives replication.