How would you preregister analyses on a team that must still explore data freely?

## The organisational problem Every analytics or research team faces the same tension. Flexible, curious analysis is how you find things; flexible analysis is also what makes claims unreliable. A leader who resolves the tension by banning exploration will get compliance theatre — plans written after the fact, or exploration relabelled as something else. A leader who resolves it by ignoring the problem will get a stream of confident findings that evaporate on contact with the next dataset. The workable answer is not less exploration. It is a **hard boundary between two kinds of claim**, with different rules and different rhetoric attached to each. ## Two tracks **Confirmatory.** Governed by a written, time-stamped plan filed before the outcome data are inspected. This is the only track allowed to produce decisions, launches or headline claims, and the only track where a p-value carries its stated meaning. **Exploratory.** Unlimited. Slice, model, transform, hunt. Its output is *candidate hypotheses*, reported in explicitly hypothesis-generating language, never as an established effect, and never as a launch justification on its own. The rhetorical discipline matters as much as the statistical one. If exploratory findings can be written up in the same voice as confirmatory ones, the boundary has no teeth. ## What a plan must contain to actually bind A plan that says "we will analyse the effect of the intervention" constrains nothing. To close the forks it needs: 1. **The primary outcome**, defined precisely enough that two analysts would compute the same number — including the window, the unit of analysis and the aggregation. 2. **Exclusion and data-quality rules**, written before anyone knows what removing a group does to the result. 3. **The model and covariates**, including transformations and how continuous variables are binned. 4. **The specific comparison** that constitutes the test, and the direction expected. 5. **The decision rule**: what result leads to what action. This is the item most often omitted and the one that most reliably prevents post-hoc reinterpretation of an ambiguous result. 6. **Secondary and exploratory questions**, listed as such, so that later interest in them is documented rather than promoted. Keep it short. A one-page plan that is actually filed beats a template so heavy that nobody fills it in. ## Making it enforceable - **Held-back data.** The most persuasive mechanism, because it does not rely on anyone's memory of what they intended. Explore on one portion; freeze the specification; run it once on a reserve that had no influence on the choice. Analysts accept it because it costs them nothing during exploration. - **Timestamps in a place the team does not control.** A commit, a ticket, a registry entry. The point is not distrust; it is that a plan that can be silently edited provides no evidence. - **Deviation logs, not deviation bans.** Real analyses hit surprises — a broken instrument, an unusable measure. Forbidding deviation guarantees silent deviation. Require that a change be recorded with the reason and the date, and that the report show both the planned and the executed analysis. - **Review at plan time, not only at result time.** Reviewing a design before data collection is where the effort actually improves the study; reviewing after the result mostly negotiates the story. - **Publish the nulls internally.** If a pre-specified analysis that finds nothing disappears from the record, the team has recreated the file drawer inside the company, and the internal knowledge base becomes as selected as any biased literature. ## The costs, named honestly A principal-level answer does not pretend this is free. - **Speed.** Writing a plan and holding back data adds days and reduces the effective sample for the confirmatory run. - **Fewer headline findings.** The visible output gets less impressive, and if leadership rewards striking results the discipline will be quietly abandoned. - **Genuine loss of some real effects.** Restricting confirmatory claims to pre-specified comparisons means some true findings sit in the exploratory bucket for a cycle before they can be confirmed. - **Overhead falls unevenly.** A large, expensive, decision-critical study justifies the full apparatus. A quick diagnostic look does not, and mandating identical rigour for both is how the policy loses credibility. ## Where I would set the threshold Scale the requirement to the stakes. Anything that will drive an irreversible decision, an external claim or a publication gets a full plan and a held-back confirmation. Routine internal analysis gets the lightweight version: name your primary outcome before you look, and label the rest exploratory. That asymmetry is what keeps the heavy process credible for the cases that need it. ## In an interview The signals being looked for are: you know that pre-specification is the fix rather than more sophisticated arithmetic; you know a plan only binds if it names outcomes, exclusions, model and decision rule; you protect exploration instead of criminalising it; and you can state what the discipline costs and who pushes back.