How do you handle a Bayesian efficacy readout whose conclusion flips between a sceptical and an enthusiastic prior?

## Why this is a leadership question When a readout is robust to the prior, nobody asks about priors. The question only arises when it is not, and at that point the statistics have already told you the important thing: **the data alone are not sufficient to decide**. Your job is to keep that fact visible rather than launder it into a single number. ## Pre-specify a community of priors The defensible structure is decided before the data are seen and written into the analysis plan. - **A sceptical prior**, centred on no effect, with a spread chosen so that the effect the study was designed to detect sits far out in the tail. It embodies the reviewer who thinks most promising results do not replicate. A common calibration is to set the spread so that the pre-data probability of seeing an effect as large as the design target is small — a few percent — and to state that rule in the plan rather than after the fact. - **An enthusiastic prior**, centred on the design effect, embodying the team that proposed the study. - **A weakly informative reference prior**, centred at no effect but wide enough to be nearly agnostic about size. Reporting the posterior under all three is a *prior sensitivity analysis*. The output is not one number but a map of which beliefs lead to which decisions. ## Reading the result **Agreement across the community.** Every prior a reasonable person could hold leads to the same decision. Say that explicitly — it is a stronger claim than any single posterior, and it forecloses the argument about whose prior was used. **Disagreement.** Two things follow. First, publish the flip rather than the winner: the conclusion depends on an assumption, and hiding that is the failure mode that destroys trust in Bayesian reporting inside an organisation. Second, locate the boundary. Report how sceptical a prior would have to be to reverse the call — equivalently, how much prior evidence against the effect the data can overcome. That converts an argument about philosophy into a quantity people can reason about: "you must already believe there is a 90% chance of no effect before this data leaves you unconvinced." ## What happens as data accumulate A large enough sample settles it. As observations pile up, the likelihood becomes sharply concentrated and swamps any prior that assigns nonzero density near the truth; posteriors from a sceptical and an enthusiastic prior converge toward the same place. By the time a readout carries 50,000 observations, two vague-to-moderate priors have become practically indistinguishable, and that is the expected behaviour. So if the conclusion still flips at that scale, the diagnosis is not "we need a better prior". It is one of: - **A near-dogmatic prior.** A prior with essentially zero density in the region the data support can never be updated into it. This is the practical form of the rule that a probability of zero stays zero forever, and it means someone specified a belief no evidence could touch. - **Weak identification.** The likelihood is nearly flat in the direction that matters — a parameter the design cannot separate from another — so extra rows do not add information about *that* quantity. The fix is design or model structure, not the prior. - **A model problem.** If the likelihood is misspecified, prior sensitivity is a symptom and re-tuning the prior treats the symptom. ## Governance, which is the real answer The technical steps are easy compared with the organisational discipline they require: 1. **Priors go in the plan, before unblinding.** A prior chosen after seeing results can be tuned to any conclusion, and a reviewer is entitled to assume it was. 2. **Informative priors must name their evidence.** Which earlier study, which population, why it transfers. "Domain expertise" without a citation is an opinion wearing a distribution. 3. **Defaults for routine work.** Most readouts should not be a negotiation. Fix a weakly informative default for the standard analysis, and require documented justification to depart from it. 4. **Sensitivity output is part of the deliverable**, not an appendix produced when challenged. 5. **Separate inference from decision.** The posterior is what you believe; the action depends on the cost of being wrong in each direction. A flip that matters under symmetric costs may be irrelevant when one error is ten times more expensive than the other, and saying so out loud often ends the argument faster than more statistics. ## What a strong answer sounds like It refuses the framing that one prior must win, pre-commits to a set, quantifies the tipping point, distinguishes a prior problem from an identification or model problem, and finishes on the decision-cost framing rather than on a posterior probability. A weak answer picks the prior that gives the cleaner story, or reaches for a flat prior as if that were neutral.