With 20 observations, a stronger prior halves your credible interval width - how do you report that?
answer
- small sample, prior carries weight
- count the prior in observations
- conjugate parameters add to the counts
- vary the prior and republish
- narrow is not the same as evidenced
basics
~20 sReport the prior explicitly and quantify its weight - a Beta prior contributes roughly its two parameters as pseudo-observations - then publish the interval under a ladder of priors. A narrow interval from a strong prior is an assumption, not evidence.
solid answer
~50 sThe narrowing is real arithmetic, not a bug: with 20 trials the prior is a large share of the total information. Take 12 successes in 20. Under a uniform prior the posterior is Beta(13, 9) and the 95% equal-tailed interval runs roughly 0.38 to 0.78. Under a Beta(40, 40) prior the posterior is Beta(52, 48) and the interval runs roughly 0.42 to 0.62 - about half as wide, and pulled toward 0.5. That prior is worth about 80 pseudo-observations against 20 real ones, so it is doing four fifths of the work. I would report the prior in the same sentence as the interval, state its effective sample size, and publish a sensitivity table across a weak, a moderate and a strong prior so a reader sees which conclusions survive. If only the strong-prior conclusion survives, the honest headline is that 20 observations do not settle the question.
go deeper
Remember the direction: a tighter prior gives a tighter posterior, and with few observations the prior can dominate. Be able to say that the interval depends on the prior at all.
Explain the conjugate mechanics - prior parameters adding to observed counts - and convert a Beta prior into a number of pseudo-observations you can compare against the real sample size.
Show the reporting protocol you would actually run: declare the prior, quantify its weight, publish a sensitivity ladder, and state which conclusions survive across all of it.
Own the standard that stops prior shopping across the organisation: priors registered before analysis where possible, provenance documented, and sensitivity tables required in any readout that drives a decision.
## Why the width moves at all The posterior is proportional to likelihood times prior. With a small sample the likelihood is diffuse, so the prior contributes a large share of the final shape - including its spread. A tight prior therefore produces a tight posterior and a narrow credible interval, whatever the data say. With a large sample the likelihood dominates and priors that differ only mildly give near-identical intervals. ## A concrete case Suppose 12 successes in 20 trials, modelled as independent Bernoulli draws with unknown rate `p`, and a Beta prior. The Beta family is conjugate for this likelihood: a Beta(a, b) prior with `k` successes in `n` trials gives a Beta(a + k, b + n - k) posterior. - **Uniform prior**, Beta(1, 1): posterior Beta(13, 9). Posterior mean about 0.59, 95% equal-tailed interval roughly `[0.38, 0.78]`, width about 0.39. - **Concentrated prior** centred at a half, Beta(40, 40): posterior Beta(52, 48). Posterior mean about 0.52, 95% equal-tailed interval roughly `[0.42, 0.62]`, width about 0.20. The second interval is about half as wide and is pulled toward 0.5. Nothing about the data changed. ## Effective prior sample size For this conjugate pair the prior parameters add directly to the counts, so `a + b` behaves like a number of prior pseudo-observations. Beta(1, 1) is worth about two; Beta(40, 40) is worth about eighty. Against 20 real trials, the strong prior supplies roughly four fifths of the total information in the posterior. That single number - prior pseudo-observations versus real observations - is the most useful thing you can put in a report, because it converts a vague worry into a ratio a reader can judge. Analogous notions exist for other conjugate pairs: a Gaussian prior on a mean with variance `sigma^2 / m` behaves like `m` prior observations. ## How to report it 1. **Name the prior wherever the interval appears.** "95% credible interval [0.42, 0.62] under a Beta(40, 40) prior" is a complete claim. The bare interval invites the reader to attribute all the precision to the data. 2. **State the effective prior sample size** next to the real sample size. Eighty against twenty is a sentence, not a footnote. 3. **Run a sensitivity analysis.** Recompute the interval under a small ladder of priors - a flat or weakly informative one, the one you actually believe, and a deliberately adversarial one that leans the other way - and publish the table. This is the standard, expected move; treating it as an admission of weakness is a mistake. 4. **Report what survives.** If every prior in the ladder puts the interval above the decision threshold, the conclusion is robust and you can say so plainly. If only the strong prior gets you there, the finding depends on the assumption, and the headline should be that 20 observations are not enough. 5. **Justify a strong prior by its provenance,** not by convenience: historical results on the same surface, a hierarchical fit across many similar units, a physical or contractual bound. "It made the interval narrow enough to ship" is not a justification. ## The trap to avoid Prior shopping - trying priors and keeping the one whose interval supports the desired call - is the Bayesian analogue of any other selective-reporting practice, and it is exactly what the sensitivity table prevents. Fix the prior before seeing the data where you can, and where you cannot, publish the full ladder rather than the winner. The converse error is treating any informative prior as illegitimate. Refusing to use knowledge you actually have wastes information and, on a small sample, gives you an interval so wide it supports no decision at all. A weakly informative prior that merely rules out absurd values - a conversion rate above 50% on a checkout page, say - is usually better practice than a flat prior, and it should still be declared. ## What good sounds like in interview Explain the mechanism (small `n` means the prior is a large share of the information), quantify it (pseudo-observations against real observations), then give the reporting protocol (declare, quantify, sensitivity ladder, report what survives). Close by naming the failure mode you are guarding against: a narrow interval read as strong evidence when it is mostly an assumption.
- How would you quantify how much the prior is contributing?For a Beta prior on a rate, the parameters add straight to the success and failure counts, so `a + b` acts as a number of prior pseudo-observations. Compare that with the real sample size: Beta(40, 40) against 20 trials means eighty pseudo-observations to twenty real ones. Report the ratio, not just the interval.
- What would make you defend the strong prior rather than widen the interval?Provenance. If it encodes many past experiments on the same surface, a hierarchical fit across comparable units, or a hard physical or contractual bound, it is information and using it is correct. If it was chosen after seeing that a weaker prior gave an inconvenient interval, it is not defensible at all.
- How much data would it take before the choice stops mattering?Roughly when the real sample size dwarfs the prior's effective sample size. With a Beta(40, 40) prior worth about eighty pseudo-observations, a few hundred trials already shrink the difference between priors to something below reporting precision, and a few thousand make it negligible.
A confident prior is a co-author who has already written most of the paragraph; the 20 data points only edit it. Say who wrote which part.
saying these in an interview costs you the question
- Presents the narrow interval as though data alone produced it
- Chooses the prior after seeing which interval it yields
- Treats a sensitivity analysis as an admission of weakness
- Claims a prior can never be overwhelmed by more data
- Insists only flat priors are ever legitimate