What does a sensitivity analysis for unmeasured confounding tell you about a causal estimate?

## The assumption you cannot test Every causal estimate from non-randomised data leans on the claim that treated and untreated units, once you adjust for what you measured, are comparable in everything else that matters. If some variable outside the dataset pushes both who gets treated and what outcome they have, the estimate is biased. Because that variable is unmeasured, no test on the observed data can confirm or refute its existence. A goodness-of-fit statistic, a narrow confidence interval, or a large sample tells you nothing about it - more data makes a confounded estimate more precisely wrong. Sensitivity analysis is the standard response. Rather than pretending the assumption is verified, it asks a quantitative what-if: suppose a hidden variable exists with specified strength; what would the corrected estimate be? Sweeping over strengths produces a tipping point - the smallest hidden bias that would move the estimate to the null or across a decision threshold. ## The historical template The method traces to the smoking and lung cancer debate. Critics argued the observed association could reflect a constitutional factor that made some people both smoke and develop cancer. Cornfield and colleagues answered arithmetically rather than rhetorically: for a hidden factor to fully explain a risk ratio near 9, that factor would have to be at least nine times more prevalent among smokers than non-smokers, and be essentially a perfect predictor of the disease. No plausible candidate came close. That is the whole idea in one move - the finding was not proved, but the escape route was priced and found absurdly expensive. ## What the output looks like A sensitivity analysis produces one of three shapes of statement: - **A tipping point on bias strength.** `The effect goes to zero only if an unmeasured variable is associated with both treatment and outcome by a factor of at least 3.` - **A bounded estimate.** `Under confounding no worse than a stated magnitude, the effect lies between 4 percent and 11 percent.` - **A comparison to known covariates.** `The required confounder would need to be stronger than age, which is the strongest predictor in the model.` This last framing is the most persuasive, because it benchmarks the hypothetical against something the audience already understands. ## What it does not do Three misreadings are common and each is disqualifying in an interview. First, a robust sensitivity analysis does not prove the effect is causal. It proves only that weak confounding cannot explain it. A strong confounder may still exist. Second, a fragile result is not thereby shown to be confounded. Fragility means the finding does not survive scrutiny, not that a hidden variable was found. Third, sensitivity analysis is not a fix. You cannot adjust away an unmeasured confounder; you can only characterise how much of one it would take. If the tipping point is low, the honest conclusion is that the design cannot answer the question and you need a better one - an experiment, a natural experiment, or measurement of the suspected variable. ## Where it sits in a credibility argument Sensitivity analysis is one leg of a three-legged defence of an observational estimate. The second leg is falsification: run the same estimator where the answer must be zero - a placebo period before the intervention could have acted, or an outcome the treatment cannot possibly affect - and confirm you get nothing. The third is external checks: does the sign and rough magnitude agree with a randomised result, a different data source, or a different identification strategy? Sensitivity analysis quantifies the residual doubt; falsification tests catch outright broken estimators; replication addresses idiosyncrasy. A practical reporting discipline follows. State the estimate, state the tipping point, name the two or three concrete variables that could plausibly play the confounder role, and say for each whether it could realistically reach the required strength. That is a defensible paragraph. `We adjusted for all available covariates` is not, because it says nothing about what was unavailable. ## Interview framing When asked to defend an observational number, the strongest opening is to concede the limit immediately and then price it: `This is not randomised, so I cannot rule out hidden confounding. What I can say is that a confounder would have to be about this strong to erase it, and here is why I think nothing in this domain is.` That answer shows you understand both the ceiling of the method and how professionals work within it.