How do you defend an adjustment set when the DAG behind it cannot be verified from data?

## The uncomfortable fact An adjustment set is valid *relative to a graph*. Change the graph and the set changes. The graph itself is not estimated from the data in any strong sense: many distinct graphs imply the same conditional independencies, and the direction of most arrows is not recoverable from association at all. So the honest statement is that your causal estimate rests on assumptions that the data can refute in a limited way and can never confirm. A lead who hides this loses credibility the first time someone asks "how do you know you controlled for everything?". A lead who states it and then shows what they did about it is doing the job. ## 1. Provenance: where the arrows came from The strongest defence of a graph is that it describes a mechanism someone actually built. Ask who decided which units got treated, and get the rule. In most business settings the assignment mechanism is a documented process — a targeting rule, an eligibility threshold, an operational policy — and its inputs are precisely the variables that sit on backdoor paths. Recovering those inputs converts a guess into a description. Record the graph as an artifact, with a note on each contested edge: who asserted it, on what basis, and what would change their mind. This turns future disagreement into a specific, reviewable claim about one edge rather than a general argument about whether causal inference works. ## 2. Refutation: run the tests the graph offers A DAG implies a set of conditional independencies, one for each d-separation. Those are testable. If the graph says two variables are independent given a third and the data show a strong association within levels of that third variable, the graph is wrong somewhere along the paths involved. Be precise about what passing means. Passing does not validate the graph, because rival graphs imply the same independencies and because absence of association can also arise from cancelling paths or from insufficient power. Report these as refutation attempts that the graph survived, not as verification. ## 3. Robustness across rival graphs This is the part most teams skip and the part that most improves a defence. Do not analyse one graph. Write down the two or three graphs that competent, informed people would actually draw for this system — usually they differ over one or two contested edges. Derive the adjustment set implied by each. Estimate under each. The result is far more informative than a single number: - If the estimate is stable in sign and roughly in magnitude across all of them, the conclusion does not depend on the disputed edge, and you can say so precisely. - If it flips, you have identified the exact assumption the decision hinges on, and you can direct effort at that one edge — measure a variable, run a small experiment, or escalate the disagreement to whoever owns the mechanism. This is also the antidote to graph-shopping. Committing the candidate graphs before estimation, ideally in writing, is what stops the analysis from quietly converging on whichever adjustment set gives the desired answer. ## 4. Sensitivity: how wrong would you have to be Assume the graph is missing an unmeasured common cause of treatment and outcome. The question is not whether one exists — one usually does — but how strong it would have to be to matter. Sensitivity analysis expresses the estimate as a function of the strength of that hypothetical confounder's relationships with treatment and with outcome, and reports the combination that would move the estimate past the threshold the decision actually turns on. That framing is decision-relevant. "An unmeasured factor would need to shift both treatment and outcome more strongly than any variable we measured in order to overturn this" is a defensible claim. "We controlled for everything we had" is not. ## 5. Say what the analysis cannot support Several conclusions should be refused outright: - An effect whose backdoor path runs entirely through an unmeasured variable is unidentified. Report it as such rather than shipping an adjusted number with a caveat nobody reads. - An adjustment set justified for one treatment does not justify reading the other coefficients in the same model as causal; each variable's identification requires its own graph argument. - Where the decision is high-stakes and reversible by design, the right answer is often to stop arguing about the graph and buy identification with randomisation or a holdout. ## 6. Make it an organisational habit At lead level the deliverable is not one defensible analysis but a standard: causal claims ship with the graph, the adjustment set derived from it, the refutation tests run, the rival graphs considered, and the sensitivity statement. Reviews then argue about edges, which is productive, instead of about estimators, which is usually beside the point. The most valuable long-run outcome is that teams start logging the inputs to their assignment mechanisms, because that is what makes the next graph defensible cheaply.