A causal forest says a discount lifts only price-sensitive new users while the overall effect is flat — do you roll out targeted?

## What the forest actually gave you A causal forest estimates `CATE(x)` by growing trees whose splits are chosen to separate units with different treatment effects rather than different outcomes, and by estimating the effect within each leaf from data that was not used to choose that leaf's splits. That second property is called honest estimation, and it matters more than the first. Without it, a split is chosen precisely because the treated-minus-control difference looked extreme among those particular observations, and then the same observations are used to report that difference. The reported effect inherits the selection, comes out exaggerated, and its interval comes out too narrow. Honesty breaks the loop by spending one subsample on the question 'where should the boundary go' and another on 'how big is the effect inside it'. The price is that each observation contributes to only one of those tasks, so honesty costs statistical efficiency in exchange for estimates you can quote. What honesty does *not* do is validate the segment. It disciplines the estimate given the discovered structure; it does not tell you that the structure will reappear next month or in a different cohort. ## The confirmation you actually need The check that settles the question is out-of-sample and prospective. Score a randomized holdout the model never touched, sort it by predicted effect, and compare the treated-minus-control gap among the top-ranked units with the gap among the rest. If the ranking carries real signal, the top group shows a materially larger incremental effect, with an interval that excludes the population average. A related check is calibration: regress the observed outcome differences on the out-of-fold predicted effects and see whether predictions of larger effects really correspond to larger realised effects, rather than to a constant. Second, check stability. Refit on a different time window and see whether the same covariates drive the top of the ranking. A segment that reshuffles between quarters is not a segment; it is the model tracking whatever noise was loudest. Third, apply judgment about the story. 'Price-sensitive new users respond to a discount' is mechanistically plausible, which raises the prior. A segment defined by an arbitrary hash bucket or a device-model string is more likely a proxy for something operational — a broken flow on one platform, a logging gap, a promotion that happened to overlap. ## The decision, not just the statistics Suppose the segment survives. Rolling out targeted is still a business decision with several moving parts. **Operational feasibility.** The segment must be computable at the moment the discount is offered, from features available then. A rule built on ninety-day aggregates cannot be applied to a user on day one. **Economics.** Compare the incremental margin from the targeted group against the discount cost plus the cost of building and maintaining the targeting rule and its monitoring. A statistically real segment can still be worth less than the engineering to serve it. **The flat overall number.** A flat overall effect with a strong positive segment implies something offsetting it. Look for the group being harmed before you launch; if it exists and is identifiable, excluding it is part of the policy, and if it exists and is not identifiable, that is a serious argument against rollout. **Shrinkage on contact with reality.** Selecting the top of a noisy ranking captures upward estimation error along with true effect, so the realised lift generally comes in below the predicted lift. Plan for that in the business case rather than treating the shortfall as a betrayal. ## How to roll out if you do Ship the targeted policy with a permanent randomized holdout inside the targeted segment, so the effect keeps being measured after launch rather than assumed. Set the targeting depth where incremental value still exceeds cost, and revisit it as the estimate updates. Define in advance what result would cause you to roll back — a holdout gap that shrinks below a threshold, or a negative reading on any monitored slice. ## The judgment being tested An interviewer at this level is watching for three things: that you do not accept a discovered segment as a result, that you know what honest estimation does and does not buy, and that you can convert a statistical finding into a decision with costs, feasibility and a rollback condition attached. Answering only the statistical half — or only the business half — is what separates a strong answer from an average one.