What policy should govern segment reporting in A/B test readouts across an experimentation team?

## Why this needs a policy at all Segment reading is the most common route by which an experimentation programme starts producing wins that do not show up in the business. Individual analysts rarely intend to mislead; the pathology is structural. A flat result is unwelcome, slicing is free, tooling makes thirty cells available in one click, and nobody records how many cells were looked at. A policy is how you make the honest path the default path rather than an act of personal discipline repeated under deadline pressure. ## A three-tier readout **Tier 1 — the decision.** The primary metric, all-up, with its interval. This is what launches or does not launch. Nothing in the tiers below overrides it. **Tier 2 — pre-declared segments.** A short list, agreed at design review, each entry carrying: the cut, the mechanism you expect, and a sample plan showing the segment has enough traffic to detect an effect worth acting on. Short is load-bearing — three declared cuts you can power beats fifteen you cannot. Tier 2 results may qualify the decision: gate a launch to a segment, hold back a rollout, trigger a follow-up. **Tier 3 — exploration.** Everything else, shown openly, in a section labelled exploratory, with **the number of cuts examined printed at the top**. Tier 3 generates hypotheses. By policy it cannot carry a launch on its own. ## The promotion rule The policy lives or dies on what it takes to move a Tier 3 observation into a decision. A workable rule: a confirmatory experiment, the segment declared in advance, powered for that segment specifically, with the decision rule fixed before the run. Expect a real effect to come back smaller than its discovery estimate, because the discovery was selected for being extreme; the confirmatory number is the one you forecast with. Tune the bar by stakes rather than applying one threshold everywhere: - **Cheap and reversible** (a copy variant shown to one segment) — a weaker bar is rational, because being wrong costs a rollback. - **Expensive or sticky** (a permanently divergent experience, a pricing rule, a headcount commitment) — insist on the full confirmatory run, because unwinding costs quarters. ## Making it cheap to comply Policies that depend on virtue decay. Build the rule into the tooling: - The experiment template asks for pre-declared segments at design time, before a launch button appears. - The analysis surface logs which cuts were pulled, and the readout renders the count automatically. Nobody has to remember to disclose. - The readout template ships with the three tiers as sections, so an exploratory finding physically cannot appear in the decision block. - Segment reads carry intervals by default, so a cell too small to say anything looks visibly uninformative rather than exciting. ## The tension to own A principal-level answer names the cost on both sides. **Too strict** and you suppress real heterogeneity: treatment effects genuinely do vary across users, and a programme that never looks will never find the segmentation that matters, while analysts route around a rule they find unreasonable and do the slicing in private spreadsheets — which is worse than doing it in the open. **Too loose** and every flat experiment yields a segment win, the roadmap fills with launches whose promised lift never materialises, and within a year leadership stops believing experiment readouts at all. That loss of credibility is the expensive failure, because it removes the programme's ability to stop bad ideas. The resolution is not a stricter threshold; it is a clear *status* distinction plus a cheap promotion path. Exploration stays fully permitted and fully visible; only its authority is constrained. ## Cultural levers - **Make flat results publishable.** If a team is judged on wins, they will find one in the grid. Review rituals that treat a well-run flat experiment as a good outcome remove most of the pressure that causes segment fishing. - **Review readouts, not just designs.** Design review catches unpowered segments; readout review catches the ones added later. - **Track the promotion record.** Count how many Tier 3 findings later replicated. A team that sees its own hit rate — often startlingly low — self-corrects faster than any policy document achieves. ## What an interviewer is listening for They want to hear that you separate *permission* from *authority*, that you have an operational promotion rule rather than a slogan, that you would build it into tooling rather than training, and that you can articulate the cost of over-correcting. A candidate who simply says "we should not p-hack" has not answered the question; the job is designing the system in which the right behaviour is also the easy one.