How do you decide whether a KPI computed only over still-active users is trustworthy enough to publish?

## The problem in one line A metric whose denominator is "users who are still here" has a survivorship filter built into its definition. The population it describes changes every period, and it changes in a way that is correlated with the metric itself — the users who leave are systematically the least engaged. That makes it an internal, self-inflicted version of the same bias that afflicts a table of surviving funds. ## The canonical failure Average session length among monthly active users rises 10% quarter over quarter. Leadership reads it as an engagement win. In fact no individual user changed behaviour at all; the bottom slice of light users stopped qualifying as active, and removing the low values from a mean raises it. The metric moved because the population moved. Worse, the same mechanism guarantees the metric *improves* fastest in the quarters where churn is worst, so it can be highest exactly when the business is doing worst. The general form: any statistic `mean(Y | still active)` is a conditional mean over a population defined by an outcome downstream of `Y`. Its trend confounds two things you need separately — how individuals changed, and who remained. ## The decision framework **1. State the population the number claims to describe.** Write it down in one sentence before anything else. "Average weekly sessions among users active this week" is honest; "average weekly sessions" is not, and the second is what usually ends up on the slide. Half of these failures die at this step. **2. Ask whether attrition can move the metric in the flattering direction.** If churned users sit systematically at one end of the distribution — and for engagement metrics they always sit at the low end — the metric drifts upward for free. If churn is unrelated to the metric, the exposure is small and a labelled survivor-conditional number may be fine. **3. Build the cohort-anchored companion.** Fix membership at the start of the window: everyone who signed up in month M, or everyone active at the start of the quarter. Follow all of them forward, and give the departed an explicit value rather than deleting them — usually zero for a volume metric, since zero sessions is what they actually did. The cohort-anchored series answers "did the people we had get better?", which is the question the reader thought they were asking. **4. Bound the uncertainty rather than pretending to remove it.** Recompute the metric twice: once with departed users held at their last observed value, once with them at the floor. These are deliberately crude sensitivity bounds, not a correction. If the conclusion holds under both, publish with confidence. If it flips, you have learned that your headline is a churn artefact. **5. Decide what you will not fix.** Some survivor-conditional metrics are genuinely the right measure. Satisfaction among current customers is a legitimate operational number for the support organisation; it just cannot be used to argue that the product is improving overall. The principal-level call is separating metrics that need correcting from metrics that need labelling. ## Making it stick organisationally - **Encode the population in the metric name.** If the definition conditions on survival, the name says so. A naming convention costs nothing and prevents the slide-deck version of the mistake, which is where the real damage happens. - **Default the metric layer to cohort anchoring.** Where the semantic layer or metric store lets you define a metric once, make the safe definition the easy one, so an analyst has to opt in to the survivor-conditional variant. - **Publish the denominator next to the number.** A metric shown without its population size hides exactly the movement that causes the artefact; when both are on the chart, a reader can see the denominator shrinking. - **Review at definition time, not at incident time.** The cheapest place to catch this is when a metric is proposed. Once a survivor-conditional number has been in a board deck for four quarters, correcting it looks like a regression and is politically expensive — which is itself an argument for getting it right early. ## The tradeoff to own Cohort-anchored metrics are less flattering, slower to move and harder to explain, and they will make quarters look worse than the survivor-conditional versions did. That is the cost, and it is real. The counter-cost is a leadership team steering on a number that improves when customers leave. A lead's job is to accept the first cost explicitly, in exchange for metrics whose direction can be trusted — and to say so before the numbers get worse, not after.