Why use a log-rank test on time-to-cancel instead of comparing day-30 cancellation rates between two onboarding flows?

## What the test asks The log-rank test is a hypothesis test for the null that two groups have **identical survival distributions** over the whole follow-up period. It is not a test of a point estimate or a single milestone; it is a comparison of two curves. ## The mechanism Walk the pooled event times in order. At event time `t_i`: - `n_i` subjects are at risk overall, of which `n_Ai` belong to group A; - `d_i` events occur, of which `d_Ai` are in group A. Under the null hypothesis, the group membership of the events at `t_i` is just a random split of `d_i` among the `n_i` subjects at risk, so the expected number of group-A events is ``` E_Ai = d_i * ( n_Ai / n_i ) ``` with a variance given by the hypergeometric distribution of that split. Sum across event times to get `O_A = sum d_Ai`, `E_A = sum E_Ai`, and `V = sum of the per-time variances`. The statistic ``` (O_A - E_A)^2 / V ``` is referred to a chi-square distribution with one degree of freedom when there are two groups (`k - 1` degrees of freedom for `k` groups). A large positive `O_A - E_A` means group A produced more cancellations than an identical-curves world predicts. Two structural features fall out of this. First, censoring is handled through the risk sets: a subject who is still active contributes to `n_i` and `n_Ai` at every event time up to its censoring time and then leaves, contributing no event. Nobody is dropped and nobody is coded as a cancellation. Second, only the **ordering** of the event times matters, not their numeric values — the test is non-parametric and assumes no distributional shape. ## Why a single-day rate is weaker **It discards timing.** Two flows can produce identical day-30 cancellation percentages while one bleeds users in the first 72 hours and the other loses them slowly across the month. Those are different product problems with different fixes, and the point comparison cannot separate them. The log-rank statistic accumulates the discrepancy at every event time, so an early divergence registers even if the totals converge later. **It forces a censoring decision you do not want to make.** To compute a day-30 rate you need each account to have 30 days of history. Accounts younger than that must either be excluded — which throws away your most recent and often most relevant users, and delays every read by a month — or counted as retained, which is a fabrication. Survival analysis takes them as right-censored and uses the exposure they do have. **It wastes information and therefore power.** Every cancellation carries a time, and the point comparison collapses all of them into a single binary. Given the same data, the whole-curve comparison detects a real difference with a smaller sample. ## Where the log-rank is weak - **Crossing curves.** The test sums signed discrepancies. If flow A is worse early and better late, the positive and negative contributions cancel and the total can land near zero even though the curves are visibly different. Always plot before testing; a crossing pattern calls for a landmark comparison or a split-window analysis instead of a single p-value. - **Differences concentrated where nobody is left.** A gap that opens only in the thin tail carries little weight because the risk sets there are tiny. - **Unequal censoring patterns.** The test assumes censoring is unrelated to the risk of the event and behaves similarly in both groups. If one flow's users disappear from the dataset for reasons tied to their likelihood of cancelling, the comparison is contaminated. - **It returns no effect size.** The output is a p-value. Pair it with something a stakeholder can act on: the two Kaplan-Meier curves with intervals, a difference in survival at a pre-specified landmark, or a difference in restricted mean event-free days over a fixed horizon. ## Weighted variants The standard log-rank weights every event time equally. Weighted variants exist that emphasise early event times — the Gehan-Breslow generalised Wilcoxon test weights each time by the size of the risk set, so differences observed while many subjects remain count for more. Choosing the weighting after seeing which one gives a smaller p-value is a multiplicity problem; fix it in advance based on where you believe a difference should appear. ## In an interview Say what the null is, describe the observed-minus-expected accumulation over risk sets, explain how censored subjects participate, and then name the two honest caveats: crossing curves and the absence of an effect size. Candidates who describe the log-rank as "a t-test for survival times" have missed both the censoring handling and the rank-based construction.