You slice a 3% drop across 40 country-by-platform cells and one is down 30% — what now?

## Two independent traps in one dashboard When a metric drops and you open a breakdown, two different failure modes are waiting. One is statistical: scanning many cells manufactures extreme-looking cells even when nothing is wrong. The other is arithmetical: the most extreme percentage often belongs to a cell far too small to have caused the aggregate move. A good answer addresses both, because fixing only one still leads to the wrong conclusion. ## Trap 1 — scanning manufactures extremes Every cell's observed change is its true change plus noise. Look at one cell and you rarely see a large noise excursion. Look at 40 and you are effectively asking for the *maximum* of 40 noise draws, which is systematically large. This is why an exploratory breakdown almost always produces something alarming, and why the alarm is not evidence in the way a pre-stated hypothesis would be: the cell was chosen *because* it was extreme, and that selection is exactly what makes the number untrustworthy. The pattern is worse when the slicing itself is improvised — try country, then platform, then country by platform, then add app version, and stop when a story appears. This is the garden-of-forking-paths problem: with enough branches, an alarming path exists in almost any dataset. Formal statistical adjustments for testing many slices at once are their own topic; the practical defences on the diagnosis side are cheaper and just as important: decide the breakdown dimensions and a minimum cell size before looking, look at the whole set of cells rather than the winner, and treat anything found by scanning as a lead that must be confirmed independently. ## Trap 2 — small cells swing hardest Relative sampling variation scales roughly with one over the square root of the cell size. A cell with 100 users has about ten times the relative noise of a cell with 10,000, because the square root of 10,000 divided by 100 is 10. So even in a completely healthy product, the smallest cells produce the biggest weekly percentage swings, purely mechanically. Sorting a breakdown by percentage change is therefore close to sorting it by inverse cell size, and it puts your smallest, least informative cells at the top of the screen every single time. The corrective is to look at cells against their own history: is a 30% swing unusual *for this cell*, given how it moves in normal weeks? A cell that routinely oscillates by 25% has not told you anything by moving 30%. ## The sizing check Before any narrative, do the multiplication. A cell holding share s of the volume and falling by p percent removes about s times p from the total. Worked on the numbers here: 0.5% of volume times a 30% fall is 0.15 percentage points of the aggregate. Against a 3 percentage point drop, that is 5% of the movement. Whatever happened in that cell, 95% of the drop happened somewhere else, so the investigation is not finished — it has barely started. This single multiplication kills most exciting-looking drilldown findings in seconds, and it reframes the search productively: to explain a 3% aggregate drop you need either a large cell moving moderately, or a moderate cell moving enormously, or a broad move spread across many cells. Sorting the breakdown to surface those candidates finds causes; sorting by percentage change finds the smallest cell you have. Attributing the full delta across all segments in a rigorous, additive way is a separate exercise with its own conventions; the sizing check here is a sanity filter, not that attribution. ## Confirming a lead When a cell survives both checks — large enough to matter, and moving far more than it normally does — confirm it before you report it: - **Persistence.** Does the cell stay down over subsequent days or weeks? Noise regresses; a real break persists. Two follow-up periods make a chance finding much less likely and cost nothing but patience. - **Mechanism.** Is there a plausible cause tied to that cell — a release on that platform, a payment provider or carrier in that country, a partner outage, a regulatory or pricing change? A finding with a mechanism you can point at is far stronger than one without. - **Coherence.** Do related metrics in the same cell move consistently? If sessions dropped but downstream conversions and revenue in that cell are untouched, the cell's drop may be a measurement artefact rather than a behavioural one. ## Reporting honestly The language matters when the finding reaches a decision meeting. 'We scanned 40 breakdowns and this was the most extreme' is a very different claim from 'we predicted this cell would be affected and it was', and stakeholders cannot tell them apart unless you say which one you did. Label exploratory findings as exploratory, state the cell's share of volume next to its percentage change, and give the confirmation status. That habit costs one sentence and prevents the expensive version of this mistake: an engineering team spending a week on a cell that was never big enough to matter.