You re-run a customer segmentation quarterly and 30% of customers changed segment - is that a problem?
answer
- two clocks, not one
- did the customers move or the definitions?
- look at where they moved, not how many
- customers near a boundary flip for free
- a re-fit is a versioned release
basics
~20 sNot by itself. Customers moving between segments is the segmentation working, as long as the segment definitions were held fixed. The expensive problem is re-fitting definitions each run, so every segment name silently changes meaning.
solid answer
~50 sThe first question is what moved: the customers, or the definitions. If the segment boundaries were held fixed and 30% of customers crossed one in a quarter, that is the signal you built the thing to see - people lapse, recover and grow, and a transition matrix of old segment against new tells you which flows matter. If instead the segmentation was re-fitted from scratch, then `Champions` may simply mean something different this quarter, and every campaign, dashboard and target keyed to that name has shifted without anyone deciding to. So I run two clocks: score membership often against frozen definitions, and re-fit the definitions rarely and deliberately - when a segment's profile no longer matches its name, when the business changes, or on an annual cycle. A re-fit is a versioned release with old-versus-new profiles and a cutover date.
go deeper
Be ready to say that customers changing segment is expected behaviour, not a bug, and that the segmentation exists to detect exactly that. Knowing that re-running the model and re-assigning customers are two different actions is the key point.
Explain the two cadences and why scoring against frozen definitions keeps movement interpretable. Be able to describe a transition matrix and what a healthy versus scattered pattern looks like.
Show the operational judgment: hysteresis for boundary cases, trigger-based rather than purely calendar-based re-fitting, and running a re-fit as a versioned release with overlapping assignments and a cutover date.
Own the tradeoff between freshness and shared vocabulary. Segment names end up in targets and dashboards, so the cost of re-briefing every consumer is what should set the cadence, and you decide who is allowed to change a definition.
## Two clocks, not one The single most useful idea here is that a segmentation has two independent cadences and conflating them causes most of the confusion: - **Scoring cadence** - how often each customer is re-assigned to a segment using the *existing* definitions. This can be monthly, weekly or nightly, as fast as the input data refreshes. It is cheap and it is the point of the exercise. - **Re-fitting cadence** - how often the definitions themselves are re-derived, which changes what each segment *means*. This should be rare, deliberate and versioned. Running them as one thing - a quarterly re-fit that also re-assigns everyone - makes it impossible to tell whether the customer base changed or the ruler did. ## Is 30% movement in a quarter alarming? Usually not, and in a retail base it is unremarkable. A single purchase moves a customer across a recency boundary. Seasonality shifts a large fraction of the base at once. Newly acquired customers have to land somewhere and then settle. The segmentation was built precisely to detect these transitions, so their presence is not evidence of a defective model. What matters is *where* people moved. Build a transition matrix: rows are last quarter's segments, columns are this quarter's, cells hold customer counts. A healthy picture shows a heavy diagonal with interpretable off-diagonal flows - Champions drifting toward At-Risk, Hibernating customers reactivating after a campaign. An unhealthy picture shows movement scattered roughly uniformly across all cells, which suggests assignments are close to arbitrary. The transition matrix is also the most actionable artefact the re-run produces: the flow out of the top segment is a retention brief with names attached. ## Boundary flapping Some movement is real and some is an artefact of drawing a line. Customers sitting close to a boundary flip back and forth between two segments each period without changing behaviour meaningfully, and if segment membership drives contact rules, those people get treated inconsistently for no reason. Two practical mitigations: require a customer to satisfy the new segment's condition for two consecutive periods before re-labelling them (hysteresis), or attach a distance-to-boundary or confidence flag so downstream consumers can choose to ignore marginal cases. Neither changes the model; both make the output usable. ## Why definition drift is the expensive kind Segment names propagate. They end up in campaign audiences, dashboards, executive narratives, retention targets and sometimes compensation. If a re-fit quietly changes what `At-Risk` means, then a quarter-over-quarter chart of At-Risk headcount is comparing two different populations, and nobody looking at it knows. That is why re-fitting is a release, not a refresh. ## Triggers for a re-fit Rather than re-fitting on a calendar alone, watch for causes: - **The profile stopped matching the name.** The group called Champions now has median spend near the base average. The label has decayed and needs redrawing. - **The business changed.** New pricing tiers, a new product line, entry into a new market, or a change to how the input variables are recorded. Any of these can make the old geometry meaningless. - **Segment sizes have drifted far from launch.** A segment that was 8% of customers and is now 24% is describing something different from what was signed off. - **A large share of customers now sit far from every segment's centre.** The population has moved into territory the definitions never covered. - **The calendar.** An annual refresh as a backstop is reasonable, because absence of triggers can also mean nobody is watching. ## Treat a re-fit as a versioned release When you do re-fit: publish the old and new profile tables side by side, provide a mapping showing where each old segment's members landed, keep the old assignment available for one overlapping period so dashboards can be reconciled, announce a cutover date, and version the definition (`v2`, effective from a stated date) so any historical analysis can state which definition it used. Re-briefing every downstream consumer is the real cost of a re-fit, and that cost - not model freshness - is what should set the cadence. ## Cadence follows purpose A strategic segmentation that shapes planning and org structure should be stable for a year or more; its value comes from everyone using the same vocabulary. An operational segmentation driving next-week's campaigns can re-score continuously and tolerate more churn, because nothing downstream depends on the labels lasting. Decide which one you are building before arguing about how often to refresh it.
- How do you tell whether the customers moved or the definitions moved?Score the new period's data against the previous definitions, held fixed, and compare that assignment with the freshly re-fitted one. If most of the movement disappears when the old definitions are used, the ruler changed rather than the customers. A second check is qualitative: read each new segment's profile against the name it inherited and see whether the description still fits.
- What does a transition matrix between two runs give you?Rows are the previous segments, columns the current ones, cells hold counts. It shows whether movement follows interpretable paths - top-tier customers drifting toward at-risk, dormant customers reactivating - or is scattered roughly uniformly, which suggests assignments are close to arbitrary. It is also directly actionable: the flow out of the most valuable segment is a retention brief with names attached to it.
- What would trigger an unscheduled re-fit before the annual one?A business change that invalidates the old geometry - new pricing tiers, a new product line, a change in how an input is recorded - or a segment whose profile has drifted so far that its name no longer describes its members. A large share of customers sitting far from every segment centre is the same signal seen from the data side.
- Some customers flip between two segments every month - how do you handle it?They are sitting near a boundary, so tiny changes cross it. Require the new condition to hold for two consecutive periods before re-labelling, or publish a distance-to-boundary flag so campaign owners can exclude marginal cases. Both keep contact treatment consistent without touching the model, which is the actual complaint behind the observation.
Two clocks: the odometer updates every trip, while the map is redrawn only when the roads actually change. Re-drawing the map every trip makes every previous journey unmeasurable.
saying these in an interview costs you the question
- Re-fits the definitions every run and renames segments each time
- Treats any membership movement as evidence of a broken model
- Never re-fits, so segments describe a market that no longer exists
- Ships new definitions without telling downstream consumers
- Compares segment headcounts across two different definitions