How do covariate shift, label shift and concept drift differ in what they break?

## One joint distribution, two factorisations Everything supervised learning does concerns a joint distribution `p(x, y)` over inputs and labels. It factorises two ways: - `p(x, y) = p(y | x) * p(x)` — the **discriminative** view: a marginal over inputs, and the labelling rule. - `p(x, y) = p(x | y) * p(y)` — the **generative** view: a class prior, and what members of each class look like. The three named failures are just "which factor moved, and which stayed put". ## Covariate shift: p(x) moves, p(y|x) holds The input population changes; the rule linking inputs to outcomes does not. Canonical case: a consumer-credit scorecard fitted on 2019 applicants is scoring 2021 applicants after the lender widened its marketing. The new applicants skew younger, thinner-file and lower-income. But an applicant with a given set of attributes still defaults at the same rate as an identical 2019 applicant would have — the risk relationship is unchanged, only who shows up is different. What this does: - The learned function is **not wrong**. If you had fitted the true `p(y | x)` perfectly, nothing would need fixing. - Real models are misspecified and fitted on finite data, so they are accurate where training density was high and shaky where it was thin. Shifting mass into the thin regions surfaces exactly those errors. - The **old held-out score no longer describes the new population**, because it averaged errors under the old input mix. Where the two populations overlap, you can re-weight the old held-out rows by the ratio of new-to-old input density to re-estimate the score under the new mix; where the new population reaches inputs the old data never covered, nothing can be re-weighted, because there is no old evidence there at all. ## Label shift: p(y) moves, p(x|y) holds The class balance changes; what each class looks like does not. This is natural when the label causes the observation rather than the other way round. Canonical case: a diagnostic model trained when a condition affected 2% of patients is deployed during an outbreak when it affects 15%. A patient who genuinely has the condition presents the same way as before — `p(x | y)` is stable — but the base rate is seven and a half times higher. What this does: - **Ranking survives.** Because the per-class feature distributions are unchanged, the true-positive and false-positive rates at any fixed threshold are unchanged, so the ROC curve and ROC-AUC are the same. - **Calibrated probabilities do not survive.** A model that learned the 2% prior systematically under-predicts risk at 15%. If the model outputs calibrated probabilities, you can correct it by multiplying each class probability by the ratio of new prior to old prior and renormalising — a cheap fix, provided you can estimate the new prior. - **Precision-flavoured metrics move on their own.** Precision, positive predictive value and the precision-recall curve all depend on prevalence, so a rising base rate raises precision at a fixed threshold with no change in the model whatsoever. Reporting that as an improvement is a classic error. ## Concept drift: p(y|x) moves The labelling rule itself changed. The same input now maps to a different outcome distribution. Canonical case: a content-moderation model where the platform rewrites its policy mid-year. The posts look identical; what counts as a violation is different. Historical labels now describe a rule that no longer exists. What this does: - The model is **wrong**, not merely evaluated on the wrong population. There is no reweighting scheme that repairs it, because there is no weighting of the old inputs that reproduces the new labelling rule. - Old labelled data becomes actively misleading for training: it teaches the superseded rule. - Recovery requires **new labels under the new rule**, and the practical question becomes how quickly you can obtain them and how much history you can still reuse. Drift also comes in gradual forms — fraud tactics evolving, language use changing — where the rule erodes rather than flips on a date. The date-stamped policy rewrite is the easy case precisely because you know exactly when the old labels stopped being valid. ## Diagnosing which one you have A useful ordering in an interview: 1. Did the **inputs** move? Compare the input distributions of the old and new populations, feature by feature and jointly. 2. Did the **outcome rate** move once you condition on inputs? If overall outcome rates moved but the rate within comparable input strata did not, you are looking at label or covariate shift, not drift. 3. Did the **rule** move? If comparable cases now get different outcomes, it is concept drift — and that usually correlates with a datable external event: a policy change, a pricing change, a regulatory change. The distinction matters because the responses differ sharply. Covariate shift argues for reweighting or for collecting data in the newly reached region. Label shift argues for a prior correction and for re-choosing thresholds. Concept drift argues for new labels and a refit — and for treating the historical file as evidence about a world that no longer exists.