A 2019 credit scorecard now scores a much wider 2021 applicant pool — do you still trust its held-out AUC?

## Restate what the old number means A held-out AUC of, say, 0.82 computed on 2019 applicants is an estimate of one specific quantity: how well the model ranks risk **among applicants drawn from the 2019 population**. It is not a property of the model in isolation. Once the population changes, the estimate has not become inaccurate — it has become an accurate answer to a superseded question. So the honest response is not "yes" or "no" but a re-statement: *that number describes the old applicant mix; here is what I can and cannot say about the new one.* ## Name the shift before choosing a fix Marketing widened, so more people with different profiles applied. The plausible reading is that the input distribution moved while the relationship between an applicant's attributes and their default risk held — an applicant with a given profile is still as risky as an identical 2019 applicant would have been. That is covariate shift, and under it the fitted mapping is not invalidated. But this reading is an assumption, not an observation, and it is the assumption the whole plan rests on. It fails if underwriting policy, pricing, or the macroeconomic environment also changed, because then the outcome rule itself moved and the historical file describes a rule no longer in force. So state the assumption explicitly, and name what would falsify it. ## Re-estimating the score under the new mix Where the two populations overlap, the old held-out rows can be reweighted toward the new input distribution. Each old row gets a weight equal to the ratio of its density under the 2021 applicant distribution to its density under 2019, and the score is recomputed as a weighted average. You do not need the two densities separately: train a classifier to distinguish 2019 rows from 2021 rows and the odds it produces give the ratio directly. Three caveats deserve to be volunteered before the interviewer asks: 1. **Support.** Reweighting can only redistribute evidence you already have. If 2021 brings applicants in a region 2019 never populated, their weight has nothing to attach to, and the reweighted score is silent about exactly the segment you are most worried about. 2. **Variance.** When a few old rows carry enormous weight, the weighted average is effectively computed on a handful of observations. The reweighted estimate can be unbiased and useless at the same time, so report its effective sample size alongside it. 3. **Assumption dependence.** The whole procedure presumes the risk relationship held. If it did not, reweighting produces a confident estimate of the wrong thing. ## The constraint that dominates: label delay Credit outcomes are slow. Whether a 2021 applicant defaults is only known after twelve to eighteen months of loan performance. So the evidence that would settle the question — a labelled holdout drawn from the 2021 population — cannot exist yet, no matter what budget you assign. This reframes the problem from a measurement exercise into a decision under acknowledged uncertainty. The available moves are about how much you risk while the evidence accrues: - **Cap exposure** where the model is extrapolating: lower limits, tighter cutoffs, or manual review for applicants in the regions 2019 did not cover. - **Keep a randomised holdout policy** so a slice of decisions is made in a way that produces unbiased outcome data later. Otherwise the only 2021 outcomes you ever observe are for applicants the model approved, and the future evaluation set is selected by the model itself. - **Use early proxies with care.** Early-stage delinquency arrives sooner than final default and is genuinely informative, but it is a different target with a different base rate, and treating it as the real label quietly changes the question again. - **Re-fit once mature labels exist** on the new population, rather than tuning on the old file forever. ## What a strong answer sounds like A weak answer is binary: "no, retrain." Retrain on what? There are no mature 2021 labels. The strong answer sequences it: state what the old number does and does not describe; name the shift you believe you have and what would disprove it; re-estimate under the new mix where overlap permits, reporting how thin that estimate is; identify the region where no evidence exists and control risk there directly; and design today's decisions so that in a year you own unbiased outcome data instead of a self-selected sample. The underlying discipline is the same one the i.i.d. assumption always demands: a score is a claim with conditions attached, and the leader's job is to keep those conditions attached to the number wherever it travels.