What is target leakage, and why does a leaky feature still score well on a held-out test set?

## The definition **Target leakage** is a defect in the *dataset*, not in the model. One or more features carry information that did not exist at the moment the prediction has to be made, and they only carry it because the outcome has already happened by the time the training table was assembled. The decisive test is temporal, not statistical. For every feature, ask one question: **at the instant this prediction is needed in production, is this exact value already written — and is it this value?** If the value is written later, or is later overwritten, the feature is leaked. ## Three shapes it takes **Post-outcome fields.** An insurance claims table carries a settled amount alongside a flag for whether a claim was ever filed. The settlement is created *by* the claim. A model predicting whether a claim will be filed learns that a non-null settlement means yes, and separates the classes almost perfectly. **Proxies and near-copies.** The leaked field does not have to be the label. A hospital record's discharge disposition code may be written or rewritten only after a readmission has occurred. It is not the readmission flag, but it is close enough that a tree can reconstruct it in one split. **Operational artefacts.** A collections dataset contains a payment-plan start date. Payment plans are only ever opened for accounts that have already defaulted, so the field is non-null on almost exactly the positive class. Nobody put it there maliciously; it is simply what the operational system records, and the modelling table inherited every column. ## Why the held-out set does not save you The intuition behind a held-out set is that the model has never seen those rows, so a high score means it generalises. That intuition assumes the *only* way a model can cheat is by memorising rows. Leakage is a different cheat: the shortcut lives in a column, and every row in the table — held out or not — was generated by the same process, after the outcome was known. The test rows carry the leak just as faithfully as the training rows, so the score is high everywhere and nothing looks wrong. This is exactly what separates leakage from **overfitting**. Overfitting is the model memorising noise in the training rows; you detect it because held-out performance falls below training performance. Leakage keeps *both* numbers high. That is why "the train and test scores agree, so we are fine" is a dangerous sentence: agreement between them rules out overfitting and says nothing at all about leakage. ## What it looks like in practice Symptoms cluster: - A score near the ceiling for a problem the business has never solved — 0.99-plus area under the ROC curve, or accuracy far above what a domain expert would guess. - One feature dominating every importance ranking, with the model collapsing to near-random when it is removed. - A field whose missingness alone is almost a perfect classifier. - A model whose live performance, once deployed, falls to roughly the base rate — because in production the leaked field is null, defaulted, or not yet written. ## Finding it The cheapest detection is not statistical. It is reading the column list against the decision moment and, where you cannot tell, asking whoever owns the source table when each field is written. A short structured pass works well: 1. Write down the **decision timestamp** — the exact event that triggers a prediction (a quote request, an admission, an account entering arrears). 2. For each feature, record when the value is first written and whether it is ever updated afterwards. 3. Flag anything written after the decision timestamp, anything updated afterwards, and anything whose existence is conditional on the outcome. 4. Confirm suspects empirically: fit a model on each suspect feature alone, and retrain the full model with the suspects dropped. A collapse from near-perfect to a plausible number confirms the diagnosis. ## Fixing it Three options, in order of preference. **Remove** the feature outright, if nothing legitimate is left in it. **Re-derive** it as of the decision moment — sometimes the underlying system does hold the earlier value, and the modelling table simply took the latest one. **Keep it** if the audit shows the value genuinely is available and unchanged at prediction time; a strong legitimate predictor is not leakage. ## The nuance that trips candidates Correlation strength is not the criterion. A credit score correlates powerfully with default and is available before the decision — that is a good feature, not a leak. A settlement amount correlates powerfully and is unavailable — that is a leak. The distinction is availability in time. Conversely, a leaked feature is not always suspiciously named; the most damaging ones are innocuous operational fields that happen to be created by the outcome.