An analyst cleared a checkout as legitimate and a dispute for unauthorised use settles against you six weeks later — which becomes the training label?

## A ladder of label sources One checkout can acquire several claims about what it was, arriving at different times and carrying different authority. | Source | Typical latency | Authority | What it is really saying | |---|---|---|---| | Rule or heuristic flag | Instant | Weakest | This matched a pattern somebody wrote | | Analyst verdict on a reviewed case | Hours | Provisional | A trained human, working from what was visible then | | Customer report of unauthorised use | Days | Strong but partial | The cardholder says they did not do it | | Filed dispute | Weeks | Not yet final | A case has been raised and may still be contested | | Settled dispute | Weeks to months | Final for training | The case resolved, and on what reason class | The ladder matters because a training pipeline that pools these into one boolean loses the ability to say why a row is positive, and loses the ability to notice when the cheap sources disagree with the expensive one. ## Which one wins, and the three qualifications The settled outcome wins as the training label. Three qualifications keep that from becoming an overstatement: 1. **Reason class matters.** A dispute filed because goods never arrived, or because the quality was wrong, is a service failure, not fraud. Only the unauthorised-use class belongs in the fraud positive class; pooling the rest teaches the model to predict logistics problems. 2. **Settled, not filed.** A filed dispute can be contested and won, and a win restores the row to legitimate. Training on filed disputes imports every reversal as a false positive. 3. **The verdict is not wrong, it is earlier.** The analyst decided from evidence available within hours. Losing to a settlement six weeks later is the normal outcome of an information gap, not a performance failure. ## What to do with the disagreement The disagreement is a signal in its own right, about the **review policy** rather than the model. Watch it as a rate, sliced by reviewer cohort and by the guidance version they were working under: - A rising rate of *cleared then disputed* means reviewers are being asked to clear too readily, or the evidence surfaced in the case view has stopped being sufficient. - A rising rate of *flagged then never disputed* is harder to read, because a flagged case is often actioned and so prevented — which is exactly the censoring problem again, in miniature. - A stable low rate is the healthy state, and it is what lets analyst verdicts be used as provisional labels at all. ## Recording it so the training set stays honest The record is append-only: the decision, then each label claim with its source, its state and the time it was observed. Nothing is overwritten, because the sequence is what lets you reconstruct what any past model was trained on. Two consequences follow that teams discover late: - **A dataset is a window plus an as-of date.** "Last July's checkouts" read in September and read again in November are different label sets. Two models compared without that stamp are not comparable. - **Provisional training is a choice with a cost.** Training on analyst verdicts lets you refresh weeks earlier than settlements allow, at the price of a label that can flip underneath a deployed model. A reasonable posture is to train on provisional labels only where the refresh urgency is real, and to keep the settled-label model as the reference the provisional one is measured against. ## The failure this prevents The failure mode is a pipeline that writes a single `isFraud` column and updates it in place. Everything downstream then sees one number with no source, no state and no time. A backfill silently rewrites history, a metric moves for reasons nobody can trace, an analyst's judgement and a settlement become indistinguishable, and nobody can answer the simplest audit question about a model in production: which labels was it actually trained on, and when were they true?