skip to content

Your fraud model is retrained only on checkouts it allowed — what does declining a transaction do to the next model's training data?

level: seniorimportance: must knowfreq 64%

answer

  1. no charge, so nothing to dispute
  2. a hole, not a negative example
  3. fitted on traffic the last policy allowed
  4. false declines are the invisible error
  5. the block hardens with every retrain

basics

~20 s

A declined checkout is never charged, so almost nothing can dispute it and no outcome arrives. The declined region stays permanently unlabelled, and each retrain learns the fraud rate only among traffic the previous policy allowed.

solid answer

~50 s

Declining creates a hole in the label supply rather than a negative example. No charge means no dispute, so apart from the rare customer who phones in and gets the case resolved by hand, a declined attempt returns no outcome at any age. Retraining on what remains estimates risk **conditional on the previous policy's allow decision**, and the loop self-confirms: whatever the old model blocked, the new one never sees evidence against, so a wrongly blocked segment stays blocked forever and its false declines are invisible. Offline metrics quietly improve as this worsens, because the hardest cases have been removed from the evaluation pool. Breaking it needs deliberately labelled traffic in the declined region — a randomised-allow slice, an analyst verdict on a sample of declines, or a challenge instead of a hard decline — plus the deciding score and policy version logged so you can tell which rows were censored by which policy.

go deeper

for a junior

Hold on to the basic asymmetry: an allowed transaction can later be disputed, a declined one is never charged, so it produces no outcome to learn from.

for a middle

Explain the conditioning — each refresh is fitted on traffic the previous policy allowed — and name at least one way to get labels back in the declined region.

for a senior

Diagnose the loop in production: a block that hardens over refreshes, offline metrics improving while loss is flat, and false declines that no dashboard can show.

for a principal

Decide whether the business funds a permanent labelled stream in the declined region, and what unmeasured false declines are worth against the loss that buying labels costs.

## Declining removes the row from the world An allowed card-not-present checkout runs to settlement and eventually earns an outcome: a settled dispute for unauthorised use, or silence past the maturation window. A declined checkout is never charged. With no charge there is nothing to dispute, so apart from the occasional customer who calls in and has the case resolved manually, a decline returns no outcome at all — not late, but never. This is qualitatively different from a delayed label. A delayed label is a row you must wait for. A declined row is a row that will never arrive, so patience does not help and neither does a longer maturation window. ## What the next model actually learns Each retrain therefore fits the risk of fraud **conditional on having been allowed by the policy that was live at the time**. The consequences compound: - **Self-confirmation.** Whatever the previous policy blocked generates no counter-evidence, so the next model has no reason to unblock it. A segment wrongly declined once — a device class, an issuing-country pattern, a shipping corridor — can stay declined indefinitely, and the block hardens with every refresh. - **Invisible false declines.** The most expensive error in card-not-present risk is turning away a real customer, and it is precisely the error the label supply cannot see. Loss shows up in a ledger; a declined good customer shows up nowhere. - **Flattering offline numbers.** The hard positives were removed from the pool before evaluation. Precision on the allowed slice rises while true performance over all attempted traffic is unmeasured, so the metric moves in the opposite direction to reality. - **Drift you cannot attribute.** When the allowed-traffic mix shifts, you cannot tell whether attacker behaviour changed or your own decisions reshaped the pool, because the pool is a function of your decisions. ## Four ways to get labels in the declined region | Source | What it returns | What it costs | Honest limit | |---|---|---|---| | Randomised-allow slice | Real settled outcomes above the cutoff | Loss you knowingly accept | Needs an owner and a loss cap; labels still mature slowly | | Analyst review of sampled declines | A human verdict in hours | Review labour | A verdict, not a settled outcome; it can be contradicted later | | Challenge instead of hard decline | Pass or abandon signal | Conversion, not loss | An abandonment is not a confirmed fraud; the challenge itself changes behaviour | | Manual customer contact on declines | A resolved case | Support handling time | Self-selected: only motivated customers call, so the sample is skewed | Only the first buys outcomes of the same kind as the rest of the training set. The others buy cheaper, weaker signals that must be recorded as a different label source rather than silently pooled with settled outcomes. ## What the decision log has to carry None of the repairs work unless each decision record carries the score, the action emitted, and the version of the policy that emitted it. Three uses depend on it: 1. Reconstructing which rows were censored by which policy, so a training set can be described honestly. 2. Identifying the randomised-allow rows, which must be flagged at decision time — you cannot recover them afterwards from the score alone. 3. Weighting by the probability the policy would have allowed the row, where the decision was stochastic, so a model can be fitted closer to the attempted distribution rather than the allowed one. ## The trap the numbers set The loop is dangerous because every visible signal says it is fine. Loss per allowed transaction falls, because losses can only be booked on allowed traffic. Offline precision and recall improve on each refresh, because the pool is easier each time. The fraud rate in training drifts down. The only number that would contradict the story — the share of declined attempts that were genuinely good customers — is unmeasurable without paying for it. That is why teams that operate this well treat a small, permanent, signed-off stream of labelled declines as part of the running cost of the system, not as an experiment they switch on once.

  • Does a step-up challenge instead of a hard decline give you a usable label?
    A partial one. A passed challenge that settles produces something close to a normal outcome, but the challenge itself changed the customer's experience. An abandoned challenge is ambiguous: a fraudster who gave up and a legitimate customer who lost patience look identical, so it is recorded as its own label source rather than as fraud.
  • Why do offline metrics improve as this bias gets worse?
    Evaluation runs on the same censored pool the training set came from. The hardest positives were declined and never entered it, so precision and recall on the allowed slice rise while performance over all attempted traffic goes unmeasured. The metric and reality move in opposite directions.
  • Can you just treat declined attempts as fraud, since the model thought they were?
    No. That labels the model's own opinion as truth and bakes every false decline in permanently — the next model is fitted to agree with the last one. Declines belong in an unknown state, and the only honest way out is to buy labels in that region.

saying these in an interview costs you the question

  • Labelling declined attempts as confirmed fraud in the training set
  • Expecting a longer maturation window to eventually resolve declined rows
  • Reading rising offline precision as proof the loop is healthy
  • Assuming false declines will surface as customer complaints
  • Believing the allowed traffic mix is a sample of attempted traffic