Your fraud team proposes allowing a random 2% of would-be-declined checkouts to earn real labels — how do you size and sign off that holdout?

## What the holdout is for A fraud policy cannot observe its own declines: nothing is charged, so no dispute arrives. The randomised-allow holdout buys that observation by letting a random slice of would-be-declined checkouts through anyway and letting them run to a real outcome. It is the only mechanism that returns outcomes of the *same kind* as the rest of the training set, which is why teams keep paying for it. Three things it buys, in order of business value: 1. **A measured false-decline rate.** The share of declines that were good customers is otherwise a number the business argues about rather than knows. 2. **Calibration above the cutoff.** Scores in the declined band are extrapolation until something up there has an outcome. 3. **Training rows where the policy operates.** A model fitted only below its own cutoff is being asked to generalise into a region it has never seen labelled. ## Sizing it as a loss budget The sizing is arithmetic, and the whole conversation with the business happens in the units of that arithmetic: - 200,000 checkouts a day, 1% discretionary decline rate, so **2,000 declines a day** - sampling rate 2%, so **40 allowed a day** - roughly 70% of the declined population genuinely fraudulent, so **28 losses and 12 confirmed good customers a day** - average loss per fraudulent transaction $120 including the dispute fee, so **$3,360 a day**, about **$100,800 over 30 days** Two corrections are usually missing from a first proposal. First, the labels mature like every other label: over a 45-day maturation window the holdout yields **1,800 sampled rows at a cost of $151,200** before the first cycle of them is usable. Second, the rate that matters is the count per maturation cycle, not the count per day — a slice that looks generous daily can still be too thin to say anything once the cycle is over. ## Guardrails, and what must never be sampled | Guardrail | Why it exists | |---|---| | Per-transaction value ceiling | A random allow on an unusually large basket can spend a month of budget in one decision | | Daily accepted-loss cap with automatic suspension | An attack spike raises the fraud share of declines, so a fixed sampling rate silently costs more | | Exclude blocks that exist for legal or compliance reasons | Those declines are not discretionary and are not yours to sample | | Exclude hard blocks on known-compromised instruments | Sampling them buys a label everyone already has | | Sampled flag written at decision time | The rows cannot be recovered later from the score alone | The holdout therefore samples only the band where the policy is exercising **discretion** — scores above the cutoff that no mandatory rule would have stopped anyway. ## Who signs it This is the part that makes the question a leadership one rather than an engineering one. Engineering can design the sampler, but the decision is to accept a quantified loss in exchange for information, and that trade belongs to whoever owns the loss line — typically the risk or finance owner, with fraud operations consulted because the sampled rows also land in the review workload. The proposal that gets signed states four numbers: the daily accepted loss, the daily cap that triggers suspension, the expected label yield per maturation cycle, and the decision the labels will be used to make. A proposal missing the fourth tends not to survive its first budget review, because a label nobody has committed to act on is a donation. ## The cheaper alternatives, and what they give up A **challenge** on the sampled slice instead of an allow spends conversion rather than loss, which is often an easier sell — but an abandoned challenge is not a confirmed fraud, so the label it returns is weaker and belongs to its own source. **Analyst review** of a sample of declines returns a verdict within hours at labour cost, useful for steering but subordinate to a settled outcome when the two disagree. Neither replaces the holdout, because neither produces the unbiased outcome distribution above the cutoff that the false-decline rate depends on. The realistic posture is a small permanent holdout plus the cheaper signals layered on top, with each label source recorded distinctly so nobody later pools a verdict with a settlement.