Which group fairness criterion should a marketplace commit to for a fraud model that routes sellers into manual verification?

## Why the criterion cannot be chosen in the abstract Group fairness criteria are not ranked. Each is a statement about a different conditional rate, and the impossibility result guarantees that committing to one lets the others drift once base rates differ. So the real question an interviewer is asking here is whether you can reason from a **decision** to a **criterion**, and then defend the choice to people who did not train the model. ## Step 1: describe the decision and both error types in plain language A fraud model scores sellers; sellers above a threshold are routed into a manual verification queue where they must supply documents and wait. Two errors exist: - **False positive** — a legitimate seller sent to the queue. Cost: delay, paperwork, held payouts, possibly abandoning the platform. The cost falls on an individual who did nothing wrong. - **False negative** — a fraudulent seller not flagged. Cost: losses absorbed by the marketplace and by defrauded buyers. The cost is diffuse and mostly institutional. That asymmetry already points somewhere: the harm that a fairness commitment most needs to bound is the burden on innocent sellers, which is the **false-positive rate computed among legitimate sellers of each group**. Committing to equalize it is the negative-label half of equalized odds, and it is a claim you can state in one sentence to a non-technical audience: *if you are a legitimate seller, your chance of being pulled into verification does not depend on your group*. ## Step 2: name what each alternative would commit you to **Demographic parity (equal flag rates).** The most common opening ask, because it is auditable without outcomes. Its cost is explicit: if the measured fraud rate genuinely differs between segments, equal flag rates means either dragging more legitimate sellers from the lower-rate segment into the queue, or letting more fraud pass in the higher-rate one. That is a coherent policy — it just has to be chosen with the price visible, not adopted because it sounds neutral. **Predictive parity (equal precision among flagged sellers).** This one serves the **reviewer**, not the seller: it means a flag means the same thing whichever segment it came from, so reviewers do not learn to discount flags from one group. Worth monitoring for exactly that reason. But a segment can enjoy healthy precision while still having a large share of its innocent members swept in, so it is a poor primary commitment for the harm at hand. **Equal opportunity (equal recall on fraudsters).** This equalizes catch rates across groups. It protects the platform's losses and, arguably, honest sellers competing against uncaught fraud — but it says nothing about who is wrongly burdened, which is the harm this system produces. ## Step 3: interrogate the label before you condition on it Everything above except demographic parity is defined relative to `Y`, the recorded fraud label. Ask where `Y` came from. If fraud is only known where someone investigated, and investigations historically concentrated on one segment, then the measured base rate in that segment is inflated and the other segment's is deflated. Criteria that condition on such a label inherit the enforcement history and dress it up as ground truth. In that world the label-free criterion — equal flag rates — is the more honest commitment, precisely because it refuses to trust the label. Being able to make this argument, and to say what evidence would settle it, is the difference between a principal answer and a competent one. ## Step 4: commit, publish, monitor, revisit Write down one primary criterion and the reasoning. Report the others on the same evaluation sample on a fixed cadence — group base rate, flag rate, recall, false-positive rate, precision — so that drift in an unconstrained rate is visible rather than discovered by a journalist. State what the commitment costs: a fairness constraint that binds always costs something on some other axis, and a team that claims otherwise has not measured it. Then name the triggers to reopen the decision: the seller population's composition shifts, the queue gets slower or the document demand heavier so a false positive costs more, or the label-generating process changes. And name who owns the call. A fairness criterion with no owner reverts to whatever the loss function happened to favour. ## What a weak answer looks like Picking a criterion in the first sentence; treating demographic parity as automatically correct or automatically naive; promising to satisfy several criteria at once; or presenting the choice as a technical result when it is a policy commitment the organisation has to be willing to defend in public.