skip to content

What do Youden's J and maximising F1 each assume when used to pick a classification threshold?

level: middleimportance: nice to knowfreq 34%

answer

  1. both stand in for a missing cost matrix
  2. one adds rates, one ignores a cell
  3. sensitivity plus specificity minus one
  4. vertical gap above the ROC diagonal
  5. true negatives never enter F1

basics

~20 s

Youden's J, sensitivity plus specificity minus one, weights both error rates equally and ignores class sizes. Maximising F1 ignores true negatives entirely and implies a trade-off that shifts with the operating point rather than a fixed cost ratio.

solid answer

~50 s

Both are stand-ins for a cost matrix nobody wrote down, and each smuggles in its own assumption. Youden's J is `sensitivity + specificity - 1`, equivalently `TPR - FPR`, and its maximum is the ROC point furthest above the diagonal. It treats one percentage point of false-positive rate as exactly as bad as one point of missed recall -- reasonable when the two classes are comparable in size, disastrous under heavy imbalance, where a point of FPR is a flood of false alarms. F1 is the harmonic mean of precision and recall. It never looks at true negatives, so it is unmoved by a mountain of easy negatives, and because precision depends on prevalence, the cut it selects moves when the class mix moves. Neither corresponds to a constant exchange rate between a miss and a false alarm. Use them when costs genuinely are unavailable, and say out loud which assumption you accepted.

go deeper

for a junior

Be able to state both formulas correctly: J is sensitivity plus specificity minus one, and F1 is the harmonic mean of precision and recall. Mixing up which pair of quantities each uses is the common slip.

for a middle

Explain what each rule implicitly assumes -- J weights the two error rates equally and ignores class sizes, F1 ignores true negatives and moves with prevalence -- and pick one for a described problem on that basis.

for a senior

Show judgment about fit: reject J on a heavily imbalanced problem because a point of false-positive rate is thousands of alerts, and note that an F1-tuned cut must be re-derived when the live class mix differs from validation.

for a principal

Push the conversation toward an explicit constraint or cost ratio the business owns, and treat any implicit rule as a placeholder you have accepted knowingly and documented rather than a defensible default.

## Why these rules exist The principled way to set a threshold is expected cost. Frequently nobody can price the errors -- a KYC identity-match score is the classic case, where a wrongly rejected genuine applicant and a wrongly accepted impostor both cost "reputation and friction" in units no one has converted to money. Youden's J and maximising F1 are the two rules people reach for instead. Neither is cost-free; each just makes the cost assumption implicitly. ## Youden's J ``` J = sensitivity + specificity - 1 = TPR - FPR ``` Sensitivity (recall, TPR) is the share of real positives you catch. Specificity is the share of real negatives you correctly leave alone, `1 - FPR`. J ranges from 0 for a coin flip to 1 for perfect separation, and it can go negative for a classifier that is worse than chance. Geometrically, `TPR - FPR` is the vertical distance from an ROC point up to the diagonal, so maximising J picks the operating point that stands highest above the chance line. **The assumption.** J adds the two *rates* with equal weight. One percentage point of TPR is worth one percentage point of FPR, regardless of how many items sit in each class. Where the classes are roughly balanced -- as in an identity-match problem where genuine and non-matching pairs are of comparable order -- that symmetry is defensible, and J is a clean, prevalence-independent choice. It also has the nice property that it does not move when the class mix changes, because TPR and FPR are computed within each class. **Where it breaks.** Suppose 1 positive per 1,000. Buying 1 point of TPR at the price of 1 point of FPR means catching a fraction of one extra positive in exchange for roughly 10 extra false positives per thousand items. J is indifferent; your review team is not. Under heavy imbalance, J systematically picks a cut far lower than any sane operating point. ## Maximising F1 ``` F1 = 2 * precision * recall / (precision + recall) ``` The harmonic mean punishes imbalance between the two: 0.9 and 0.1 give F1 of 0.18, not 0.5. Sweeping the threshold and keeping the cut with the highest F1 is probably the most common default in practice. **The assumptions.** Three, and all of them are worth naming. First, **true negatives never enter the formula**. Precision and recall are both built from TP, FP and FN. That is exactly why F1 is popular on rare-positive problems -- adding a million easy negatives leaves it unchanged -- and exactly why it is silent about how much needless work the negatives generate. Second, **it is prevalence-dependent**. Precision depends on the class mix, so the F1-maximising cut on a validation set with 5% positives is not the F1-maximising cut on live traffic with 15% positives. J does not have this problem; F1 does. Third, and most subtly, **F1 does not correspond to a constant exchange rate between a false positive and a false negative**. The expected-cost rule says "a miss is worth k false alarms" for a fixed k, which is a straight line in the confusion counts. F1's contours are not straight lines, so the trade-off it implies changes as you slide along the curve. Maximising it is a defensible convention, not a derivation. ## Choosing between them, and the third option - Classes of comparable size, both errors matter symmetrically, no costs available -- **J** is a reasonable, explainable default, and it is stable under changes in the class mix. - Rare positives where the negatives are cheap to ignore and you care about the quality of the flagged set -- **F1** is the more sensible of the two, with the caveat that it must be re-derived if the class mix shifts. - Neither, when the business can state a **constraint** instead of a cost: "at least 85% precision, take whatever recall that buys" turns threshold choice into a constrained search with no implicit weighting at all. When a constraint is available, prefer it: it is the one option whose assumption is written on the tin. A weighted F-beta (`F2` favouring recall, `F0.5` favouring precision) is the usual middle ground when you can express a preference but not a price. It is still an implicit cost assumption; it is just one you chose deliberately. ## What a good answer sounds like Name the formula, name the assumption, and say why it does or does not fit the problem in front of you. "I would use J because the classes here are roughly balanced and I have no cost data" is a good answer. "I maximised F1" with no justification is the answer that invites the follow-up about what you assumed a false positive was worth.

  • Why is Youden's J unaffected by the class mix while the F1-maximising threshold is not?
    J is built from TPR and FPR, and each is computed within one class, so changing how many positives and negatives you have leaves both untouched. F1 uses precision, whose denominator mixes true and false positives, so it moves with prevalence. That makes J stable across populations and makes an F1-tuned cut something you must re-derive whenever the class mix changes.
  • When would you use F2 rather than F1 to pick a cut?
    When recall matters more than precision but you cannot price the difference. F-beta weights recall by beta squared relative to precision, so F2 favours catching positives and F0.5 favours keeping the flagged set clean. It is still an implicit cost assumption -- you have just chosen it deliberately rather than defaulting into the equal weighting F1 imposes.
  • Is there a threshold rule that avoids an implicit cost assumption altogether?
    A hard constraint comes closest: fix the precision the product must deliver, or fix the volume the reviewers can absorb, then take the best cut satisfying it. The assumption is then stated explicitly by the business rather than hidden in a formula. It does not make the trade-off disappear -- you still accept whatever recall falls out -- but nobody has to guess what was assumed.

saying these in an interview costs you the question

  • Defines Youden's J as precision plus recall minus one
  • Treats maximising F1 as objectively correct
  • Believes F1 accounts for true negatives
  • Uses J on heavily imbalanced data without comment
  • Cannot name any assumption behind the chosen rule

context

resolving links…