With a 2% positive rate, how do you decide between class weights, resampling, and moving the operating point?

## The three levers are not interchangeable They act at three different moments and have three different costs. - **Moving the operating point** happens after training. The model and its scores are untouched; you only change the score above which you act. Nothing about the fit is at risk, and the change is reversible in seconds. - **Reweighting** happens during training. It changes the objective, so it changes the fitted parameters. The scores are no longer on the natural probability scale. - **Resampling** happens before training. It changes the data the model sees, so it changes the fit *and* the effective prior, and it can throw information away or reuse the same rows many times. An interviewer asking this wants to hear the ordering logic, not a preference. ## The diagnosis that comes first Fit the model with no intervention, then ask two separate questions on a validation split. 1. **Does it rank?** Look at whether high-scoring cases really do contain most of the positives. If the ordering is good, the model has learnt the signal, and the low flag rate is purely a cut-off problem. 2. **Is the operating point wrong?** If yes, that is a decision-layer problem with a decision-layer fix. A large majority of "the model does not detect the rare class" complaints resolve at step 2. That is the single most useful thing to say: the default cut-off of 0.5 is an arbitrary inheritance, and on a 2% base rate almost nothing will ever exceed it. You do not have a training problem; you have a cut-off that nobody chose deliberately. Which value to choose, and from what — expected cost, a required recall, the size of the review queue — is a threshold-selection exercise in its own right. ## When reweighting is the right lever Reweighting earns its place in three situations. First, when the fit is genuinely degenerate rather than just conservatively scored. Some learners collapse under extreme imbalance: a shallow tree may never make a split that isolates the rare class, a margin-based learner may place the boundary where no positive is on the correct side, and there is then no cut-off that recovers useful behaviour because the ordering itself is uninformative. Second, when you have a real cost asymmetry and you want the model to internalise it during fitting, so that it spends its limited capacity on the errors that matter. A cut-off can only re-slice the ranking the model already produced; the weight can change what the model chose to learn. Third, when the downstream consumer wants a single decision, not a score, and calibration is irrelevant. The cost is that the outputs stop being calibrated, and that a class with very few rows now dominates the objective, which raises variance and the risk of fitting noise in those few rows. ## When resampling is the right lever Undersampling the majority is defensible when the majority is enormous and training time or memory is the binding constraint — dropping 90% of an overwhelming negative class can turn an overnight fit into a ten-minute one at little cost in signal, because the marginal negative row carries almost no information. It shifts the prior, so the same calibration caveat applies, and it discards data you paid for. Oversampling by duplication is the weakest of the three: for most losses it is equivalent to weighting, but it inflates row counts, corrupts cross-validation if applied before splitting, and interacts badly with anything that subsamples rows. If duplication is what you mean, weight instead. Synthesising new minority points is a different mechanism with its own tradeoffs and belongs to that discussion, not this one. ## Stacking, and how to compare fairly Applying all three at once is the classic mistake. Weighting 49:1 *and* balancing the classes by resampling *and* dropping the cut-off compounds three tilts toward the rare class, and the result flags almost everything. Pick one primary lever, and treat any second one as a deliberate, measured addition. Whatever you pick, the comparison protocol has to be honest. Resampling and reweighting are training-time choices, so they must be applied inside the training fold only, never to validation or test data. Evaluation always happens on data with the real class balance — otherwise you are measuring performance on a population that does not exist. And report the comparison at a matched operating point or across the full range of them, because a reweighted model compared at a fixed 0.5 cut-off against an unweighted one is not a comparison of models at all; it is a comparison of cut-offs. ## The answer that lands "Move the operating point first, because it is free, reversible and keeps the scores meaningful. Reweight when the fit itself is broken or when I know the cost ratio. Resample when the data volume forces my hand. And I never do more than one of them without measuring what the second one added."