A model trained on 1:1 undersampled data outputs 0.5 when the live positive rate is 4%. How do you fix the probabilities?

## Why balancing breaks the probabilities By Bayes' rule, the odds a row is positive factor into a prior term and a likelihood-ratio term: `odds(x) = [pi / (1 - pi)] * [f1(x) / f0(x)]` where `pi` is the class prior and `f1`, `f0` are the feature distributions within the positive and negative classes. Randomly dropping majority rows to reach a 1:1 ratio changes `pi` from 0.04 to 0.5 and, in expectation, leaves `f0` and `f1` untouched - a uniform random subsample of the negatives has the same shape as the full set of negatives. So the model trained on the balanced data estimates the right likelihood ratio and the wrong prior. Its odds are the correct odds multiplied by a constant, or equivalently its log-odds are the correct log-odds plus a constant shift. This is not a modelling failure and it is not noise. It is a deterministic offset with a closed-form inverse. The symptom is unmistakable: scores pile up around the middle of the range, the mean predicted probability across the population is roughly 0.5 while only 4% of rows are positive, and the reliability diagram sits far below the diagonal everywhere. ## The correction Let `pi_train` be the positive rate the model was trained at and `pi_true` the rate it will meet. For a predicted probability `p`: ``` odds = p / (1 - p) factor = (pi_true / (1 - pi_true)) * ((1 - pi_train) / pi_train) odds_fixed = odds * factor p_fixed = odds_fixed / (1 + odds_fixed) ``` With a 1:1 training set the second bracket is 1 and the factor collapses to the true prior odds, `0.04 / 0.96 = 0.0417`. Then: - `p = 0.5` gives odds 1, corrected odds 0.0417, `p_fixed = 0.04` - the base rate, exactly as it should be for a row the model finds unremarkable. - `p = 0.9` gives odds 9, corrected odds 0.375, `p_fixed = 0.27`. - `p = 0.2` gives odds 0.25, corrected odds 0.0104, `p_fixed = 0.010`. For a linear model on the log-odds scale this is simply subtracting a constant from the intercept; the coefficients are already right. ## What the correction does and does not touch - **Ranking is untouched.** The map is strictly increasing in `p`, so every pairwise comparison survives and AUC is unchanged to the decimal. Do not expect the correction to improve discrimination - it cannot. - **Decision cutoffs move.** A rule written against balanced scores no longer means what it did once the scale is restored; the cut has to be re-derived on the corrected scale. - **Aggregate quantities become usable.** Summing corrected probabilities over a population now estimates the expected number of positives; summing the balanced ones would have predicted half the book. ## When the closed form is not enough The derivation rests on one assumption: the resampling changed only the class proportions. That holds for uniform random removal of majority rows. It does **not** hold when the sampling was stratified or non-uniform, when the minority side was expanded with invented rows whose feature distribution differs from real positives, or when the training snapshot was drawn from a different period than the scoring population. In those cases the shift is no longer a single constant, and forcing the closed form gives a curve that is right at one point on the range and wrong elsewhere. The robust alternative, and a good belt-and-braces step even when the assumption holds, is to fit a monotone recalibration map on a **held-out sample drawn at the natural prevalence** - not on the balanced data. That sample has to be large enough to contain a workable number of positives, which is the real constraint at a 4% rate: ten thousand rows carry only about four hundred positives, and the top bins of the reliability diagram will still be thin. ## Verifying Evaluate on a natural-prevalence holdout, never on the balanced one, because the balanced set will make uncorrected scores look fine and corrected ones look terrible. Plot the reliability diagram on that holdout and check three things: the mean corrected probability across all rows should land near 0.04; the diagram should track the diagonal in the score region where decisions are made; and the per-bin counts should show whether the high-score bins hold enough positives to say anything at all. ## The prior question worth asking If probabilities are the deliverable, the cleanest option is often not to rebalance at all. Training on the natural distribution with the full data produces scores that need no correction, and modern learners handle a 4% positive rate without help. Undersampling buys training speed and sometimes better behaviour from a particular learner; it costs data and it costs the meaning of the output. Prior correction is how you buy the meaning back, and it is worth knowing precisely so the choice to rebalance is made with eyes open.