Class Weights and Thresholds

## What is actually being changed A supervised model is fitted by minimising a loss summed over training rows. For binary log loss the per-row term is ``` loss_i = -[ y_i * log(p_i) + (1 - y_i) * log(1 - p_i) ] ``` and the training objective is `L = sum_i loss_i`. Class weighting replaces that with ``` L = sum_i w(y_i) * loss_i ``` where `w(1)` is one constant applied to every positive row and `w(0)` another applied to every negative row. This is important to say out loud in an interview: **the data set is untouched**. Nothing is duplicated, deleted or synthesised. The only thing that changes is how much each mistake costs the optimiser. That is why the technique is usually described as *cost-sensitive learning* rather than a sampling technique. ## Why the rare class gets ignored without it Take a 98/2 split and the simplest possible model, an intercept only, so every row gets the same predicted probability `p`. For log loss with a logistic link, the derivative of a row's loss with respect to the row's score (the log-odds `z`) is exactly `p - y`. Summing over rows, the unweighted gradient is ``` 2 * (p - 1) + 98 * (p - 0) ``` which is zero at `p = 0.02`. The unweighted optimum *is* the base rate; the model is behaving correctly, it simply has almost no reason to predict anything large. Now apply weights and the gradient becomes ``` w_pos * 2 * (p - 1) + w_neg * 98 * p ``` which is zero when `p = (w_pos * 2) / (w_pos * 2 + w_neg * 98)`. If the two classes carry equal total weight, that is `p = 0.5`. In log-odds terms the whole score surface is shifted by `log(w_pos / w_neg)`. With a richer model the shift is not a clean constant, but the direction is the same: the fitted boundary moves so that the rare class occupies more of the input space. ## The balanced rule The common automatic rule sets `w_c = n / (k * n_c)`, where `n` is the number of rows, `k` the number of classes and `n_c` the count of class `c`. On 98 negatives and 2 positives this gives `w_pos = 100 / (2 * 2) = 25` and `w_neg = 100 / (2 * 98) = 0.51`, a ratio of 49:1, which is exactly `n_neg / n_pos`. Each class ends up carrying total weight `n / k = 50`, and the total weight over the whole data set still sums to `n`. That last property matters when a penalty term is involved: because the data term keeps its original scale, the strength of L1 or L2 regularisation relative to the data is unchanged. If instead you hand-set weights of 1 and 49, the total weight mass grows to roughly twice `n`, and the same penalty is now relatively weaker — a silent change in regularisation that people rarely account for. Balanced is a default, not an answer. It targets a 50/50 effective prior, which is the right target only if the cost of a false negative really is `n_neg / n_pos` times the cost of a false positive. When you actually know the costs, encode them; when you do not, the ratio is a knob to tune, not a formula to trust. ## Relation to duplicating rows For an integer weight `w`, adding a minority row `w` times reproduces the weighted loss exactly, so the two are equivalent for a plain full-batch fit. They diverge as soon as anything counts rows rather than weight: bootstrap draws in a bagged ensemble, row subsampling in a booster, a minimum-samples-per-leaf constraint in a tree, mini-batch composition, and cross-validation splits, where duplicated rows leak the same observation into training and validation folds. Weighting is the safer expression of the same intent. ## What it buys, and what it costs It buys a decision region tilted toward the rare class — higher recall, lower precision — and it can genuinely change the fit for learners whose structure depends on class mass, such as a tree whose split criterion is computed on weighted counts, or a margin-based learner deciding where to place the boundary. What it usually does **not** buy is new information: the ordering of scores often barely improves, so a ranking metric may look nearly identical before and after. It costs you two things. First, the effective sample for the rare class is still those few rows, now shouting louder, so variance and overfitting risk on that class rise. Second, the outputs are no longer calibrated to the real base rate, which breaks anything downstream that treats a score as a probability.