Reweighting the loss versus oversampling the rare class: what is identical and what actually differs?

## The part that is genuinely the same Suppose you want to optimise a rebalanced objective in which each class contributes loss mass proportional to some target rather than to its raw count. There are two ways to get there. Draw examples from the natural distribution and multiply each one's loss by a weight; or change the sampler so that examples are drawn at the target rate and leave the loss alone. Written as expectations, these are the same integral - multiplying a term by `w` and multiplying its sampling probability by `w` both scale its contribution by `w`. So the expected gradient is identical, and any claim that one of them "optimises something different" is wrong at the level of the objective. The interesting differences all live in the estimator, not the target. A deep network is trained with noisy mini-batch estimates of that gradient, and the two routes produce estimates with very different behaviour. ## Gradient variance and batch composition With loss weighting, the sampler is untouched, so a class that makes up one row in ten thousand appears in roughly one batch in thirty at batch size 256 - and when it appears, its term is multiplied by a large weight. The estimate is unbiased and noisy: most steps carry no signal about that class at all, and the occasional step carries a very loud one. Symptoms are loss spikes, clipping firing, and sensitivity to the learning rate. With oversampling, rare rows appear in nearly every batch at weight 1, so each step carries a modest, consistent signal. Variance drops. The price is that there are still only 20 distinct rare rows; oversampling shows the same rows repeatedly, and repeated exposure at full weight is a fast route to memorising them. Augmentation is what converts repetition into something useful, which is why oversampling tends to be much more effective for images and audio, where strong augmentation exists, than for short text where it does not. There is a specifically deep-learning consequence too. Any layer that normalises across the batch dimension - batch normalisation computes a per-channel mean and variance over the examples in the batch (and over spatial positions for a convolutional feature map) - sees a different input distribution when you change the batch's class mix. Resampling therefore changes those statistics and the running estimates used at inference; loss weighting does not. If you are debugging why a resampled run behaves differently at inference than during training, this is the first place to look. ## What an epoch means This one trips people up in real projects. With loss weighting, one epoch is one pass over the real dataset, and comparing runs is easy. With oversampling by duplication, one pass is longer - potentially much longer - and it revisits the rare rows many times. Anything keyed to epochs silently changes meaning: the learning-rate schedule, the early-stopping patience, the checkpoint cadence, the number of times a given rare row has been fitted. Two runs reported as "20 epochs" may differ by an order of magnitude in gradient steps and by far more in exposure to the rare class. Compare on gradient steps or on examples consumed, not on epochs. Undersampling the majority instead is the mirror image: passes become cheap and fast, variance is low and the batch mix is controlled, but you throw away most of your data, and on a head class with genuine internal diversity that is a real loss of information. Reweighting always keeps every row. ## How to choose in practice A reasonable default is: start with a moderate loss weight, because it is a one-line change that leaves the data pipeline, the epoch semantics and the batch statistics alone. Move to a rebalanced sampler when the weights you need are extreme enough to make training unstable, since the sampler converts a variance problem into a repetition problem, and repetition is easier to manage with augmentation than spikes are to manage with clipping. Many practical setups do both at partial strength: sample toward the target rate to keep the rare class present in every batch, and apply a residual weight for the remaining gap, which keeps both the variance and the duplication moderate. Whichever you pick, two things stay true. Both shift the model's implied class prior, so probability outputs need recalibrating on data drawn at the natural rate before any downstream system reads them as rates. And neither invents new rare-class examples - they only change how often and how loudly the ones you have are heard.