Is a log-loss of 0.31 good when the positive class occurs 12% of the time?

## Why the raw number cannot be judged Log-loss is a mean penalty in log units. It has a floor of 0 for perfect forecasts and no ceiling, and crucially the *achievable* value depends entirely on how predictable the target is and how common it is. A 0.31 might be outstanding on one dataset and worse than doing nothing on another. So the first thing to do with any reported log-loss is ask: compared with what? ## The constant base-rate forecaster The standard reference is the dumbest non-degenerate model available: ignore every feature and predict the overall positive rate `b` on every row. Because that forecast is the same everywhere, its expected log-loss is `-(b*log(b) + (1 - b)*log(1 - b))` With `b = 0.12`: - `ln(0.12) = -2.1203`, so the positive rows contribute `0.12 * 2.1203 = 0.2544` - `ln(0.88) = -0.1278`, so the negative rows contribute `0.88 * 0.1278 = 0.1125` - total: **0.367** So the model at 0.31 is better than the do-nothing forecaster, but not by much. A convenient way to express this is a **skill score**: `1 - (model loss) / (reference loss)`, here `1 - 0.31/0.367 = 0.155`. Read it as "the model removes 15.5% of the reference's loss". Zero means no skill, 1 means perfect, and negative means the features are actively hurting you. The same discipline applies to the Brier score. The constant base-rate forecaster's Brier score is `b*(1 - b)^2 + (1 - b)*b^2 = b*(1 - b)`, which at `b = 0.12` is **0.1056**. Any Brier score above that is a model doing worse than a single number. ## Trap 1: the base rate moves the yardstick The reference loss shrinks as the base rate gets more extreme. At a 12% rate the floor-setting reference is 0.367; at a 9% rate it is 0.303; at a 1% rate it is 0.056. So a model reporting a log-loss of 0.10 on a 1% target is *worse than useless* even though 0.10 looks like a much smaller number than 0.31. Never compare log-losses across datasets or across classes with different prevalences. ## Trap 2: drift makes month-over-month comparisons lie Suppose last month the model reported 0.31 at a 12% base rate and this month it reports 0.26 at a 9% base rate. The raw number improved, but the reference also fell from 0.367 to 0.303. Skill went from `1 - 0.31/0.367 = 0.155` to `1 - 0.26/0.303 = 0.14`. The model actually got slightly *worse* at the job; all that happened is that the world got easier to predict. This is one of the most common ways an evaluation dashboard tells a comfortable lie, and it is why monitoring should track skill against a rolling baseline rather than the raw loss. ## Other references worth having The constant base-rate forecaster is the floor, not the bar. Two more useful comparisons: - **The incumbent.** If a model, a rule set or a human process is already in production, its score on the same rows is the number that decides whether the new model ships. - **A deliberately simple model.** A single-feature or shallow model gives a sense of how much of the achievable signal is cheap. If your ensemble's skill barely exceeds it, the complexity is not paying for itself. Also note the reference must be computed on the *same* evaluation rows as the model, with the base rate measured on those rows, not on the training set. Computing the baseline from a different prevalence quietly biases the comparison. ## How to report it Never ship a bare proper score. Report the model's loss, the base rate of the evaluation set, the reference loss implied by that base rate, and the skill score. Four numbers, one line, and any reader can tell in a second whether the model is worth anything. That habit also protects you in the review where someone asks whether 0.31 is good, because the answer is already on the slide.