Why can the same regression model post very different held-out R-squared on two test splits?

## The denominator belongs to the data, not the model ``` R2 = 1 - SS_res / SS_tot = 1 - (mean squared error) / (variance of the target on the scored rows) ``` Only the numerator is about the model. The denominator is a property of the rows you chose to score on. Change those rows and you change the score without touching a single coefficient. ## A worked case An employee-tenure model predicts months of tenure with a root mean squared error of 5 months — the same 5 months in both scenarios below. - **Broad hold-out**, employees drawn from across the company, tenure standard deviation 12 months. Variance is 144. `R2 = 1 - 25/144 = 0.83`. - **Narrow hold-out**, restricted to one job family where nearly everyone leaves in a similar window, tenure standard deviation 7 months. Variance is 49. `R2 = 1 - 25/49 = 0.49`. Same model, same predictions, same errors in months. The reported quality collapses because on the narrow split the mean was already a good answer — a constant is only 7 months off on average there, so beating it is a much higher bar. It runs the other way too. Deliberately or accidentally include a few extreme rows and the target spread widens, the denominator inflates, and the same model looks better. A hold-out with wide spread is a generous grader. ## Why this matters in a real review Two teams score the same fitted model and report 0.71 and 0.44. Neither made an arithmetic mistake. They held out different periods, and the two periods had different target spreads — perhaps one covered a reorganisation and the other a quiet quarter. The disagreement is not about the model at all, and an hour spent auditing the code finds nothing, because there is nothing to find. The resolution is procedural: agree one frozen hold-out that both teams score against, or compare in the units of the target instead. The same argument kills cross-project comparison. R-squared 0.6 on a noisy human-behaviour target can be a strong result; 0.6 on a sensor-calibration target where the physics is nearly deterministic is a failure. There is no threshold that transfers between problems, because the denominator is a different quantity in each. ## Pair it with an absolute error The practical habit is to never report a held-out R-squared alone. Report three things: 1. The R-squared, so a reader sees the comparison against the constant baseline. 2. A mean absolute error or root mean squared error **in the target's units** — months, dollars, degrees. This is the number that keeps its meaning when the split changes, and the one a stakeholder can weigh against the cost of being wrong. 3. The target's mean and spread on the scored rows, which is the denominator made visible. With those three, the narrow-split story reads correctly at a glance: "R-squared 0.49, MAE 4 months, hold-out spread 7 months" says clearly that the model is off by about four months on a population that varies by seven — modest lift, honestly described. The bare 0.49 says nothing you can act on. ## What R-squared is still good for This is not an argument to stop using it. Within one fixed hold-out, R-squared is a clean way to compare candidate models against each other and against the do-nothing baseline, and its zero point carries real meaning: below it you should ship the constant. The rule is narrow and absolute: compare R-squared **only** between models scored on the same rows. ## Diagnostic reflex When a held-out R-squared moves and you did not change the model, look at the denominator before you look at anything else. Compute the target's standard deviation on both splits. If it moved, the score's movement is at least partly arithmetic, and the model's error in units will tell you whether anything real happened. If the spread is stable and the score still moved, only then is there a modelling question to answer.