How do you stratify cross-validation folds when the regression target is continuous?

## Why regression folds need it at all Stratified k-fold is usually taught as a classification device, and the reason is mechanical: stratification balances a *discrete* key across folds, and regression has no discrete key. The underlying motivation carries over unchanged though. A validation fold whose target distribution does not resemble the population produces a metric that does not resemble the population metric, and the fold-to-fold scatter you then see is split noise rather than model behaviour. The symptom is loudest with a right-skewed target — student test scores where most of the mass sits mid-range and a thin tail runs high, house prices, claim amounts, session durations. With 300 rows and 10 folds, the twenty highest-value rows are dealt at random across the folds; one fold may get five of them and another none. The fold that got five is asked to predict values the training data barely covers, and its error is large; the fold that got none is scored only on the easy middle. ## The construction Bin the target, then stratify on the bin. Concretely: sort the target, cut it at the quartiles (or deciles, depending on sample size), attach the bin index to each row as a temporary label, and run stratified k-fold on that label. Each fold then holds roughly a quarter of its rows from each quartile of the target distribution. Use *quantile* bins — equal numbers of rows per bin — rather than equal-width bins. On a skewed target, equal-width bins put almost every row into the first bin and a handful into the last, which reproduces the problem you were trying to solve. Quantile bins are equal-count by construction, so each bin has enough rows to be spread across the folds. The bin label exists only for the splitter. It is discarded afterwards; the model still trains on the continuous target with a regression loss, and the metrics are still regression metrics. ## How many bins More bins mean a tighter match between the fold and population target distributions, and fewer rows per bin. The binding constraint is the same one that limits k for a rare class: a bin with fewer rows than there are folds cannot be represented in every fold. As a rule of thumb, keep the number of bins small enough that each bin holds several times k rows — with 300 rows and 10 folds, quartiles give 75 rows per bin, comfortably enough; twenty bins would give 15 per bin, which is already marginal. Quartiles or quintiles are a good default; deciles are worth it only on larger samples. ## Why per-fold R-squared is the most sensitive metric R-squared on a fold is one minus the ratio of the fold's squared error to the fold's total squared deviation from its own mean target. The denominator is the fold's target variance. Two folds with exactly the same absolute prediction error will report different R-squared values if one has a wider spread of targets than the other, and the narrow fold will look worse. Balancing the target distribution across folds removes that source of movement. Metrics defined purely on absolute error — mean absolute error, root mean squared error — are less exposed to it but still move when one fold holds a disproportionate share of extreme values, because large targets tend to carry large errors. ## Is using the target to build the folds a form of leakage? No, and this is worth being able to answer crisply. Leakage means information from the held-out rows reaches the model — through a feature, a fitted parameter, a preprocessing statistic. Binning the target uses the labels only to decide which row goes into which fold. No feature is derived from it, no parameter is fitted on held-out values, and the model never sees a validation row's target. It is exactly the same operation stratified classification folds perform, and nobody calls that leakage. The rule to hold onto is that the split may look at the label; the model may not. ## When to skip it With a large sample and a target that is not badly skewed, random folds already match the population closely and stratifying on bins changes nothing measurable. It is worth reaching for when the sample is small, when the target is heavily skewed or multimodal, when a rare high-value region is what the business actually cares about, or when you have noticed fold metrics moving more than the model plausibly does.