How does a regression tree decide where to split when the target is continuous?
answer
- no classes, so no Gini
- impurity becomes spread around the mean
- weighted child variance, biggest drop wins
- leaf predicts the mean of its rows
- squared error means outliers steer the cut
basics
~20 sA regression tree scores each candidate split by the sample-weighted variance of its two children and keeps the largest variance reduction. Each leaf then predicts the mean of the training targets that land in it.
solid answer
~50 sImpurity for a continuous target is spread rather than class mixture, so Gini and entropy are replaced by within-node variance - equivalently, the sum of squared deviations from the node's mean. For each candidate cut the learner computes `(n_L/n)*var(left) + (n_R/n)*var(right)` and takes the largest drop from the parent's variance. Because the mean is the constant that minimises squared error, each leaf predicts the mean of its training targets. There is a neat identity: minimising weighted child sum-of-squares is the same as maximising `(n_L * n_R / n) * (mean_L - mean_R)^2`, so the tree is really hunting for cuts that separate the target means while keeping both sides reasonably large. If you switch the criterion to absolute error instead, leaves predict the median and the tree becomes far less sensitive to extreme targets, at higher computational cost.
go deeper
Recall that with a numeric target the tree measures spread instead of class mixture, and that the prediction at a leaf is just the average of the training values that reached it.
Be able to write the weighted-variance score and compute a variance reduction from two child means and variances, and to say why the mean is the leaf prediction under squared error.
Demonstrate that you have been burned: extreme targets hijacking a root split, heteroscedastic targets pulling every cut into the noisy region, and the decision of when an absolute-error criterion is worth its training cost.
Own the choice of what the tree should optimise at all. Argue when squared error matches the business loss, when a robust or asymmetric objective matters more, and what a fleet of teams reporting incomparable variance reductions costs the organisation.
## Same machinery, different impurity A regression tree is grown by exactly the procedure a classification tree uses: enumerate candidate cuts, score each by the size-weighted impurity of the two children, keep the best, recurse. The only thing that changes is what "impurity" means. With a continuous target there are no class proportions to square or take logs of, so homogeneity becomes **low spread of the target values**. The standard choice is the within-node **variance**, or equivalently the **sum of squared errors** about the node's mean: ``` SSE(node) = sum over rows of (y_i - mean_y)^2 score(split) = (n_L/n)*var(left) + (n_R/n)*var(right) gain(split) = var(parent) - score(split) ``` This is often called **variance reduction**, and it is the regression counterpart of information gain. ## A concrete node Suppose a node holds 400 completed food-delivery orders and the target is elapsed time in minutes, averaging 34 with a variance of 180. A candidate cut on "kitchen prep time above 12 minutes" sends 150 orders left (mean 47, variance 90) and 250 right (mean 26, variance 70). The weighted child variance is `(150/400)*90 + (250/400)*70 = 33.75 + 43.75 = 77.5`, so the variance reduction is `180 - 77.5 = 102.5`. A competing cut on courier vehicle type that leaves both children near mean 34 will reduce almost nothing, however intuitively meaningful the feature is. The criterion only rewards **separation of the target's central value**, weighted by how many rows sit on each side. ## The identity worth knowing Total sum of squares decomposes into within-group plus between-group parts, and for a two-way split the between-group term is ``` (n_L * n_R / n) * (mean_L - mean_R)^2 ``` Since the parent's total is fixed while you search, **minimising the weighted within-child SSE is identical to maximising that expression**. Two consequences fall out immediately. First, the tree wants a large gap between the two child means. Second, it wants the split to be reasonably balanced, because `n_L * n_R` collapses when one side is tiny - a 5-row child with a wildly different mean earns far less than a 150-row child with a moderately different one. This is the regression version of the same size-weighting instinct that stops classification trees from chasing tiny pure children. ## Why the leaf predicts the mean The constant `c` minimising `sum (y_i - c)^2` is the arithmetic mean, so once the criterion is squared error the leaf prediction is forced: the mean is the value that makes the leaf's own impurity as small as possible. Change the criterion to mean absolute error and the minimising constant becomes the median, so the leaf predicts the median instead. The two choices go together; predicting the median while splitting on variance would be internally inconsistent. ## What this criterion is bad at - **Outliers.** Squared error is quadratic, so one order that took 210 minutes because the restaurant closed mid-shift can, on its own, make a split look excellent. The absolute-error criterion is the principled remedy; it is also markedly slower, because a running mean updates in constant time while a running median does not. - **Scale dependence of the number.** A variance reduction of `102.5` is in minutes-squared. It cannot be compared to a variance reduction computed on a different target, on a log-transformed target, or on a differently-sized node. It is a ranking device inside one node, nothing more. - **Heteroscedasticity.** If the target's spread genuinely grows with its level - late orders are both slower and more variable - the criterion will keep splitting the high-variance region simply because that is where squared error lives, not because those cuts carry the most information. - **Nothing forces monotonicity or smoothness.** The fitted function is a set of constants; two adjacent regions can predict values in any order. ## Interview-ready summary "Impurity is variance instead of Gini. Score each cut by the row-weighted variance of the two children, take the biggest reduction, and let each leaf predict the mean of its rows. Equivalently, the tree is maximising `n_L*n_R/n * (mean_L - mean_R)^2`, which is why it prefers big separations from reasonably balanced cuts - and why a handful of extreme targets can hijack the search."
- Why does a leaf predict the mean rather than the median of its training targets?Because the criterion and the prediction must agree. The mean is the constant that minimises the sum of squared deviations, so under a squared-error impurity it is the value that makes the leaf's own impurity smallest. Switch the criterion to mean absolute error and the minimising constant becomes the median, and the leaf should predict that instead.
- A few targets in the node are extreme outliers. How does that distort variance-reduction splitting?Squared error weights a deviation quadratically, so a handful of extreme values dominate both the parent's impurity and every candidate's score. The tree will happily spend its root split isolating those rows, because doing so removes an enormous amount of squared error while explaining almost nothing. Switching to an absolute-error criterion, with leaves predicting the median, removes most of that pull at a real cost in training time.
- Two teams report variance reductions of 102 and 4 at their root splits. Can you conclude anything?Nothing at all. Variance reduction is in squared units of whatever the target is, so a model predicting minutes and a model predicting a probability produce numbers that are not on the same scale. It also scales with node size and with the target's own variance. The quantity is only meaningful as a ranking of candidate cuts within one node.
saying these in an interview costs you the question
- Says regression trees use Gini or entropy on the target
- Averages child variances without weighting by node size
- Thinks the leaf fits a line rather than a constant
- Says the cut is chosen to balance the two child sizes
- Compares variance reductions across different targets