Your gradient-boosted trees stop growing far short of the depth cap - which parts of the regularized objective are refusing the splits?

## What the gain number is In a regularized boosting objective each leaf's best achievable contribution is `-0.5 * G^2 / (H + lambda)`, where `G` is the sum of the loss gradients over the rows in the leaf and `H` the sum of the Hessians. Splitting one leaf into two replaces one such term with two, and adds one leaf to the tree, which costs `gamma`. So the improvement from a candidate split is ``` gain = 0.5 * [ G_L^2/(H_L+lambda) + G_R^2/(H_R+lambda) - G^2/(H+lambda) ] - gamma ``` where `G = G_L + G_R` and `H = H_L + H_R`. This is not an impurity measure borrowed from a classification tree; it is the literal reduction in the objective the model is minimising, penalty included. That is the elegance of the regularized formulation: growth decisions and leaf values come out of the same expression. ## The three controls that suppress growth **The per-leaf cost, `gamma`.** Charged once for every leaf, so any single split must earn more than `gamma` to be worth making. It acts as pre-pruning with an explicit price tag. This is why the marginal leaf is the one that fails: the first splits in a tree separate large, strongly signalled regions and produce gains far above `gamma`, but by the time you are considering a fortieth leaf you are carving up a region holding few rows and little residual pressure. The improvement it offers is small, `gamma` is unchanged, and the split stops paying for itself. Growth halts naturally at whatever tree size the data can afford. **The L2 penalty on leaf scores, `lambda`.** It appears in the denominator of every term in the gain, so raising it shrinks all three quantities. It does not shrink them evenly: a child with a small summed Hessian has its `G^2/(H+lambda)` term suppressed hardest, so it is exactly the thin, few-row branches whose gains collapse first. Raising `lambda` therefore prunes the sparse periphery of the tree before it touches the well-supported trunk. **A floor on child curvature.** A split can also be refused before its gain is even compared, if either child's summed Hessian falls below a required minimum. For squared error this is effectively a minimum row count. For log loss it is stricter and more interesting, because rows the model already predicts confidently contribute almost no curvature - a child can hold plenty of rows and still be judged unsupported if the model has already learned them. ## A refusal, concretely Suppose the best candidate on a node scores a gain of 0.4 and the configured minimum gain is 1.0. The split is not made, the node stays a leaf, and the branch is finished. Two things follow that people miss. First, this is **pre-pruning**: the decision is taken during growth, not by trimming a fully grown tree afterwards. Second, the search is **greedy** - having refused this node, the split finder never looks at what could have been split beneath it. A weak split masking a strong pair of grandchildren is invisible. That is the price of the threshold, and it is why some implementations offer the alternative of growing to a depth cap while permitting negative-gain splits and pruning back afterwards. ## Negative gains are normal Gain can be negative. If the children's combined objective improvement is smaller than `gamma`, or if `lambda` flattens the children's terms enough, the split makes the penalized objective worse and is rejected. A candidate who insists gain is always non-negative is thinking of an unpenalized impurity criterion, where any split weakly reduces impurity by construction. ## Diagnosing it Shallow trees are a symptom, not a diagnosis. Work through it in order. 1. **Look at training loss, not just validation loss.** If both are poor and still falling when boosting stops, the model is under-fitting and structural regularization is a prime suspect. If training loss is excellent and validation is not, the trees are shallow for some other reason and loosening the penalty will only cost you. 2. **Count leaves per tree.** Trees pinned well under the depth cap with a stable, small leaf count point at a threshold or a curvature floor, not at data volume. 3. **Look at the distribution of the gains being accepted.** A dense cluster of rejected candidates just under the threshold means you are clipping real structure; a long empty stretch below it means the threshold is not binding and the signal genuinely runs out. 4. **Change one control at a time.** Lower the minimum gain, or lower the L2 penalty, or lower the child-curvature floor - separately, re-checking held-out loss each time - so you learn which one was binding instead of just landing on a configuration that happens to work. And keep the alternative honest: shallow trees plus many boosting rounds is a legitimate, often better-generalising model than deep trees plus few. Strong structural regularization is a choice, not automatically a bug. The bug is only when both curves say the model has not finished learning.