In gradient boosting, how does a second-order Taylor expansion set each leaf's weight?

## The objective a boosted tree actually minimises Gradient boosting builds an additive model one tree at a time. After `t-1` rounds every row `i` carries a running raw score `s_i` (the *margin*, turned into a probability or a prediction by whatever link the loss uses). Round `t` searches for a tree `f` that reduces ``` obj = sum_i loss(y_i, s_i + f(x_i)) + Omega(f) ``` The regularized formulation makes the complexity term explicit: ``` Omega(f) = gamma * T + 0.5 * lambda * sum_j w_j^2 ``` `T` is the number of leaves in the new tree, `w_j` is the single score leaf `j` outputs, `gamma` is a fixed cost charged for each leaf and `lambda` is an L2 penalty on the leaf scores. Notice what is being penalised. A tree has no feature coefficients, so there is nothing there to shrink; the penalty falls on how many leaves the tree has and how large a number each of them emits. ## Why expand, and why to second order The loss is generally not something you can minimise over a whole tree in closed form. So approximate it locally. A second-order Taylor expansion of `loss(y_i, s_i + z)` around `z = 0` is ``` loss(y_i, s_i) + g_i * z + 0.5 * h_i * z^2 ``` where `g_i = d loss / d s` and `h_i = d^2 loss / d s^2`, both evaluated at the current score. The first term does not depend on the new tree, so it drops out of the optimisation. What is left is a quadratic in the tree's output. Stopping at first order would leave `g_i * z` only, a linear function with no minimum: you would need an externally chosen step size or a line search per leaf. Keeping `h_i` supplies the curvature, which is exactly what turns the update into a **Newton step** rather than a gradient step. Two practical consequences follow. First, the step size is derived, not guessed, and it adapts per leaf to how sharply the loss is bending there. Second, the whole machinery becomes loss-agnostic: supply two derivatives and the identical split-finding and leaf-fitting code works for squared error, log loss, Poisson, ranking objectives, or a custom loss. This is the formulation XGBoost introduced, and LightGBM and CatBoost score their splits with the same gradient-and-Hessian machinery. ## Grouping the sum by leaf The sum runs over rows, but a tree only has as many free numbers as it has leaves. Let `I_j` be the set of rows falling into leaf `j`; every one of them gets `f(x_i) = w_j`. Regrouping, ``` obj = sum_j [ G_j * w_j + 0.5 * (H_j + lambda) * w_j^2 ] + gamma * T G_j = sum over I_j of g_i H_j = sum over I_j of h_i ``` The `lambda` from the penalty has merged straight into the quadratic coefficient. Each leaf is now an independent scalar quadratic `a*w + 0.5*b*w^2` with `b > 0`, whose minimum sits at `w = -a/b`: ``` w_j = -G_j / (H_j + lambda) ``` Substituting back, the best objective that leaf can reach is `-0.5 * G_j^2 / (H_j + lambda)` — a quantity worth remembering, because the score used to compare candidate splits is built directly from it. ## Reading the two sums `G_j` is the total pressure on the leaf: which way, and how hard, the loss wants these rows' scores to move. `H_j` is the total curvature the leaf carries, often called its *cover*. Because both are sums over rows, they scale with how many rows land in the leaf. For log loss with `p = 1/(1 + exp(-s))`, `g = p - y` and `h = p*(1-p)`. The Hessian is at most 0.25, hit at `p = 0.5`, and collapses toward zero for rows the model is already confident about. So a leaf full of rows the model has already got right contributes very little curvature, even if it holds many of them. For squared error written as `0.5*(y - s)^2`, `g = s - y` and `h = 1`, so `H_j` is just the row count and the leaf weight reduces to the mean residual shrunk by `n/(n+lambda)` — the familiar case, recovered as one instance of the general formula. ## What lambda actually does `lambda` sits in the denominator, so it damps the leaf's output, and it damps it *relative to the curvature already there*. Take a leaf holding three rows against one holding three thousand. The three-row leaf's `H` may be a fraction of a unit, so a `lambda` of 1 dominates the denominator and pulls the output most of the way back toward zero. The three-thousand-row leaf's `H` is in the hundreds; the same `lambda` moves it by well under a percent. That asymmetry is the point: the penalty is a prior that thinly supported leaves should not shout, and it costs nothing where the evidence is thick. ## Where candidates go wrong The two derivatives are taken with respect to the **prediction**, not with respect to any tree parameter — there is no differentiable parameter in a split threshold. `lambda` does not penalise features or splits, only leaf magnitudes. And the second-order term is not an optional speed optimisation: without it there is no closed-form leaf weight and no closed-form split score to compare candidates with.