How does gradient boosting build a regression model stage by stage?
answer
- It starts from one number
- Each stage looks at what is left
- The tree's target is the leftover error
- Earlier trees are frozen, never refitted
- The ensemble is a sum, not a vote
basics
~10 sGradient boosting starts from a single constant prediction, then repeatedly fits a shallow tree to the errors the model still makes and adds that tree to the running total. Earlier trees are never refitted.
solid answer
~40 sIt is forward stagewise additive modelling. Stage zero is a constant `F0` -- the mean of the target for squared-error regression. At each stage you compute what the current model still gets wrong on every training row, fit a shallow tree with those leftovers as its target, give each leaf one constant value, and add that tree into the running sum: `F_m = F_{m-1} + h_m`. Then you recompute the leftovers against the improved model and go again. Two details matter. The target changes every round, so no tree after the first ever sees the original labels. And every tree already added is frozen -- the procedure is greedy and one-pass-per-stage, which is what *stagewise* means. The base learners stay shallow because boosting works by many small corrections, not one big one.
code
python · 23 linesmileage = [15, 30, 42, 55, 60, 72, 88, 95, 110, 130]
price = [22.0, 19.5, 18.0, 16.0, 15.5, 13.0, 11.5, 10.5, 8.0, 6.5]
def best_stump(x, r):
best = None
for t in x:
left = [ri for xi, ri in zip(x, r) if xi <= t]
right = [ri for xi, ri in zip(x, r) if xi > t]
if not left or not right:
continue
ml, mr = sum(left) / len(left), sum(right) / len(right)
sse = sum((v - ml) ** 2 for v in left) + sum((v - mr) ** 2 for v in right)
if best is None or sse < best[0]:
best = (sse, t, ml, mr)
return best[1], best[2], best[3]
F = [sum(price) / len(price)] * len(price) # F0 = the global mean
for stage in (1, 2, 3):
resid = [y - f for y, f in zip(price, F)] # negative gradient of squared error
t, ml, mr = best_stump(mileage, resid)
print(stage, 'split at', t, 'error before', round(sum(r * r for r in resid), 2))
F = [f + (ml if xi <= t else mr) for xi, f in zip(mileage, F)]
print('final predictions', [round(f, 2) for f in F])go deeper
Be ready to state the loop out loud: start from a constant, compute what is still wrong, fit a small tree to that, add it, repeat. Knowing that the trees are summed rather than averaged is the discriminating detail here.
An interviewer expects you to explain where F0 comes from, that the tree's target is recomputed every round, and that each leaf contributes one constant. Be able to justify shallow base learners in bias-variance terms rather than as a convention.
Show you can reason about the consequences: training cannot be parallelised across stages, inference cost grows linearly with the number of trees, and base-learner depth is a claim about how many of your features genuinely interact. Tie depth choice to the feature structure of a real problem.
Own the tradeoff of choosing a sequential ensemble at all: it usually wins on tabular accuracy but costs retraining time, makes the model a long ordered list rather than a parallel committee, and couples serving latency to the stage count. Be ready to say when that price is not worth paying.
## The shape of the algorithm Gradient boosting builds one model out of many small ones, added **one at a time**, where each new piece is fitted to what the sum of the previous pieces still gets wrong. The formal name for this is *forward stagewise additive modelling*: the final prediction is a plain sum, ``` F_M(x) = F0 + h_1(x) + h_2(x) + ... + h_M(x) ``` and the stages are built left to right, with every earlier term **frozen** once it is added. ### Stage zero: the constant model Before any tree exists, the model is a single number `F0` -- the constant that best fits the target under the chosen loss. For squared-error regression that constant is the arithmetic mean of the training targets; for a binary problem trained with log loss it is the log-odds of the training positive rate. Getting this right is not cosmetic: every later stage is measured against it, and starting from a sensible base rate means the first tree already works on real structure rather than on the overall level of the target. ### One stage, in four moves Suppose we are predicting the resale price of a used car and the model so far is `F_{m-1}`. 1. **Compute what is left over.** For each training row, form the current error. Under squared-error loss written as `L = 0.5 * (y - F)^2`, the quantity that drives the next stage is exactly `r_i = y_i - F_{m-1}(x_i)` -- the ordinary residual. A car whose true price is 16.0 and whose current prediction is 14.5 contributes a residual of `+1.5`. 2. **Fit a shallow tree to those leftovers.** The tree is trained with the residuals as its target, so it partitions the feature space into a handful of regions where the leftover error behaves similarly -- say, high-mileage cars whose prices the model is still overestimating. 3. **Choose one number per leaf.** Each terminal region gets a single constant, chosen to reduce the loss for the rows that fall into it. Under squared error that constant is the mean of the residuals in the leaf. 4. **Add it in.** `F_m(x) = F_{m-1}(x) + h_m(x)`. The residuals are then recomputed against the new, slightly better model, and the next stage begins on a different set of leftovers. Because step 1 is recomputed every round, the *target changes at every stage*. This is the single detail that separates boosting from every averaging ensemble: no tree after the first ever sees the original labels. ### Why the trees stay shallow The base learner in boosting is deliberately weak -- typically a tree of depth two to about six, and in the earliest formulations a single split. Two reasons. First, boosting is a **bias-reduction** machine. Its power comes from taking many small, correcting steps, and that only works if each step is small. A fully grown tree can drive the residuals of the training set to nearly zero in one stage; there is then nothing left for stage two to learn except noise, and the ensemble collapses into a single overfitted tree. Second, the depth of the base learner caps the **interaction order** the ensemble can express. A tree of depth one is a sum of single-feature effects -- an additive model, no interactions at all. A tree of depth two can express pairwise interactions, depth three can express three-way interactions, and so on. Choosing depth is therefore a statement about how many features you believe genuinely interact in your problem, not just a knob. ### Why the sum, and not a vote Averaging ensembles train their members independently, on perturbed copies of the data, and combine them by averaging or voting; their members are individually strong and the combination cancels variance. Boosting trains members in strict sequence, each one conditioned on all of its predecessors, and combines them by **adding**. That is why the ensemble members cannot be trained in parallel across stages, why the ensemble keeps improving on the training set essentially forever, and why the ensemble needs to be watched rather than simply grown. ### What you actually store and serve The trained model is an ordered list: the constant `F0` and then the trees. At prediction time every tree is evaluated and the outputs are summed -- the order of summation does not matter at inference, only at training. This is why inference cost grows linearly with the number of stages, and why a boosted model with a thousand shallow trees can still be cheaper to evaluate than one very deep tree's worth of memory traffic would suggest. ### The common mistakes Candidates who have only read a summary say the trees are trained on the labels and then averaged. They are trained on leftovers and then summed. Candidates also sometimes claim earlier trees are refitted as the ensemble grows -- they are not; the greedy, one-pass-per-stage nature of the procedure is exactly what *stagewise* (as opposed to full *stepwise* re-optimisation) means.
- Why are the base learners in gradient boosting kept shallow?Boosting reduces bias by taking many small corrections, so each step must be small. A fully grown tree would drive the training leftovers to nearly zero in one stage, leaving later stages nothing but noise to fit. Depth also caps interaction order: depth one gives a purely additive model, depth two allows pairwise interactions, and so on.
- What is the initial model F0, and does its choice matter?It is the constant that best fits the target under the chosen loss -- the mean of the targets for squared error, the log-odds of the positive rate for log loss. A poor choice is not fatal, since later stages can correct it, but it costs stages and starts the first tree fitting the overall level of the target instead of real structure.
- Can the base learner be something other than a tree?In principle yes -- any weak regressor that can be fitted to the leftovers works, and the stagewise loop is unchanged. Trees dominate in practice because they handle mixed feature types, non-linearity and interactions without scaling or manual transformation, and because a shallow tree is a naturally weak, cheap learner.
It is like planing a board: each pass shaves off only what is still uneven, and no pass undoes the one before it.
saying these in an interview costs you the question
- Says every tree is trained on the original labels
- Claims the trees are trained in parallel and averaged
- Thinks earlier trees are refitted as the ensemble grows
- Believes deeper base trees always make boosting better
- Describes the combination as a vote rather than a sum