skip to content

In AdaBoost, why is a one-split decision stump the standard weak learner?

level: juniorimportance: should knowfreq 50%

answer

  1. the loop is supposed to do the work
  2. high bias, low variance, very cheap
  3. an edge over chance, nothing more
  4. one feature, one threshold, two outputs
  5. no interactions without extra depth

basics

~20 s

Boosting supplies accuracy through many rounds, so it needs a high-bias, low-variance base learner only slightly better than chance. A stump, one feature and one threshold, is the cheapest such learner and refits fast each reweighted pass.

solid answer

~50 s

AdaBoost supplies the model capacity itself, through hundreds of rounds of reweighting and voting, so the base learner should contribute as little capacity as possible. A depth-1 stump — pick one feature, pick one threshold, predict two constants — is about the weakest useful classifier: heavily biased, barely variable, and cheap enough to refit on every new weight distribution. The only requirement is that each round's stump beat chance **on the current weighted data**, meaning weighted error below 0.5; that keeps the vote weight positive and the training-error bound shrinking. Stumps also make the ensemble a purely additive function of single features, with no interaction terms, which is why practitioners move to depth 2 or 3 when the target genuinely depends on feature combinations. A strong base learner, by contrast, drives weighted error near zero immediately and the sequence stops being a correction process.

go deeper

for a junior

Recall that a stump is a one-split tree and that boosting wants many weak learners rather than one strong one. Being able to say "slightly better than a coin flip" is the bar here.

for a middle

Explain the requirement precisely: weighted error below 0.5 on the current round's distribution, which is what keeps the vote weight positive and the training-error bound below 1 each round.

for a senior

Talk about depth as a real complexity knob — what interactions stumps cannot express, why a too-strong base learner collapses the sequence, and how you would choose depth from held-out evidence rather than convention.

for a principal

Own the framing that base-learner capacity and round count are two ways of buying the same thing, and argue which one you would spend given constraints on training cost, interpretability and how much interaction structure the domain really has.

## What "weak" actually means A **weak learner** is a classifier whose accuracy only has to exceed random guessing by some margin. Formally, in each AdaBoost round the learner must achieve **weighted error `e_t` below 0.5 on the weight distribution of that round** — not on the original, uniformly weighted data. That distinction matters: as boosting proceeds, the weighted problem gets progressively nastier because the weight piles onto the rows earlier learners could not fit, so "better than chance" is a fresh requirement every round, not a one-off property of the model class. Why does 0.5 matter so specifically? The round's vote weight is `alpha_t = 0.5 * ln((1 - e_t) / e_t)`, which is positive exactly when `e_t < 0.5`, zero at `e_t = 0.5`, and negative above it. And the training-error bound is a product of the per-round factors `2 * sqrt(e_t * (1 - e_t))`, each of which is strictly below 1 only when `e_t` is away from 0.5. Writing `e_t = 0.5 - g_t`, the bound shrinks like `exp(-2 * sum of g_t^2)`: a small but persistent edge, compounded over many rounds, is enough to drive training error toward zero. ## Why a stump fits that role A **decision stump** is a decision tree of depth 1: choose one feature, choose one threshold, and predict one class on each side. It has essentially two degrees of freedom. - **It is high-bias and low-variance.** Boosting is a bias-reduction machine — each round exists to fix what the ensemble still gets wrong. Feeding it a base learner that is already low-bias leaves nothing for the sequence to do, and the residual capacity turns into variance instead. - **It is very cheap.** The base learner is refitted from scratch on a new weight distribution every round, hundreds of times. Scanning one feature at a time for the best weighted split is the cheapest fitting problem in the tree family. - **It is naturally a feature selector.** Each round commits to a single feature-threshold rule, so the sequence of chosen stumps reads as an ordered list of what mattered. This was the point of the classic face-detection cascade: depth-1 stumps over simple rectangle features, boosted into a stage, with early stages rejecting most windows so later stages only see hard candidates. The stump was chosen precisely because it could be evaluated in a handful of operations. - **It always beats chance on a two-class problem** as long as some feature carries any signal at all under the current weights — and if none does, that is diagnostic information you want, not something to paper over with a stronger learner. ## What depth buys and costs Stumps have one real structural limitation: an additive combination of single-feature step functions **cannot express feature interactions**. If a target depends on "high pressure *and* low temperature" in a way that neither variable predicts alone, no number of stumps will capture it; the ensemble is an additive function of the individual features (nonlinear in each one, since it is a sum of step functions, but with no cross terms). The standard fix is depth. A depth-`d` tree can represent interactions among up to `d` features, so depth 2 or 3 is common when interactions are expected. The cost is symmetric: deeper base learners drive weighted error down fast, alphas become large early, and the ensemble reaches low training error in few rounds — which sounds good but means the smooth, incremental correction that gives boosting its generalisation behaviour has been replaced by a handful of aggressive fits. Depth is therefore a genuine complexity knob, and it is chosen the same way any complexity knob is: by held-out performance, not by taste. ## The other failure direction What happens if the base learner is *too strong* — say, a fully grown tree? It can drive weighted error to nearly zero on the first round. Then `alpha` is enormous, the reweighting step barely has anything left to redistribute, and the "ensemble" is effectively one overfitted tree with decoration. Boosting a strong learner is not an error the algorithm catches for you; it just quietly stops being boosting. ## How to say it in an interview One sentence: *the base learner supplies the shape, the boosting loop supplies the power, so you want the weakest learner that still has an edge* — and a stump is the canonical example, with depth 2 or 3 reserved for problems where interactions are real.

  • What breaks if you boost a fully grown decision tree instead of a stump?
    The first learner can drive weighted error close to zero, so it earns a huge vote weight and the reweighting has almost nothing left to redistribute. The result is effectively one overfitted tree with a few decorative extras rather than a sequence of corrections, and the incremental error reduction that gives boosting its behaviour never happens.
  • When would you deliberately use depth-2 or depth-3 trees rather than stumps?
    When the target depends on feature combinations. A stump ensemble is additive in single features, so it cannot represent an effect that only appears when two variables move together. Depth `d` buys interactions among up to `d` features; you pick the depth on a held-out set, since each extra level also makes each round a stronger, less "weak" learner.
  • Does the weak-learning condition apply to the original data or the reweighted data?
    The reweighted data, freshly each round. A learner that looks fine on the uniform training set can fail to beat chance once the weight has concentrated on the rows previous rounds could not fit. Persistent weighted errors drifting toward 0.5 are the signal that the base learner has run out of edge on the hard region.

saying these in an interview costs you the question

  • Says a stronger base learner always gives a better ensemble
  • Thinks the weak learner must beat chance on the original unweighted data
  • Claims a stump ensemble is a linear model in the raw features
  • Believes stumps can capture feature interactions given enough rounds
  • Cannot state any numeric requirement on the weak learner's error

context

resolving links…