Gradient boosting

Boosting builds trees sequentially. Each new tree fits the residuals (or gradients of the loss) of the ensemble so far. Predictions add up. That is the opposite of bagging, which trains trees independently and averages them.

The big idea

F₀(x) = base score (e.g. mean of y)
F_m(x) = F_{m−1}(x) + η · h_m(x)

h_m is a small tree. η is the learning rate (shrinkage).

Key knobs:

Why shallow trees? A depth-10 tree already models complex interactions alone. Boosting a forest of deep trees stacks capacity on capacity and overfits faster. A depth-4 tree is a weak learner — slightly better than the current residual guess — and the ensemble becomes strong by addition.

Early stopping watches a validation metric every few rounds and freezes the ensemble when improvement stalls. You set n_estimators deliberately too high and let stopping pick the effective size. Without it, people either underfit (too few trees) or overfit (train for thousands of rounds at high η).

Walking through three points

Targets y = [10, 20, 30], start F₀ = 20. Residuals [-10, 0, 10]. Tree 1 predicts roughly those residuals. With η = 0.5, F₁ ≈ [15, 20, 25]. New residuals [-5, 0, 5]. Tree 2 shrinks them again. After enough rounds F_M hugs y on train. Early stopping on validation prevents hugging noise.

Default gain / split-count importances are useful but biased toward high-cardinality features. SHAP (SHapley Additive exPlanations) attributes each prediction to features using game-theoretic shares — slower, much more trustworthy for stakeholder conversations. Use global SHAP to debug leakage (one feature dominates) and local SHAP to explain a single denied loan.

import numpy as np

y = np.array([10.0, 20.0, 30.0])
F = np.full(3, y.mean())  # 20, 20, 20
eta = 0.5
resid = y - F
h1 = resid.copy()         # toy: tree predicts residuals exactly
F = F + eta * h1
print("F1", F, "resid", y - F)

Output:

F1 [15. 20. 25.] resid [-5.  0.  5.]

Common mistakes

• Boosting deep trees. Capacity stacks on capacity. Keep them shallow.
• Skipping early stopping. You either underfit or memorize.
• Trusting gain importance on a high-cardinality ID. Use SHAP.

Check yourself

1. For squared error, what does the next tree fit? Answer: The residual y − F.
2. Why is a smaller learning rate usually better? Answer: Each tree is a smaller correction, so you need more trees but you overfit less.

Key takeaways

• Gradient boosting adds shallow trees on loss gradients with shrinkage.
• Early-stop on validation. Set n_estimators high and let stopping pick the size.
• Shallow trees are the point. Deep trees overfit faster here than in a forest.
• SHAP is the credible explanation. Gain importance is a rough hint.

Further learning

Next: XGBoost, LightGBM and CatBoost.

StatQuest’s gradient boost series is the clearest visual of residual fitting: