Trees and gradient boosting

For structured tabular data — rows of numbers and categories with a label — gradient boosted decision trees (GBDTs) are still the default that wins Kaggle-style leaderboards and a huge fraction of production ranking and risk models. Deep nets dominate images, audio and text. On churn tables, click logs and credit features, start with LightGBM or XGBoost unless you have a reason not to.

A decision tree is a program that learns threshold rules. Ensembles of trees average or boost many such programs so no single brittle flowchart owns the prediction. Pair this with feature engineering — boosting amplifies good columns and leaked ones equally — and with evaluation so you early-stop on a metric that matches the decision.

Concepts in this section

Read them in order. Each one builds on the one before.

Why GBDTs beat deep nets on most tabular problems

Trees handle mixed types, missingness and irregular decision boundaries without heavy preprocessing. Useful signal is often piecewise thresholds and interactions of a few columns — the trees' inductive bias. Datasets are "small" by deep-learning standards (10⁴–10⁷ rows), where overparameterized nets overfit unless heavily regularized. Training is fast on CPU, and iteration speed matters more than FLOPs.

Choose LightGBM for the first production model. Revisit nets if you have many millions of rows, mostly dense learned features, or a team already operating a deep ranking tower.

At work, a LightGBM model with a clean feature pipeline, early stopping and a SHAP review is a complete delivery for many tabular launches. Log the best iteration and metric in the model registry. Freeze the threshold from validation, not a default 0.5.

Interview angle: explain bagging versus boosting, why boosting uses shallow trees, what early stopping does, how you would tune learning rate and n_estimators together, when you would pick CatBoost, and how SHAP differs from gain importance.

Common mistakes

• Dual-writing a leaked column into a boosted model. SHAP will show it. The metric will lie until you catch it.
• One shared deep tree as the production model. Use a forest or a GBDT.
• Switching to a neural net because it is fashionable. Need a concrete reason: scale, unstructured inputs, or an existing deep stack.

Key takeaways

• Trees split by impurity or variance reduction. Depth controls overfitting.
• Forests bag trees and subsample features to cut variance. OOB scores help.
• Gradient boosting adds shallow trees on loss gradients with shrinkage. Early-stop on validation.
• LightGBM / XGBoost / CatBoost differ mainly in speed and categorical handling. Features matter more.
• GBDTs remain the tabular default. Deep nets need a concrete reason.

Further learning

XGBoost: A Scalable Tree Boosting SystemTree boosting is a highly effective and widely used machine learning method. In this paper, we describe a scalable end-to-end tree boosting system called XGBoost, which is used widely by data scientists to achieve sta…arXiv.org

ISL chapters on trees, bagging, random forests and boosting:

An Introduction to Statistical LearningAn Introduction to Statistical Learning LightGBM ParametersParameters — LightGBM 4.7.0.99 documentationlightgbm.readthedocs.io