Bagging and random forests

A single deep tree overfits. Bagging trains many trees on bootstrap samples of the data and averages their predictions. Each tree overfits differently. The ensemble smooths them out.

The big idea

A random forest adds feature subsampling: each split considers only a random subset of features (max_features="sqrt" is a common default for classification). That decorrelates trees further.

Averaging reduces variance. A single tree is interpretable as a flowchart but unstable. A forest is accurate and robust, harder to read as one diagram, still gives feature importances. Prefer forests over lone trees for almost any real tabular task that is not literally a policy you must print as rules.

More trees in a forest almost monotonically help until the benefit flattens. Unlike boosting, you rarely overfit by adding trees if each is trained with bagging. Compute budget and latency are the usual stop conditions. n_estimators in the low hundreds is common.

OOB is not a substitute for a time-based test when the world drifts. It only estimates IID generalization of the forest. Still keep a final test set.

Walking through a forest

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0, stratify=y)

rf = RandomForestClassifier(
    n_estimators=300,
    max_depth=None,
    min_samples_leaf=2,
    n_jobs=-1,
    oob_score=True,
    random_state=0,
)
rf.fit(X_tr, y_tr)
print("oob", round(rf.oob_score_, 3), "test", round(rf.score(X_te, y_te), 3))

Typical output is OOB and test both around 0.95 on this dataset. The two numbers agreeing is the point: OOB was a decent stand-in for a holdout, on IID data.

Boosting usually wins pure accuracy on tabular benchmarks. Forests are friendlier defaults when you want fewer knobs and decent OOB estimates. Keep both in the bakeoff. The gap tells you whether features or capacity are the bottleneck.

Common mistakes

• Treating OOB as a time-based test. Drift needs a later holdout.
• Stopping at 10 trees. Variance is still high. Hundreds is the usual range.
• Reading a forest as one flowchart. Use importances or SHAP, not a single path.

Check yourself

1. About what fraction of rows is out-of-bag for one tree? Answer: About 37%.
2. Why does sampling features at each split help? Answer: Trees become less correlated, so averaging cuts more variance.

Key takeaways

• Bagging averages trees trained on bootstrap samples, which cuts variance.
• Random forests also subsample features at each split.
• OOB scores help, but they are not a substitute for a drifted-world test set.
• Forests are the robust baseline. Boosting is the usual accuracy winner once you invest in tuning.

Further learning

Next: gradient boosting.

Random forest intuition: