Decision trees

For structured tabular data, predictions are often rules in disguise: thresholds on a few columns, then interactions ("new user and mobile and no verified email"). A decision tree is literally a program that learns those rules from data.

The big idea

A tree partitions the feature space with axis-aligned splits: if logins_14d < 3 and tickets_30d ≥ 2 then churn else …. At each leaf it predicts the mean of y (regression) or the class distribution (classification).

Deep trees memorize the training set. Control capacity with max_depth, min_samples_leaf, or pruning. A single deep tree has low bias and high variance — great fit, fragile generalization. See overfitting.

Neural nets can approximate the same functions, but they need more data, careful normalization and usually more ops cost to get there. On a 200-column warehouse extract with missing cells and mixed categoricals, trees are the path of least resistance.

Walking through Gini

Binary labels in a node: 80 negatives, 20 positives → p = 0.2.

Gini = 1 − (0.8² + 0.2²) = 1 − 0.68 = 0.32
Entropy = −(0.8 log₂ 0.8 + 0.2 log₂ 0.2) ≈ 0.72 bits

A split that yields children (70N, 5P) and (10N, 15P) drops weighted Gini sharply. That split is preferred over one that barely changes class mix.

from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0, stratify=y)

tree = DecisionTreeClassifier(max_depth=3, random_state=0)
tree.fit(X_tr, y_tr)
print(round(tree.score(X_te, y_te), 3))
print(export_text(tree, feature_names=list(load_breast_cancer().feature_names))[:400])

Typical output is a test accuracy around 0.9 and a short flowchart of worst radius and similar columns. Depth 3 stays readable. Depth 20 is a memorization engine.

Libraries allow monotone constraints: predicted risk must not decrease when past_defaults increases. Use them when a stakeholder will reject any model that violates domain monotonicity. Constraints reduce variance in that direction and sometimes cost a bit of AUC — usually worth it for credit and pricing.

Common mistakes

• Shipping a single deep tree. It is interpretable as a flowchart and unstable. Prefer a forest or a boosted ensemble.
• Ignoring min_samples_leaf. Tiny leaves memorize one row.
• Treating Gini and entropy as rivals that change the story. They usually pick similar splits. Depth and leaf size matter more.

Check yourself

1. Node with 80 negatives and 20 positives. What is Gini? Answer: 1 − (0.8² + 0.2²) = 0.32.
2. Why does a deep tree overfit? Answer: Each extra split can isolate a few training rows, so leaves memorize noise.

Key takeaways

• Trees split by impurity or variance reduction. Depth controls overfitting.
• Gini and entropy measure how mixed a node is. The greedy split picks the biggest purification.
• A single deep tree is a fragile flowchart. Ensembles come next.
• Monotone constraints encode business rules the tree is not allowed to break.

Further learning

Next: bagging and random forests.

Decision tree intuition: