Feature scaling and leakage-safe pipelines

If income is in the tens of thousands and logins are in single digits, a linear model treats the two very differently. The large-scale feature dominates the penalty and the gradient steps. Feature scaling puts features on comparable footing. The catch is that the scaler must learn its numbers from the training data only. If it sees the test data, information leaks into training.

The big idea

Standardization rescales each feature to mean 0 and standard deviation 1:

z = (x − mean) / std

The mean and standard deviation come from the training split. Min–max scaling maps each feature to the range 0 to 1 using the training minimum and maximum.

Tree models mostly do not need scaling, because they split on thresholds. Linear models and neural nets do need it.

Walking through a scaled income feature

Say the training mean of income is 100,000 and the standard deviation is 40,000.

• An income of 140,000 becomes (140,000 − 100,000) / 40,000 = 1.0.
• An income of 60,000 becomes (60,000 − 100,000) / 40,000 = −1.0.

Now income and logins are both in "standard deviations from average", so the penalty and the gradient treat them on equal terms.

Here is a complete pipeline on the built-in breast cancer dataset. The scaler is part of the model, so it is fit on training data only.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0, stratify=y
)

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000, C=1.0)),
])
pipe.fit(X_train, y_train)
print("accuracy", round(pipe.score(X_test, y_test), 3))

Output:

accuracy 0.958

Calling pipe.fit fits the scaler on the training rows, then fits the classifier on the scaled rows. Calling pipe.score applies the same scaler to the test rows. Saving the whole pipeline also saves the scaler, so serving uses the same transform.

Leakage

Scaling the full dataset before splitting puts test statistics into training. The model then benefits from information it should never have seen, and the test score looks better than the real performance. The pipeline prevents this, because the scaler is refit inside each training step and only applied to the held-out rows.

Common mistakes

• Scaling before splitting. Fit the scaler on training rows only.
• Saving only the model. At serving time, the same scaler must run first. Save the pipeline.
• Scaling for tree models. Trees ignore monotone scaling, so the step adds complexity without benefit.

Check yourself

1. The training mean income is 100,000 and the standard deviation is 40,000. What is the standardized value for 60,000? Answer: (60,000 − 100,000) / 40,000 = −1.0.
2. Which rows should the scaler's mean come from? Answer: The training split only.

Key takeaways

• Unscaled features make the penalty and the gradient unfair to small-scale features.
• Standardize with training statistics, then apply the same transform everywhere.
• Put the scaler inside a pipeline so it cannot leak test data.
• Save the pipeline, not just the model, so serving matches training.

Further learning

The scikit-learn user guide covers pipelines, scalers and penalties in detail:

User GuideSupervised learning- Linear Models- Ordinary Least Squares, Ridge regression and classification, Lasso, Multi-task Lasso, Elastic-Net, Multi-task Elastic-Net, Least Angle Regression, LARS Lasso, Or...scikit-learn

Next, look at how raw columns become useful features: features and data preparation.