Missing values and pipelines

Real data has gaps, and the gaps often carry information. A missing last-payment date can mean the customer never paid. Filling every gap with the average hides that signal. This page covers how to flag missing values and fill them, and why the fill value must be learned inside the training pipeline.

Missingness can be a signal

Three common strategies, in order of usefulness:

1. Indicator plus impute. Add a flag column such as is_missing_last_payment, then fill the numeric column with the median. Categorical columns can use a "__MISSING__" level.
2. Let the model handle it. LightGBM and XGBoost learn a default direction for missing values. Still make the serving path match.
3. Fill with a domain value. Use 0 for counts where zero means "none". Do not use 0 for a continuous sensor, where zero is a real reading.

Walking through a median fill

Suppose the training data has one missing value. Fit the median on training rows only, then apply it everywhere:

import numpy as np

train = np.array([12.4, np.nan, 9.0, 15.0])
test = np.array([np.nan, 30.0])

median = np.nanmedian(train)                # fitted on training data only
print("fill value:", median)

train_filled = np.where(np.isnan(train), median, train)
test_filled = np.where(np.isnan(test), median, test)
missing_flag = np.isnan(test).astype(int)   # keep the signal too
print(train_filled)
print(test_filled, missing_flag)

Output:

fill value: 12.4
[12.4 12.4  9.  15. ]
[12.4 30. ] [1 0]

The median of the three known training values is 12.4, and that number is used for both the training gap and the test gap. The test set never influenced it. The flag records which test row was missing.

Keep the imputer inside the pipeline

If you compute the mean on the full dataset, the test rows shape the fill value, and your validation score is optimistic. The fix is to put the imputer and the scaler inside a pipeline, so they are fitted only when the model is fitted.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression

numeric = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
])
categorical = Pipeline([
    ("impute", SimpleImputer(strategy="most_frequent")),
    ("onehot", OneHotEncoder(handle_unknown="ignore")),
])
preprocess = ColumnTransformer([
    ("num", numeric, ["age", "logins_14d", "spend_30d"]),
    ("cat", categorical, ["plan", "country"]),
])
clf = Pipeline([
    ("prep", preprocess),
    ("model", LogisticRegression(max_iter=1000, class_weight="balanced")),
])
clf.fit(X_train, y_train)

Common mistakes

• Filling continuous sensors with zero. Zero may be a real reading. Use the median and a flag.
• Imputing before the split. The fill then uses information from the test rows.
• Dropping the flag. A filled value without the indicator hides the fact that the value was missing.

Check yourself

1. Why add a missing-indicator next to the filled value? Because the missingness can predict the label, and filling alone hides it.
2. Why fit the median on training rows only? Otherwise the test rows influence the fill value, which leaks information into validation.

Key takeaways

• A missing value can be a signal, so keep a flag next to the fill.
• Choose the fill by meaning: median for most numbers, 0 only when zero means none.
• Fit imputers and scalers inside a pipeline, on training data only.
• Serialize the whole pipeline, so serving applies the same fitted steps.

Further learning

The fill is only correct if the time windows behind each feature are correct too: point-in-time features.