Target encoding without leakage

Target encoding replaces each category with the average label for that category. A city with 10,000 values can become one number: the historical positive rate in that city. It is powerful, but computed naively it memorizes the labels it is trained on. Out-of-fold statistics and smoothing are what make it safe.

The naive version leaks

The naive formula takes the mean label for each category over all training rows:

encode(c) = mean(y | category = c)   # computed on all training rows

A rare category with one row gets an encoding equal to that row's own label. The model then sees a perfect signal that will not exist at prediction time.

Out-of-fold encoding

Compute the statistic on one set of rows, and apply it to another. For each cross-validation fold, compute the category means on the other folds only, then encode the held-out fold with them. Every row is encoded by statistics that never saw its own label.

For the final model, compute the means on the full training set, with smoothing as described below. Use the same fitted means at serving time.

Smoothing toward the global mean

Rare categories have noisy means. Shrink each one toward the global mean:

encode(c) = (n_c · mean_c + m · mean_global) / (n_c + m)

Here n_c is the number of rows in category c, and m is a prior strength, often between 10 and 100. Rare categories lean on the global rate. Common categories keep their own mean.

rare city, 4 rows, mean 0.5:     (4 × 0.5 + 10 × 0.2) / 14     = 0.286
common city, 1,000 rows, mean 0.5: (1000 × 0.5 + 10 × 0.2) / 1010 = 0.497

Checking for leakage

A permutation check is a simple test. Shuffle the labels, rebuild the encoding, and train again. If the encoding is leak-free, the model should lose all its signal, and the AUC should fall to about 0.5. If the AUC stays high on shuffled labels, the encoding is memorizing.

Common mistakes

• Encoding on the full training set before cross-validation. The validation folds have then influenced their own encodings.
• No smoothing. Rare categories get extreme encodings that the model over-trusts.
• Unseen categories at serving. Fall back to the global mean, and log how often it happens.

Check yourself

1. Why does a rare category move toward the global mean? With few rows, m dominates the formula, so the encoding leans on the global rate.
2. What should a permutation check show for a leak-free encoding? After the labels are shuffled, the model's AUC should fall to about 0.5.

Key takeaways

• Target encoding is strong for high-cardinality categories, but it can memorize labels.
• Compute encodings out of fold, so no row is encoded by its own label.
• Smooth rare categories toward the global mean with a prior strength m.
• Test for leakage with a shuffled-label permutation check.

Further learning

Another common source of trouble in training data is the rare positive class, and it changes the metric as well as the weights: class imbalance.