Encoding categories and text

Models need numbers, but a country or a plan is a label. Encoding turns each label into numeric columns. Text can be handled the same way, by counting the words it contains. This page covers the common encodings, what each one costs, and where each one breaks.

Common encodings

One-hot in practice

Take plan with three levels: free, pro and team. One-hot gives three columns:

free → [1, 0, 0]
pro  → [0, 1, 0]
team → [0, 0, 1]

A linear model can drop one column, because the other two imply it. A tree does not need that. A city column with 10,000 levels would add 10,000 columns, which is why hashing or target encoding is used there instead.

from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer

cat = ["plan", "country"]
num = ["logins_14d", "log_spend"]

pre = ColumnTransformer([
    ("num", "passthrough", num),
    ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), cat),
])

Text as word counts

The simplest text representation is a bag of words: one column per word, and each value is the count of that word in the document. Word order is lost, but for many tasks, such as routing tickets or flagging spam, the words are enough.

docs = ["refund my order", "order not received", "refund refund"]
vocab = sorted({w for d in docs for w in d.split()})
counts = [[d.split().count(w) for w in vocab] for d in docs]

print(vocab)
for row in counts:
    print(row)

Output:

['my', 'not', 'order', 'received', 'refund']
[1, 0, 1, 0, 1]
[0, 1, 1, 1, 0]
[0, 0, 0, 0, 2]

Common words such as "the" get large counts in every document and carry little signal. TF-IDF fixes this by multiplying each count by an inverse document frequency, which is lower for words that appear in many documents. TF-IDF with a linear model is a strong baseline for ticket routing and spam. Keep it in your evaluation sheet before you try embeddings.

Common mistakes

• One-hot on a column with thousands of levels. The matrix becomes very wide and sparse. Hash it, or target-encode it with care.
• Using ordinal codes for unordered categories. A linear model will treat the distance between codes as meaningful.
• Crashing on unseen categories in production. Set the encoder to ignore unknown levels, or map them to an "other" bucket.

Check yourself

1. How many columns does one-hot give for a three-level category, for a linear model with a reference level dropped? Two. The third is implied by the others.
2. Why does ordinal encoding mislead a linear model on country? It implies that countries have an order and equal spacing, which they do not.

Key takeaways

• One-hot encoding is the safe default for moderate numbers of levels.
• Hashing bounds the width for huge vocabularies, at the cost of collisions.
• Bag-of-words and TF-IDF turn text into sparse counts, and they are strong baselines.
• Plan for unseen categories at serving time.

Further learning

Missing values are the next problem, and they often carry information of their own: missing values and pipelines.