Text analysis: tokens, stems and synonyms

The words in a search index are not the raw text. They are the output of an analyzer, a chain of steps that cleans the text before it is indexed. The same analyzer must run on every query. If the document says "running" and the query is "run" without stemming on either side, the match is lost.

The big idea

Think of a kitchen prep station. Ingredients are washed, chopped and labeled before they go in the pantry. A recipe that calls for "chopped onion" must find onions that were prepped the same way.

The picture breaks because prep is lossy. Once a word is stemmed, you cannot recover the original spelling from the index.

Walking through it

An analyzer usually has three stages.

1. Character filters clean the raw text, such as stripping HTML and normalizing Unicode.
2. The tokenizer splits the text into tokens. Languages such as Chinese and Japanese need dictionary-based tokenizers because they do not use spaces.
3. Token filters lowercase, remove stop words, stem, add synonyms and fold accents.

Here is a trace with a typical English analyzer:

Input:   "The running shoes at Café"
Split:   [The, running, shoes, at, Café]
Lower:   [the, running, shoes, at, café]
Stop:    [running, shoes, café]          ("the", "at" removed)
Stem:    [run, shoe, café]               (with a Porter-style stemmer)
Fold:    [run, shoe, cafe]               (accent removed)

The index stores run, shoe and cafe. A query for "Running Shoe" goes through the same steps and matches.

Teams usually index one field several ways. The analyzed title supports full-text matching. A title.keyword copy, which is not analyzed, supports sorting and exact counts. An edge n-gram copy supports prefix lookups.

The trade-off

Stemming and synonyms improve recall, because more relevant documents match. They can also merge unrelated words. Porter stemming maps both "university" and "universe" to univers, so a search for one can return the other.

Common mistakes

• Changing the analyzer without reindexing. The index keeps the terms the old analyzer produced, so the change does nothing for existing documents.
• Using different analyzers for indexing and querying. Matches disappear silently.
• Using a standard tokenizer on Chinese or Japanese text. It will not find word boundaries.
• Stemming a field that holds identifiers. Part numbers and SKUs should use an exact, unanalyzed field.

Check yourself

1. You switch to a gentler stemmer. Why do existing documents still match the old way?

Answer: The index stores the terms the old stemmer produced, so every document must be reindexed to apply the change.

2. Why is "university" matching "universe" a problem?

Answer: Aggressive stemming merges unrelated words, so searches return results the user did not mean.

Key takeaways

• An analyzer turns text into the terms the index stores.
• Index time and query time must use the same analyzer.
• Stemming and synonyms raise recall but can merge unrelated words.
• Any analyzer change requires a full reindex, so plan it carefully.

Further learning

Next, see how matching documents are ranked in BM25 ranking.