WordPiece, SentencePiece and special tokens

BPE is not the only subword trainer. WordPiece and SentencePiece (often with a Unigram language model) are the cousins you will meet on BERT-style and Llama-style model cards. In practice you inherit whatever the base model used. Never mix tokenizers across a fine-tune.

The big idea

WordPiece (BERT) has a similar merge spirit, but the training objective picks merges that maximize likelihood of the training data under a language-model-ish score, not raw pair frequency. WordPiece often marks continuation pieces with ## (playing → play + ##ing).

SentencePiece trains on raw text without requiring pre-tokenized words. It can use BPE or a Unigram LM. Unigram starts from a large candidate vocab and prunes pieces to maximize a unigram language-model likelihood — a different inductive bias than greedy BPE merges. Llama and many multilingual models use SentencePiece-style pipelines. GPT-family models stick to byte-level BPE with a regex pre-tokenizer.

BPE is deterministic and easy to reason about. Unigram can assign multiple segmentations and samples during training, which sometimes helps rare languages.

Walking through specials and templates

Special tokens sit outside normal text: <|endoftext|>, [BOS], [EOS], [PAD], and chat markers like <|user|> / <|assistant|>. They get reserved IDs and must not be split by BPE.

Chat templates (Jinja in Hugging Face, and similar) wrap roles and tool calls into an exact string before tokenization. A one-character mistake in the template changes token IDs and can tank a fine-tune. Always tokenize with the model's official template.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
ids = tok.encode("Tokenization is unbelievably useful")
print(tok.convert_ids_to_tokens(ids))

Typical output, with Ġ marking a leading space in GPT-2's scheme:

['Token', 'ization', 'Ġis', 'Ġunbeliev', 'ably', 'Ġuseful']

GPT-2's pattern roughly keeps spaces with the following word and splits punctuation. That means "Hello world" and "Hello world" (two spaces) tokenize differently. Code indent spaces can become many tokens. Tabs may differ from spaces. JSON and HTML burn tokens on structural characters.

When building eval sets for code models, normalize formatting or you measure tokenizer noise instead of model skill. Some modern pre-tokenizers are tuned for code (different splits on camelCase).

Reserve IDs at vocab-build time for things you will need later: pad, bos/eos, and a block of unused slots for future chat and tool markers. Expanding vocab after pretraining means new embedding rows with no gradient history. Initialize from mean embeddings or subword averages and train them during SFT. Cleaner to plan specials early.

Chat templates compose specials into a contract. Every missing newline is a different token sequence. Fine-tunes that "don't follow instructions" are often template mismatches.

Domain adaptation sometimes adds tokens for new tool names or languages. Append rows to the embedding and LM head, initialize from the mean of piece embeddings that spell the new string, train with a slightly higher LR on new rows, and never reshuffle existing ids. Every published checkpoint depends on that stability.

Common mistakes

• Mixing a BERT WordPiece tokenizer with a GPT BPE checkpoint. The ID spaces are unrelated.
• A missing newline in the chat template. Token IDs change. Instruction following collapses.
• Expanding vocab without an init plan for the new rows. Random rows have no meaning.

Check yourself

1. What does Ġ mean in GPT-2 tokens? Answer: A leading space attached to the piece.
2. Why reserve unused special IDs at vocab build time? Answer: So later chat or tool markers do not need new embedding rows with no training history.

Key takeaways

• WordPiece and Unigram/SentencePiece are cousins of BPE with different training criteria.
• Inherit the base model's tokenizer. Never mix.
• Pre-tokenization, specials and chat templates decide the ID sequence as much as the merge list does.
• Plan reserved specials early. Expanding vocab later needs careful embedding init.

Further learning

Next: tokenizer pathologies.