Guide

Transformers and language models

The pretraining data pipeline

Frontier pretraining is mostly data engineering. Raw web pages become clean, deduplicated, weighted text through extraction, filtering, deduplication and mixing. The data recipe moves quality as much as the architecture does.

119 of 187 · 2 min read

Most of the work in pretraining happens before the first gradient step. Raw crawl data is noisy, and most of it gets thrown away. A pipeline extracts the text, filters it, removes duplicates and mixes the sources in chosen proportions. Each step changes what the model learns.

The pipeline

The stages

Crawl and extract. Common Crawl publishes large web snapshots. Raw HTML carries navigation, boilerplate and spam. Extractors turn each page into plain text. Most of the bytes die in the next step.

Filter. Heuristic filters drop documents that are too short, in the wrong language, very repetitive or full of spam. Learned quality classifiers, trained on curated text versus random web pages, rank what remains. Aggressive filtering makes the corpus smaller, but each token is more useful.

Deduplicate. Exact hashing removes identical pages. Near-duplicate methods such as MinHash catch mirrors and boilerplate clones. Duplicates waste capacity, and repeated text can make the model memorize it.

Mix. Labs weight the sources. Web text, code, math, books and multilingual text are sampled at chosen rates. The weights are a real design choice, so record the mixture with the model.

Case study. Hugging Face's FineWeb write-up walks through its extraction and filtering choices. Reading it teaches more about modern pretraining than another architecture diagram does. The architecture is fairly standard now, and data recipes still move the results.

Walking through a mixture

Suppose the total budget is 1 trillion tokens, and the mixture is 70% web, 15% code, 5% math and 10% books.

web:   0.70 × 1,000B = 700B tokens
code:  0.15 × 1,000B = 150B tokens
math:  0.05 × 1,000B =  50B tokens
books: 0.10 × 1,000B = 100B tokens
total:                 1,000B tokens

The percentages are about proportions. The token counts are what the model actually sees.

Common mistakes

  • Filtering too little. Assuming the model will sort out the noise. Low-quality text takes up training compute and teaches bad habits.
  • Counting duplicates as new data. Repeats cost compute and add memorization without adding knowledge.
  • Confusing the share with the amount. A 20% code share of 500 billion tokens is 100 billion code tokens, not 20 billion.

Check yourself

  1. A mixture is 20% code, and the budget is 500 billion tokens. How many code tokens? Answer: 100 billion.
  2. Why deduplicate near-duplicates, not only exact copies? Answer: Mirror sites and boilerplate clones differ slightly, so exact hashing misses them. They still waste capacity.

Key takeaways

  • Pretraining quality depends heavily on the data pipeline.
  • Extraction, filtering, deduplication and mixing each change the result.
  • Mixtures set how often each source is seen. Document them.
  • Compute the token counts from the shares before planning a run.

Further learning

The FineWeb write-up documents each filter and its effect:

FineWeb: decanting the web for the finest text data at scaleThis page explains how the FineWeb dataset was created, shows visual summaries, and provides direct links to download the 15‑trillion‑token web‑scale text collection or its FineWeb‑Edu subset. No i...huggingface.co

Next, estimate what a run costs before you start it: compute budget and the 6ND rule.

Pretraining a language model

Data, objective and compute: how a base model is trained on trillions of tokens.

Tokenization

Byte-pair encoding in depth: how vocabularies are built and why tokenization causes strange model behavior.

Next-token prediction

A language model learns by guessing the next token of real text. The text supplies the label, so no human annotation is needed, and the guesses reward learning grammar, facts and patterns.