Pretraining a language model

Pretraining turns random weights into a base language model. The model runs next-token prediction on a large, filtered text mixture until the loss levels off for the compute budget. Everything later, such as supervised fine-tuning, preference tuning and tool use, sits on that base. Post-training can teach manners and tools, but it cannot cheaply replace a weak base.

That is why labs spend very large amounts of compute on pretraining, and why open data mixtures matter. They make the data half of the recipe available, even if you cannot match frontier compute. As an engineer, your job is to understand what the recipe optimizes, so you can choose a base, estimate the cost and debug a run.

Concepts

Step	Concept	What it answers
1	Next-token prediction	What is the training objective, and why does it teach knowledge?
2	The pretraining data pipeline	How does raw web text become training data?
3	Compute budget and the 6ND rule	How many FLOPs and GPU-days does a run need?
4	Training dynamics, schedules and checkpoints	What do healthy curves look like, and how does a run recover from a spike?
5	Evaluating a pretraining run	How do you know a run is good, without fooling yourself?

Working in practice

Treat the data mixture, the token budget and the learning-rate schedule as first-class hyperparameters, on par with the architecture.
Estimate 6ND before proposing a run, and compare it with your GPU budget at a realistic utilization.
Log tokens seen, not epochs. Epochs mean little on large, deduplicated web data.
A base checkpoint is not a chat model. Judge it on completions, and wait for supervised fine-tuning before judging it as an assistant.
Read scaling laws before arguing for a bigger model at a fixed budget.

The distributed training page covers how one model is spread across many devices, through sharding, pipeline stages and tensor splits.

For a hands-on anchor, reproduce GPT-2 (124M) following Karpathy's walkthrough. The run exercises the same loop as a large lab run, at a much smaller scale: data loading, checkpointing, validation loss and utilization.

Common mistakes

Treating the architecture as the main lever. At a fixed size, data quality often matters more.
Comparing runs at different token counts.
Assuming the base model answers like an assistant.

Key takeaways

Pretraining is next-token cross-entropy on a large mixture, and the labels come from the text itself.
The data pipeline (extract, filter, deduplicate, mix) shapes quality as much as the architecture does.
Training cost is about 6ND FLOPs, so an 8-billion-parameter model on 15 trillion tokens needs on the order of 20,000 GPU-days.
Warmup, decay and checkpoints keep runs alive, and honest validation keeps the conclusions right.