Next-token prediction

Pretraining rests on one objective: given the tokens so far, predict the next one. The correct next token is the label, so nobody annotates anything. That simple game forces the network to pick up syntax, facts and patterns, because each of them lowers the loss.

The big idea

For a sequence of tokens x₁ … xₙ, the model maximizes:

sum over t of log P(x_t | x_1 … x_(t−1))

Maximizing this is the same as minimizing the average cross-entropy on the training text. The loss at each position is the negative log of the probability the model gave to the token that really came next.

Walking through one guess

Say the text is "The capital of France is Paris". At the position before "Paris", the model gives these probabilities:

The loss is the negative log of the probability for the true token. This script computes it, and it runs as written.

import numpy as np

vocab = ["Paris", "Lyon", "the", "other"]
probs = np.array([0.90, 0.04, 0.02, 0.04])   # model's guess for the next token

target = vocab.index("Paris")                 # the token that really came next
loss = -np.log(probs[target])
print(f"loss for 'Paris': {loss:.3f}")
print(f"loss if the text had said 'Lyon': {-np.log(probs[1]):.3f}")

Output:

loss for 'Paris': 0.105
loss if the text had said 'Lyon': 3.219

A confident correct guess costs almost nothing. A confident wrong guess costs a lot. Training pushes the probabilities toward the text.

Why it teaches knowledge

Predicting "Paris" after "The capital of France is" requires geographic facts stored in the weights. Predicting the next line of a Python function requires syntax, and often the logic of the algorithm. Predicting a dialogue requires social patterns. None of these needs a separate head. All of them reduce the same loss, so they are stored in one shared network.

Post-training then steers that stored knowledge into assistant behavior. It rarely adds much new factual knowledge compared with pretraining.

The limits are real. A fact seen once may be memorized in a brittle way. Conflicting text produces confused probabilities. Events after the data cutoff are simply absent.

Common mistakes

• Treating a base model as an assistant. A base model continues text. Prompt it like a chat, and you get a continuation, not an answer.
• Confusing loss with benchmark accuracy. Loss falls smoothly during training. Benchmark scores can lag or jump.
• Thinking the label is a human answer. The label is the next token in the data, so the quality of the data is the quality of the labels.

Check yourself

1. The model gives the true next token probability 0.5. What is the loss in nats? Answer: −ln 0.5 ≈ 0.693.
2. Why does a base model not chat well out of the box? Answer: It was trained only to continue text. Post-training teaches it to follow instructions.

Key takeaways

• The next token is the label, so pretraining needs no human annotations.
• The loss is the negative log of the probability given to the true token.
• Predicting text well requires facts, syntax and patterns, all stored in shared weights.
• Next, see how the training text is prepared: the pretraining data pipeline.

Further learning

Language Models are Few-Shot Learners (GPT-3 paper)Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this…arXiv.org

nanoGPT (Andrej Karpathy)The simplest, fastest repository for training/finetuning medium-sized GPTs. - karpathy/nanoGPTGitHub

3Blue1Brown's eight-minute explainer shows a language model as a function that scores every possible next word, and how pretraining tunes it.