Evaluating a pretraining run

A pretraining run can look better for reasons that have nothing to do with the model. A run that saw more tokens shows lower loss. A benchmark leak shows a higher score. Evaluation means tracking the right numbers and comparing runs fairly.

Three measures

• Validation loss or perplexity on held-out text from the same mixture. This is the main health check. It is cheap and steady.
• Downstream benchmarks, run periodically. They are noisy early in a run, and they can disagree with the loss.
• Contamination checks. Benchmark questions that appear in the training text inflate the scores. Check for them before you trust a jump.

Walking through a contamination check

Suppose a benchmark has 200 questions, and the training shards contain 3 of them verbatim. That is 1.5%. The score on the clean 197 questions is the honest number. Report both: the score with the contaminated items removed, and the full score.

The usual steps are:

1. Measure n-gram overlap between training shards and benchmark items.
2. Remove or flag overlapping items, and record how many were affected.
3. Add canary strings to any text you publish, so future leaks can be detected.
4. For claims that matter, prefer evaluations that were never published.

A jump of several points after a data change is a reason to check contamination first, before you celebrate.

Compare runs fairly

Compare runs at the same number of training tokens. A run with twice the tokens and the same loss is the worse run per token.

Data changes often matter more than architecture changes at a fixed size. So test one change at a time, using a small proxy model:

1. Fix a small model, much smaller than the one you plan to ship.
2. Train on data mixture A and on mixture B for the same token budget.
3. Compare validation loss and a small benchmark suite.
4. Scale up the winner.

Do not change the architecture and the data in the same experiment. You will not know which change helped.

Common mistakes

• Comparing loss across different token counts. Equal loss at different token counts is not equal quality.
• Trusting a public benchmark jump without a contamination check.
• Changing data and architecture together. One variable per experiment.

Check yourself

1. Two runs reach the same loss, but run A used twice as many tokens. Which is better per token? Answer: Run B, because it reached the same loss with fewer tokens.
2. A benchmark score jumps 5 points after a data change. What do you check first? Answer: Contamination. Check whether benchmark items appear in the training text.

Key takeaways

• Validation loss is the primary health check for a base model.
• Benchmarks are noisy correlates, so check contamination before trusting them.
• Compare runs at equal token counts.
• Test one change at a time, using a small proxy model.

Further learning

Scaling laws explain how loss moves with parameters and tokens: scaling laws.

Back to the overview: pretraining a language model.