Chinchilla and compute-optimal training

Kaplan-era runs often grew parameters faster than data. DeepMind's Chinchilla paper asked a sharper question: given a fixed training FLOP budget, how should you split it between model size and token count? The answer that stuck is to scale both roughly equally, at about 20 tokens per parameter.

The big idea

Training FLOPs track C ≈ 6 N D. If the budget is fixed, the product N D is fixed, so a bigger model means fewer tokens and a smaller model means more tokens.

N D = C / 6 = constant
D_opt ≈ 20 · N

A compute-optimal 70B model then wants on the order of 1.4T tokens. That is the Chinchilla model itself. It can beat a much larger model trained on too little data, for example a Gopher-class 280B compute budget spent on a too-wide, under-trained net.

Walking through an equal-compute split

Suppose the budget C = 6 N D is fixed, so N D is constant.

• Kaplan-flavored choice: huge N, small D → an under-trained giant.
• Chinchilla: shrink N, raise D until D ≈ 20 N → lower loss at the same FLOPs.

GPT-3 (175B on ~300B tokens) sits near ~1.7 tokens/parameter. Through this lens it was undertrained.

The paper fitted the frontier three ways. When they disagree, trust IsoFLOP grids on your stack.

1. Fix the model, vary tokens — final loss versus D for each N.
2. IsoFLOP profiles — for several fixed C, train many (N, D) pairs with 6ND ≈ C, pick the best loss, and plot optimal N(C).
3. Parametric fit — assume a form L(N, D), fit coefficients, then minimize under the FLOP constraint.

Constants shift with data quality, tokenizer and sparse versus dense. DeepSeek and others re-ran Chinchilla-style analyses for mixture-of-experts and found different optimal ratios. Do not ship a 20× MoE plan blindly.

Chinchilla-style analyses often assume unique tokens. In practice you may epoch a high-quality subset more than once. Mild repeats help. Heavy repeats waste capacity and risk memorization. If your "15T tokens" are 3× over the same 5T unique documents, your place on the curve is not the same as 15T unique.

Common mistakes

• Treating 20× as a law for every architecture. Re-measure with IsoFLOPs for MoE or hybrids.
• Minimizing train loss when you will serve billions of tokens. That is a different optimum. See over-training.
• Counting repeated documents as fresh tokens. Repeats are not the same as unique data.

Check yourself

1. You have a fixed train budget C = 6ND. You can train 100B params on 100B tokens, or 10B params on 1T tokens. Which is closer to Chinchilla-optimal? Answer: 10B on 1T, because that is 100 tokens/parameter, nearer 20 than the other split's 1 token/parameter.
2. A 70B model at D ≈ 20N wants about how many tokens? Answer: About 1.4 trillion.

Key takeaways

• For fixed train compute, scale N and D roughly equally.
• A useful rule is D_opt ≈ 20 N.
• IsoFLOP grids on your own stack beat slogans copied from another lab.
• Compute-optimal for training is not the same as cheapest to deploy.

Further learning

Next: over-training for inference. For the FLOP accounting, see the 6ND rule.

Training Compute-Optimal Large Language Models (Chinchilla)We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequen…arXiv.org