Scaling laws

If loss fell randomly with scale, frontier training would be gambling. Empirically, for transformer language models, cross-entropy loss is a smooth power law in model size, data and compute. Labs fit curves on small runs and aim large runs at a target loss. Scaling laws do not replace taste in data or architecture. They answer the budgeting question: given FLOPs, should you buy parameters or tokens?

Think of them as the unit economics of learning. Training orgs track FLOPs to shave another 0.01 off loss. The functional forms are empirical, the constants depend on the recipe, and the decisions still need judgment. Without the curves you are guessing.

Concepts in this section

Read them in order. Each one builds on the one before.

Test-time compute is a later axis: extra inference FLOPs after the weights are frozen. Classic Kaplan plots assumed one forward pass per answer. Reasoning models broke that.

A two-minute allocation

You have 1e23 FLOPs. Do you train 100B parameters on 100B tokens, or 10B parameters on 1T tokens?

Training cost tracks C ≈ 6 N D, so both splits spend the same FLOPs. Chinchilla says pick near D ≈ 20 N. The 10B-on-1T split is 100 tokens per parameter, much closer than 1 token per parameter. Then qualify: if you will serve the model heavily, over-train a smaller net still further. That is the 2022 correction plus the later product reality.

Chinchilla-optimal training sits near twenty tokens per parameter. Over-trained product models can sit two orders of magnitude higher on that ratio when inference economics dominate. A forward pass is about 2N FLOPs per token.

Common mistakes

• Pitching "we will 10× the model" without 10× the tokens or a new objective. Kaplan already showed you cannot only grow N.
• Using train-optimal size as the serving plan. Lifetime cost is often decode, not train.
• Trusting a discontinuous benchmark as a scaling cliff. Check a continuous metric and contamination.

Key takeaways

• LM loss scales as a power law in N, D and C over wide ranges, which enables predictive budgeting.
• Chinchilla: compute-optimal training uses on the order of 20 tokens per parameter. Earlier giants were often undertrained.
• Products often over-train smaller models to cut inference cost, not train FLOPs.
• Proxy runs plus IsoFLOPs (and μP) extrapolate large-run loss only when data and architecture match.
• Test-time compute is a new scaling axis once weights are frozen.

Further learning

A short lecture digest on applying scaling laws (IsoFLOPs, over-training ratios, μP):

Scaling Laws for Neural Language Models (Kaplan et al.)We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spa…arXiv.org Training Compute-Optimal Large Language Models (Chinchilla)We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequen…arXiv.org How to Scale Your Model (Google DeepMind)Training LLMs often feels like alchemy, but understanding and optimizing the performance of your models doesnHow To Scale Your Model