Kaplan power laws

If language-model loss bounced around as you added parameters or data, a large training run would be a gamble. Empirically it does not. For transformer language models, cross-entropy loss is a smooth power law in model size, data and compute. Labs fit those curves on small runs and aim a large run at a target loss.

The big idea

OpenAI's Scaling Laws for Neural Language Models (Kaplan et al., 2020) trained many transformers and saw, schematically:

L(N) ≈ L∞ + (N_c / N)^{α_N}
L(D) ≈ L∞ + (D_c / D)^{α_D}
L(C) ≈ L∞ + (C_c / C)^{α_C}

N is non-embedding parameters, D is training tokens, C is training FLOPs, and L∞ is an irreducible loss floor. The constants depend on the recipe. The shape is the point: bigger models plus more data give predictably lower loss.

Think of this as the unit economics of learning. A training org tracks FLOPs to shave another 0.01 off loss the way a business tracks cost to serve a customer. The analogy breaks when the data mixture or architecture family changes, because then you are no longer on the same curve.

Walking through what Kaplan recommended

Three takeaways shaped the GPT-3 era:

• Scale beat cleverness over wide ranges. Architecture tweaks mattered less than N and D.
• Model shape (depth versus width) mattered less than total non-embedding parameters, within reason.
• Their compute-optimal allocation recommended scaling parameters faster than data. The community trained very large models on relatively modest token counts.

GPT-3 is the concrete case: 175 billion parameters on about 300 billion tokens, or roughly 1.7 tokens per parameter. Through a later Chinchilla lens that is undertrained. It does not make GPT-3 a failure. It explains why smaller models with more tokens later punched above their weight.

The functional forms are empirical, not physical constants. They describe one recipe family. A better optimizer, a better mixture or a sparse architecture can beat a previous frontier at equal FLOPs. Within one recipe, ignoring the curve is how you burn a cluster on an under-trained giant.

Common mistakes

• Treating the exponents as laws of nature. They shift with tokenizer, data quality and dense versus sparse models.
• Reading "scale beats cleverness" as "architecture never matters". Shape is secondary to N inside a family. Switching families (for example mixture-of-experts) is a new curve.
• Planning a product on train loss alone. Lower cross-entropy on web text is not the same as a safer or more helpful assistant.

Check yourself

1. Kaplan plotted loss against three scale axes. What are they? Answer: Parameters N, tokens D and train FLOPs C.
2. GPT-3 trained 175B parameters on ~300B tokens. About how many tokens per parameter is that? Answer: About 1.7, far below the later Chinchilla rule of thumb of 20.

Key takeaways

• Transformer LM loss scales as a power law in N, D and C over wide ranges.
• That smoothness is why labs can budget a large run from smaller proxy runs.
• Kaplan-era allocation scaled parameters faster than data, which left some giants undertrained.
• The curves describe a recipe family. They do not replace judgment about data or architecture.

Further learning

Next: Chinchilla and compute-optimal training.

Scaling Laws for Neural Language Models (Kaplan et al.)We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spa…arXiv.org