Over-training for inference

Chinchilla answers a training question: given FLOPs, how do you reach the lowest training loss? Once a model is served billions of times, the lifetime compute is dominated by inference. Then a smaller model trained past the Chinchilla ratio can win on dollars per token, even if some train FLOPs look "wasted."

The big idea

Llama 3 8B on about 15T tokens is roughly 1900 tokens per parameter — two orders of magnitude above 20×, on purpose.

Distillation and pruning push the same idea further: spend train compute to buy cheap decode. Scaling laws still guide the proxy sweeps. The objective now includes expected decode volume.

Walking through a product choice

You have a serving budget and a latency cap.

1. A Chinchilla-optimal 70B may have better train loss.
2. An over-trained 8B may have worse train loss but far lower cost per token and better p95 latency.
3. If you serve a lot, pick (2). If you need one strong research checkpoint and will barely serve it, pick (1).

A builder's decision tree, written as an objective first:

• Fixed train budget, maximize checkpoint quality: aim near Chinchilla-optimal D/N for your architecture family.
• Fixed serving budget, maximize quality at latency L: over-train a smaller model (or distill) until decode cost fits. Spend leftover compute on data.
• Unknown future serving volume: prefer a slightly smaller over-trained open base you can fine-tune. Optionality beats a one-shot giant.

Fine-tuning FLOPs are tiny next to pretrain. Pick the best base rather than the bigger number on the badge. An 8B over-trained open model often beats a sparsely trained 30B on real apps.

Training scale is also not the only axis. Test-time compute spends extra FLOPs per answer after the weights are frozen. For many math and coding tasks you can trade train parameters for test-time tokens along a Pareto front. That breaks the old "one forward pass equals one answer" assumption baked into classic Kaplan plots.

Common mistakes

• Citing Chinchilla in a design doc whose real goal is $/query. Write the objective first.
• Assuming a larger badge always wins in production. Token count and over-training matter as much as parameter count.
• Ignoring latency SLOs. A train-optimal 70B can lose to an over-trained 8B on p95 even if train loss favored the 70B.

Check yourself

1. Llama 3 8B trained on ~15T tokens. About how many tokens per parameter is that, and is that Chinchilla-optimal? Answer: About 1900 tokens/parameter, far above 20×, so it is over-trained on purpose.
2. When does an over-trained smaller model beat a Chinchilla-optimal larger one? Answer: When inference volume or latency dominates lifetime cost.

Key takeaways

• Chinchilla is train-optimal. Products often over-train smaller models to cut inference cost.
• Over-trained open 8B-class models can beat sparsely trained larger ones on real apps.
• Distillation spends train compute to buy cheap decode.
• Test-time compute is a second scaling axis once weights are frozen.

Further learning

Next: extrapolating scaling laws. For the serving-cost side, see cost per token.