Extrapolating scaling laws

Labs rarely guess a 1e24-scale loss from first principles. They train a grid of small models, fit L(N, D), and extrapolate. That works when the proxy and the final run share data, tokenizer and architecture family. It fails when you treat a weekend 125M curve as a budget commitment.

The big idea

Reliability depends on three matches:

• Tokenization and data mixture between proxy and final run.
• Sensible learning-rate and batch transfer as width grows.
• Not extrapolating across architecture families blindly. Mixture-of-experts is not dense.

μP (maximal update parametrization) and related hyperparameter-transfer schemes scale initialization and learning rate with width, so optimal settings found on a small model stay near-optimal when you widen. When it works, you avoid retuning Adam's learning rate at 100× scale. That is a practical companion to loss scaling laws.

Walking through a shaky plot

Early papers claimed sharp "emergence" of skills at certain scales. Follow-ups argued many plots were metric artifacts: discontinuous pass/fail measures on tasks that actually improve smoothly in log-likelihood or continuous scores.

The careful stance:

• Cross-entropy and well-chosen continuous metrics usually look smooth.
• Thresholded accuracies can look like phase changes even when underlying competence does not.
• Some capabilities still appear practically unusable until a scale threshold. That is engineering emergence, not a metaphysical discontinuity.

Do not plan a product on a single discontinuous benchmark without checking contamination and how soft the metric is.

Cross-entropy on web text is not MMLU. Still, within a model family and mixture, lower validation loss usually correlates with better average downstream scores. That is why labs monitor loss continuously and only periodically run expensive eval suites. When the correlation breaks, suspect contamination, a mixture shift that hurts the benchmark domain, or a metric discontinuity.

For product work, map the chain: train FLOPs → val loss → capability evals → user metrics. Scaling laws speak to the first links. They are silent on retention and trust.

What the curves do not tell you

• Data quality ceilings. Power laws eventually bend when you exhaust high-quality tokens or recycle too hard. Synthetic data and filtering change the curve.
• Alignment and product metrics. Lower CE loss is not safer or more helpful chat. Post-training dominates those axes.
• Latency SLOs. Train-optimal size can lose to a smaller over-trained model on p95.
• Algorithmic breakthroughs. Better architectures, optimizers or objectives can beat a previous frontier at equal FLOPs.

Progress still moves the frontier through more compute, better data, over-training recipes, mixture-of-experts (more total parameters per active FLOP) and test-time compute.

If your only curve is a weekend 125M experiment, treat extrapolation as a hypothesis, not a budget. Constants move with data and architecture. The accounting structure does not: train FLOPs track 6ND, and a forward pass is about 2N FLOPs per token.

Common mistakes

• Extrapolating across architecture families. A dense IsoFLOP grid does not price an MoE run.
• Planning a launch on a cliff-shaped accuracy plot. Check a continuous metric first.
• Stopping at val loss. Product metrics live several links downstream.

Check yourself

1. Why can a task look "emergent" on a plot even if the model improved smoothly? Answer: A pass/fail metric can stay near zero until a threshold, then jump, while log-likelihood improved all along.
2. What does μP try to transfer from a small model to a wide one? Answer: Optimal hyperparameters such as learning rate, by scaling init and LR with width.

Key takeaways

• Fit L(N, D) on matched proxy runs. IsoFLOPs make the train optimum visible.
• μP is the usual companion for transferring learning rates across width.
• Emergent-ability cliffs are often the metric, not a discontinuity in competence.
• Scaling laws budget train loss. They do not pick alignment, latency or product metrics.

Further learning

Back to the map: scaling laws.

A short lecture digest on IsoFLOPs, over-training ratios and μP:

How to Scale Your Model (Google DeepMind)Training LLMs often feels like alchemy, but understanding and optimizing the performance of your models doesnHow To Scale Your Model