Training dynamics, schedules and checkpoints

A pretraining run lasts weeks. The loss curve tells you whether it is healthy, the learning-rate schedule shapes most of that curve, and occasional spikes will happen. The practical skill is recognizing a healthy run, and recovering quickly when one is not healthy.

What healthy looks like

Validation loss tracks training loss, and both fall smoothly. Spikes, which are sudden jumps, are common at scale. Their causes include optimizer state that has been corrupted, a bad batch of data, or numerical problems.

The usual response is to roll back to the last good checkpoint. If the same data keeps causing the spike, skip that range of data and resume.

The learning-rate schedule

The standard recipe has two phases:

1. Linear warmup. Ramp from near zero to the peak rate over thousands of steps.
2. Decay. Either cosine decay to a small fraction of the peak, or WSD (warmup, stable, decay). WSD keeps the rate constant for most of the run, then decays it briefly at the end. That makes mid-run comparisons and continued training easier.

Here is a worked example with an illustrative peak rate of 3e-4, 2,000 warmup steps and 50,000 total steps:

step 500:    500 / 2,000 = 25% of warmup, rate = 0.25 × 3e-4 = 7.5e-5
step 1,000:  50% of warmup, rate = 0.5 × 3e-4 = 1.5e-4
step 26,000: the midpoint of the cosine decay (steps 2,000 to 50,000),
             where the cosine gives half the peak, rate = 1.5e-4

Steps 1,000 and 26,000 both land at 1.5e-4, from different phases of the schedule.

Stability tools

Mixed precision (BF16 or FP8) saves memory and time. Gradient clipping caps the size of rare huge updates, and a common threshold is a global norm of 1.0. RMSNorm and careful initialization keep activations in range. Fused kernels speed up the math.

The batch size also has a limit. Past a certain size, a bigger batch adds compute without speeding up each step much, so each extra token helps less.

Stages of a run

A typical curriculum has three parts:

1. Main phase at a short context length, on the bulk mixture.
2. Long-context phase. Raise the rotary base, or apply YaRN-style scaling, and upweight long documents. A short run of this phase is what lets a model use a long prompt well. See long-context techniques.
3. Optional anneal. Lower the rate, and shift the mixture toward the highest-quality sources.

Checkpointing

Save a checkpoint every few minutes of wall-clock time, and also every set number of tokens. Keep the last several checkpoints. When a spike corrupts the optimizer state, you can roll back to an earlier checkpoint, not just the latest one. The distributed training page covers the parallelism that makes these runs possible.

Common mistakes

• Judging the run by one spike. Look at the trend over many steps.
• Advertising a context window without the long-context phase. The number may not reflect real use.
• Choosing the warmup length carelessly. Too short causes early spikes. Too long wastes tokens at a tiny rate.
• Keeping only the latest checkpoint. One corrupted save can cost days.

Check yourself

1. Warmup lasts 2,000 steps to a peak of 3e-4. What is the rate at step 500? Answer: 500 / 2,000 × 3e-4 = 7.5e-5.
2. Why keep several checkpoints instead of only the latest? Answer: If the latest one holds corrupted optimizer state, an earlier good checkpoint lets you resume.

Key takeaways

• A healthy run has smooth loss curves, with validation tracking training.
• Warmup protects the start of the run, and cosine or WSD decay settles the end.
• Clipping, mixed precision and careful normalization keep training stable.
• Schedule a long-context phase, and checkpoint often enough to roll back.

Further learning

Next, check whether a run actually got better: evaluating a pretraining run.