Compute budget and the 6ND rule

Before anyone proposes a training run, someone should estimate its cost. The rule of thumb is short: C ≈ 6ND. Here N is the number of non-embedding parameters, and D is the number of training tokens. It is accurate enough to tell you whether a plan fits the budget.

The big idea

A forward pass costs about 2 FLOPs per parameter per token. The backward pass costs about twice the forward pass. Add them, and each token costs about 6 FLOPs per parameter:

C ≈ 6 · N · D

Compute scales with the product of model size and data size. Doubling either one doubles the cost.

Walking through an 8B model on 15T tokens

Take a model with 8 billion parameters trained on 15 trillion tokens.

C = 6 · 8×10⁹ · 15×10¹² = 720×10²¹ = 7.2×10²³ FLOPs

An H100 GPU is rated on the order of 10¹⁵ FLOP/s for dense BF16 or FP8 at peak. Real training reaches only part of that peak. For this estimate, assume 4×10¹⁴ FLOP/s per GPU:

time = 7.2×10²³ / 4×10¹⁴ = 1.8×10⁹ GPU-seconds
     ≈ 20,800 GPU-days

That is a few thousand GPUs running for several weeks. The estimate leaves out restarts, cool-downs and lower utilization, so the real run takes longer.

Here is the same arithmetic as a script that runs as written:

N = 8e9           # parameters
D = 15e12         # training tokens
C = 6 * N * D     # total FLOPs
gpu_flops = 4e14  # effective FLOP/s per GPU (assumed)
gpu_days = C / gpu_flops / 86_400
print(f"{C:.2e} FLOPs, about {gpu_days:,.0f} GPU-days")

Output:

7.20e+23 FLOPs, about 20,833 GPU-days

Tokens and steps

The number of optimizer steps depends on the global batch:

tokens per step = batch size × sequence length × gradient accumulation × data-parallel size
steps = D / tokens per step

For 100 billion tokens and a global batch of 2 million tokens per step, there are 100e9 / 2e6 = 50,000 steps. If each GPU can hold only a small micro-batch, gradient accumulation adds several micro-steps before each update. The math is the same, and the hardware is smaller.

Common mistakes

• Using peak FLOPs. Plan with the achieved rate, which is the peak times MFU.
• Dropping the backward pass. The factor is 6, not 2.
• Comparing losses at different token counts. A run that sees more tokens usually has lower loss.

Check yourself

1. A 1-billion-parameter model trains on 100 billion tokens. Roughly how many FLOPs? Answer: 6 · 10⁹ · 10¹¹ = 6×10²⁰.
2. Two runs have equal FLOPs. One has twice the parameters. How do their token counts compare? Answer: The bigger model sees half as many tokens, because C = 6ND and N doubled.

Key takeaways

• The training cost is about 6ND FLOPs.
• Use achieved throughput, not peak, when you estimate time.
• Convert FLOPs to GPU-days before proposing a run.
• Compare runs at equal token counts.

Further learning

Scaling laws predict how loss depends on N and D at a fixed budget: scaling laws.

Next, watch the run itself: training dynamics, schedules and checkpoints.