Transformer parameter and FLOP counts

When a paper says "7B transformer," decode L, d, heads, vocab, tied embeddings, GQA, MoE before comparing to another 7B. Two formulas do most of the napkin math: non-embedding parameters ≈ 12 L d², and forward FLOPs per token ≈ 2N.

The big idea

For a classic dense decoder block with multi-head attention (separate Q/K/V) and a standard 4d MLP (not SwiGLU):

Attention:  4 · d²          # W_Q, W_K, W_V, W_O
MLP:        2 · d · (4d) = 8 d²
Norms:      ~4d             # negligible
Per layer:  ~12 d²
L layers:   ~12 L d²
Embeddings: V d  (+ optional untied LM head another V d)

SwiGLU and grouped-query attention change the constant. GQA shrinks K/V matrices. SwiGLU changes the 8d² term.

A useful rule from the scaling literature: forward FLOPs per token ≈ 2N. Each parameter participates in roughly one multiply-add (counted as 2 FLOPs) in the forward pass. Backward training is about 2× the forward compute for gradients, totaling ≈ 6N FLOPs per token trained. That is the constant in pretraining and scaling laws. See also the 6ND rule.

Walking through a 7–8B model

Take d = 4096, L = 32:

12 · 32 · 4096² = 12 · 32 · 16,777,216 ≈ 6.44e9

Plus embeddings V d: with V = 128,000, that is ≈ 0.52e9. Total near 7B if the LM head is tied. Closer to 8B if untied, or if SwiGLU/wider MLP bumps the layer constant. Llama 2 7B used d=4096, L=32. Llama 3 8B is in the same ballpark with a larger vocab.

Interview napkin math: "7B params → ~14 GFLOPs forward per token." Then multiply by tokens, and by 3 for train (fwd+bwd ≈ 3× fwd with the 6N vs 2N convention).

The 2N rule is a forward-pass average. Breaking it down for one token in a dense block:

For short n, MLP plus projections dominate. For long n, attention's n² term wins. That is why long-context prefill is expensive even when parameter count is fixed.

At inference:

• Prefill processes n prompt tokens in parallel: ~2N n FLOPs, compute-bound.
• Decode generates one token at a time: ~2N FLOPs per step, but often memory-bound because you stream weights and read the growing KV cache.

Activation memory during training holds residuals, attention intermediates and MLP expansions for backward. Activation checkpointing (recompute versus store) trades FLOPs for VRAM. Inference avoids most of that but must store the KV cache: roughly 2 · L · n · n_kv_heads · d_k · bytes.

Llama-2-7B-class: L=32, n_kv_heads=32, d_k=128, FP16. Cache per token ≈ 2·32·32·128·2 ≈ 0.5 MB. At 4k context ≈ 2 GB just for KV, before weights (~14 GB FP16). GQA with 8 KV heads cuts that about 4×.

Common mistakes

• Comparing two "7B" models without L, d, vocab and tying. The badge is not the architecture.
• Using 2N for training cost. Training is about 6N per token.
• Forgetting KV cache when sizing decode memory. At 4k context it can be gigabytes.

Check yourself

1. d = 4096, L = 32. About how many non-embedding parameters? Answer: 12 · 32 · 4096² ≈ 6.44B.
2. Why is decode often memory-bound if it is only ~2N FLOPs per token? Answer: You stream the weights (and read a growing KV cache) for a tiny matmul.

Key takeaways

• Classic non-embedding params ≈ 12 L d². Forward ≈ 2N FLOPs/token. Train ≈ 6N.
• Embeddings add V d, twice if the LM head is untied.
• Long n makes attention's n² term dominate even at fixed N.
• KV cache is often the hidden decode-memory cost.

Further learning

Next: encoder vs decoder transformers. For serving math, see inference math.