Tensor cores and peak FLOPs

The useful math in an LLM is matrix multiply-accumulate. CUDA cores handle general arithmetic and address math. Tensor cores are the units that chew small matrix tiles at much higher throughput, in the precisions they support.

The big idea

Think of CUDA cores as a general workshop and tensor cores as a punch press that only stamps tiles. LLM layers are tiles, so the press is where the datasheet numbers come from.

The picture breaks when the software never emits the matching instructions. An FP8 or FP4 peak is fiction until a library actually runs that MMA path on your GPU.

Walking through it

Datacenter generations, as of when this was written:

For H100 SXM, dense tensor-core peaks commonly cited:

Use the dense number unless you actually deploy sparse weights.

The roofline ridge is peak FLOP/s divided by peak bytes/s. For H100 BF16:

Ridge ≈ 989e12 / 3.35e12 ≈ 295 FLOP/byte

Kernels below that intensity are bandwidth-limited. Kernels above it can approach the compute roof. Decode at small batch sits far left of the ridge. Large matmuls in prefill sit nearer the roof.

Hopper can mix precisions, for example FP8 compute with higher-precision accumulation, under libraries marketed as Transformer Engine. Lower precision raises the compute roof and shrinks weight traffic. Accuracy comes from calibration and kernels, not from casting a tensor. Engine flags only help if the kernel path exists.

Real kernels rarely sustain 100% of peak. Use peaks as upper bounds, then believe measured tokens/s.

Common mistakes

• Quoting the sparse number for a dense model. That silently doubles the roof.
• Treating peak FLOPs as chat speed. Single-stream decode is set by HBM bandwidth.
• Assuming a dtype flag is enough. No matching GEMM means no speedup.

Check yourself

1. Why is the H100 BF16 ridge about 295 FLOP/byte? Answer: 989e12 FLOP/s divided by 3.35e12 bytes/s.
2. A slide quotes ~1,979 TFLOPS of BF16 on H100. What did it likely assume? Answer: 2:4 structured sparsity. The dense peak is about 989 TFLOPS.

Key takeaways

• Tensor cores dominate useful LLM math. CUDA cores do the glue.
• Believe dense peaks unless you deploy sparsity.
• The ridge point tells you whether a kernel is compute-bound or memory-bound.
• New precisions only count when the library emits the matching instructions.

Further learning

The roofline model turns these peaks into a picture. Next, see how chips talk in GPU interconnects.