Communication rooflines

Multi-GPU serving adds another roof: network or NVLink bandwidth. If a tensor-parallel shard must all-reduce an activation every layer, you can define intensity against communicated bytes instead of HBM bytes. When collective time exceeds compute time, you are communication-bound.

The big idea

Splitting a casserole across two ovens only helps if the hallway between kitchens is fast. If you spend longer carrying pans than baking, a third oven makes dinner later. The picture breaks when the model does not fit on fewer GPUs: then you pay the hallway because you have no choice.

Walking through an all-reduce

Napkin: all-reduce an 8 MB tensor over a link that sustains ~300 GB/s effective:

t_comm ≈ 8e6 / 300e9 ≈ 27 µs

If the layer's local compute is ~20 µs, tensor parallel is already hurting. NVLink-class in-node links keep this viable longer than PCIe or cross-node RDMA. Same roofline thinking, different bandwidth number.

More tensor-parallel GPUs can slow tokens/s when you are communication-bound. That is why interactive serving prefers modest TP (2-way on a node is common for a 70B) and avoids TP across nodes when it can.

Use the model as a lie detector on any claimed speedup:

1. Compute a lower bound time from bytes/bandwidth or FLOPs/peak (whichever is larger).
2. Compare to claimed tokens/s.
3. If a blog claims batch-1 decode near peak TFLOPS on a 70B, intensity math says no.
4. If prefill is far below the compute roof, look for launch overhead, poor GEMM shapes, or the CPU.

Roofline assumes the kernel runs long enough to be limited by FLOPs or bytes. If the CPU launches a 3 µs kernel every 10 µs, measured performance sits under both ceilings. Horace He calls this the overhead regime. Fixes: fuse, compile, CUDA graph, or batch work so each launch does more. See launch overhead.

Common mistakes

• Adding TP ranks because the model "should be faster on more GPUs" without timing the collective.
• Comparing NVLink napkins to multi-node RDMA and wondering why tokens/s collapsed.
• Ignoring overhead and concluding the roofline math is wrong when the host never fed the GPU.

Check yourself

1. Local compute is 20 µs and the all-reduce is 27 µs. Are you compute-bound? Answer: no; communication is the longer term, so TP is hurting.
2. Why can more TP GPUs reduce tokens/s? Answer: extra ranks add collective bytes; if you were already communication-bound, the extra tax wins.

Key takeaways

• Collectives have their own roofline, with interconnect bandwidth in the denominator.
• When t_comm > t_compute, more TP can slow decode.
• In-node NVLink stays viable longer than PCIe or cross-node RDMA.
• Overhead can put you under every roof — then compile and graph before buying silicon.

Further learning

The scaling book's roofline chapter is written for this LLM use case.

How to Scale Your Model: All About RooflinesWhen we run algorithms on hardware, weHow To Scale Your Model