Reasoning with latency and bandwidth

The latency table tells you what one access costs. Engineering questions are usually about many accesses. "Can this endpoint call the profile service for each of 200 items?" is a question about multiplication. This page shows how to do that arithmetic, and how to tell latency from bandwidth.

Walking through it: three quick estimates

Is a hash map with 50 million entries fast? Each entry is about 32 bytes with overhead, so the table is about 1.6 GB. That is far bigger than any cache, so a random lookup goes to DRAM at about 100 ns:

10,000,000 lookups × 100 ns = 1 s

A 50,000-entry map is about 1.6 MB and fits in L2 or L3. The same 10 million lookups cost maybe 10 to 20 times less. The code and the big-O are identical. The tier is what changed.

Should this service call another service in a loop? Suppose each call is a same-datacenter round trip of about 0.5 ms, and you make 200 of them one after another to render one page:

200 calls × 0.5 ms = 100 ms of pure waiting before any work happens
batched into 1 call: 0.5 ms

This is the N+1 query problem measured in physical units.

Can we read the file on every request? If the file is in the OS page cache, a 10 KB read is a memory copy of a few microseconds. If it is not cached, it is an SSD read of about 100 µs. On a network filesystem or S3, it is 1 to 50 ms. The code is the same in all three cases. The tier is not.

Bandwidth is not latency

Latency is one access. Bandwidth is the total rate when many accesses run at once. Rough figures for 2025 to 2026 hardware:

The link between them is Little's law: throughput equals concurrency divided by latency. Each outstanding cache miss moves one 64-byte line per 100 ns:

One miss at a time:  64 B / 100 ns = 0.64 GB/s
To reach 64 GB/s:    64 GB/s × 100 ns / 64 B = 100 misses in flight

A single core can track only a few dozen outstanding misses. A thread that chases pointers through DRAM gets a tiny share of the machine's bandwidth. A sequential scan lets the prefetcher keep many lines in flight, so it gets far more. The same law applies to an SSD: at 100 µs per read, you need a queue depth of 64 to reach 640,000 IOPS.

Memory-bound or compute-bound?

For each byte you load, how much computation do you do? That ratio is arithmetic intensity. A chip has a ridge point, the ratio of its compute rate to its memory bandwidth. Workloads below the ridge are memory-bound. Workloads above it are compute-bound.

NVIDIA H100 (SXM):  ~990 TFLOP/s dense BF16, ~3.35 TB/s HBM
Ridge point:        990e12 / 3.35e12 ≈ 295 FLOPs per byte

LLM decode, batch size 1, 7B parameters in FP16:
  weights read per token:  7e9 × 2 bytes = 14 GB
  compute per token:       ≈ 2 × 7e9 = 14 GFLOP
  intensity:               14 GFLOP / 14 GB = 1 FLOP per byte
  time per token ≥ 14 GB / 3.35 TB/s ≈ 4.2 ms  → at most ~240 tokens/s

At 1 FLOP per byte, the GPU's arithmetic units sit idle more than 99% of the time. Generating one token for one user is a memory-bandwidth problem. That is why inference servers batch many users together, and why quantization speeds decoding almost in proportion. The full model is covered in the roofline model.

Common mistakes

• Treating bandwidth as the fix for latency. A 100 Gbit link does not make one 50 ms round trip faster.
• Forgetting concurrency. Doubling the number of threads does nothing for a memory-bound job that already saturates bandwidth.
• Estimating with the wrong tier. Cached and uncached reads differ by about 1,000 times.

Check yourself

• Why does a single pointer-chasing thread reach only a small share of DRAM bandwidth? It can keep only a few misses in flight, and Little's law ties throughput to concurrency.
• How many calls does a 200-item loop cost at 0.5 ms each, and how would you fix it? 100 ms in series. Batch the calls into one request.

Key takeaways

• Multiply the per-access cost by the number of accesses on the critical path. That product is usually the answer.
• Latency and bandwidth are separate limits. Little's law links them through concurrency.
• Single-user LLM decoding is limited by memory bandwidth, not by FLOPs.

Further learning

Next, see why the 64-byte cache line decides which access patterns are fast, in cache lines and locality.