Latency numbers every engineer should know

Some operations take a nanosecond and others take milliseconds. The gap between them matters more than the exact figures. These classic numbers, popularized by Jeff Dean and Peter Norvig, show that gap. Hardware has moved on since, but the ratios have held up well.

Make the gaps human-sized

Scale the table so that an L1 cache reference takes one second. Everything else stretches by the same factor:

L1 cache reference         0.5 ns   →  1 second
Main memory reference      100 ns   →  about 3 minutes
Datacenter round trip      500 µs   →  about 12 days
California → Netherlands → California  150 ms  →  about 9 to 10 years

Now the cost of a design choice is easy to feel. A request that waits on a memory lookup is a blink. A request that waits on a cross-continent call is a decade.

Three things to take away

• Memory is far faster than the network. A datacenter round trip (500 µs) is about 5,000 times a main memory reference (100 ns). A design that makes several cross-region round trips per request can't be low-latency, however fast the servers are.
• Sequential beats random. On a spinning disk, one seek (10 ms) costs about as much as reading half a megabyte sequentially (20 ms per MB). Many small random reads pay the seek every time. That is why log-structured and append-only designs work.
• CPU is cheap compared with the network. Compressing 1 KB takes about 3 µs. Sending 1 KB over 1 Gbps takes about 10 µs. When bandwidth is the constraint, compressing first often wins.

Common mistakes

• Treating the table as exact. Measure on your own hardware. Keep the ratios, not the decimals.
• Assuming a network call costs about the same as a function call. The gap is thousands of times.
• Putting a cross-region call in the hot path. Every request pays it. Move it off the request path if you can.

Check yourself

1. Using the table, how many times slower is a datacenter round trip than a main memory reference? 500 µs ÷ 100 ns = 5,000 times.

2. Why does a spinning disk punish many small random reads? Each read pays a 10 ms seek, which costs about as much as reading half a megabyte in one sequential pass.

Key takeaways

• Memory, disk and network differ by orders of magnitude. Ratios matter more than exact times.
• A datacenter round trip is thousands of times slower than memory.
• Sequential reads and batched writes beat many small random operations.
• Compression is usually cheap enough to pay for itself when bandwidth is scarce.

Further learning

The original table, with commentary and updated versions linked in the comments:

Latency numbers every programmer should knowLatency Numbers Every Programmer Should Know. GitHub Gist: instantly share code, notes, and snippets.Gist

Next, use these ratios to size a real request in the estimation chain.