Machines and programs
The memory hierarchy
From registers to SSDs to the network, the latency numbers that dominate performance, and why the tier your data sits in matters more than the code you write.
A modern core can add two numbers in about a third of a nanosecond. Fetching those numbers from main memory takes about 100 ns, roughly 300 to 400 times longer. An SSD is about 1,000 times slower again, and a server on another continent is slower still. Most performance problems come down to which tier your data was sitting in.
Computers handle this with a hierarchy: small, fast, expensive storage close to the core, and large, slow, cheap storage further away. Each level caches the one below it. The numbers below are the ones to remember, and they change little from one hardware generation to the next.
| Operation | Typical latency | In CPU cycles at ~4 GHz | If 1 ns were 1 second |
|---|---|---|---|
| Register access, simple ALU op | ~0.3 ns | 1 | 0.3 s |
| L1 cache hit | ~1 ns | ~4 | 1 s |
| Branch mispredict | ~5 ns | ~15–20 | 5 s |
| L2 cache hit | ~4 ns | ~12–15 | 4 s |
| L3 cache hit | ~10–40 ns | ~40–150 | 10–40 s |
| Main memory (DRAM) | ~100 ns | ~400 | 1.5 minutes |
| Mutex lock/unlock (uncontended) | ~20 ns | ~80 | 20 s |
| Compress 1 KB with a fast codec | ~1–3 µs | thousands | ~1 hour |
| NVMe SSD random 4 KB read | ~20–100 µs | ~100,000 | ~1 day |
| Round trip within a datacenter | ~100–500 µs | ~1 million | ~2–6 days |
| Read 1 MB sequentially from NVMe | ~200–500 µs | ~1 million | ~2–6 days |
| HDD seek | ~5–10 ms | ~30 million | 2–4 months |
| Round trip, US East to US West | ~60–70 ms | ~250 million | ~2 years |
| Round trip, cross-continent (US to Europe or Asia) | ~70–150 ms | ~500 million | 2–5 years |
The right-hand column gives intuition. If an L1 hit took one second, a DRAM access would be a coffee break, an SSD read would cost a day, and a request to Europe would take years.
Why it matters
Most code is written as if every variable access were equally cheap. It isn't. A hash map with 50 million entries and a hash map with 50,000 entries can run the same code 10 to 20 times apart, because only the small one fits in cache. The same logic applies to a loop that calls another service once per item instead of once per page.
Concepts in this section
Read them in order. Each builds on the one before.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Reasoning with latency and bandwidth | How do you turn the table into a cost estimate? |
| 2 | Cache lines and locality | Why does walking an array beat following pointers? |
| 3 | Array of structs vs struct of arrays | How should you lay out records in memory? |
| 4 | False sharing and NUMA | Why can adding threads make code slower? |
Common mistakes
- Reasoning only in big-O. A search over a pointer-based tree with 30 cache misses can lose to a scan over a small, contiguous array.
- Benchmarking with hot caches. A 10 KB input in a loop measures L1, not production behavior.
- Ignoring the network tier. A synchronous cross-region call costs more than a million L1 hits.
- Adding threads to a memory-bound job. If one thread already saturates memory bandwidth, more threads just compete for it.
Key takeaways
- Memorize the orders of magnitude: L1 about 1 ns, DRAM about 100 ns, NVMe 20 to 100 µs, datacenter round trip 0.1 to 0.5 ms, cross-continent 70 to 150 ms. Each step is roughly 100 to 1,000 times slower than the one before.
- Caches move whole lines of 64 bytes (128 on Apple Silicon). Access that is sequential and contiguous is fast. Pointer chasing pays full latency at every hop.
- Layout and traversal order can change speed several times over with identical arithmetic.
- Throughput depends on concurrency and latency together. Multi-socket and multi-core machines add their own traps.
Further learning
Scott Meyers' talk is the classic introduction to why CPU caches matter for ordinary code, with benchmarks of traversal order, false sharing and data layout.
Latency numbers every programmer should know is a short reference for the table above.
Gallery of Processor Cache Effects shows each effect with a small benchmark.
Related
How code runs
From source file to running instructions. Covers the CPU's loop, compilers, interpreters and JITs, branch prediction and the system call boundary.
Back-of-the-envelope estimates
Turn users and features into QPS, storage and bandwidth numbers in two minutes, and use them to decide what the design actually needs.
The roofline model and arithmetic intensity
Predicting performance from FLOPs and bytes: compute-bound vs memory-bound. Start here for the map, then work the examples.