Data building blocks
Caching
Where to put a cache, how to read and write through it, how to size it from hit-rate math, and how to keep it from serving stale data or melting down.
A cache stores the result of expensive work, so the next request can skip the work. It trades freshness for speed and lower load. A Redis read takes a fraction of a millisecond. The database query it replaces might take 10 ms and a share of a scarce primary's CPU. For read-heavy systems, a cache is the most effective scaling tool you have.
Caches are also a steady source of subtle bugs. They serve stale data after writes. They turn one expiring key into thousands of simultaneous database queries. And a cache that quietly absorbs 99% of your load becomes a single point of failure the moment it restarts empty.
The pages below take the problem in order: where caches live, how data gets in, how to size them, how they go stale, and how they fail.
Concepts on this topic
| Step | Concept | What it answers |
|---|---|---|
| 1 | Cache layers | Where can a cache sit, and what does each layer cost you? |
| 2 | Cache patterns | Who fills the cache, and how do writes flow? |
| 3 | Hit rate and sizing | How much load does a cache actually remove? |
| 4 | Redis specifics | How does the default shared cache behave? |
| 5 | Cache invalidation | How does a cache learn that its data changed? |
| 6 | Cache stampede | What happens when a hot key expires? |
| 7 | Cache failure modes | How do hot keys, missing keys and cold starts break a cache? |
| 8 | Memcache at Facebook | How does one large operator handle all of this? |
Common mistakes
- Caching before measuring. Adding Redis to fix a missing database index hides the real problem.
- A global key for per-user data. Caching one user's response under a shared key shows that data to other users. This is a security bug.
- Forgetting invalidation for a second writer. Another service that writes the same table must also invalidate the cache.
- Treating the cache as storage. A cache can lose data at any time. Keep the source of truth in the database.
What to answer in a design doc
Every caching decision should state three things:
- What staleness is acceptable for this data.
- What happens to the database when the cache is empty.
- How you'll know it works. Track hit rate, hit and miss latency, evictions per second and memory.
In interviews, name the cache layer and pattern, give a TTL and an invalidation plan, do the hit-rate arithmetic, and mention stampede and hot-key protection before you're asked.
Key takeaways
- Caches sit at every layer, from the browser down to the database buffer pool.
- Cache-aside with delete-on-write and a jittered TTL is the default pattern.
- Hit rate drives database load. Going from 90% to 99% cuts misses tenfold.
- Invalidation gives bounded staleness, not strong consistency. Read exact values from the database.
- Protect against stampedes, hot keys, penetration and cold starts before they happen.
Further learning
The Scaling Memcache at Facebook paper is the best long read on caches under real load.
Redis documentation covers the server in depth.
ByteByteGo blog (Alex Xu) has short, clear diagrams of cache designs.
Optimal Probabilistic Cache Stampede Prevention (Vattani et al.)cache_stampede.pdfcseweb.ucsd.eduThe original NSDI '13 talk on the memcache design:
An MIT lecture on the same paper:
Related
Back-of-the-envelope estimates
Turn users and features into QPS, storage and bandwidth numbers in two minutes, and use them to decide what the design actually needs.
Databases and data modeling
Choose between relational, document, wide-column, key-value, graph and analytical stores by listing your access patterns first, then model the data to fit them.
CAP, PACELC and consistency models
What "consistent" means once data lives on several replicas, from linearizable to eventual, what each guarantee costs, and how to pick one per feature.