Fundamentals
Availability, latency and the tail
Nines, percentiles and the math of composing services. Why p99 matters more than the average, how fan-out amplifies the tail, and how to design for both uptime and speed.
Two questions sit under every system design. Is it up? Is it fast? Both sound like yes-or-no questions, and both are really statistics. "Up" is a fraction of time or requests that succeed. "Fast" is a spread of response times, and the slow end of that spread matters most.
The pages below give you the math behind both answers. You'll learn what each extra nine costs in downtime, how availability multiplies across services, why averages mislead, and how fan-out makes a rare slow call the common case.
Consider a checkout that is 99.95% available on its own but calls five services at 99.9% each. The whole request is only about 99.45% available, roughly 4 hours of failure a month. Every hard dependency lowers the ceiling. The same math shows up again in the speed question, where one slow call out of many can become the user's experience. The pages below build on each other, so read them in order.
Concepts on this topic
| Step | Concept | What it answers |
|---|---|---|
| 1 | Nines and downtime | How much downtime does a target allow? |
| 2 | MTBF and MTTR | Should you prevent failures or recover faster? |
| 3 | Percentiles and tail latency | Why is the average the wrong number? |
| 4 | Composing availability | What happens when services depend on each other? |
| 5 | Tail latency amplification | Why does fan-out make slow calls common? |
| 6 | Taming the tail | How do you keep answering when some parts are slow? |
| 7 | Latency budgets | How does one end-to-end target split across hops? |
Common mistakes
- Quoting one latency number. Name the percentile and where you measured it. "40 ms" can mean almost anything.
- Counting only your own uptime. Your service's availability is capped by the hard dependencies it can't run without.
- Trusting redundancy blindly. Two replicas only help when their failures are independent.
Design-time order
Pick availability and latency targets before you draw the architecture. The targets decide how many hard dependencies you can afford, how much redundancy you need, and whether you need multiple regions.
In an interview, these numbers make requirements concrete. Saying that a read path targets p99 under 200 ms and 99.99% availability, with at most two hard dependencies, signals experience.
Key takeaways
- Four nines allow about 4.3 minutes of downtime a month, so automation matters more than heroics.
- Series dependencies multiply availability. Parallel replicas help only when their failures are independent.
- Averages hide pain. Use p50, p99 and p99.9.
- Fan-out amplifies the tail. With 100 parallel calls, a backend's p99 becomes the user's median.
- Recovery time is often the cheapest path to another nine.
Further learning
The Tail at Scale (Dean and Barroso) is the paper behind most of these pages.
Gil Tene's talk explains why most latency measurements are wrong, including coordinated omission and averaged percentiles.
Related
Back-of-the-envelope estimates
Turn users and features into QPS, storage and bandwidth numbers in two minutes, and use them to decide what the design actually needs.
SLIs, SLOs and error budgets
Define reliability in numbers users care about, then use error budgets and burn-rate alerts to balance shipping speed against stability.
Resilience patterns
The toolkit for keeping one failure from becoming an outage, from timeouts and retries to circuit breakers, bulkheads, load shedding and cells. Start here for the resilience map.