Latency and percentiles

Before you can make something faster, you have to say how slow it is. "The API feels slow" is not a measurement. Latency is how long one operation takes, as the caller sees it. Percentiles describe the spread of many latencies, and they tell the truth that averages hide.

The big idea

Latency distributions are not neat bell curves. They have a long right tail, caused by garbage collection pauses, cache misses, lock waits, retransmitted packets and queues. A few requests take much longer than most.

Describe them with percentiles:

• p50 (median): the typical request. Half are faster.
• p99: 1 in 100 requests is slower than this.
• p99.9 and max: the tail, where the worst bugs hide.

Walking through why averages lie

Suppose 100 requests take 20 ms each, except one that takes 2,000 ms.

mean = (99 × 20 ms + 1 × 2,000 ms) / 100
     = (1,980 + 2,000) / 100
     = 39.8 ms

The mean says 39.8 ms, but no real request took 39.8 ms. The median is 20 ms, which describes 99 of the 100 requests. The mean is pulled up by one outlier. Percentiles keep the typical case and the tail apart.

Why the tail matters more than it seems

Users make many requests. A page that makes 30 backend calls hits a p99 event on at least one of them with probability 1 − 0.99^30.

1 − 0.99^30 = 1 − 0.740 = 0.26   → about 26% of page loads

A "1% problem" per call becomes a problem on roughly a quarter of page loads.

Fan-out amplifies the tail. In "The Tail at Scale", Jeff Dean and Luiz Barroso described the same effect for search. If one server has a 1% chance of a slow response, a request that waits on 100 such servers in parallel is slow whenever any one of them is slow.

1 − 0.99^100 = 1 − 0.366 = 0.63   → about 63% of requests

Search, feed ranking and any scatter-gather service live with this every day.

Coordinated omission

A load generator that waits for each response before sending the next request can hide the problem it is meant to find. During a stall, it stops sending. The stalled requests never show up in its results, so the latency graph looks fine.

Use open-loop tools that send requests at a fixed rate, such as wrk2, k6 with arrival-rate executors, or Vegeta.

Common mistakes

• Reporting only the average. Show p50, p99 and max together.
• Averaging percentiles across hosts. Merge histograms first.
• Trusting a load test that waits for each response. It hides stalls.
• Ignoring the tail because most requests are fast. Multiply the tail rate by the number of calls per user action.

Check yourself

1. A service has p50 = 20 ms and p99 = 1.9 s. Which number tells you about the typical user? The p50. The p99 says 1 in 100 requests is slow, which may still be a lot of users.
2. A page makes 30 backend calls, each with a 1% chance of being slow. About how often does a page see a slow call? About 26% of the time, from 1 − 0.99^30.

Key takeaways

• Latency is a distribution. Report p50, p99 and max, not just the mean.
• Averages hide outliers. Percentiles show them.
• Each call in a fan-out adds its tail risk. Page-level latency is worse than any single call's p99.
• Merge histograms before computing percentiles across machines.

Further learning

Gil Tene's talk is the clearest explanation of why averages and many load tools mislead, including coordinated omission. It changes how you read latency graphs.

Next, read Amdahl's and Little's laws to predict what a speedup or a pool-size change will buy you.