Percentiles and tail latency

Ask a service how fast it is, and you might hear "40 ms." That number could be the average, the typical request or the worst one. Latency is a spread of values, and the interesting part is the slow end. Percentiles describe the spread honestly. The p99 is the value that 99% of requests beat.

The big idea

Averages describe nobody. Say 100 requests come in. 98 take 50 ms and 2 take 2,000 ms. The average is 89 ms, but no request took 89 ms. Most users had a 50 ms experience. Two users waited two seconds, and the average hid them.

Percentiles keep that information:

• p50 (median): half of requests are faster. This is the typical experience.
• p99: 1 in 100 requests is slower. It sounds rare until you multiply by traffic.
• p99.9: 1 in 1,000 requests is slower. It matters most in high-volume systems.

Walking through it

Work through the example with the numbers from above.

mean = (98 × 50 + 2 × 2,000) / 100 = (4,900 + 4,000) / 100 = 89 ms
p50  = 50 ms
p99  = 2,000 ms

Now scale the slow tail up. At 10,000 requests per second, a p99 of 1 in 100 means 100 slow requests every second.

Users also make many requests. A single page load might trigger 30 API calls. If each call has a 1% chance of being slow, a page load hits at least one slow call about 1 − 0.99^30 of the time:

0.99^30 = 0.740
1 − 0.740 = 0.26  → about 26% of page loads include a slow call

The heaviest users also get the slowest requests. People with large carts or long histories generate expensive queries. They are often the most valuable customers, which is why some services set targets at p99.9.

Common mistakes

• Reporting only the average. Always name the percentile and where it was measured.
• Averaging percentiles across servers. The mean of ten hosts' p99 values is not the fleet's p99. Merge the underlying histograms, such as Prometheus histograms or HdrHistogram, and compute from the merged data.
• Coordinated omission. A load generator that waits for a slow reply before sending the next request silently skips the slow samples. The p99 it reports looks better than the truth. Gil Tene's talk linked below explains this.

Check yourself

1. In the example, what is the p99? 2,000 ms. The mean is 89 ms, but 2 of the 100 requests took 2,000 ms.
2. Why can't you average the p99 values from ten servers? The average of those values isn't the fleet's p99. Merge the histograms and compute the percentile from the merged data.

Key takeaways

• Averages hide the slow requests. Use p50, p99 and p99.9.
• Slow requests multiply with traffic. One in 100 is 100 per second at 10,000 requests per second.
• Aggregate histograms, not averages of percentiles.
• Watch for coordinated omission, which makes measured latency look better than it is.

Further learning

Gil Tene's talk is the best explanation of why most latency measurements are wrong. It covers coordinated omission, averaging percentiles and reading histograms.

Next, read about composing availability, which combines the nines from the earlier pages across several services.