Histograms and fleet percentiles

A latency percentile such as p99 says that 99% of requests finished faster than this value. It sounds easy to compute across a fleet, and it is not. The common shortcut is to take each pod's p99 and average them. That number is meaningless. To get a correct fleet-wide p99, add up the histogram buckets across pods first, and only then take the percentile.

The big idea

Percentiles are not linear, so you cannot average them. Imagine two pods. Pod A serves 10 requests per second with a p99 of 50 ms. Pod B serves 1,000 requests per second with a p99 of 800 ms. Averaging gives 425 ms, which describes neither pod. The fleet's p99 is driven almost entirely by Pod B, because it serves 99 times the traffic.

Walking through it

A histogram stores counts for buckets, such as "requests under 0.1 s", "under 0.25 s", "under 0.5 s" and so on. These counts can be added across pods, because they are simple totals. Percentiles cannot be added, but bucket counts can.

The right order is:

1. Sum the bucket rates across all pods, keeping the le (bucket boundary) label.
2. Take the quantile of the summed buckets.

histogram_quantile(
  0.99,
  sum by (le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

sum by (le) merges the pods. histogram_quantile then estimates the p99 from the merged buckets. If you write avg(histogram_quantile(...)) per pod instead, you are back to averaging percentiles.

A summary computes its quantiles inside each process and sends those numbers out. That is accurate for one process, but the results cannot be combined across instances. For a fleet-wide percentile or a latency SLO, use a histogram.

Common mistakes

• Averaging p99 across pods. It ignores traffic volume. The busiest pod should count the most.
• Using summaries for a fleet SLO. You cannot merge them, so the number will be wrong for the fleet.
• Dropping the le label in the sum. Then the buckets are no longer there to estimate the percentile.

Check yourself

1. Pod A has p99 = 50 ms at 10 requests per second. Pod B has p99 = 800 ms at 1,000 requests per second. Why is their average misleading? It gives equal weight to two pods with very different traffic. Pod B carries almost all the requests.
2. Why can a histogram be merged across pods but a summary cannot? Bucket counts are plain totals that add up. Quantiles computed inside each process cannot be added together.

Key takeaways

• Never average percentiles across instances.
• Sum histogram buckets by le first, then call histogram_quantile.
• Use histograms for fleet latency and SLOs. Summaries stay per process.

Further learning

The SLO side of latency is in SLOs. For the metric types, see metric types and rate.