Utilization and the knee

A queue forms whenever work arrives faster than it can be served, even for a short stretch. Utilization ρ (the Greek letter rho) is the fraction of time a resource, such as a CPU, thread, connection or disk, is busy. As ρ rises, waiting time rises with it, and the rise accelerates near 100%.

Walking through the knee

Take a service that takes S = 50 ms when it is idle.

Check one row. At 90% utilization, W = 50 / (1 − 0.9) = 50 / 0.1 = 500 ms. Going from 80% to 90% busy doubles the latency. Going from 90% to 95% doubles it again.

That curve is the knee. Latency looks fine for a long time, then a small bump in traffic doubles p99. This is why "CPU was only 85%" is not reassurance. For latency-sensitive services, plan for 70 to 80% on the scarce resource, which leaves room for variance and for failures.

Variance moves the knee earlier. If service times have a long tail, with occasional queries 10 times slower, the same average utilization gives a worse p99 than the table shows. Caching, fixing slow queries and isolating expensive paths are capacity work too.

Common mistakes

• Trusting averages. The average hides the tail, and the tail is where users feel the queue.
• Planning at 95%. The system is already in the steep part of the curve.
• Ignoring variance. Heavy-tailed service times push the knee to lower loads.

Check yourself

1. At 90% utilization with S = 50 ms, what is the time in the system? Answer: 500 ms, from 50 / (1 − 0.9).
2. Why is "CPU was only 85%" not reassurance? Answer: Latency climbs steeply near the knee, so 85% can already be past the point where p99 starts moving.

Key takeaways

• Latency rises steeply as utilization approaches 100%.
• Plan for roughly 70 to 80% on the scarce resource.
• Heavy-tailed service times bring the knee forward.
• Use the knee to decide how much headroom to keep.

Further learning

Next, use Little's law to turn these numbers into pool sizes: Little's law and pool sizing.