CPU load and the run queue

Load average is the number everyone looks at first, and it is widely misread. On Linux it is not a CPU percentage. It counts the tasks that are either running or waiting for something uninterruptible, usually disk. The run queue is the more direct signal: the processes that are ready to run and waiting for a CPU core.

The big idea

Picture a single barista. Load average counts the people at the counter and the people waiting for a delivery in the back room. A load of 8 on an 8-core box could mean every core is busy and healthy, or that all 8 tasks are stuck waiting on the disk. The two cases need different fixes, so the number alone does not decide anything.

Check vmstat 1. Look at:

• r (runnable): processes waiting for a CPU.
• b (blocked): processes waiting on I/O.
• wa: percentage of CPU time spent idle while I/O is pending.

If r is much larger than the core count, the box is CPU-saturated. If b is high and wa is 30% or more, look at the disk first.

Walking through it

Say vmstat 1 5 shows r around 40 on an 8-core machine, with id near 0 and wa near 0. That is CPU saturation: 40 runnable tasks for 8 cores. Queueing theory explains why this hurts. As utilization approaches 100%, wait time grows fast, so latency can climb while average utilization still looks like "only 70%".

The next step is to find which process burns the CPU, with pidstat 1 3, and then profile it. Going to iostat would waste time, because the disk is not the problem here.

Steal time (%st in top or mpstat) is different. It means the hypervisor ran someone else's virtual machine on one of your vCPUs. In a cloud, high steal points to a noisy neighbor or an oversubscribed host. Scale out or change the instance family. You cannot fix it from inside the guest.

Common mistakes

• Equating load average with CPU percent. A load of 8 on 8 cores can be fully busy or fully blocked on disk. Check r and b.
• Looking only at averages. One hot core with the others idle points to a single-threaded bottleneck. mpstat -P ALL 1 shows each core.
• Ignoring steal in the cloud. It makes the guest look slow for reasons inside the guest cannot fix.

Check yourself

1. Load average is 8 on an 8-core box. What do you check to know whether the CPU is the problem? The run queue (r) and blocked tasks (b) in vmstat. Many runnable tasks means CPU saturation. Many blocked tasks means I/O.
2. %st is 20%. Can you fix it by tuning the application? No. The hypervisor is taking CPU time. Move to another host or instance type.

Key takeaways

• Load average counts runnable plus uninterruptible tasks. It is not CPU percent.
• A long run queue (r) that exceeds the core count means CPU saturation and queueing latency.
• Steal time is hypervisor time you cannot tune away from inside the guest.

Further learning

The scheduler behind these queues is in CPU scheduling. For the next idea, see memory and page cache.

The USE Method (Brendan Gregg)The USE Methodbrendangregg.com