Linux performance analysis

A production box can be slow in many ways that look the same from the application. p99 jumps from 80 ms to 2 s, health checks flap, and timeouts cascade. The instinct is to open the app dashboard and guess at a bad query, a GC pause or a slow dependency. Often the cause is lower in the stack. The CPU run queue is deep. The page cache is thrashing. A disk is at 100% utilization. The NIC is dropping packets. Or Kubernetes throttled the container so hard it looks like a hung process.

This topic is a repeatable way to find the resource at fault in about a minute. Then you go deep, with profiling, only when the bottleneck is inside a process.

Walking through the method

The pages build from the method to the tools to the container traps:

In practice

When a host is slow on call, follow the same routine each time:

1. Separate the resource from the app. Spend the first minute on the checklist before you read business logic.
2. Prefer rates and saturation over single gauges. Utilization without queue depth misleads, and throttle counts matter in Kubernetes.
3. Write the timeline down. Was there a deploy, a traffic spike, or a neighboring job? Check dmesg for OOM kills.
4. Profile only once the resource is confirmed. A flame graph of an I/O-bound process wastes time.
5. Fix the source. A raised limit is a mitigation. Capacity or an efficiency fix closes the incident.

Interview questions in this area often ask you to explain USE in a minute, read a vmstat line, explain why load average is not CPU percent, and describe the throttling trap. A clear answer to all four shows production experience.

Common mistakes

• Going to the application first. Sixty seconds of resource checks rule out the host before you read business logic.
• Reading one gauge. Utilization without queue depth can mislead. Check saturation and throttle counts too.
• Profiling too early. A flame graph of an I/O-bound process points at the wrong code.
• Raising a limit and stopping. A higher limit is a mitigation. The closeout is capacity or an efficiency fix.

Key takeaways

• Start with USE: utilization, saturation and errors for each resource.
• The 60-second checklist is a fast USE pass with stock tools.
• Load average, memory available, disk await and CPU throttling each have a trap. Read the paired numbers.
• Profile only after the resource is confirmed. In containers, check the cgroup budget before the app.

Further learning

The general method for measuring performance is in performance and profiling.

Brendan Gregg's LinuxCon talk maps where vmstat, perf and the tracers fit together. His LISA talk introduces the USE method.

Linux Performance (Brendan Gregg)A collection of documents, slides, and videos about Linux performance, mostly created by Brendan Gregg, and with a focus on performance analysis.brendangregg.com Flame GraphsHomepage for Flame Graphs: a visualization for stack traces.brendangregg.com