Guide

When a host is slow and you have a minute, you need a fixed order of commands, not a search for the right flag. Netflix's performance team popularized a sequence that runs in under a minute with stock tools. It is a fast pass of the USE method. It does not replace the method. It makes the first pass quick.

Walking through it

Run these in order. Each one checks a different resource.

  1. uptime: load averages and time since boot. A very recent boot can mean a crash loop.
  2. dmesg -T | tail -n 40: the kernel's log. OOM kills, disk errors, NIC resets and thermal events show up here.
  3. vmstat 1 5: run queue (r), blocked tasks (b), swap (si, so), and the CPU split (us, sy, id, wa).
  4. mpstat -P ALL 1 3: utilization for each CPU core. One hot core with others idle often means a single-threaded bottleneck.
  5. pidstat 1 3: which processes use the CPU. Add -d for per-process disk I/O.
  6. iostat -xz 1 3: disk utilization, await (average wait) and queue size.
  7. free -m: read the available column, not free.
  8. ss -s: a summary of socket counts. A large pile of TIME-WAIT is usually fine. A large pile of CLOSE-WAIT means the app is not closing its sockets.
  9. top or htop: confirm the picture.

The pieces fit as a script. This sequence needs the sysstat package for mpstat, iostat and pidstat:

Bash
uptime
dmesg -T | tail -n 40
vmstat 1 5
mpstat -P ALL 1 3
pidstat 1 3
iostat -xz 1 3
free -m
ss -s

Here is how a reading turns into a next step. Say vmstat shows r around 40 on an 8-core box, with id near 0 and wa near 0. The next step is pidstat and profiling. iostat would be a waste. If r is low, b is high and wa is 30% or more, the disk is the place to look first.

Common mistakes

  • Running the commands but not writing down the timeline. Note what changed: a deploy, a traffic spike, a neighboring job.
  • Reading free without available. The free column is almost always low on a healthy box.
  • Skipping dmesg. Kernel messages are the fastest way to see an OOM kill.

Check yourself

  1. vmstat shows b at 12 and wa at 45%. Where do you look first? At the disk. Many tasks are blocked on I/O, so run iostat -xz 1.
  2. Why run dmesg before the others? It can show an OOM kill or a hardware error, which explains symptoms no other command reveals.

Key takeaways

  • Nine commands in a fixed order give a fast USE pass with stock tools.
  • Read r and b in vmstat before blaming the CPU or the disk.
  • Use the first pass to pick the next tool. Profile only after the resource is confirmed.

Further learning

The resource-by-resource details are on the CPU load and memory pages. The Netflix write-up is the original.

Linux Performance Analysis in 60,000 Milliseconds (Netflix)netflixtechblog.com