Linux and networking
The 60-second checklist
Nine standard commands that give a fast USE pass over a Linux host in about a minute, and how to read what each one shows.
When a host is slow and you have a minute, you need a fixed order of commands, not a search for the right flag. Netflix's performance team popularized a sequence that runs in under a minute with stock tools. It is a fast pass of the USE method. It does not replace the method. It makes the first pass quick.
Walking through it
Run these in order. Each one checks a different resource.
uptime: load averages and time since boot. A very recent boot can mean a crash loop.dmesg -T | tail -n 40: the kernel's log. OOM kills, disk errors, NIC resets and thermal events show up here.vmstat 1 5: run queue (r), blocked tasks (b), swap (si,so), and the CPU split (us,sy,id,wa).mpstat -P ALL 1 3: utilization for each CPU core. One hot core with others idle often means a single-threaded bottleneck.pidstat 1 3: which processes use the CPU. Add-dfor per-process disk I/O.iostat -xz 1 3: disk utilization,await(average wait) and queue size.free -m: read theavailablecolumn, notfree.ss -s: a summary of socket counts. A large pile ofTIME-WAITis usually fine. A large pile ofCLOSE-WAITmeans the app is not closing its sockets.toporhtop: confirm the picture.
The pieces fit as a script. This sequence needs the sysstat package for mpstat, iostat and pidstat:
uptime
dmesg -T | tail -n 40
vmstat 1 5
mpstat -P ALL 1 3
pidstat 1 3
iostat -xz 1 3
free -m
ss -sHere is how a reading turns into a next step. Say vmstat shows r around 40 on an 8-core box, with id near 0 and wa near 0. The next step is pidstat and profiling. iostat would be a waste. If r is low, b is high and wa is 30% or more, the disk is the place to look first.
Common mistakes
- Running the commands but not writing down the timeline. Note what changed: a deploy, a traffic spike, a neighboring job.
- Reading
freewithoutavailable. Thefreecolumn is almost always low on a healthy box. - Skipping
dmesg. Kernel messages are the fastest way to see an OOM kill.
Check yourself
vmstatshowsbat 12 andwaat 45%. Where do you look first? At the disk. Many tasks are blocked on I/O, so runiostat -xz 1.- Why run
dmesgbefore the others? It can show an OOM kill or a hardware error, which explains symptoms no other command reveals.
Key takeaways
- Nine commands in a fixed order give a fast USE pass with stock tools.
- Read
randbinvmstatbefore blaming the CPU or the disk. - Use the first pass to pick the next tool. Profile only after the resource is confirmed.
Further learning
The resource-by-resource details are on the CPU load and memory pages. The Netflix write-up is the original.
Linux Performance Analysis in 60,000 Milliseconds (Netflix)netflixtechblog.comRelated
Linux performance analysis
The USE method and the 60-second checklist for finding why a box is slow, covering CPU, memory, disk, network and the container limits that look like application bugs.
The USE method
A checklist for finding a bottleneck: for every resource, check utilization, saturation and errors before you guess at the application.
CPU load and the run queue
What load average really counts, how the run queue shows CPU saturation, and why steal time in the cloud is a problem you cannot fix inside the guest.