Linux and networking
The USE method
A checklist for finding a bottleneck: for every resource, check utilization, saturation and errors before you guess at the application.
When a production box is slow, the application dashboard rarely says why. The USE method is a fixed checklist that asks the same three questions of every hardware and OS resource. It was popularized by Brendan Gregg. You run it before you guess at a bad query or a GC pause.
Concept
A resource is anything the kernel schedules or meters: CPUs, memory, disks, network interfaces, and software pools such as thread or connection pools. For each one, utilization is how busy it is (0 to 100%). Saturation is how much work is queued and waiting. Errors are failed operations, such as drops, timeouts or OOM kills.
The big idea
Think of a checkout line. Utilization is how many cashiers are working. Saturation is how long the queue is. Errors are customers who walked out. A checkout can have all cashiers busy and no queue, which is fine. A long queue with idle cashiers means the bottleneck is elsewhere.
The rule is simple. If utilization is high and saturation is high, that resource is a bottleneck. If errors are non-zero, investigate them even when utilization looks fine. A dying disk can show low utilization while returning I/O errors.
Walking through it
The table gives one signal per cell for each resource:
| Resource | Utilization | Saturation | Errors |
|---|---|---|---|
| CPU | Busy percentage from mpstat | Run-queue length (vmstat column r) | Thermal throttling, machine checks |
| Memory | Used, excluding reclaimable cache | Swap in and out, major page faults | OOM killer events |
| Disk | %util in iostat | Queue depth and high await | I/O errors in dmesg |
| Network | Bytes per second against link speed | Socket backlog overflows | Retransmits, drops, RX and TX errors |
Start at the hardware and OS resources, then move up to application pools. Skipping the bottom is how people spend a day tuning SQL while the disk sits at 100% from a noisy neighbor's logging.
Common mistakes
- Reading utilization alone. A busy CPU that is not queueing is working fine. Saturation is the part that predicts latency.
- Skipping errors because utilization looks normal. Errors are the early warning for hardware.
- Memorizing commands instead of questions. The value is the three questions. The commands change between systems.
Check yourself
- A disk shows 30% utilization, but the application times out and
dmesgshows I/O errors. What does USE say to do? Investigate the errors first. Errors matter even when utilization looks low. - A CPU is at 95% busy, and the run queue is empty. Is it saturated? Not by the USE definition. Work is not queued. Check the next resource or the application.
Key takeaways
- USE asks utilization, saturation and errors of every resource.
- High utilization plus high saturation means a bottleneck.
- Check errors even when utilization looks fine.
Further learning
The fast version of this checklist is the 60-second checklist. The methodology is in Brendan Gregg's write-up.
The USE Method (Brendan Gregg)The USE Methodbrendangregg.comRelated
Linux performance analysis
The USE method and the 60-second checklist for finding why a box is slow, covering CPU, memory, disk, network and the container limits that look like application bugs.
The 60-second checklist
Nine standard commands that give a fast USE pass over a Linux host in about a minute, and how to read what each one shows.
Metrics, dashboards and alerting
Golden signals, RED and USE, Prometheus histograms and alerts that page you only when users are hurting.