Disk and network saturation

Disk and network saturation look alike from the application. Requests get slower, and timeouts start. The host-level counters tell them apart. Disk saturation shows as queued I/O that takes longer to finish. Network saturation shows as lost packets, retransmits and connections that never get accepted.

Disk: utilization, await and queue

iostat -xz 1 is the main tool for disks. Three columns matter:

• %util: the fraction of time the device had at least one I/O in flight. Near 100% means the device is busy. It does not always mean full, because NVMe drives handle many requests at once. On a single spinning disk it usually does mean full.
• await: the average time, in milliseconds, from a request entering the queue to its completion. Queue time plus service time.
• Queue depth (avgqu-sz): how much work is piled up.

Rising await together with a growing queue is saturation. Application symptoms include high %wa in vmstat, processes stuck in state D, and slow fsync calls. Databases and Kafka are sensitive to this, so the storage type matters. A network-attached disk behaves differently from a local SSD.

Network: retransmits, drops and overflows

For networks, saturation and errors matter more than raw bandwidth until you are near the link speed.

• Retransmits: nstat -az | grep Retrans or ss -ti. Rising retransmits under load mean packet loss or heavy congestion.
• Listen overflows: ListenOverflows and ListenDrops in nstat climb when the accept queue is full and new connections are dropped. The fix is a larger backlog, both in the kernel (somaxconn) and in the application, or faster accepts.
• NIC drops and errors: ip -s link and ethtool -S.
• Connection states: ss -s for a summary.

For a single request, combine host numbers with curl -w timings. A 200 ms TCP connect points at the network or overload. A fast connect with a 200 ms first byte points at the application or its backend.

Walking through it

A worked example from the disk side. A write-heavy service slows down after a deploy. iostat -xz 1 shows %util pinned near 100%, await climbing from one sample to the next, and the queue size growing. That is a saturated device. The next step is to find which process writes the most, with pidstat -d 1, and then decide whether to move the workload to faster storage or reduce the writes.

Common mistakes

• Reading %util as "full" on NVMe. Check await and queue size too.
• Raising the listen backlog without accepting faster. A bigger queue only delays the drops.
• Blaming the network for a slow first byte. If connect is fast, the delay is in the application.

Check yourself

1. %util is 100%, await is 3 ms, and the queue is empty. Is the disk saturated? Not by the signals that matter. The device is busy, but work is not waiting. Check the application for the latency.
2. ListenOverflows is rising. What does it mean? The accept queue is full, so new connections are being dropped before the application accepts them.

Key takeaways

• Disk saturation is rising await plus a growing queue. %util alone can mislead.
• Network saturation shows as retransmits, drops and listen overflows.
• Use curl -w to split a slow request into connect time and first-byte time.

Further learning

For the mechanics of the TCP handshake and its failures, see TCP handshake and reliability. For the next idea, see profiling.

Linux Performance (Brendan Gregg)A collection of documents, slides, and videos about Linux performance, mostly created by Brendan Gregg, and with a focus on performance analysis.brendangregg.com