Async I/O

Most server time is spent waiting: for a database, a disk, or a remote API. A thread that blocks while it waits holds memory and a slot in the scheduler, and it can't serve anything else. A server that gives one thread per connection runs out of threads long before it runs out of CPU. This was the "C10K problem" of handling 10,000 concurrent connections, and async I/O is the answer.

The idea: ask the operating system to tell you when a connection is ready, and let one thread serve many connections in turn. Waiting becomes cheap, and the thread stays busy with work that is ready.

The big idea

Three layers work together:

1. The kernel watches many file descriptors at once. Mechanisms include select, poll, epoll on Linux, and kqueue on BSD and macOS. Newer io_uring on Linux submits and completes I/O through shared ring buffers.
2. An event loop in your process asks the kernel what is ready, runs the callback or task for each, and repeats.
3. Async functions let you write the callbacks as straight-line code. Each await marks a point where the function can pause and let the loop run something else.

Concepts in this topic

Choosing a model

Async I/O is for many connections that mostly wait. It is not a way to speed up CPU work. A loop that runs a heavy computation blocks every other connection, so CPU-bound work belongs on a thread pool or in another process.

• Event loop (Node.js, Python asyncio, Nginx). Best for many connections, short handlers, and one or two cores per process. Run several processes for more cores.
• Threads with a pool (Java servlets, many Python and Ruby servers). Simple to reason about, and the blocking code you already have still works. Each thread costs memory and scheduling.
• Green threads (Go goroutines, Erlang processes, Java virtual threads). Write blocking-style code, and let the runtime multiplex many lightweight tasks onto a few OS threads.

Common mistakes

• Calling a blocking function, such as a synchronous file read or a CPU-heavy JSON parse, inside an event loop. Every other connection stalls behind it.
• Starting thousands of tasks with no limit, which fills memory with work the downstream can't absorb.
• Assuming async makes a slow query faster. It only frees the thread while the query runs.

Key takeaways

• Async I/O lets one thread serve many connections by waiting on the kernel for readiness.
• An event loop runs whichever task is ready, and await marks where a task yields.
• Keep blocking and CPU-heavy work off the loop.
• Bound the work in flight, so the system degrades gracefully under load.

Further learning

Next, see blocking I/O and epoll for the kernel side of how one thread waits on many sockets.