CPU scheduling

A machine with 8 cores might have 600 threads that want to run. The scheduler is the kernel component that decides which thread runs on which core, and for how long. Most threads are not running at any moment. A web server thread spends much of its life waiting for the network, so the scheduler's real job is to hand the cores to whichever threads are ready.

The big idea

Each thread is in one of a few states. It is runnable (ready to use a core), running (on a core now) or blocked (waiting for I/O, a lock or a timer). A blocked thread uses no CPU until its wait ends.

The scheduler runs a thread until one of three things happens: it blocks, its time slice runs out (a few milliseconds is typical), or a higher-priority thread wakes up.

Walking through the cost

Suppose a service makes 10,000 context switches per second on one core. At about 3 µs each:

switches per second  = 10,000
cost per switch      = 3 µs
time lost each second = 10,000 × 3 µs = 30,000 µs = 30 ms
share of the core    = 30 ms / 1,000 ms = 3%

At 100,000 switches per second, the same arithmetic gives 300 ms of every second, or 30% of the core, spent switching. You can see this with vmstat 1: a huge cs column while CPU is busy and little real work gets done.

CPU-bound vs I/O-bound

• CPU-bound work (compression, large JSON parsing, image resizing) keeps a core busy. More threads than cores only adds switching.
• I/O-bound work (most web services) spends its time waiting. You can run far more concurrent tasks than cores, because most are blocked at any moment.

Little's law gives the number. If a request takes 50 ms in total but only 2 ms of CPU, one core can handle about 1000 / 2 = 500 requests per second. At that rate, about 500 × 0.05 = 25 requests are in flight at once. That is how many concurrent tasks it takes to keep the core busy, and it drives the choice between thread pools and async I/O in async I/O.

Priorities and fairness

Nice values run from -20 (highest priority) to +19 (lowest). nice -n 10 ./batch-job politely deprioritizes a batch job. Each nice step changes a thread's CPU weight by about 1.25 times, so two CPU-bound threads at nice 0 and nice 5 share a core roughly 75/25.

Linux has used several fair-share schedulers. Since Linux 6.6 (2023), the default is EEVDF (earliest eligible virtual deadline first). It keeps the older idea: threads get CPU in proportion to their weight, and threads that sleep often get a quick turn when they wake.

Container CPU throttling

Containers do not get CPU shares only. They get a quota per period, by default 100 ms. A container with cpu: 500m (half a core) gets 50 ms of CPU per 100 ms.

quota per period   = 50 ms  (0.5 cores × 100 ms)
5 busy threads on 5 cores:
  each burns 10 ms → 50 ms used in the first 10 ms of wall time
  the container is throttled for the remaining 90 ms

The average CPU looks low, but every request that lands in the throttled window waits. You see this as p99 latency spikes at low average CPU. Check nr_throttled in the cgroup's cpu.stat file.

Common mistakes

• Running far more CPU-bound threads than cores. The extra threads spend time switching, not computing.
• Judging a container by average CPU. Throttling hides inside a low average. Check nr_throttled.
• Choosing a worker count by habit. For CPU-bound work, start with about one worker per core. For I/O-bound work, use Little's law and measure.

Check yourself

1. A container shows 40% average CPU, but its p99 latency spikes every few seconds. What should you check? Throttling in cpu.stat. Bursts can use the whole quota early in each period.
2. Why can 1,000 I/O-bound threads run on 8 cores? Most of them are blocked waiting at any moment, so few need a core at once.

Key takeaways

• The scheduler gives cores to runnable threads. Blocked threads use no CPU.
• A context switch costs about 1 to 5 µs plus cache damage. Too many threads wastes throughput.
• Little's law turns "how long is a request in CPU and in total" into "how many concurrent tasks I need".
• Container CPU limits are quotas per 100 ms period. Short bursts can throttle a service at low average CPU.

Further learning

Next, see how a service handles being told to stop in signals and graceful shutdown. For how I/O-bound work uses few threads, read async I/O.