Container throttling and cgroups

Containers share the host kernel, and their resource limits are enforced by cgroups, the kernel feature that groups processes and meters their CPU and memory. The host can look fine while a container is in trouble. Its CPU is throttled, or it is OOM-killed, and nothing on the node looks wrong. That is why the node-level view can hide a container problem.

The big idea

Think of a monthly phone plan with a data cap that resets every 100 ms. Your average use can be far under the cap, but if you use it all in the first 20 ms of each window, the rest of the window your phone is blocked. The average is fine. The bursts are not.

CFS quota works the same way. A CPU limit does not average the usage over minutes. It caps the CPU time inside each short period.

Walking through it

Say a container has a CPU limit of 500m, half a core. With the usual 100 ms period, the container may run for 50 ms of every 100 ms. That is its budget. A request that needs 15 ms of CPU uses 30% of the budget. Three such requests arriving together use 45 ms of the 50 ms, and a fourth has to wait for the next period.

Check the throttling counter, not the utilization chart:

cat /sys/fs/cgroup/cpu.stat   # look for nr_throttled and throttled_usec

In Prometheus with Kubernetes, the matching metric is container_cpu_cfs_throttled_seconds_total. A rising rate there means the limit is too low for the bursts.

Memory works differently. Hitting the memory limit triggers an OOM kill (exit 137) even when the node has free RAM. kubectl describe pod shows OOMKilled. free on the host still looks fine, because the limit belongs to the cgroup.

Fixing it

1. Mitigate. Raise the CPU limit, or add replicas, so the budget covers the bursts.
2. Confirm. Throttling should drop, and p99 should return to its normal range.
3. Fix the cause. Make the expensive path cheaper, or set requests from measured usage with margin, and alert on the throttle rate instead of only on percent utilization.

Common mistakes

• Trusting average CPU. A pod at 45% average can be throttled hard in each period.
• Reading the node instead of the container. Node averages hide one hot pod.
• Setting memory limits without measuring the working set. Check the real peak before you pick the number.

Check yourself

1. A container has a 500m limit and averages 45% of it. Can it still be throttled? Yes. Bursts can use the whole 50 ms budget within a period, while the average stays low.
2. A pod exits with 137, and the node has plenty of memory. Why? The pod hit its cgroup memory limit, which the node's free memory does not change.

Key takeaways

• A CPU limit is a budget per short period, so averages can hide throttling.
• Watch cpu.stat or container_cpu_cfs_throttled_seconds_total, not only utilization.
• Memory limits cause OOM kills even when the node has free RAM.

Further learning

Requests, limits and their effect on scheduling are in Kubernetes in production. The memory side is in memory, page cache and OOM.