Little's law and pool sizing

Little's law connects three numbers. The average number of requests in a system equals the arrival rate times the average time each request spends there. It holds for any stable system, whatever its internals.

Walking through a pool

Suppose the service handles λ = 1,000 requests per second, and each request takes W = 0.2 seconds.

L = λ × W = 1,000 × 0.2 = 200

About 200 requests are in flight at any moment. Your thread pool, connection pool and memory must all hold that many. A pool of 50 threads would queue 150 requests, and their latency would include the wait in line. The law does not say how the 200 are spread out. It only says the average is 200.

Stateful limits bind first

Stateless app pods scale out until a shared dependency stops them. Plan that dependency explicitly.

Connection pools need explicit arithmetic. Suppose each app pod opens 20 database connections, and you scale to 50 pods.

50 pods x 20 connections = 1,000 connections

The database must accept 1,000 connections, or it runs out of memory or hits max_connections. App CPU can look fine the whole time. Fewer pods with sensible pool sizes, or a connection pooler such as PgBouncer, usually fixes it.

For partitioned data stores, one hot key beats the cluster average. A launch that hashes badly onto one shard fails far below the cluster's average capacity.

Common mistakes

• Guessing pool sizes. Use Little's law with the measured rate and latency.
• Scaling app pods to fix a database-bound outage. More pods add more queries and connections, and make it worse.
• Load-testing with uniform random keys. Real traffic has hot keys, and they set the limit.

Check yourself

1. The service runs at 500 requests per second, and each takes 0.4 seconds. How many are in flight? Answer: 200, from 500 × 0.4.
2. Why can adding app pods make a database-bound outage worse? Answer: Each pod adds connections and queries, so the database hits its connection limit or CPU ceiling sooner.

Key takeaways

• Little's law gives the average in-flight work: L = λ × W.
• Size thread pools and connection pools from that number, with headroom.
• Connection counts multiply across pods. Check the database's limit first.
• Hot keys and stateful limits usually bind before stateless compute does.

Further learning

Next, prove the numbers under realistic load: load testing for the knee.