Finding the bottleneck

Scaling is a search. You find the bottleneck, remove it, and then find the next one. Two quantitative tools make the search faster. Amdahl's law tells you how much a speedup is worth. Little's law and utilization tell you how work queues up when a resource is busy.

The big idea

A fix helps only in proportion to the share of time it touches. If the database accounts for 60% of a request's time, making the application code instant still leaves 60%. Amdahl's law puts a number on that limit.

speedup = 1 / ((1 − p) + p / s)

p = fraction of the work you improve
s = how much faster that part becomes

Queues add a second limit. As a resource gets busier, waiting grows much faster than the load. Little's law gives the number of requests in flight, and utilization sets how long they wait.

Walking through it

Amdahl on latency. A request takes 200 ms: 120 ms in the database and 80 ms in application code. You rewrite the application to be 10 times faster.

new time = 120 + 80 ÷ 10 = 128 ms
speedup  = 200 ÷ 128 ≈ 1.56×   (not 10×)

Amdahl on throughput. The database saturates at 10,000 queries per second. Each request makes 2 queries. The system tops out at 5,000 requests per second, however many app servers you add. The 21st app server does nothing.

Little's law on connections. Each request holds a database connection for 50 ms. At 2,000 requests per second, about 100 connections are busy on average (2,000 × 0.05). A pool of 20 per server across 5 servers is right at the edge.

This explains a common incident. The database slows from 50 ms to 500 ms per query. In-flight requests rise tenfold, the connection pools run out, and requests start failing, even though traffic has not changed.

Utilization sets the queue. In a simple single-server queue, the wait grows with ρ / (1 − ρ), where ρ is utilization:

Latency does not degrade gently as load rises. It falls off a cliff. That is why capacity plans aim for 50 to 70% at peak.

The database is usually the bottleneck. The app tier is stateless and cheap to copy. The database is stateful and shared by everything. When a system gets slow, check the database first: slow queries, missing indexes, lock contention, connection exhaustion and replication lag. A missing index can turn a 1 ms lookup into a 2-second table scan, and that looks exactly like a capacity problem. The databases page covers how to find these.

Common mistakes

• Optimizing a part that is not the bottleneck. Amdahl's law says the gain is capped.
• Running servers at 90% CPU to save money. Tail latency and incident time usually cost more.
• Buying infrastructure for an index problem. Check the query plan before adding capacity.

Check yourself

1. A request takes 200 ms, 120 ms of it in the database. What is the most you can gain by making application code instant?

   Answer: about 1.67× (200 ÷ 120), because the database time stays the same.

2. Why keep utilization near 50 to 70%?

   Answer: waiting grows as ρ / (1 − ρ), so at 90% utilization queueing delay is about 9 times the service time.

Key takeaways

• Amdahl's law caps the gain from speeding up one part of a request.
• A saturated shared resource limits throughput, however many clients you add.
• Little's law gives the number of requests in flight, which sizes pools.
• Latency rises sharply near full utilization, so plan for headroom.
• The database is usually the bottleneck, and a missing index often looks like a capacity problem.

Further learning

Next, see all of this applied to a real plan, in growing a marketplace.