Amdahl's and Little's laws

Before you spend a week on an optimization, ask two questions. How much of the total time does this part take? And how many requests will be in flight once it is faster? Amdahl's law answers the first. Little's law answers the second. Both need only arithmetic, and both stop you from working on the wrong thing.

Amdahl's law: the part you speed up sets the limit

Speeding up one part of a system only helps in proportion to how much time that part takes. If a fraction p of the time is sped up by a factor s, the overall speedup is:

overall speedup = 1 / ((1 − p) + p / s)

The part you did not touch, 1 − p, stays the same. It is the ceiling.

Walking through a database speedup

A request spends 60% of its time in one database call. You add an index that makes that call 3 times faster.

p = 0.60, s = 3
speedup = 1 / (0.40 + 0.60 / 3)
        = 1 / (0.40 + 0.20)
        = 1 / 0.60
        ≈ 1.67×

Even an infinitely fast database gives 1 / 0.40 = 2.5×. The other 40% of the request is the limit.

Walking through a parallel job

A batch job is 95% parallel. You move from 1 core to 16.

speedup = 1 / (0.05 + 0.95 / 16)
        = 1 / (0.05 + 0.059)
        = 1 / 0.109
        ≈ 9.1×

Not 16×. No number of cores can beat 1 / 0.05 = 20×. The serial 5%, such as startup, a single-threaded merge or a lock, dominates at scale.

The practical rule: profile first, then attack the biggest share of time. Making a part that takes 5% of the time 10 times faster saves at most 4.5%.

Little's law: how many requests are in flight

Little's law relates three quantities in any stable system:

L = λ × W

L = items in the system (concurrency)
λ = arrival rate (items per second)
W = average time each item spends in the system

It makes no assumption about how latencies are spread out, which makes it a strong back-of-the-envelope tool.

Walking through a connection pool

A service receives λ = 2,000 requests per second, and each takes W = 150 ms on average.

requests in flight  L = 2,000 × 0.150 = 300

Suppose each request holds a database connection for 40 ms of that time.

connections in use  = 2,000 × 0.040 = 80 on average

A pool of 50 connections cannot cover an average of 80. Requests wait for a free connection. The waiting adds to W, which raises L, which makes the queue longer. That loop is how a pool-size setting turns into a p99 cliff.

Little's law also explains thread-per-request servers. If a downstream call grows from 50 ms to 2 seconds, the same traffic needs 40 times as many requests in flight. The servers run out of threads long before the CPU is busy. See capacity planning for the fleet-level version of this arithmetic.

Common mistakes

• Speeding up the wrong part. Check the share first. A 10× gain on 5% of the time saves little.
• Expecting linear speedup from more cores. The serial part sets the ceiling.
• Setting a pool size by habit. Compute L from λ and W at peak.
• Using averages for W in a queue. A few slow requests hold connections for much longer. Check the tail too.

Check yourself

1. A request is 20% database time. If the database becomes infinitely fast, what is the most the request can speed up? 1 / 0.80 = 1.25×. The other 80% is unchanged.
2. Traffic is 500 requests per second, and each request takes 0.2 s. How many requests are in flight on average? 500 × 0.2 = 100.

Key takeaways

• Amdahl's law: the fraction you do not speed up caps the total gain.
• Little's law: concurrency equals arrival rate times time in system.
• Use both before writing code. They predict whether a change is worth making and how many workers or connections you need.

Further learning

Next, read flame graphs and CPU profiling to find out which part of a request deserves the speedup.