Retry budgets and retry amplification

Retries multiply load when the system can least afford it. Each layer that retries adds a multiplier. A few layers can turn one user click into dozens of calls to the part that is already struggling.

Walking through the multiplication

A request flows from a mobile app through an API gateway and a service to a database. Each layer retries a failed call up to three times, so each makes four attempts in total.

database call fails persistently
service makes 4 attempts per gateway call
gateway makes 4 attempts per app call     → 4 × 4  = 16 database calls
app makes 4 attempts per user action      → 4 × 16 = 64 database calls per click

A database that normally handles 1,000 queries a second can receive 64 times that load during the failure. Every query times out, and every timeout is retried, so the load stays high even after the original cause is gone.

Two rules that prevent it

1. Retry at one layer. Usually the layer closest to the failure, or the one that knows the operation is idempotent. The other layers should fail fast and pass the error up.

2. Use a retry budget. A per-request cap is not enough. Cap retries as a fraction of total traffic. Google's SRE book describes a per-client budget in which retries may add at most 10% to the request rate. When the budget runs out, failures return immediately.

Normal traffic: 1,000 requests/s
Budget: retries may add 10%, so at most 100 retries/s
Worst case: 1,000 + 100 = 1,100 requests/s, about 1.1 times normal

In steady state the budget is barely used. During an outage it limits the amplification to about 1.1 times, instead of 64 times.

The AWS SDKs use a version of this as a client-side retry token bucket. Each retry spends tokens, and each success refills a little. When the bucket is empty, the SDK stops retrying.

Common mistakes

• Retrying at every layer for safety. Every layer adds a multiplier. Safety comes from one good layer, not several weak ones.
• Relying on a per-request cap alone. Thousands of clients each within their cap can still overwhelm a service.
• Retrying past the deadline. Work that finishes after the caller left adds load and helps nobody.

Check yourself

1. With four attempts at each of three layers, how many database calls can one click cause during a failure? 4 × 4 × 4 = 64.

2. What does a 10% retry budget cap? Retries can add at most 10% to the normal request rate. Worst case, load rises to about 1.1 times normal.

Key takeaways

• Retries multiply across layers. Three layers of four attempts make 64 calls.
• Retry at one layer, and let the others fail fast.
• A retry budget caps total retry load, not just the retries of one request.
• During an outage, a budget keeps the amplification near 1.1 times.

Further learning

For the wider set of patterns, including circuit breakers and load shedding, see resilience.