Retry budgets and hedging

A retry recovers from a transient failure, such as one dropped packet or one overloaded instance. Retries also amplify a persistent failure. The job is to retry only when it helps, and to space retries so they do not arrive in a synchronized wave. The full rules for retries are in idempotency. This page covers the budget and the hedge.

The big idea

A retry is safe only when two things hold. The operation must be idempotent, so running it twice has no extra effect. The error must be retryable, such as a timeout, a 503 or a 429. A retry that breaks either rule can double a charge or add load to a dying service.

Retries also need a budget. A retry budget caps retries at a small share of normal traffic, about 10%. During a brief blip, retries absorb the failure. During a real outage, the budget runs out, and retries stop adding load.

Walking through it

Backoff with jitter. Without randomness, every client that failed at the same moment retries at the same moment. Jitter spreads them out. The cap doubles on each attempt, and the actual sleep is random below the cap:

BASE, CAP = 1.0, 30.0

for attempt in range(6):
    cap = min(CAP, BASE * 2 ** attempt)  # the ceiling for this attempt
    print(f"attempt {attempt}: sleep somewhere in [0, {cap:g}] s")

Output:

attempt 0: sleep somewhere in [0, 1] s
attempt 1: sleep somewhere in [0, 2] s
attempt 2: sleep somewhere in [0, 4] s
attempt 3: sleep somewhere in [0, 8] s
attempt 4: sleep somewhere in [0, 16] s
attempt 5: sleep somewhere in [0, 30] s

Retry at one layer. If three layers each make four attempts, one user request can make 4 × 4 × 4 = 64 calls at the bottom when everything fails.

Hedged requests. A hedge is different from a retry. If a read has not returned by its p95 latency, send a second copy to another replica and take whichever answers first. Google's "The Tail at Scale" paper showed that hedging after a short delay cuts tail latency a lot, for a few percent more load. Hedging is a latency tool for idempotent reads. It is not a failure recovery tool, and it is wrong for writes, because a hedge runs the operation twice.

Common mistakes

• Retrying at every layer. Pick one layer that owns the retry policy.
• Retrying non-idempotent operations. A retried payment can charge twice.
• Retrying past the deadline. Once the budget is gone, a retry only adds load.
• Hedging writes. A hedge sends the operation twice.

Check yourself

1. Three layers each retry three times, so each makes four attempts. How many calls reach the bottom for one user request when everything fails?

   Answer: 64, because 4 × 4 × 4 = 64.

2. Why hedge reads but not payments?

   Answer: a hedge sends a second copy, which is safe only when running the operation twice has no extra effect.

Key takeaways

• Retry only idempotent operations, and only on retryable errors.
• Use exponential backoff with jitter, so retries spread out.
• Cap retries with a budget of about 10% of normal traffic.
• Retry at one layer, and never past the deadline.
• Hedging cuts tail latency for idempotent reads, at a small extra load.

Further learning

Retries and timeouts only bound one caller's damage. To stop calls to a dependency that is clearly failing, read circuit breakers.