Metastable failures and retry storms

Some of the worst outages have no single cause. A brief trigger pushes the system into a bad state, and the system's own behavior keeps it there after the trigger is gone. Bronson et al. named these metastable failures in a 2021 HotOS paper.

The big idea

The pattern has three parts:

1. A stable state. The system works, with some spare capacity.
2. A trigger. A load spike, a deploy, a cache flush or a 30-second network blip.
3. A sustaining effect. Something amplifies load, so the system stays overloaded after the trigger ends.

Retries are a common sustaining effect. A request path with several layers, each retrying three times, can multiply the load on the database several times over. The arithmetic is on the retry budgets page.

Walking through a cache flush

A cache has a 95% hit rate. Only 5% of reads reach the database, and the database is sized for that 5%. Now the cache is flushed.

Normal:   1,000 reads/s → 50 reach the database (5%)   fine
Flushed:  1,000 reads/s → 1,000 reach the database      20 times the load it was sized for

The database falls over. It is also the thing that must answer queries to refill the cache. While it is overloaded, the cache can't refill, so the hit rate stays near zero. The trigger was a brief flush. The sustaining effect is the cache miss path itself.

Other sustaining effects follow the same pattern:

• Failover storms. Overloaded nodes miss heartbeats, get declared dead, and their load moves to survivors, which then miss heartbeats too.
• Queue backlogs. A queue builds up during an outage. When the consumer recovers, it processes stale work whose clients have already given up, while new requests wait.

What breaks the loop

Recovery usually means shedding load hard. Turn off retries, block traffic at the edge, or restart with a cold cache and let traffic back in slowly. The longer-term defenses are in resilience patterns: retries at one layer only, exponential backoff with jitter, a retry budget, circuit breakers, and early load shedding, so the work you accept finishes.

Common mistakes

• Assuming recovery starts when the trigger ends. Check whether load is still above capacity.
• Adding retries to fix errors during overload. Retries add load at the moment the system has none to spare.
• Processing stale queued work. Drop requests whose callers have already given up.

Check yourself

1. A cache flush is over. Why might the database stay overloaded? Its capacity was sized for the 5% of reads that miss the cache. Refilling the cache needs queries it can't serve while it is saturated.

2. What is the first defense during a metastable outage? Shed load hard: stop retries, block traffic at the edge, or let traffic back in slowly.

Key takeaways

• A metastable failure keeps going after its trigger is gone, because something amplifies load.
• Common amplifiers are retries, cache misses, failovers and stale queues.
• Break the loop by cutting load, not by waiting.
• Design for it with retry budgets, backoff with jitter and early shedding.

Further learning

The paper that named these failures describes the pattern in detail, with examples from production systems.

Next, see how retry math turns one slow call into many in retry budgets and amplification.