Resilience patterns

Large outages rarely start large. They start with one slow disk, one overloaded cache node, or one dependency whose latency drifts from 50 ms to 5 seconds. What turns that into a site-wide outage is the rest of the system's reaction. Callers wait too long, retry too eagerly, queue too much, and keep sending traffic to something that cannot serve it.

Resilience patterns are the deliberate choices that keep a failure small. Each one keeps a problem inside one dependency, one feature, one cell or one customer.

Why failures spread

A slow dependency is more dangerous than a dead one. A dead dependency fails fast, and your code handles that. A slow one holds your threads, connections and queue slots while it works through its backlog. Your callers then hold theirs, and the failure climbs the call graph. Cascading failures walks through the mechanism with numbers.

Concepts in this topic

Read them in order. Each one builds on the last.

Backpressure, which is how a slow consumer tells a fast producer to slow down, is covered in backpressure. The rule that applies everywhere: every queue needs a bound and a policy for what happens at the bound.

Testing resilience

A mechanism that has never fired is a hypothesis. Fault injection turns it into evidence.

• In tests. Tools such as Toxiproxy sit between your service and a dependency, adding latency, dropping connections or limiting bandwidth. Check that timeouts fire, breakers open and fallbacks render.
• In production. Netflix's Chaos Monkey (2011) randomly ended production instances during business hours, which forced teams to build services that survive instance loss. Managed tools such as AWS Fault Injection Service and Gremlin run controlled experiments with stop conditions.
• Game days. A scheduled exercise where a team breaks something on purpose, such as a failed database or a dead zone, and watches whether alerts, runbooks and automation work. These often reveal a broken alert channel or a deleted runbook script.

Start small: one dependency, in staging, with a clear hypothesis. For example: "If pricing returns errors, product pages render without prices, and the error rate stays under 1%." The incident response and disaster recovery pages cover the human side.

Common mistakes

• Assuming a slow dependency is harmless. Slowness consumes your threads just as errors do.
• Adding a fallback that is never tested. It will run for the first time during an emergency.
• Retrying at every layer. Multiple layers of retries multiply load on the failing dependency.
• Leaving queues unbounded. An unbounded queue hides overload and turns it into long waits.

Key takeaways

• Cascades start with slowness. Retries and inefficiency under load finish the job.
• Every call gets a timeout, and every request carries a deadline that downstream calls respect.
• Circuit breakers stop calls to a failing dependency. Bulkheads cap how much capacity any one dependency can use.
• Shed load early and by priority, so accepted work still meets its deadline.
• Limit blast radius through architecture, with cells and static stability, before anything fails.
• Prove resilience with fault injection and game days.

Further learning

Netflix engineers explain how prioritized load shedding in their API gateway keeps playback working during spikes:

Peter Vosshall's re:Invent talk is the clearest explanation of how AWS uses regions, zones, cells and shuffle sharding to limit blast radius:

Start with cascading failures to see the mechanism behind the rest of this topic.