Limiting blast radius

The earlier patterns react to failure at runtime. The most robust systems also limit the blast radius, which is the set of users one failure can affect. They do this in the architecture, so the limit holds before anything goes wrong. Three techniques do most of the work: static stability, cells and shuffle sharding.

The big idea

A system is statically stable if it keeps working through a failure without having to change anything. A cell is a complete, independent copy of a stack that serves a fixed group of customers. Shuffle sharding gives each customer a random subset of workers, so one bad customer cannot take down everyone who shares a shard.

Each technique answers a different question. Static stability asks whether the system can react to a failure at all. Cells ask how many customers share one failure. Shuffle sharding asks how many customers share the same workers.

Walking through it

Static stability. A service needs the capacity of 6 servers and runs in 3 availability zones. Compare two designs:

Reactive design: 2 servers per zone = 6 total
  lose one zone → 4 servers left, must launch 2 more under stress,
                  using a control plane that may itself be impaired

Static design:   3 servers per zone = 9 total (50% more)
  lose one zone → 6 servers left, still at full capacity, no action needed

The static design costs 50% more servers. It removes dependence on the autoscaler at the moment it is most likely to be overloaded. The same idea applies to config, where a service keeps running on its last known config, and to credentials, which can be cached long enough to ride out an identity outage.

Cells. A thin routing layer maps each customer to a cell. A bad deployment, a poison request or a database failure touches one cell. With 20 equal cells, that is about 5% of customers instead of all of them. Deployments roll out cell by cell, so a bad change is caught while its blast radius is small. Slack has described moving to a cellular design after outages in which one degraded zone affected everyone.

Shuffle sharding. Plain sharding gives each customer one shard. If one customer floods a shard, every customer on it is hit. Shuffle sharding gives each customer a random combination of workers:

8 workers, plain sharding into 4 shards of 2:
  a bad customer takes down its shard → 25% of customers affected

8 workers, shuffle sharding with 2 workers per customer:
  distinct pairs = C(8, 2) = 8 × 7 ÷ 2 = 28
  another customer shares both workers with the bad one: 1/28 ≈ 3.6%
  everyone else keeps at least one healthy worker

At AWS scale the numbers get large. Route 53 gives each hosted zone 4 name servers from a pool of 2,048. That is C(2,048, 4), about 730 billion combinations, so one customer's attack almost never takes out every name server another customer depends on. This pattern needs clients that retry across their own workers.

Common mistakes

• One big cell for simplicity. A bad deploy then touches every customer.
• Assuming redundancy works mid-outage. Scaling up during an incident depends on the same control plane that is failing.
• Shuffle sharding without client retries. Clients must try their other workers when one fails.

Check yourself

1. Why is 9 servers (3 per zone) more robust than 6 (2 per zone), when 6 is enough at normal load?

   Answer: losing a zone leaves 6 servers in the static design, at full capacity with no action. The reactive design drops to 4 and must launch servers through a control plane that may be impaired.

2. With 20 equal cells, roughly what share of customers does one bad cell affect?

   Answer: about 5%, which is 1 of 20 cells.

Key takeaways

• Static stability keeps the system working through a failure without any action.
• Cells limit how many customers one failure can touch, and let you roll out changes gradually.
• Shuffle sharding makes it unlikely that one bad customer affects another customer's workers.
• Blast radius is a design choice, so make it before the outage.

Further learning

Peter Vosshall's re:Invent talk explains how AWS uses regions, zones, cells and shuffle sharding to limit blast radius:

The Amazon Builders' Library article on shuffle sharding walks through the combinatorics in more detail:

Workload isolation using shuffle-sharding (Amazon Builders' Library)How shuffle-sharding isolates workloads to limit blast radius, giving each customer a near-unique shard so one bad actor can't take down everyone.aws.amazon.com

For regions, the multi-region page extends this idea across geography.