Game days and region drills

A runbook you have never run is a guess. A game day is a planned exercise where the real team runs a failure scenario with the real runbook, against written success criteria. A region evacuation drill is a game day that shifts traffic and data dependence out of one region.

The big idea

A game day is a fire drill. Everyone practices the same steps, you time them, and you find the stuck door before the fire does.

The fire drill analogy breaks in one place. A real game day can move real traffic and cost real money. Scope the blast radius first. Start with a staging cell or a small canary slice, and bring executives in before a full production drill.

How to run one

1. Declare the scope. A staging cell, a canary slice of production traffic, or full production with executive awareness.
2. State the hypothesis with a number. For example: fail over payments to the second region within 20 minutes, with RPO at most 1 minute.
3. Assign roles exactly as in a real incident.
4. Run the runbook. Shift traffic in DNS, promote the replica, drain queues and verify writes.
5. Measure the real RTO and RPO. File every gap as a P0 or P1 item.
6. Reset the environment, and write a short after-action note.

How often? Critical paths need a drill at least once a year, and preferably twice. Restore tests run quarterly.

Walking through a hypothesis

The table is the output of the drill. Each gap becomes a ticket with an owner and a date.

Include degraded dependencies

Dependencies fail even when your region is healthy. A good drill includes more than "our zone is gone":

• The authentication provider is slow or erroring.
• Object storage returns 503s.
• The DNS provider is slow to update.

For each, document the degraded mode: read-only mode, cached auth, or queued writes for later. DR is often product behavior under partial failure, not only a database promotion.

Common mistakes

• Only testing a full region loss. Partial failures are more common.
• No success criteria. Without a number, every drill looks like a pass.
• Skipping the after-action note. The gaps are lost by the next quarter.
• Running the drill with different people. It must use the real on-call roles.

Check yourself

1. Why write the hypothesis, with a number, before the drill? Without a target, you cannot tell pass from fail, and the drill proves nothing.
2. Besides a lost zone, what should a drill test? A degraded dependency, such as auth or object storage returning errors.

Key takeaways

• A runbook is a guess until a drill measures it.
• Write a numeric hypothesis, run the real runbook, and time it.
• Test degraded dependencies, not just a missing zone.
• Every gap gets an owner and a date.

Further learning

Next, see how to test failures in production-like conditions in chaos engineering.