Failover testing

The most dangerous multi-region system is one that has never actually failed over. Replication may have broken months ago without anyone noticing. The standby may lack capacity because nobody updated its scaling settings. A hard-coded endpoint may still point at the primary. The runbook may name a dashboard that no longer exists. You find all of this during a real disaster, at 3 a.m., under pressure.

The big idea

A failover mechanism can work exactly as designed and still cause a disaster, if the situation is not the one it was designed for. Failover is safe only when you have practiced it, checked its assumptions and fenced the old primary.

GitHub's October 2018 incident is the classic example.

Walking through it

A network maintenance problem cut connectivity between GitHub's US East Coast hub and its primary data center for 43 seconds. The database orchestration tool, Orchestrator, reacted by its own logic and promoted replicas on the West Coast. When connectivity returned, both coasts had accepted writes that the other did not have.

43 s     connectivity lost between the East Coast hub and the primary data center
         Orchestrator promotes West Coast replicas (correct by its own logic)
         both coasts accept writes that the other does not have
after    connectivity returns; reconciling the two sets of writes takes
         just over 24 hours of degraded service

GitHub chose data integrity over availability. The failover mechanism did what it was built to do. What failed was the assumption that a cross-country failover was safe and routine, for a system that had never been exercised that way. The application's latency to the cross-country database was also far higher than the design had anticipated.

What good practice looks like

• Fail over regularly, for real. Netflix runs regional evacuation exercises, historically called Chaos Kong, in which it moves all traffic out of an AWS region and confirms the others absorb it. Many teams run quarterly game days that fail over a production database.
• Run active-active where you can. Then failover is a routing weight change, and normal traffic exercises it every day.
• Monitor replication lag and standby capacity as first-class SLOs, with alerts.
• Fence the old primary. After failover, the old primary must not accept writes when it comes back. Revoke its credentials, or flip a flag in a consensus store, so you never have two primaries.
• Practice failback. Returning to the original region is a risky migration in its own right.

Common mistakes

• Counting an untested standby as protection. Test it, or treat its behavior as unknown.
• Failing over without fencing. Two primaries accept conflicting writes, and reconciliation can take a day.
• Skipping failback planning. The return trip is where new failures show up.

Check yourself

1. Why is a standby without fencing dangerous when the old primary returns?

   It can accept writes again, so both regions hold writes the other lacks, as happened at GitHub.

2. What is the advantage of running active-active?

   Failover becomes a routing change that normal traffic exercises all the time, so it is not a rare, untested event.

Key takeaways

• An untested failover plan is a guess, so practice it on purpose.
• Fence the old primary, so it can't accept writes after failover.
• Monitor replication lag and standby capacity, and alert on both.
• Practice failback too, since returning is a migration.

Further learning

GitHub's post-incident analysis describes the 2018 failover in detail, including the decisions made during reconciliation.

October 21 post-incident analysis (GitHub)In-depth analysis of the incident that impacted GitHub services on October 21 and 22.The GitHub Blog

Return to the multi-region overview to see how these pages fit together.