Failover and split brain

When the leader dies, a follower must take its place. This is failover. Every step is a source of incidents. Failover is where most replication problems show up in production, because it happens under pressure and with partial information.

The big idea

Failover has four steps.

1. Detect the failure. Usually a heartbeat timeout. A 30-second timeout means at least 30 seconds of failed writes. A 3-second timeout means a long garbage-collection pause or a network blip can trigger a failover you did not need, and that failover causes its own downtime.
2. Choose the new leader. Ideally the follower with the most recent data.
3. Reconfigure clients. Applications, connection poolers and other replicas must learn the new leader's address.
4. Fence the old leader. If the old leader was only partitioned and not dead, it may still accept writes. Two nodes accepting writes is split brain, and it corrupts data in ways that are hard to undo.

Fencing blocks the old leader from writing. You can revoke its storage access, shut it down, or use a fencing token: a number that rises with each election, so storage rejects writes from a stale leader.

Tools such as Patroni (Postgres) and Orchestrator (MySQL) automate this. They rely on a consensus store such as etcd, ZooKeeper or Consul to decide who is leader. That is why such clusters use an odd number of nodes. A majority must agree, so 3 nodes survive 1 failure and 5 survive 2. See consensus.

Walking through it

Setup: one Postgres primary handles 2,000 write transactions per second. Two asynchronous replicas sit in other zones. Replica A lags about 50 ms, and replica B lags about 300 ms, because it also runs heavy reports.

T = 0 s    The primary loses power. Its last commit is not on replica A's 50 ms tail.
T ≈ 10 s   The leader key in etcd expires. Patroni promotes the most up-to-date replica, A.
Lost:      2,000 writes/s × 0.05 s ≈ 100 acknowledged transactions
           (if B had been promoted: about 600. At a 5-second lag burst: about 10,000)
Later:     The old primary rejoins as a replica. Its unreplicated tail is discarded.

The takeaway: if losing 100 orders is unacceptable, use synchronous replication and accept a millisecond or two per commit. Monitor lag and alert on it. Keep reporting workloads from making the best failover candidate fall behind.

GitHub's October 2018 outage is the standard case study. A 43-second network partition between its East Coast hub and its primary data center caused replicas on the West Coast to be promoted. Writes that had reached the old primary but not replicated could not be merged cleanly with new writes. Restoring consistency took about 24 hours of degraded service.

Common mistakes

• Using a heartbeat timeout that is too short. A brief pause triggers an unnecessary failover.
• Promoting the wrong follower. Choose the one with the most recent data, not the first one healthy.
• Skipping fencing. Without it, a partitioned old leader can keep writing.
• Reusing auto-increment IDs across systems. A failover can reissue them.

Check yourself

1. Replica lag is 50 ms and writes arrive at 2,000 per second. About how many acknowledged writes can a failover lose?

   Answer: about 100 (2,000 × 0.05).

2. Why does a consensus-backed cluster use an odd number of nodes?

   Answer: a majority must agree. Three nodes survive one failure, and five survive two.

Key takeaways

• Failover has four steps: detect, choose, reconfigure and fence.
• Asynchronous replication can lose the writes that were acknowledged just before the leader failed.
• Fencing prevents split brain, so a stale leader cannot write.
• Odd-sized consensus groups let a majority decide who is leader.

Further learning

The GitHub post-incident analysis is the clearest public account of how replication failures unfold:

GitHub: October 21 post-incident analysisIn-depth analysis of the incident that impacted GitHub services on October 21 and 22.The GitHub Blog

Next, see why a user can read stale data from a replica right after writing, in replication lag anomalies.