Replication

Replication keeps a copy of the same data on several machines. It sounds like a backup problem, but it is really a coordination problem. Every write has to reach every copy. The copies are connected by an unreliable network, and machines fail in the middle of it. Most serious database incidents, such as lost writes after a failover, a user who cannot see their own comment, or two servers that both think they are in charge, come from how replication handles those facts.

Why replicate

Four reasons, and they pull in different directions:

1. Availability. If one machine dies, another holds the data and keeps serving. A single server at 99.9% availability is down about 8.8 hours a year.
2. Read scaling. If reads outnumber writes 20 to 1, five read replicas multiply read capacity roughly fivefold.
3. Latency. A replica in Frankfurt serves European users in about 10 ms, instead of a 90 ms trip to Virginia. See multi-region.
4. Durability. A disk can die for good. Data on three machines in three availability zones survives what one machine cannot.

These goals pull against each other. Read scaling wants many replicas that answer without coordination. Durability wants writes confirmed on several machines before acknowledging. Availability wants writes to keep working when some copies are unreachable. Each topology below makes a different compromise.

Concepts in this topic

Read them in order. Each one builds on the last.

Choosing a topology

Default to single-leader. Reach for multi-leader when writes must be accepted in several regions or offline, and you have a conflict plan. Choose a leaderless store when you need very high write availability, and your data can tolerate concurrent updates.

Common mistakes

• Treating a replica as a backup. A bad DELETE replicates too.
• Alerting only on whether replicas are up. A replica 20 minutes behind is a failover candidate that will lose 20 minutes of data. Alert on lag.
• Reading from a replica right after a write without a plan. Users will see old data.
• Choosing multi-leader without a conflict plan. Ask what happens when the same row changes in two regions within the replication delay.

Key takeaways

• Replication buys availability, read scale, latency and durability, and every copy is a coordination problem.
• Single-leader is the default, because one leader gives a single write order and no conflicts.
• Failover is where incidents happen. Detection timeouts, the choice of follower and fencing the old leader all matter.
• Lag causes user-visible anomalies, which have standard fixes.
• Leaderless quorums give tunable freshness, but they are not linearizable.

Further learning

Martin Kleppmann's Cambridge lecture is a concise walkthrough of why replication is hard, from the author of the standard textbook:

Designing Data-Intensive Applications (Kleppmann)Designing Data-Intensive Applications (DDIA) — an O’Reilly book by Martin Kleppmann (The Wild Boar Book)dataintensive.net PostgreSQL: High Availability, Load Balancing, and ReplicationChapter 26. High Availability, Load Balancing, and Replication Table of Contents 26.1. Comparison of Different Solutions 26.2. Log-Shipping Standby Servers 26.2.1. Planning …PostgreSQL Documentation

Start with single-leader replication to see how one leader orders every write.