Multi-leader replication

In multi-leader replication, more than one node accepts writes. Each leader sends its changes to the others, usually asynchronously. A write is local and fast, and it reaches the other leaders later.

Multi-leader earns its complexity in three situations: multi-region deployments, offline-capable clients and collaborative editing. A phone app with a local database is effectively a leader that syncs when it reconnects. Calendar and notes apps work this way, and so do shared document editors.

The big idea

The cost is write conflicts. Two leaders accept different writes to the same record at about the same time, and neither knows about the other. Something has to decide which write wins, or how to combine them.

Walking through it

A conflict within the replication delay:

t=0 ms    US leader: set price = 10
t=0 ms    EU leader: set price = 12
t=80 ms   each leader sends its change to the other

Both leaders now hold two different values for the same record.
A last-write-wins rule keeps the one with the later timestamp, here 12.
The write of 10 disappears, and no error is raised.

There are four common ways to handle it.

• Avoid conflicts. Send all writes for a record to its home leader, such as the region where that user lives. This is the most common real-world answer. It makes each record behave as if it had a single leader.
• Last write wins (LWW). Attach a timestamp and keep the highest. It is simple, but it silently discards one write. With skewed clocks, the "last" write may actually be earlier.
• Merge with domain rules. For a shopping cart, take the union of the items. For a counter, add the deltas.
• Use data types that merge automatically. CRDTs and operational transforms are designed for this. They are covered in conflict resolution.

Common mistakes

• Going multi-leader without a conflict plan. Conflicts then get resolved by accident.
• Trusting clocks for LWW. Skewed clocks can keep the older write.
• Letting two regions write the same record. A home-region rule removes most conflicts.

Check yourself

1. What is the simplest way to avoid write conflicts?

   Answer: route every write for a record to one home leader, so the record behaves like single-leader.

2. Why is last write wins risky when clocks are skewed?

   Answer: the write with the later timestamp may have happened earlier, so the newer write can be the one discarded.

Key takeaways

• Multi-leader lets several nodes accept writes, which suits multiple regions and offline clients.
• Concurrent writes to one record are conflicts, and they need a plan.
• A home leader per record avoids most conflicts.
• Last write wins is simple, but it discards writes silently and depends on clocks.

Further learning

Next, see how leaderless stores handle writes without any leader, in leaderless quorums.