What consensus costs

Consensus is not free. Every committed write waits for a majority to store it, and each node must flush to disk before it answers. Where the nodes sit decides how long that takes. This page puts numbers on the cost, so you can decide when consensus is worth it.

The big idea

A committed write has two costs. The first is a round trip from the leader to a majority. The second is a durable disk write, called fsync, on each node that answers.

In one data center, that adds up to roughly 1 to 5 ms. Across regions, the speed of light dominates. The farther apart the nodes, the longer every write takes.

Walking through it

Three nodes: a leader in us-east, one follower in us-west at about 65 ms, and one in eu-west at about 75 ms. The leader needs only one follower ack, because two of three nodes is a majority.

Commit latency  = fastest follower round trip + fsync
                = min(65, 75) + 2
                = about 67 ms

Client in eu-west adds its own trip to the leader:
                + 75 ms
                = about 142 ms per write

Now five nodes: two in us-east, one in us-west, one in eu-west and one in ap-northeast, which is about 160 ms away. The leader needs two follower acks. The other us-east node answers in about 1 ms. So the second ack comes from us-west at about 65 ms.

Placement matters more than the node count. Keep a majority close together and writes are fast, but a regional outage takes the majority with it. Spread the nodes out and every write pays the long trip.

Reads can be cheaper. A lease read lets the leader answer from its own state while it holds a time-based lease. A ReadIndex read has the leader confirm it is still leader with one heartbeat round. Both stay linearizable. A read from a follower without either check can be stale.

Throughput is limited by the leader. It sends every entry to every follower. A 5-node group sends each entry 4 times. A 7-node group sends it 6 times. Batching helps, but one group is still one leader. You scale by running many groups.

Common mistakes

• Adding nodes for extra safety. Each extra node adds messages and disk flushes to wait for. Five nodes already survive two failures.
• Running on slow disks. Slow fsync delays commits and can cause missed heartbeats and spurious elections.
• Putting application data in etcd. Its default storage quota is 2 GB, and it is built for small, critical data.

Check yourself

1. Why does a 5-node group send each entry 4 times? The leader sends it to each of the four followers.
2. With the leader in us-east, what is the commit latency for a 3-node group using a us-west follower at 65 ms? About 67 ms: the 65 ms round trip plus about 2 ms of disk flush.

Key takeaways

• A committed write costs a majority round trip plus a disk flush.
• Across regions, distance sets the latency floor.
• Placement is a trade-off between speed and surviving a regional outage.
• One Raft group has one leader, so scale by running many small groups.

Further learning

Operations that span several consensus groups need a transaction protocol on top. See distributed transactions.

For the cluster-level picture, read Consensus and Raft again with these numbers in mind.