Globally consistent databases

Some data must look the same from every region. A global username registry is the classic case: two people must not both claim the same name. Worldwide inventory for a flash sale and financial balances accessed from anywhere are others. Several databases can provide this, but each pays for it in latency, because a consistent write must be acknowledged by a quorum that spans regions.

The big idea

Consistency across regions is not free. A write has to reach enough replicas that its result can't be lost or contradicted. When some of those replicas are far away, the write waits for them. The speed of light sets the floor, and no software removes it. So use global consistency surgically, for the few tables that truly need it, and keep everything else regional.

Walking through it

Say a write must be acknowledged by replicas in Virginia, Oregon and Ireland, and a quorum is two of three. A write that starts in Virginia has one local vote. It needs one more, so it must wait for the nearest other region, about 60 to 80 ms away.

single-region commit (in-region quorum)     about 2 to 5 ms
cross-region quorum (Virginia to nearest)   about 60 to 80 ms
ratio                                        60/5 = 12x up to 80/2 = 40x

So a globally consistent write is roughly 12 to 40 times slower than a regional one. Every write pays this cost, not every request, which is why the set of tables that need it should stay small.

The systems differ in how they do it:

The home-region pattern from the previous page fits CockroachDB's REGIONAL BY ROW closely. Spanner's TrueTime keeps the commit wait short, and clocks and ordering explains why clocks matter here.

Common mistakes

• Making everything globally consistent. Most tables are fine as regional, and the latency cost lands on every write.
• Assuming all reads are slow too. Spanner's reads can often be served locally when slightly stale data is acceptable.
• Ignoring write latency in user-facing flows. A single form submit with three globally consistent writes adds three cross-region waits.

Check yourself

1. Why does a cross-region quorum cost at least one round trip per write?

   Only one of the two votes needed is local, so the commit waits for the nearest other region to agree.

2. Which tables should use global consistency?

   Only the few that need one answer everywhere, such as a username registry or a worldwide inventory count. Keep the rest regional.

Key takeaways

• A globally consistent write waits for a cross-region quorum, so it is much slower than a local write.
• Spanner, CockroachDB, Aurora Global and DynamoDB make different trade-offs between consistency, writes and reads.
• Use global consistency only where the data needs it.
• Keep everything else regional, and let home regions handle the rest.

Further learning

Google's Spanner paper describes the globally distributed database and the TrueTime design behind it.

Next: cells and regional isolation, which limits what a bad change can break.