Resharding without downtime

Data grows, and shards fill up. Moving data while the system keeps serving traffic is the hardest part of sharding. The best defense is planning for it early: pick a layout where growth means moving whole pieces, and when you must change the layout, migrate in steps you can reverse.

The big idea

Moving stock from a small shop to a bigger one is easier if the stock is already packed in crates. You move crates, not individual items. The label on each item never changes.

The picture breaks when crates are too big to move or too small to balance. That is the reason for choosing the number of logical partitions carefully.

Three strategies for growth

1. Many logical partitions up front. Create far more partitions than machines, for example 1,024 logical shards on 8 machines, which is 128 each. To add machines, move whole partitions. The key-to-partition mapping never changes. Kafka, Elasticsearch, Riak and Redis Cluster, which uses 16,384 hash slots, work this way.

Notion split its Postgres data into 480 logical shards spread over 32 databases, and later moved them to 96 machines without rehashing any keys.

2. Dynamic splitting. When a range partition passes a size threshold, split it in two and maybe move half. HBase defaults to around 10 GB per region, and CockroachDB to 512 MiB. Partition count follows the data.

3. Partitions per node. Each node owns a fixed number of partitions, and new nodes take over some of them. Cassandra's virtual nodes work this way.

The migration playbook

When the scheme itself has to change, the usual steps look like this:

Each step can be reversed until the last one. Three rules keep it safe:

• Backfills must be idempotent. Running one twice must give the same result.
• Backfills must not overwrite newer rows. Compare versions or updated_at before writing.
• Shadow reads must reach zero mismatches. Compare old and new answers, and log every difference, before cutting over.

Here is the second rule in action. A backfill copies order 881 from the old shard, but that row was updated after the copy started and dual-written with a newer version. The backfill sees the older version and skips it, so the newer row survives.

For a large system, the whole sequence can take weeks.

Common mistakes

• Backfilling over newer dual-written rows. The migration then quietly reverts recent changes.
• Cutting reads over before shadow reads match. Users see wrong data.
• Changing the key and the scheme in one step. Each change is hard to reverse on its own.
• Decommissioning the old shards too soon. Keep them until the rollback window has passed.

Check yourself

1. Why does a fixed set of logical partitions make adding machines cheap?

Answer: The key-to-partition mapping never changes, so you only move whole partitions between machines.

2. During a backfill, a copied row is older than a dual-written row. What should happen?

Answer: The backfill should skip it, because the newer dual-written row is the correct one.

Key takeaways

• Plan for growth by using many logical partitions, so moves are whole pieces.
• Dynamic splitting makes the partition count follow the data size.
• Migrate with dual writes, an idempotent backfill, shadow reads and then a cutover.
• Keep every step reversible until the final cleanup.

Further learning

Next, see how to generate IDs that stay unique across shards in unique ID generation.