Choosing a shard key

The shard key matters more than the partitioning algorithm. It decides which queries stay on one shard and whether any single shard ends up with most of the load. Choose it before you write any code, and choose it from the queries you run most often.

The big idea

Picking a shard key is like deciding how to file a company's paperwork. File by customer, and each customer's papers sit in one drawer. File by month, and every new paper goes in this month's drawer, which becomes the busiest drawer in the office.

The picture breaks when you need to look across customers. That cost is covered in secondary indexes when sharded.

Four tests

A good shard key has four properties.

1. High cardinality. Far more distinct values than shards. Sharding by country gives at most about 200 partitions, and one of them is the United States.
2. Even traffic. Rows are not enough. A key with even row counts but skewed traffic still makes hot shards.
3. Query locality. The most frequent queries include the key, so they touch one shard.
4. Stability. Changing a record's key means moving it to another shard.

Walking through a key choice

Take an orders table sharded by customer_id across 64 shards.

"My orders" for customer 42     → one shard, fast
"All orders with SKU X"         → all 64 shards, merged

If "all orders with SKU X" is rare, the design may be fine. If it runs on every product page, the design is wrong. Check the query list before you commit to a key.

The single hot value

Even a perfect key fails when one value is hot: a celebrity account, a viral post, or a flash-sale product. In DynamoDB, a single partition key tops out at roughly 1,000 writes per second, no matter how much capacity the table has.

Fixes, from least to most invasive:

• Cache reads in front of the shard. Hot reads are the easy case. See caching.
• Salt write-hot keys. Write to post:123#0 through post:123#9, chosen at random, so writes spread over 10 partitions. Reads must fetch all 10 and combine them, for a like counter by summing.
• Isolate the key. Give it dedicated capacity, or treat it specially in the application.

Salting multiplies read cost by 10, so salt only keys you know are hot. For how feeds handle celebrities, see news feed.

Common mistakes

• Choosing a low-cardinality key such as country or status.
• Choosing by row count. Traffic is what overloads a shard.
• Sharding by time. Writes then all hit the newest shard.
• Assuming you can change the key later cheaply. Changing it means moving every row.

Check yourself

1. Why is status a poor shard key for orders?

Answer: It has very few values, so data and writes pile into a handful of shards.

2. You salt a like counter into 10 keys. What does a read cost now?

Answer: Ten reads, one per suffix, combined by summing. The read cost goes up tenfold.

Key takeaways

• Choose the shard key from your most frequent queries.
• Test every candidate for cardinality, even traffic, query locality and stability.
• A single hot value needs its own fix: cache it, salt it or isolate it.
• Salting multiplies read work, so salt only the keys you know are hot.

Further learning

Next, see what happens to queries on other fields in secondary indexes when sharded.