File sync design

A sync client keeps a folder the same on every device. The naive way is to upload whole files on every change. At Dropbox scale that wastes most of the storage and bandwidth. The better way is to split files into chunks, store each chunk under its hash, and upload only what the server doesn't already have.

The big idea

Think of a recipe card. The card lists ingredients by name, and the jars of ingredients sit on a shared shelf. Two recipes that use the same flour share one jar. A new recipe version is just a new card that points to the jars it needs.

The analogy holds for storage. Versions are small lists of hashes, and the chunks are shared. The analogy breaks when you dedupe across users. Then a client can learn whether someone else already has a file, which is a privacy issue.

Walking through it

Scale first. Assume 10 million users with 20 GB each.

Logical data: 10M × 20 GB = 200 PB
At ~$0.023/GB-month in S3 Standard: 200,000,000 GB × 0.023 ≈ $4.6M / month

That number is why deduplication matters. Dropbox moved most of its data off S3 onto its own erasure-coded system, Magic Pocket, around 2016.

Why content-defined chunks. Fixed-size chunks fail on insertions. Add one byte at the start of a file, and every chunk boundary moves, so every chunk hash changes. Content-defined chunking (CDC) sets boundaries from the bytes themselves, using a rolling hash. An insertion changes only the chunk it lands in. The boundaries resync right after it.

Real systems use larger targets, typically 1 to 8 MB for cloud sync, because per-object overhead punishes tiny chunks.

The sync protocol.

1. The client chunks a changed file and hashes each chunk.
2. It asks the metadata service which hashes are already stored. Only the missing chunks upload, directly to object storage through presigned URLs.
3. It commits a new version, (file_id, version, [hash1, hash2, ...]), with the base version it started from. If another device committed first, the base doesn't match. The client saves a "conflicted copy" instead of overwriting, which is what Dropbox does.
4. Other devices hear about the change through a notification, fetch the new chunk list, and download only the chunks they lack.

Data model. The metadata database is sharded by a namespace, such as a user's root folder. It holds files, versions, chunk lists and a change journal that clients read from a cursor. Chunks live in a store keyed by hash. A reference count or periodic sweep deletes chunks that no version still uses.

Common mistakes

• Fixed-size chunks for files edited in the middle. One insertion changes every later chunk hash, and the dedup savings disappear.
• Committing without a base-version check. Two devices can then overwrite each other silently. Check the base version and save a conflicted copy.
• Cross-user dedup without thought. If the server dedupes across accounts, a client can probe whether someone else has a file. Dedupe within one account, or make the client prove it has the content.

Check yourself

1. Why does content-defined chunking beat fixed-size chunks after an insertion? Boundaries come from the content, so they resync after the edit. Fixed boundaries shift, and every later hash changes.
2. How does the client avoid re-uploading chunks the server already has? It sends the chunk hashes first, and the metadata service says which ones are missing.

Key takeaways

• Store chunks by hash, and represent a file version as a list of hashes.
• Content-defined chunking keeps boundaries stable across insertions, so edits upload little.
• Commit versions against a base version. Save a conflicted copy instead of overwriting.
• Cross-user deduplication can leak whether a file exists, so weigh that before enabling it.

Further learning

Inside the Magic Pocket describes Dropbox's move off S3 and its storage design.

Next, return to the blob storage overview to see how these pages fit together.