Write amplification

An application writes 10 MB. How much does the disk actually write? In an LSM tree, the answer can be many times larger, because compaction keeps rewriting data as it moves down the levels. That multiplier is write amplification. It decides how much disk bandwidth and SSD endurance you need.

The amplification triangle

Every storage engine pays some mix of these three costs. The RUM conjecture (Athanassoulis et al., 2016) says an access method can optimize at most two of read overhead, update overhead and memory (space) overhead, at the expense of the third. Compaction strategies sit at different points on that triangle.

Walking through it: write amplification for leveled compaction

Assume 1 TB of live data, RocksDB defaults (L1 = 256 MB, fanout 10), and a stream of small random writes.

Step 1: count the levels. Find the first level big enough to hold 1 TB:

L1 = 256 MB
L2 = 2.56 GB
L3 = 25.6 GB
L4 = 256 GB
L5 = 2.56 TB   ← first level large enough to hold 1 TB, so it is the last

Step 2: follow one byte from the application to the bottom level.

WAL append                                  1
Memtable flush to L0                        1
L0 → L1 merge                              ~1
L1 → L2: rewrites ~10 bytes of L2 per byte ~10
L2 → L3                                    ~10
L3 → L4                                    ~10
L4 → L5                                    ~10
                                           ----
Total write amplification                  ~43

Each "~10" comes from the fanout. When one file from level N merges into level N+1, it overlaps about 10 times as much data below it, and all of that is rewritten.

Step 3: translate to hardware. At 10 MB/s of application writes:

Device writes = 10 MB/s × 43          = 430 MB/s
Per day       = 430 MB/s × 86,400 s   ≈ 37 TB/day

A 3.84 TB data-center SSD rated for one drive write per day can absorb about 3.84 TB per day. This workload would wear it out about 10 times faster than its rating. That is why write amplification is a first-class tuning concern.

Treat 43 as an upper bound

Treat ~43 as a back-of-envelope upper bound. Real deployments often measure less. Not every merge overlaps a full 10 times the data. RocksDB's dynamic level sizing keeps levels proportioned to the actual data. Skewed workloads overwrite hot keys while they are still in the memtable, so those writes never reach disk.

Ways to reduce it include:

• A larger fanout, which means fewer levels but more read work per merge.
• A larger L1.
• Size-tiered or universal compaction.
• Key-value separation (the WiscKey idea, and RocksDB's BlobDB). Large values go into a separate log, so compaction moves only keys and pointers.

Compare with a B-tree

A B-tree also amplifies writes. Updating one 100-byte row eventually rewrites its whole 8 KB page. That is a WA of 80× for that change, although several changes to the same page between checkpoints share one page write. B-trees are not automatically better on writes. Their cost comes as random I/O, where an LSM pays with sequential I/O.

When compaction falls behind

The most common LSM production problem is compaction falling behind. If writes arrive faster than compaction can merge them, L0 files pile up and reads slow down. RocksDB then throttles writes, called write stalls, or stops them entirely to protect read performance. Watch pending compaction bytes and the L0 file count, and leave enough disk bandwidth and CPU for compaction.

Common mistakes

• Budgeting only the application's write rate. Size disk bandwidth and SSD endurance for the amplified rate.
• Treating 43× as a measured number. It is an upper bound for one example setup. Measure your own.
• Ignoring write stalls. A backlog in compaction shows up first as slow writes, then as slow reads.

Check yourself

• What does an application write of 10 MB/s become on the device with a WA of about 43? About 430 MB/s of device writes.
• Why does a B-tree also amplify writes, even though it is not an LSM? A small row change rewrites its whole page. Checkpoints can batch several changes to one page, but each small change still costs a page-sized write.

Key takeaways

• The RUM conjecture says you cannot minimize read, write and space overhead at the same time.
• Leveled compaction has low read and space amplification, and high write amplification.
• Each level of fanout 10 multiplies writes by about 10. Across five levels, that is about 43× for 1 TB.
• Budget disk bandwidth and SSD endurance for the device writes, not the application writes.

Further learning

Revisit compaction strategies to compare the options in this table, or go back to LSM storage engines for the overview. For the in-place alternative, see B-tree storage engines.