Page cache and fsync

When your program calls write, the kernel copies the bytes into the page cache, a region of RAM holding recently used file data. The call returns right away. The kernel writes the page to disk later, on its own schedule, often after a few seconds. If the machine loses power before that, the write is gone, even though the call succeeded.

Durability requires an explicit request: fsync. It asks the kernel to write the file's dirty pages and metadata to the device, and to wait until the device reports completion.

The big idea

There are three durability points, and most bugs come from confusing them:

• After write: the data is in RAM. A process crash keeps it, a power cut may not.
• After fsync: the data and metadata needed to read it are on stable storage, as far as the device reports.
• After fsync on the directory: a new or renamed file's name is on stable storage too.

Skipping the last one loses files that appear to have been saved correctly. A new file's directory entry can be lost, even when its contents were flushed.

Walking through it

The pattern for a durable write has three steps: write, flush the data, then flush the name when you create or rename a file. The full version appears in durable file writes. Here is the core call sequence:

import os
fd = os.open("journal.log", os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644)
os.write(fd, b"txn 42 commit\n")
os.fsync(fd)                     # data and metadata reach the device
os.close(fd)

fdatasync does the same, but skips metadata that isn't needed to read the data back, such as timestamps. It's cheaper, and databases use it on the log when the file size didn't change. Note that appending grows the file, which is metadata, so fdatasync still has to flush it in that case.

The fsync gate

In 2018, PostgreSQL users discovered a painful failure. When the kernel's fsync reported an I/O error, the kernel cleared the dirty flags on the failed pages. A later fsync then returned success, even though the data was never written. PostgreSQL retried the flush, saw success, and carried on. Data was silently lost.

The fix was to treat an fsync error as fatal. A failed fsync must crash the process, so recovery replays from the log instead of trusting the cache. Retrying fsync is not a safe response. Any code that depends on fsync should treat its error as "this data is not on disk, and you can't know which part".

O_DIRECT

Opening a file with O_DIRECT bypasses the page cache and writes straight to the device. Databases use it to control their own caching, as they already do with the buffer pool. It has rules: buffers and offsets must be aligned to the device's block size, and the performance gain depends on whether the application caches better than the kernel does. Use it when you have measured both.

Common mistakes

• Calling write and assuming the data is durable before fsync.
• Forgetting to fsync the directory after creating or renaming a file.
• Retrying fsync after an error and trusting the second result.
• Assuming fsync makes data durable on every device. Drives with volatile write caches may lie unless they have power-loss protection.

Check yourself

• Why is a successful write not enough for durability? Answer: the data is in the page cache. The kernel may write it later, so a power cut before then loses it.
• Why must a failed fsync crash the process, not retry? Answer: after an error the kernel may have cleared the dirty pages, so a retry reports success for data that was never written.

Key takeaways

• write returns when data reaches RAM. fsync is the durability point.
• Flush the directory too when creating or renaming a file.
• Treat fsync errors as fatal, and recover from a known-good source.
• O_DIRECT bypasses the cache, but it needs alignment and measurement.

Further learning

Next, see durable file writes for the full pattern for replacing a file safely.