Replication lag anomalies

Followers apply the leader's changes a little late. This delay is replication lag. Normally it is milliseconds. Under load, during a long query on the replica, or after a network hiccup, it can grow to seconds or minutes. A read from a lagging replica shows the past, and users notice three anomalies.

The big idea

Each anomaly breaks one simple promise. Read-your-writes says a user sees their own changes. Monotonic reads says a user never sees time go backward. Consistent prefix reads says that causally related writes are seen in order. You do not need full linearizability to fix these. Cheaper rules fix the anomalies users actually notice.

Walking through it

Read-your-writes. A user edits their profile, the page reloads, and the read goes to a replica that has not applied the edit. The old name appears, and they edit it again. The fix is to read from the leader for data the user just changed. A more precise fix records the leader's log position after the write, and serves reads only from replicas that have replayed past it.

This sketch uses the Postgres log position (LSN). It assumes primary and replica are database handles:

// After a write, remember the leader's log position in the user's session
session.minLsn = (await primary.one("SELECT pg_current_wal_lsn() AS lsn")).lsn

// On read, use a replica only if it has replayed past that position
async function readUserProfile(userId: string, session: Session) {
  const replica = pickReplica()
  const { caughtUp } = await replica.one(
    "SELECT pg_last_wal_replay_lsn() >= $1::pg_lsn AS caught_up", [session.minLsn])
  const db = caughtUp ? replica : primary
  return db.one("SELECT * FROM users WHERE id = $1", [userId])
}

Monotonic reads. A user refreshes a comment thread twice. The first read hits a replica with 50 ms of lag and shows a new comment. The second hits a replica with 2 seconds of lag, and the comment disappears.

Read 1 → replica A (50 ms lag):  comment visible
Read 2 → replica B (2 s lag):    comment gone   ← time went backward

The fix is to pin each user to one replica, for example by hashing the user ID. Move them only if that replica fails. One replica applies changes in order, so the user's view never goes backward.

Consistent prefix reads. An observer sees an answer before the question that prompted it. This happens when the two writes went to different partitions with different lag. The fix is to keep causally related writes in the same partition, so one log orders them. Tracking causal dependencies explicitly is the other option, covered in clocks and ordering.

Common mistakes

• Sending every read to replicas, including reads right after a write. Route the user's own recent writes to the leader.
• Pinning users only sometimes. Pinning must be consistent, or monotonic reads still break.
• Assuming writes in different partitions keep their order. They do not, unless something orders them.

Check yourself

1. Why does pinning a user to one replica fix monotonic reads?

   Answer: a single replica applies changes in order, so the user never moves back to an older view.

2. How does the log-position check decide where to read?

   Answer: the replica serves the read only if it has replayed past the position of the user's last write. Otherwise the read goes to the leader.

Key takeaways

• Replica lag causes three anomalies: read-your-writes, monotonic reads and consistent prefix reads.
• Route a user's own recent writes to the leader, or wait for the replica to catch up.
• Pin each user to one replica to keep their reads moving forward.
• Keep causally related writes in one partition, so their order is preserved.

Further learning

Next, see how several leaders can accept writes at once, and how conflicts are settled, in multi-leader replication.