Process pauses and clock jumps

A failing node doesn't always crash. Sometimes it freezes for a while and then carries on as if nothing happened. A paused process does not know it was paused. It wakes up holding beliefs that the rest of the cluster revoked while it was frozen. A crash erases a process's state. A pause leaves that state in place, and the state is now wrong.

Causes of pauses

• Garbage collection. A stop-the-world GC on a large JVM heap can pause for seconds. Modern collectors (G1, ZGC, Shenandoah) reduce this a lot, but long pauses still happen under memory pressure.
• VM live migration or steal. The hypervisor suspends your VM to move it, or a noisy neighbor starves your vCPU.
• Swapping and page faults. Touching a page that was swapped out blocks the thread on disk I/O.
• Synchronous disk I/O. An fsync on a busy disk, or a hiccup on a network-attached volume, can block for seconds.
• Manual stops. SIGSTOP, a debugger, or a laptop lid closing in a developer environment.

Clock jumps

NTP can step the wall clock forward or backward. A process that measures a timeout by subtracting two wall-clock readings may decide that a lease expired an hour ago, or that it will expire in an hour. Wall-clock time is fine for showing dates. It is risky for measuring intervals. See time, clocks and ordering.

Walking through a pause

A leader holds a lease for 10 seconds. Then it pauses.

t=0s    Leader A is granted a lease that lasts 10 s
t=2s    A starts a 15-second garbage collection pause
t=10s   The lease expires. The cluster elects leader B
t=12s   B accepts writes
t=17s   A wakes up. It still believes it is the leader.
        A sends a write. The storage must reject it.

A crashed leader would have lost its state, and nothing bad would happen. The paused leader is the dangerous one. It acts on stale beliefs, and the only thing that can stop it is a check at the storage layer. The fix is covered in fencing tokens.

Common mistakes

• Measuring intervals with wall-clock time. Use a monotonic clock for elapsed time. It doesn't jump when NTP steps the system clock.
• Assuming a process knows it was paused. It doesn't. Its view of time and ownership can be out of date the moment it resumes.
• Trusting a check made just before a write. A pause can land between the check and the write.

Check yourself

1. Why is a paused leader more dangerous than a crashed one? A crashed leader stops acting. A paused leader wakes up with stale beliefs and may act on them.

2. Why does a monotonic clock suit timeouts better than wall-clock time? It only moves forward at a steady rate, so an interval measured with it is not distorted by clock steps from NTP.

Key takeaways

• A pause is not a crash. The paused process keeps its state and its wrong beliefs.
• GC, VM migration, swapping and disk stalls all cause pauses.
• Clock jumps can make leases look expired or valid when they are not.
• Protect shared resources with checks at the resource, not at the client.

Further learning

Next, see how to decide that a node is gone in timeouts and failure detectors.