Clock drift and NTP

Every server keeps time with a small quartz crystal. The crystal ticks at a nominal rate, but real crystals run slightly fast or slow. That error is called drift. Left alone, the clocks on a fleet of servers spread apart by seconds within days. NTP, the Network Time Protocol, pulls clocks back toward correct time. It helps, but it isn't exact.

The big idea

Think of a wristwatch that gains a minute a week. You reset it by checking a station clock. Between checks, it drifts. The more often you check, the closer you stay. The reset is only as good as your guess about how long the check took to reach you.

That guess is the weak point. NTP assumes the request and reply took equal time. If the network is faster one way, NTP can be wrong by half the difference, and it can't see that error.

Walking through it

Drift adds up quickly. A quartz clock with a worst-case drift of 200 ppm, the bound Google's Spanner paper assumes, behaves like this:

200 ppm = 200 / 1,000,000 = 0.0002 seconds of error per second

per minute: 0.0002 × 60      = 12 ms
per hour:   0.0002 × 3,600   = 0.72 s
per day:    0.0002 × 86,400  = 17.3 s

Even 20 ppm is about 1.7 seconds a day. Most servers drift less than the worst case, but "less" still adds up.

NTP's accuracy depends on the network:

Here is the asymmetry problem. If the request took 1 ms and the reply took 9 ms, NTP's estimate is off by (9 − 1) / 2 = 4 ms, and it never knows.

When NTP finds the clock is off, it either slews the clock gradually, so time keeps moving forward, or it steps the clock to the correct value, which can move it backward. Leap seconds add jumps too. Many large operators smear the extra second over several hours.

Common mistakes

• Treating NTP as exact. It's accurate within the path's asymmetry. Measure the offset and alert when it grows.
• Comparing timestamps across machines as if they're exact. Two events 2 ms apart on different servers can be in the wrong order.
• Ignoring steps and leap seconds. A duration computed from wall-clock readings can come out negative. The monotonic clock page explains the fix.

Check yourself

1. How far can a clock with 200 ppm drift be off after one day? About 17 seconds, since 200 ppm is 0.0002 seconds of error per second.
2. Why can NTP be wrong even while it's working? It assumes the request and reply took equal time. If the path is asymmetric, the error is half the difference, and NTP can't detect it.

Key takeaways

• Clocks drift, and NTP only corrects them within the network's limits.
• A timestamp is an estimate with an error bar, not an exact time.
• Clocks can step backward, and leap seconds cause jumps.
• Monitor clock offset on every node. It's cheap and catches bugs that are otherwise hidden.

Further learning

Kleppmann's lecture on physical time covers quartz drift, leap seconds and monotonic clocks with real examples.

How and why the leap second affected Cloudflare DNS shows what a clock step does to real code.

Next, read about monotonic and wall-clock time, which explains which clock to use for each job.