Time, clocks and ordering

On one machine, "which happened first?" is easy. You compare timestamps, or read the lines of a log in order. Across machines, that breaks. Each server has its own clock. The clocks disagree by milliseconds or sometimes seconds, and they can jump without warning. Two updates made 3 ms apart on two servers can carry timestamps in the wrong order. A database that keeps "the latest" write then throws away the one that actually came last.

Ordering sits under nearly every distributed problem. Replicas must apply writes in the same order. A reply must not appear before its question. A lock must not go to two holders. You can order events without trusting clocks, or you can bound how wrong the clocks can be. The pages below go in that order.

Concepts on this topic

Step	Concept	What it answers
1	Clock drift and NTP	How wrong can a clock get, and how is it corrected?
2	Monotonic vs wall-clock time	Which clock measures durations safely?
3	Last-write-wins and clock skew	How does skew silently lose writes?
4	Lamport clocks	How can counters respect cause and effect?
5	Vector clocks	How do you detect concurrent events?
6	Hybrid logical clocks and TrueTime	How do you keep time close to real time, and bounded?

Which approach fits

Approach	Clock needs	Ordering it gives	Used by
Wall clock with LWW	NTP, hope	None reliable. Loses writes under skew	Cassandra, many caches
Lamport clock	None	Total order consistent with causality	Raft terms, many protocols
Vector clock	None	Exact causality, detects concurrency	Dynamo, Riak, CRDT libraries
HLC	Bounded skew	Causality plus near-real-time	CockroachDB, YugabyteDB, MongoDB
TrueTime	GPS and atomic clocks, explicit ε	External consistency	Spanner

Total order and consensus

The tools above order events after the fact. Many systems need more. Every node must deliver the same messages in the same order, and once message 17 is delivered, nothing can be inserted before it. That is total order broadcast. It needs reliable delivery to all non-faulty nodes, and the same order everywhere.

Lamport timestamps look like they solve this, but they don't. A node that receives timestamp 5 can't know whether another node is about to send timestamp 4. It would have to hear from every node first, and one dead node would block everyone.

Total order broadcast is equivalent to consensus. A replicated log is one such broadcast. Problems such as electing one leader, granting one lock holder and assigning gap-free sequence numbers all need consensus. That's why systems like ZooKeeper and etcd are the standard tools. The consensus page shows how it's done.

In practice

Never time a duration with wall-clock time. Search for Date.now() - or time.time() - used for timeouts, and switch to monotonic clocks.
Don't order events across machines by their timestamps unless the system was designed for it. Hybrid logical clocks with a maximum offset, or TrueTime, are the designed options.
Be suspicious of last-write-wins. Find out which writes can conflict, and whether losing one is acceptable.
Monitor clock offset. Alert when a node's NTP offset grows past a threshold.
Use sequence numbers inside one partition. A Kafka offset or a database sequence is a total order with no clocks involved.
In interviews, expect "whose clock?" When you say "we'll order by timestamp," have the answer ready: a per-conversation sequence assigned by the partition leader, as in the chat system case study.

Key takeaways

Clocks drift, and NTP corrects them only within the network's limits.
Use monotonic clocks for durations, and wall clocks only for display.
Last-write-wins with physical timestamps silently discards writes inside the skew window.
Lamport clocks give an order that respects causality. Vector clocks also detect concurrency.
Hybrid logical clocks and TrueTime bound the uncertainty, and TrueTime waits it out.
Total order broadcast, a replicated log and consensus are equivalent problems.