Distributed systems
What goes wrong in distributed systems
Partial failure, unreliable networks, paused processes and timeouts, and the fencing tokens and failure detectors that let you build correct systems on top of them.
On a single machine, things mostly work or mostly don't. A function call returns or throws. If the process crashes, everything in it crashes together, and you restart it. Distributed systems break that model. A request leaves your service and nothing comes back. Did the other side crash before doing the work, after doing it, or is it just slow? Is the network down, or only the path from you to it? You often can't tell, and your code has to be correct anyway.
This topic is a catalog of what goes wrong: networks, processes and clocks, and the strange failures that appear when they interact. It also covers the tools that keep a system correct despite all of it. Later topics, including clocks, consensus and distributed transactions, respond to the problems here.
The eight fallacies
In 1994, Peter Deutsch at Sun Microsystems listed seven false assumptions that engineers new to distributed systems make. James Gosling added an eighth a few years later. Every one still causes outages.
| Fallacy | What actually happens |
|---|---|
| 1. The network is reliable | Packets drop, links flap, switches reboot, cables get cut |
| 2. Latency is zero | Same rack about 0.1 ms, cross-AZ about 1 ms, cross-continent 70 to 150 ms |
| 3. Bandwidth is infinite | A 10 Gbps link moves about 1.25 GB/s; replicating a 2 TB node takes about 30 minutes at best |
| 4. The network is secure | Traffic inside the VPC can be sniffed or spoofed, which is why service meshes use mTLS |
| 5. Topology doesn't change | Pods reschedule, IPs change, DNS caches go stale |
| 6. There is one administrator | Cloud providers, network teams and SaaS vendors all change things without telling you |
| 7. Transport cost is zero | Serialization burns CPU, and cross-AZ and egress traffic is billed per GB |
| 8. The network is homogeneous | Clients on 3G, proxies that strip headers, MTU mismatches |
The first two drive everything else. If the network is unreliable and latency is unbounded, a missing reply carries almost no information.
Reading order
| Step | Concept | What it answers |
|---|---|---|
| 1 | Partial failure and the unknown outcome | Why does silence not tell you what happened? |
| 2 | Network faults and gray failures | What exactly does the network do wrong? |
| 3 | System models | What should a correct algorithm assume? |
| 4 | Process pauses and clock jumps | Why can a process act on beliefs that are no longer true? |
| 5 | Timeouts and failure detectors | How do you decide a node is gone? |
| 6 | Fencing tokens | How do you make a lock safe when its holder pauses? |
| 7 | Metastable failures and retry storms | Why does an outage keep going after its trigger ends? |
The main rule
Safety must never depend on timing. Timeouts, clocks and pauses can slow a system down or delay progress. They must not cause it to corrupt data. Every technique in the table above exists to keep that rule true.
Common mistakes
- Reading silence as death. A missing reply has at least four possible causes. Check which one you have before you act.
- Retrying without idempotency. Retries are only safe when a repeat has the same effect as the first attempt.
- Trusting a lease without a fence. An expired lease can still look valid to a paused holder.
- Picking timeouts by guess. Measure the dependency's latency first.
Key takeaways
- Distributed systems fail partially. A missing reply can mean a lost request, a crashed server, a paused server or a lost reply.
- Networks drop, delay, duplicate and reorder messages. Gray failures pass health checks while failing real traffic.
- Processes pause without knowing it, and clocks jump. Design for both.
- Correctness must hold under any timing. Timing only affects how fast the system makes progress.
Further learning
Start with partial failure and the unknown outcome. The reading lists at the end of each page cover the sources behind this topic.
Related
Resilience patterns
The toolkit for keeping one failure from becoming an outage, from timeouts and retries to circuit breakers, bulkheads, load shedding and cells. Start here for the resilience map.
Consensus and Raft
How a cluster agrees on one value or one log despite crashes and partitions, how Raft works, and where etcd, ZooKeeper and consensus-backed databases fit.
Time, clocks and ordering
Why wall clocks lie across machines, how skew silently loses writes, and how Lamport clocks, vector clocks, hybrid logical clocks and TrueTime order events you can't time directly.