Distributed systems
Partial failure and the unknown outcome
A remote call can fail in several ways that look identical from the caller's side. Learn why silence tells you so little, and why a retry can charge a customer twice.
On one machine, a function call either returns or throws. Across a network, a request can leave your service and nothing comes back. Did the other side crash before doing the work, after doing it, or is it just slow? The defining feature of distributed systems is partial failure. Some parts are broken while others work, and the working parts can't reliably tell which is which.
Concept
A timeout means your code stopped waiting. It does not mean the work failed. The other side may have done all of it.
The big idea
Imagine a payment service calling a card processor. It sends "charge $40" and waits. After 5 seconds of silence, five things could be true:
From your side, all five look the same: silence. You cannot tell a slow node from a dead one, or "the work didn't happen" from "the work happened and the reply was lost."
Walking through the cases
| What really happened | Card charged? | If you retry |
|---|---|---|
| 1. Request lost on the way out | No | Charged once. Good. |
| 2. Processor crashed before handling it | No | Charged once. Good. |
| 3. Processor paused, will handle it later | Yes, soon | Charged twice |
| 4. Processor charged, reply lost | Yes | Charged twice |
| 5. Processor charged, reply stuck in a queue | Yes | Charged twice |
If you retry, you risk a double charge in cases 3 to 5. If you don't retry, you risk no charge at all in cases 1 and 2. Neither choice is safe on its own.
The only safe retry is one where a repeat has no extra effect. That is what idempotency provides. Stripe's Idempotency-Key header exists for case 4.
Note
The two generals problem formalizes this. Two armies must attack at the same time and can only talk by messengers who might be captured. General A sends "attack at dawn." To know it arrived, B must acknowledge. To know the acknowledgement arrived, A must acknowledge that, and so on forever. No finite protocol over a lossy channel gives both sides certainty. Real systems accept probabilistic certainty and make operations safe to repeat.
Common mistakes
- Treating a timeout as a failure. Record it as "unknown", then decide what to do.
- Retrying a non-idempotent call. A retry can charge a card twice.
- Assuming you can check the result later. You can only do that if the other side lets you look up the operation by a key you sent.
Check yourself
1. A call times out. Did it fail? Unknown. It may have failed, or it may have succeeded and lost its reply. Treat it as unknown.
2. Why can't a caller tell a slow node from a dead one? Both produce silence, and nothing in the lack of a reply says which one you have.
Key takeaways
- A timeout is an unknown outcome, not a failure.
- A missing reply can mean the request was lost, the server crashed, the server paused, or the reply was lost.
- A retry is safe only when the operation is idempotent.
Further learning
Next, see how the network creates these cases in network faults and gray failures.
Two generals lecture (Kleppmann) explains why no protocol can give certainty over a lossy network, and how real systems live with it.
Related
Network faults and gray failures
Packets are lost, delayed, duplicated and reordered, and partitions can be one-way or partial. Learn the faults to design for, including gray failures that pass health checks.
Idempotency and retries
Making operations safe to retry, with idempotency keys, deduplicating consumers and jittered backoff, so at-least-once delivery never means a double charge.
What goes wrong in distributed systems
Partial failure, unreliable networks, paused processes and timeouts, and the fencing tokens and failure detectors that let you build correct systems on top of them.