Observability and reliability
Disaster recovery and chaos engineering
RPO and RTO, backups that restore, game days and region drills, and deliberate failure injection, so confidence comes from measured recovery. Start here for the map of disaster recovery pages.
Disks fail. Regions go down. Someone runs the wrong command. A latent bug waits for a rare race. Disaster recovery (DR) is the set of designs and rehearsals that limit how much data you lose, and how long you are down, when the unlikely happens. Chaos engineering is how you prove the recovery paths work on an ordinary afternoon, not for the first time at 03:00.
DR without drills is documentation. A backup you have never restored is a hope. When a disaster is unfolding right now, switch to incident response. The design patterns are in the multi-region and resilience pages.
Read in order
| Step | Concept | What it answers |
|---|---|---|
| 1 | RPO and RTO | How much data and time can each system afford to lose? |
| 2 | Backups that restore | How do you know a backup is real? |
| 3 | Game days and region drills | How do you test recovery without a real disaster? |
| 4 | Chaos engineering | How do you find weak spots before they find you? |
Common mistakes across the topic
- Treating replicas as backups. A replica copies the bad write, and the delete, too.
- Assuming backups work. Only a timed restore proves it.
- Keeping the only copy in the same account. Ransomware or a bad destroy can wipe it.
- Setting one company-wide RTO. Each service needs its own tier.
- Buying multi-region before multi-AZ. Get the basics right first.
Build a DR card for each critical service
A one-page DR card answers "are we ready?" It lists:
- The RPO and RTO for the service.
- Where backups live, and who owns the restore.
- The date of the last successful restore, and how long it took.
- The failover command or runbook link.
Keep it next to the on-call cheat sheet. A vague "we have replicas" is not an answer.
Order your investments by risk
- Automated, verified backups with an off-account copy.
- Multi-AZ for stateful primaries.
- Practiced promotion and point-in-time recovery.
- Chaos experiments on common node failures.
- Only then, multi-region active paths.
Skipping to step 5 while step 1 is red is a classic reliability failure. It looks sophisticated and fails on the first restore.
Key takeaways
- RPO is how much data you can lose. RTO is how long you can be down.
- A backup counts only after an encrypted, off-account copy restores within the RTO.
- Game days and region drills turn runbooks into measured times.
- Chaos engineering follows a loop: hypothesize, inject, measure, fix, automate, with blast-radius limits.
Further learning
The principles document is short, and it is the shared vocabulary for chaos experiments.
Principles of Chaos EngineeringPRINCIPLES OF CHAOS ENGINEERING - Principles of chaos engineeringprinciplesofchaos.orgGitLab's database outage postmortem is the canonical cautionary tale about backup assumptions.
Postmortem of database outage of January 31 (GitLab)Postmortem on the database outage of January 31 2017 with the lessons we learned.GitLab
Netflix's early Simian Army post explains why they kill instances on purpose during business hours. The 2017 paper expands the principles with Netflix's experimental framing.
Chaos Engineering (arXiv)Modern software-based services are implemented as distributed systems with complex behavior and failure modes. Many large tech organizations are using experimentation to verify the reliability of such systems. We use …arXiv.org
An AWS re:Invent talk on how Netflix's practice moved from Chaos Monkey to safer, experiment-driven fault injection:
Related
Multi-region architecture
Active-passive, active-active and cell-based designs for latency, disaster tolerance and data residency, with the recovery, consistency and cost trade-offs each one carries.
Resilience patterns
The toolkit for keeping one failure from becoming an outage, from timeouts and retries to circuit breakers, bulkheads, load shedding and cells. Start here for the resilience map.
Incident response
Run an incident with clear roles, severity and communication. Mitigate first, debug second, then hand off to the postmortem. Start here for the map of incident pages.