Observability and reliability
RPO and RTO
The two numbers that shape every recovery plan. RPO is how much data you can lose, RTO is how long you can be down, and each one maps to a recovery tier with a price.
Every recovery plan starts with two numbers. RPO is how much data you can afford to lose, measured as time. RTO is how long you can afford to be down. Both are business choices written as engineering targets, and together they decide which recovery tier you pay for.
Concept
RPO (recovery point objective): an RPO of 5 minutes means you may lose up to about 5 minutes of writes. RTO (recovery time objective): an RTO of 1 hour means service must be back within about an hour. Both are targets. They are not measured values until a drill measures them.
The big idea
Think of a fire escape plan. How much unsaved work can you lose, and how long can you be out of the building? Those two answers decide whether you need smoke alarms, a second staircase, or an off-site copy of the files.
The plan analogy breaks in one place. A plan is a belief until a drill tests it. For DR, the measured RPO and RTO matter more than the numbers you wrote down.
Matching systems to targets
| System | Typical RPO | Typical RTO | What it implies |
|---|---|---|---|
| Marketing site | Hours | Hours | A nightly backup and a rebuild may be enough |
| SaaS application database | Minutes | Minutes to 1 hour | Continuous backup, point-in-time recovery, a practiced failover |
| Payments ledger | Near zero | Minutes | Synchronous replication, multi-site, careful failover |
| Analytics warehouse | Hours | Day | Batch replay from object storage |
The recovery tiers
| Tier | RPO and RTO, roughly | Cost shape |
|---|---|---|
| Backup, then restore into empty infrastructure | Hours or more, hours or more | Storage and engineer time |
| Automated PITR with a warm replica | Minutes, tens of minutes | Replica compute and storage |
| Multi-AZ automatic failover | Seconds to minutes, minutes | Twice the AZ footprint |
| Multi-region active-passive | Seconds to minutes, minutes after promotion | Cross-region data and idle capacity |
| Active-active multi-region | Near zero for many failures | Highest engineering and infra cost, plus conflict design |
Trade-off
Active-passive vs active-active. Active-passive is simpler: one writer region, and a promotion you have rehearsed. Active-active shortens RTO for a regional loss, but it needs conflict resolution and traffic on both sides all the time. For most product databases, start with multi-AZ, then add cross-region active-passive if the RTO requires it.
Walking through the money
Suppose one hour of checkout downtime costs about $50k at peak. A warm cross-region standby costs about $8k a month.
$8,000 / $50,000 per hour = 0.16 hours, about 10 minutes of downtimeThe standby pays for itself if it prevents about 10 minutes of peak-hour outage a month. That only holds if failover actually meets the RTO in drills. Without drills, the standby is a subscription to false comfort.
Example
A B2B SaaS company agreed with Customer Success on RPO 15 minutes and RTO 1 hour for its main database. That ruled out a nightly dump and led to point-in-time recovery plus a cross-region read replica, with a quarterly promotion drill. Analytics stayed on nightly snapshots with a 24-hour RPO. One company, two tiers, each chosen on purpose.
Common mistakes
- One RPO and RTO for the whole company. One tier then fits half the systems badly.
- Estimating RTO in your head. Measure it in a drill.
- Buying active-active before you need it. Its cost comes with conflict design.
- Going multi-region before multi-AZ. Most failures are zonal.
Check yourself
- A system has an RPO of 15 minutes. How much recent writes can you lose? Roughly the last 15 minutes of writes.
- A drill took four hours. The target is one hour. What is your RTO? Measured RTO is four hours. The one-hour target is a wish until a drill meets it.
Key takeaways
- RPO bounds data loss. RTO bounds downtime.
- Set both for each service, then fund the tier that meets them.
- Measured numbers from drills beat the targets you wrote down.
Further learning
Next, learn what makes a backup real, and how to restore from one, in backups that restore.
Related
Disaster recovery and chaos engineering
RPO and RTO, backups that restore, game days and region drills, and deliberate failure injection, so confidence comes from measured recovery. Start here for the map of disaster recovery pages.
Multi-region architecture
Active-passive, active-active and cell-based designs for latency, disaster tolerance and data residency, with the recovery, consistency and cost trade-offs each one carries.
SLIs, SLOs and error budgets
Define reliability in numbers users care about, then use error budgets and burn-rate alerts to balance shipping speed against stability.