MTBF and MTTR

A service can be unavailable for two reasons. It breaks often, or each break lasts a long time. Two measurements track these. MTBF (mean time between failures) is how often things break. MTTR (mean time to recovery) is how long each outage lasts, from start to full recovery.

The big idea

Think of a bus that breaks down once a month. If each repair takes a day, riders wait a lot. If each repair takes an hour, the same breakdowns cost far less. The breakdown count didn't change. Only the repair time did.

Prevention is hard, because you must predict failures before they happen. Recovery is more within reach. Fast detection, one-click rollback and automated failover all shorten the time a failure lasts.

Walking through it

A service fails about once a month, so MTBF ≈ 720 hours. Each incident takes 4 hours to resolve.

Baseline:             720 / (720 + 4)     = 99.45%
Halve failure rate:  1,440 / (1,440 + 4)  = 99.72%
Cut MTTR to 20 min:   720 / (720 + 0.33)  = 99.95%

Halving the failures gains about a quarter of a nine. Cutting recovery from 4 hours to 20 minutes gains almost a full nine. Recovery wins because the number of outages is the same either way, but each one is much shorter.

Break MTTR into parts. Each part has a different fix.

• Time to detect: how long before anyone knows.
• Time to mitigate: how long before users are safe, through a rollback, a failover or load shedding.
• Time to fix: how long before the root cause is gone.

Mitigation should take minutes even when the real fix takes days. Most incident reviews find the biggest savings in detection and mitigation.

Common mistakes

• Spending all the effort on prevention. Preventing half of all failures is hard. Shortening each outage often gives a bigger return.
• Measuring MTTR as one number. Detection, mitigation and repair have different owners and different fixes. Measure each one.
• Assuming a fix ends the outage. Users are still affected until mitigation or recovery is done.

Check yourself

1. A service fails monthly and takes 4 hours to recover. What happens if recovery drops to 20 minutes? Availability rises from about 99.45% to about 99.95%.
2. Which change adds more nines: halving the failure rate or cutting MTTR from 4 hours to 20 minutes? Cutting MTTR. Halving failures adds about a quarter of a nine.

Key takeaways

• Availability = MTBF / (MTBF + MTTR).
• You can't stop every failure, but you can make each one short.
• Break MTTR into detect, mitigate and fix, and invest most in detection and mitigation.
• Rollbacks and failover are often the cheapest path to another nine.

Further learning

The Google SRE book chapter on embracing risk explains how teams balance reliability against change.

Next, read about percentiles and tail latency, which covers the other half of the picture: speed.