Error budgets

An error budget is 1 − SLO. A 99.9% SLO allows 0.1% of events to be bad within its window. The budget turns "how reliable should we be?" into "how much failure can we spend this month?" Spending it is a product decision: ship faster with more risk, or slow down.

Walking through 99.9% over 30 days

There are two ways to count the budget.

Time-based
  Window          30 days = 30 x 24 x 60 = 43,200 minutes
  Budget          0.1% x 43,200 = 43.2 minutes of full outage

Request-based
  Requests        100,000,000 in 30 days
  Budget          0.1% x 100,000,000 = 100,000 failed requests
  Spike           50,000 failed requests = half the budget, in one incident

Request-based counting usually matches user pain better for APIs. An eight-minute incident that fails 50,000 requests burns half the budget. A quiet hour with no traffic barely moves the number.

Request-based or window-based?

Also choose a window type. A rolling window looks at the last 30 days, every day, so there is no end-of-month cliff. A calendar window matches business reports. Pick one and document it.

Low traffic breaks the math

A new internal API with 200 requests a day cannot honestly claim 99.9% over a week. One failed request in a day's 200 is 0.5%, which is five times the 0.1% allowed. Options:

• Use a longer window until traffic grows.
• Measure success in multi-minute buckets, so sparse traffic does not swing the number.
• Start with a looser target, such as 99% or 99.5%.

Synthetic probes help here. A probe every minute from several regions catches a total outage even at low traffic. It can miss failures that only affect some payloads, so teams often run both.

Writing the policy

A policy says what happens as the budget falls. Agree on it with product before the first bad week.

Service: checkout API
SLO: 99.9% availability (request-based), rolling 30 days
When remaining budget < 25%: reliability work outranks features
When budget exhausted: no feature deploys until a documented exception

A typical ladder looks like this:

• More than 50% left: ship normally.
• Under 25% left: tighten review, delay risky migrations, prioritize reliability fixes.
• Exhausted: freeze feature deploys for that service. Ship only fixes and security patches, until the budget recovers or the window rolls forward.
• Exceptions: an executive can override for a critical launch. Log it, and pay the budget back later.

Common mistakes

• Having an SLO and no policy. The dashboard changes nothing.
• Setting a target so strict the budget is always empty. The team learns to ignore it.
• Judging a low-traffic service by its percentage. One failure can look like a disaster.

Check yourself

1. What is the budget for 99.9% over 30 days, in minutes of full outage? Answer: 43.2 minutes.
2. A service gets 200 requests a day. Why is 99.9% over a week hard to hold honestly? Answer: one failed request in a day's 200 is 0.5%, which is five times the 0.1% allowed.

Key takeaways

• Error budget equals 1 minus the SLO.
• For 99.9% over 30 days: 43.2 minutes of full outage, or 100,000 failed requests per 100 million.
• Write the policy before the first bad week.
• Low traffic needs a longer window, multi-minute buckets, or a looser target.
• Next, burn-rate alerts show how fast the budget is going.

Further learning

Ben Treynor's Google Cloud Next talk explains error budgets as the shared control loop between shipping speed and reliability.