Observability and reliability
SLIs, SLOs and error budgets
Define reliability in numbers users care about, then use error budgets and burn-rate alerts to balance shipping speed against stability.
"The API should be reliable" is not a goal you can staff, prioritize or page on. SLIs, SLOs and error budgets turn reliability into a measured product decision: how much unreliability will we accept this month in exchange for shipping faster?
Concepts in reading order
| Step | Concept | What it answers |
|---|---|---|
| 1 | SLIs, SLOs and SLAs | What do we measure, and what is the target? |
| 2 | Error budgets | How much failure can we spend, and what happens when it runs out? |
| 3 | Burn-rate alerts | How do we get paged early without getting paged for noise? |
| 4 | Latency SLOs | How do we measure "fast enough" with histograms? |
| 5 | Rolling out SLOs | How do we start on a team that has none? |
Read step 1 first. The rest build on the ratio it defines.
Why these three ideas work together
The SLI measures. The SLO sets the target. The error budget is the room the target leaves for failure. Burn-rate alerts tell you how fast that room is being used. Each one depends on the one before it, which is why this topic is ordered that way.
Without a target, every outage becomes an argument about whether it mattered. With one, a team can say whether a launch is safe to ship, and product can trade reliability for speed on purpose.
Use it with incident response. Responders quote the burn rate in the channel, so everyone hears urgency in budget terms.
Common mistakes
- An SLO with no policy. Without consequences, the target is a dashboard, not a decision.
- Too many SLIs. Pick one availability and one latency measure for the main journey.
- Measuring what is easy to scrape. Measure what users notice.
- A target stricter than the system can reach. The team learns to ignore it.
In practice
Use SLOs to decide whether to ship on a Friday, to prioritize performance work when the budget is low, and to design alerts that track user pain. In design reviews, be able to compute the 43.2-minute budget for 99.9% over 30 days, explain request-based versus time-based counting, and derive why 14.4x shows up in fast-burn alerts.
Key takeaways
- SLI is the measurement, SLO is the target, SLA is the contract.
- Pick SLIs from user journeys, and keep the set small.
- 99.9% over 30 days allows 43.2 minutes of full outage, or 0.1% of requests.
- Error budgets need a written policy to change behavior.
- Alert on multi-window burn rates, not only on monthly breaches.
Further learning
Start with SLIs, SLOs and SLAs. The Google SRE book's chapter on SLOs is the reference for the definitions.
Service Level Objectives (Google SRE book)SRE SLO book to understand service level objective meaning and the various service level terminilogy including sla slo sli to improve service reliability.sre.googleRelated
Availability, latency and the tail
Nines, percentiles and the math of composing services. Why p99 matters more than the average, how fan-out amplifies the tail, and how to design for both uptime and speed.
Metrics, dashboards and alerting
Golden signals, RED and USE, Prometheus histograms and alerts that page you only when users are hurting.
Incident response
Run an incident with clear roles, severity and communication. Mitigate first, debug second, then hand off to the postmortem. Start here for the map of incident pages.