Observability and reliability
SLIs, SLOs and SLAs
Measure reliability as a ratio of good events to total events. Learn the three terms, and how to pick measures from what users notice.
Reliability is hard to staff or page on until it has a number. You need a measure of what works, and a target for it. Those two ideas have names: the measure is an SLI, and the target is an SLO.
Concept
SLI (service level indicator) is a measurement of one part of service quality. SLO (service level objective) is a target for that measurement over a window, such as "99.9% of requests succeed over 30 days." SLA (service level agreement) is a contract with consequences, such as credits or penalties. Internal teams usually want SLOs without customer-facing SLAs.
The ratio
Write every SLI as a ratio:
SLI = good events / total eligible eventsSome traffic should not count. Exclude load balancer health checks, because they succeed and make the ratio look better than users see. Exclude synthetic probes you already track separately, unless the SLO is those probes. Exclude internal admin routes if product agrees.
Choosing SLIs from user journeys
Start from what the user notices, not from what is easy to scrape.
| User complaint | Candidate SLI | Good event (example) |
|---|---|---|
| "The site is down" | Availability | Responses that are not 5xx and did not time out |
| "It's slow" | Latency | Requests that finish under 300 ms |
| "My data is stale" | Freshness | Reads served from data newer than N minutes |
| "The number is wrong" | Correctness | Results that match a shadow check or invariant |
For a request-driven API, availability and latency are the usual pair. For checkout, prefer a journey measure such as "payment confirmed within 5 seconds of submit." A payments service can be green while inventory is stuck, and the user still fails.
Trade-off
Too many SLIs means nobody looks at them. One availability and one latency SLO for the main journey beat eight microservice SLOs that never change behavior.
Not every SLI is a request
Batch and data pipelines need different measures.
- Freshness. "95% of partition hours are searchable within 15 minutes of event time." A partition is good if its newest event is within the threshold of now.
- Completeness. "99.9% of expected input files are processed each day." Compare expected counts with actual counts.
- Correctness. Use shadow reads, checksums, or reconciliation jobs. Alert on the mismatch rate.
Common mistakes
- Measuring what is easy to scrape. Measure what the user feels.
- Counting health checks as traffic. The good ratio gets inflated.
- Building eight SLOs for one journey. Keep the set small.
Check yourself
- An API returns a 5xx on 1 request in every 1,000 for a month. What is its availability SLI? Answer: 999 / 1,000 = 99.9%.
- Why is a checkout journey SLI better than each service's success rate? Answer: it measures what the user experiences, including the parts of the journey that sit between services.
Key takeaways
- SLI is the measurement, SLO is the target, SLA is the contract.
- Pick SLIs from user journeys, and write them as good over total.
- Pipelines need freshness, completeness and correctness measures.
- Next, error budgets turns the target into a budget you can spend.
Further learning
The Google SRE book's chapter on SLOs gives the core definitions and intuition.
Service Level Objectives (Google SRE book)SRE SLO book to understand service level objective meaning and the various service level terminilogy including sla slo sli to improve service reliability.sre.googleRelated
SLIs, SLOs and error budgets
Define reliability in numbers users care about, then use error budgets and burn-rate alerts to balance shipping speed against stability.
Availability, latency and the tail
Nines, percentiles and the math of composing services. Why p99 matters more than the average, how fan-out amplifies the tail, and how to design for both uptime and speed.
Error budgets
The error budget is the failure your SLO allows. Turn it into numbers, and agree on a policy for what happens when it runs low.