Observability and reliability
Rolling out SLOs on a team with none
Start with one measured journey, let the data set the first target, then add alerts and a policy. A strict target that the team ignores is worse than none.
Starting with a 99.99% target and a freeze policy is the fastest way to make SLOs unpopular. Roll them out in order instead. Measure first, set the target from history, then attach alerts and consequences.
Concept
Before you start. Read error budgets and burn-rate alerts. This page assumes you know what the policy and the alerts are for.
Six steps
- Instrument one user journey with a clear good-over-total measure. The target can be provisional.
- Measure for a few weeks. Draw an aspirational target as a line on the graph, with no consequences yet.
- Set the SLO from history. Pick a target slightly better than the recent bad weeks, but still achievable. For internal APIs, 99.5% or 99.9% is common.
- Add burn-rate alerts and a dashboard that shows the budget remaining.
- Write the error budget policy with product. Walk through it once, when the budget first gets close to empty.
- Iterate. An SLO is a product decision, not a permanent constant. Review it after major architecture changes.
A made-up example
A checkout team measures availability for three weeks. The weekly good ratios are 99.3%, 99.6% and 99.4%. The worst week, 99.3%, is the reference point. The team sets its first SLO at 99.5%. It is better than the bad week, and most weeks already meet it. The team then adds burn-rate alerts before it attaches any freeze rule.
The failure mode to avoid
An SLO set so strict that the budget is always empty teaches the team to ignore it. An ignored SLO is worse than none, because it trains everyone to distrust the reliability process. If the budget is always empty, loosen the target or fix the system. Do not keep both.
Common mistakes
- Setting the target from hope. Measure first.
- Rolling out many journeys at once. Start with one, and add more later.
- Treating the target as permanent. Review it when the architecture changes.
Check yourself
- Why measure for a few weeks before attaching consequences? Answer: you see the real error pattern, so the target reflects what is achievable instead of a guess.
- The budget is always empty. What is the likely problem? Answer: the target is stricter than the system can reach, so the team has learned to ignore it.
Key takeaways
- Start with one user journey.
- Let the data set the first target.
- Add alerts first, then the policy with consequences.
- Review the SLO when the system changes.
Further learning
The SRE Workbook's chapter on implementing SLOs covers rollout patterns in more detail.
Implementing SLOs (Google SRE Workbook)Learn to use Service Level Objectives (SLOs) for continuous improvement in reliability and customer satisfaction, and discover the importance of SLOs.sre.googleRelated
SLIs, SLOs and error budgets
Define reliability in numbers users care about, then use error budgets and burn-rate alerts to balance shipping speed against stability.
Error budgets
The error budget is the failure your SLO allows. Turn it into numbers, and agree on a policy for what happens when it runs low.
Burn-rate alerts
A burn rate compares your error rate with the pace that would use the budget exactly over the window. Page on fast burns over two windows, to catch problems early without noise.