Observability and reliability
Designing on-call
Sustainable rotations, executable runbooks, toil budgets and a paging policy that keeps humans healthy while production stays covered.
On-call is the commitment that when production needs a human, a specific person answers, with a laptop, the context, and the authority to mitigate. Done well, it is a finite, paid duty that keeps services alive and teaches engineers how their systems behave. Done badly, it is unpaid night work, a pager that cries wolf, and a fast path to burnout.
Designing on-call is systems design applied to people. You size the rotation, set a paging policy, attach every page to a runbook, budget the toil, and check the health of the people the same way you check latency. The goal is coverage without heroics. Heroics do not scale. Processes and automation do.
Walking through the topic
The first two pages set up the rotation and the paging rules. The hub's own content then covers the cross-cutting ideas. The last four pages cover runbooks, toil, health, and who owns the pager.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Rotation design | How many people, and how are handoffs and coverage defined? |
| 2 | Paging policy | Which signals wake a human, and how fast must they be answered? |
| 3 | Executable runbooks | What does a tired person do first, for each alert? |
| 4 | Toil and its budget | How much of the time goes to repetitive work? |
| 5 | On-call health and fatigue | Are people sleeping, and how often are they paged? |
| 6 | Ownership models for on-call | Who should carry the pager for each system? |
This topic assumes you already page on symptoms that matter, which is covered in metrics and alerting. It also assumes you know how to run an incident once awake, which is covered in incident response. Here the subject is the rotation itself.
Common anti-patterns
- Hero rotation. The same two people take every hard week, and the knowledge stays with them.
- Silent secondary. The backup is never paged, and fails when the primary is unavailable.
- Page-as-ticket. Non-urgent work pages people at night, and they learn to ignore the pager.
- No link to error budgets. Product ships risk, while on-call takes the pages with no authority to slow releases.
- Runbook theater. Thorough docs that nobody opens, because the alerts do not link to them.
Key takeaways
- Size the rotation for sustainability. Use written handoffs, and a shadow week for new people.
- Keep pages rare, tied to user symptoms, and linked to a runbook.
- Runbooks should be checklists, drilled regularly, and kept current.
- Cap toil at about half of the team's time, and measure it.
- Review on-call health regularly, and treat overload as a reliability defect.
- Choose developer-owned or SRE ownership on purpose, and avoid the failed middle.
Interview angle
Be ready to explain how you would cut pages in half without ignoring real outages. The answer combines SLO burn-rate alerts, deleting noisy causes, runbooks, and toil automation. Have a story ready about a bad page you deleted and a good page you kept.
Further learning
Google's chapter on being on-call sets the baseline for shift design and escalation. Its companion chapter on toil defines the term. The SRE Workbook's chapter adds practical policy detail.
Being On-Call (Google SRE book)Proven strategies for on-call engineers to ensure reliable services and maintain sustainable workloads in IT operations.sre.google Eliminating Toil (Google SRE book)Learn what toil is in SRE, how it affects operational work, and why minimizing it is crucial for efficiency and morale.sre.google On-Call (Google SRE Workbook)How Google Site Reliability Engineering teams structure on-call rotations, site operations, structure approaches to handle incidents and production issues.sre.googleSpotify's HumanOps talk is a concrete story of evolving rotations, ownership and follow-the-sun.
Related
Incident response
Run an incident with clear roles, severity and communication. Mitigate first, debug second, then hand off to the postmortem. Start here for the map of incident pages.
Owning your code in production
On-call, alerts, runbooks, and the loop that makes you trustworthy — and better at building systems that survive users.
Metrics, dashboards and alerting
Golden signals, RED and USE, Prometheus histograms and alerts that page you only when users are hurting.