Observability and reliability
On-call health and fatigue
On-call costs sleep and focus. Healthy programs pay for it, cap consecutive nights, allow handoffs when sick, and review the load every month. Overload is a reliability defect.
On-call costs sleep, focus and family time. Companies that pretend it does not lose their best operators first. A healthy program treats the people on the rotation as part of the reliability system. If they burn out, the system loses its humans.
The big idea
Think of a car's engine temperature gauge. If the gauge climbs, you do not just turn off the light. You look for the cause. Fatigue in on-call is the same kind of signal. If the rotation is exhausting people, the fix is in the engineering and the scope of the service, not in pep talks.
Walking through it
Healthy programs usually combine some of these:
- Extra pay, time off in lieu, or both, for nights and weekends.
- A hard cap on consecutive on-call nights.
- The right to hand off when sick or deeply tired, without it counting against anyone.
- An on-call health review every month or quarter. It checks page volume, after-hours load, open toil, whether runbooks helped, and whether anyone dreads their week.
The review should ask the question directly. Are people sleeping? Are they swapping shifts in a hurry? Are they muting alerts instead of fixing causes? These are the early signs of burnout, and each one is a production risk.
If the review shows chronic overload, the fix is structural: reduce pages, improve reliability, split the service, hire, or drop 24-hour coverage for components that do not need it. Telling people to be more resilient does not fix a rotation.
A review should also check whether the same people always take the hard weeks. If they do, the knowledge stays with two people, and the rotation is fragile. Shadowing and rotating the hard shifts spread that load.
Common mistakes
- Treating silence as health. Alerts that nobody answers are a warning, not calm.
- No way to hand off when sick. People then work through illness and make mistakes.
- Reviewing the numbers without acting on them. A review that changes nothing teaches people to stop reporting.
Check yourself
- Two people keep swapping shifts at short notice. What is the signal, and what do you check? It is a burnout sign. Check page volume, after-hours load and whether the load falls on the same people.
- Why is overload a reliability defect, not a personal problem? Exhausted operators make slower decisions and miss signals. The system's ability to respond drops.
Key takeaways
- On-call has a cost in sleep and focus, and the program should pay for it.
- Cap consecutive nights, and allow handoffs when someone is sick.
- Review health monthly or quarterly, and fix the cause with scope or engineering, not with pep talks.
Further learning
The SRE Workbook's on-call chapter covers actionable alerts, playbooks and psychological safety. Spotify's HumanOps talk tells a concrete story about evolving rotations.
On-Call (Google SRE Workbook)How Google Site Reliability Engineering teams structure on-call rotations, site operations, structure approaches to handle incidents and production issues.sre.googleRelated
Designing on-call
Sustainable rotations, executable runbooks, toil budgets and a paging policy that keeps humans healthy while production stays covered.
Toil and its budget
Toil is manual, repetitive work that scales with the service and leaves nothing behind. Google's guidance is to keep it under about half of an SRE's time, and to automate the top offenders first.
Owning your code in production
On-call, alerts, runbooks, and the loop that makes you trustworthy — and better at building systems that survive users.