Runbooks and toil

A runbook is a checklist for a human under stress: what the alert means, what to check, how to mitigate, who to call, link to dashboards. Write them as you wish they had existed at 2 a.m.

Google SRE culture encodes a related idea: operational work should not consume the whole job. Teams aim to keep enough engineering time to automate away toil — repetitive operational work that does not add lasting value. As an IC, your version is: when something wakes you, leave the system better before you drop the pager.

A useful personal SLO: after any page that took more than fifteen minutes, file at least one improvement (runbook line, alert tune, retry, dashboard link) within a week. Small, boring improvements compound into a quiet service. Quiet services are how you earn larger scope.

Walking through a minimum viable runbook

## Alert: export_failure_rate_high
Symptom: >2% export jobs failed for 10 minutes
Dashboard: ...
Mitigate:
1. Check last deploy; rollback flag admin_csv_export if correlated
2. Check object-store status page
3. If backlog > 10k, scale worker pool to N (command ...)
Escalate: #fraud-oncall if errors mention FraudSvc
Follow-up: file ticket, link incident

After a painful page, budget time in the next few days to:

• Fix the bug or add a guardrail.
• Delete or tune the noisy alert.
• Fill the runbook gap.
• Add a test or chaos check if it will recur.

If the same alert pages every week with the same mitigation, automating or eliminating it is engineering work, not "extra."

A quiet shift is not wasted if you use it. Read the dashboards like a map: normal diurnal shape for QPS and errors; which dependencies appear in every trace; which alerts are noisy; which are silent but should exist; where the runbook lies. Shadow a senior on-call once before your first solo shift. Ask them to narrate why this alert matters, what they check first, when they escalate.

During incidents, follow the graph: symptom → dashboard → probable subsystem → recent deploy or dependency health → mitigate. The debugging habits transfer. Production just raises the cost of random changes.

Common mistakes

• Being a hero every night instead of fixing toil.
• A missing runbook on an alert you introduced.
• Treating a weekly identical page as fate. Automate or delete it.

Check yourself

A page took 40 minutes. What do you file within a week?

At least one improvement: runbook line, alert tune, retry, or dashboard link.

The same alert pages every week with the same fix. Is that on-call luck?

No. Automating or eliminating it is engineering work.

Key takeaways

• Write runbooks for humans under stress: symptom, checks, mitigate, escalate.
• After a long page, file one improvement within a week.
• Recurring identical pages are toil. Automate or delete them.
• Quiet shifts are for learning dashboards, not only for luck.

Further learning

Next: production postmortems.