Executable runbooks

A runbook, also called a playbook, is a checklist for handling one alert. It is not architecture documentation, and it is not a novel. Its reader is a person who was just woken up, is not thinking clearly, and needs the next step to be obvious. A good runbook removes decisions from the middle of the night.

The big idea

Think of an aircraft checklist. Pilots do not reason from first principles when an engine warns them. They follow a short list, in a fixed order, that someone tested in advance. A runbook is that list for an alert.

Walking through it

Each runbook has the same six sections:

1. What users feel. One sentence.
2. Severity, and when to declare an incident.
3. Quick checks. Three to five commands or dashboard links.
4. Mitigations in order. For example, roll back, fail over, shed load, or turn off a feature flag.
5. Escalation. Who knows this part of the system.
6. False-positive notes. When it is safe to silence, and for how long.

Here is a quick-check and mitigation sequence from a payments-latency runbook. The commands are for an example namespace and deployment:

curl -fsS https://status.internal/payments/slo | jq .burn_rate_1h   # 1. confirm the symptom is real
kubectl -n payments rollout history deploy/payments-api | head     # 2. check the most recent deploys
kubectl -n payments rollout undo deploy/payments-api               # 3. roll back if the deploy is under an hour old

One team cut its median time to mitigate from about 40 minutes to about 12. The change was simple. Every page had to link to a runbook with a "first five minutes" section. The insight was not better engineers. The runbook removed decision thrash at 2 a.m., when judgment is weakest.

Keep runbooks close to the service repository, or in a wiki that the pager links to directly. Link each alert to its section, not to a general page. Stale runbooks are worse than none, so run a quarterly drill. The primary practices one alert against a staging failure, then fixes whatever in the document was wrong.

Common mistakes

• Runbooks nobody opens. If the alert does not link to the right section, the runbook is theater.
• Long background at the top. Put the first step first. Context can come later.
• No drill. A runbook that has never been used during practice is a guess.

Check yourself

1. Why does a runbook start with "what users feel"? It confirms the alert is real and worth acting on, before anyone starts running commands.
2. The runbook says "roll back if the deploy is under an hour old". Why is that a good rule? It removes the judgment call at 2 a.m. The responder knows the step, and the first action is the cheapest mitigation.

Key takeaways

• A runbook is a checklist for one alert, written for a tired reader.
• Use the six sections, with the mitigations in order.
• Link every page-level alert to its runbook section, and drill the runbooks.

Further learning

The practice of running an incident once awake is in incident response. The paging side is in paging policy.