Observability and reliability
Incident response
Run an incident with clear roles, severity and communication. Mitigate first, debug second, then hand off to the postmortem. Start here for the map of incident pages.
Production is broken, and customers are angry or about to be. In the next hour your job is not to find a clever root cause. Your job is to restore service, keep people informed, and leave a timeline someone can learn from later.
Incident response is the practiced system for doing that under stress. It covers severity, roles, a first-minutes checklist, mitigation and a communication rhythm. Teams without it get heroics, duplicate work and longer outages.
An incident is an unplanned disruption that needs coordinated response beyond normal ticket work. Usually users are affected, the error budget is burning fast, security is involved, or data may be wrong.
Read in order
| Step | Concept | What it answers |
|---|---|---|
| 1 | Severity and roles | How bad is it, and who decides? |
| 2 | The first fifteen minutes | What do you do in the first quarter hour, in order? |
| 3 | Mitigate before root cause | Which fixes come before understanding the cause? |
| 4 | Communication and handoffs | How often do you update, and what do you say? |
Common mistakes across the topic
- No incident commander. Five people debug the same thing while nobody decides.
- Root cause before mitigation. Users keep failing while you explain.
- A silent status page. Rumor fills the silence.
- Declaring all clear too early. A premature RESOLVED destroys trust when errors return.
- No backup commander. On a bad night, nobody can take over.
Tooling you need before the incident
You do not need an expensive platform. You need defaults:
- Paging with escalation policies, such as PagerDuty or Opsgenie (on-call design).
- An incident channel that opens automatically, with the commander, severity and links pinned.
- A timeline kept by a scribe, exported into the postmortem.
- A status page you have practiced publishing.
- Runbooks linked from alerts, with the rollback command, flag name and dashboard URLs.
Keep emergency credentials in the secrets system, not in plain text in a runbook. The secrets management procedures cover break-glass access.
Key takeaways
- An incident needs severity, a commander and a written timeline.
- In the first fifteen minutes: acknowledge, assess, declare, assemble, communicate, then mitigate.
- Prefer rollback, flags, failover and load shedding over deep debugging.
- Update on a cadence, with impact and next-update times.
- Rehearse realistic failures in a game day.
Further learning
Google's SRE chapter on managing incidents covers roles and the split between technical and coordination work.
Managing Incidents (Google SRE book)Principled incident management can limit disruptions and restore normalcy. Learn about effective strategies and processes for managing incidents.sre.googlePagerDuty's public incident response documentation is the most widely copied practical playbook.
PagerDuty Incident Response documentationA collection of information about the PagerDuty incident response process. Not only how to prepare new employees for on-call responsibilities, but also how to handle major incidents, both in preparation and after-work.PagerDuty Incident Response Documentation
A talk on how to act as incident commander, covering decisions, delegation and call hygiene under pressure:
Related
Blameless postmortems
Learning from failure with timelines, contributing factors and action items that actually get done — without turning people into the root cause.
Designing on-call
Sustainable rotations, executable runbooks, toil budgets and a paging policy that keeps humans healthy while production stays covered.
SLIs, SLOs and error budgets
Define reliability in numbers users care about, then use error budgets and burn-rate alerts to balance shipping speed against stability.