Observability and reliability
Incident severity and roles
How to rate an incident's severity so everyone shares the same urgency, and how to split the work into commander, operations, communications and scribe roles so one person is not doing everything.
Before anyone starts debugging, two questions need answers. How bad is this? And who decides? Severity gives everyone the same sense of urgency. Roles stop five people from doing the same job while nobody makes the call.
Concept
An error budget is how much unreliability your SLO allows in a period. When an incident burns that budget fast, its severity usually goes up. The commander uses that signal, along with user impact, to set the level.
The big idea
Severity works like triage in an emergency room. A nurse sorts patients by how urgently they need care, so a doctor does not spend ten minutes on a scratch while someone else is bleeding. Triage is quick, and it is revised as the facts change.
The triage analogy breaks in one place. ER triage is done by trained staff against fixed criteria. Incident severity must be set by a named person, and it has to be written down, or everyone argues about adjectives.
Severity levels
| Severity | User impact | Examples | Response |
|---|---|---|---|
| SEV1 | Most customers cannot complete a core journey, or there is data loss or a breach | Checkout failing 100%, primary database unreachable in both regions | All-hands, executive visibility, continuous updates |
| SEV2 | Significant degradation for many users, or a secondary journey is down | p99 at 10 times the SLO, payments at a 20% error rate | On-call plus owners, urgent mitigation |
| SEV3 | Limited impact, or a workaround exists | One tenant broken, a non-critical admin UI down | Business hours, the owning team |
| SEV4 | Minor, cosmetic or internal | Wrong empty-state copy, a staging outage | A ticket, not an incident channel |
If everything is SEV1, nothing is. Calibrate with real past events. Publish the table in your runbook, and let the commander set the severity and change it as facts change.
Note
Security incidents often run a parallel track with legal and forensics. Still declare a severity, appoint a commander and contain the exposure. Do not paste secrets or customer data into a public channel.
The four roles
- Incident commander (IC). Owns decisions, coordinates, timeboxes work and sets the severity. The IC does not dig through logs while commanding. Anyone trained can be an IC.
- Operations lead. Drives diagnosis and mitigation hands-on, and proposes actions for the IC to approve.
- Communications lead. Drafts internal updates, status page text and customer messages.
- Scribe. Keeps the timeline, with UTC timestamps, actions, hypotheses and graph links.
On a small team, one person can be IC and scribe. On a SEV1, split them. If the IC starts SSHing into machines, appoint a new IC. The role is coordination, not heroics.
Important
The IC's decision stands for the duration of the incident. Debate is welcome. Deadlock is not. Revisit the choice in the postmortem.
Walking through a declaration
A good declaration is one line that says severity, impact, start time and the commander:
Declaring SEV2 - checkout 5xx ~15% since 14:02 UTC - IC @youAnyone reading that line knows how urgent the incident is, when it started, and who to talk to. A vague "checkout looks bad" gives none of that.
Call hygiene
- One voice at a time on the bridge. The IC names the next speaker.
- Paste long logs and graph links into the channel, not into the call.
- Move speculation to a thread. Keep the main channel for decisions and state changes.
- When someone joins, give a 30-second summary. Do not restart the story for each arrival.
Common mistakes
- Rating everything SEV1. Then the real SEV1 looks like noise.
- The IC gets into the logs. Appoint a new IC so someone is coordinating.
- Endless debate. The IC decides and revisits later.
Check yourself
- The team disagrees mid-incident about a fix. Who decides? The incident commander. Debate is welcome, but a deadlock must end with the IC's call.
- What should a declaration contain? The severity, the impact with a time, and the name of the commander.
Key takeaways
- Severity gives everyone one shared sense of urgency.
- Four roles keep the work from piling onto one person: IC, operations, comms and scribe.
- The IC decides, and the scribe keeps the record.
Further learning
Next, see the first fifteen minutes in order in the first fifteen minutes.
Related
Incident response
Run an incident with clear roles, severity and communication. Mitigate first, debug second, then hand off to the postmortem. Start here for the map of incident pages.
Designing on-call
Sustainable rotations, executable runbooks, toil budgets and a paging policy that keeps humans healthy while production stays covered.
SLIs, SLOs and error budgets
Define reliability in numbers users care about, then use error budgets and burn-rate alerts to balance shipping speed against stability.