Observability and reliability
The first fifteen minutes
The order of actions in the first fifteen minutes of an incident, with a worked timeline for a database connection pool failure that ends in a rollback.
The first fifteen minutes decide whether an incident takes one hour or four. The goal is to know who owns it, how bad it is, and to say so out loud. Then pick one mitigation and time-box it.
Concept
RED is a dashboard view for a service: request rate, error rate, and duration (latency). It is the fastest way to see whether users are affected, and how much. A quick look at RED data beats a deep dive in the first minutes.
The big idea
The first minutes are about shape, not depth. You are not finding the cause yet. You are answering four questions: what is broken, for whom, who owns it, and what is the first thing to try.
The steps
- Acknowledge the page so others know a person owns it.
- Assess impact. Which journey, what percentage, which regions or tenants? Use RED dashboards and a quick trace or log sample.
- Declare in the agreed channel, with severity, impact and the commander.
- Assemble. Page the owning service's on-call, dependency owners if the symptoms point there, and a comms lead for SEV1 or SEV2.
- Communicate early, even if you only know symptoms.
- Pick one mitigation hypothesis and time-box it. Investigate in parallel only if it does not block the mitigation.
Tip
Use UTC timestamps and paste graph links. "A few minutes ago" fades by the postmortem. "14:02 UTC" survives.
Walking through a timeline
This example is a database connection pool failure. Read it as the timeline a scribe would keep.
| Time (UTC) | What happened | Why it matters |
|---|---|---|
| 14:02 | CheckoutHighErrorRate pages, and on-call acknowledges | A human owns it now |
| 14:04 | Checkout 5xx is about 18%, and payments dependency errors are up. CPU is fine. Declared SEV2 | Impact is a number, and the channel is open |
| 14:07 | Traces show payments calls failing fast with connection pool timeout. Active connections are at the max, and a deploy at 13:55 added a heavier query | Points to the pool, not the CPU |
| 14:10 | Two tracks: roll back the payments deploy, or raise the pool size | Adding pods would add connections to the same exhausted pool, so scaling is out. The IC chooses rollback |
| 14:18 | Rollback complete. Error ratio falling. Status: mitigation deployed | Users recover before anyone fixes the SQL |
| 14:25 | Error ratio below 0.5%. State moves to MONITORING | Stay in the channel |
| 14:55 | Stable for 30 minutes. RESOLVED, and the postmortem gets an owner | The incident closes with follow-ups recorded |
What went right: a symptom alert, a quick declaration, and a rollback before anyone rewrote SQL under stress. A worse version has five engineers running EXPLAIN on the same query while customers retry, no commander, and a silent status page.
Example
Rough budget math for the postmortem: 20 minutes at about 18% failure is a large bite of a monthly budget if the service's error rate normally sits near 0.1%. That framing supports the freeze-or-fix conversation under your SLO policy, without anyone needing to be blamed.
Common mistakes
- Deep diving before mitigation. Users fail while you explain.
- Waiting to declare. A declaration brings the right people and sets the urgency.
- Vague timestamps. Write the time, in UTC.
- Forgetting to page the owners. Dependencies cannot help if nobody calls them.
Check yourself
- Why declare an incident before you know the cause? The declaration assembles the right people and sets the urgency. The cause can come later.
- Why use UTC timestamps in the channel? Times like "a few minutes ago" are lost by the postmortem, while a UTC time stays exact.
Key takeaways
- The first fifteen minutes are about impact, owner and one mitigation.
- Declare early with severity, impact and the commander.
- Keep a timeline with UTC times and graph links.
Further learning
Next, see how to pick the mitigation in mitigate before root cause.
Related
Incident response
Run an incident with clear roles, severity and communication. Mitigate first, debug second, then hand off to the postmortem. Start here for the map of incident pages.
Observability: logs and traces
Structured logs, distributed tracing and OpenTelemetry, so you can ask new questions of a running system instead of guessing which dashboard to open.
Metrics, dashboards and alerting
Golden signals, RED and USE, Prometheus histograms and alerts that page you only when users are hurting.