Owning projects
Handling a page
Acknowledge, assess blast radius, mitigate first, escalate when needed, communicate. Diagnose after users are safe.
When the pager fires, a simple sequence beats heroics. Deeper incident command lives in incident response. Here is the craft you need as an individual contributor on a product team.
- Acknowledge so others know someone is on it.
- Assess blast radius: who is hurt, how bad, is it still spreading? Check the dashboard linked from the alert before changing things.
- Mitigate first, fix root cause second. Roll back, flip a flag, shed load, failover — restore user happiness.
- Escalate when you are past your depth, the blast radius is large, or mitigation is unclear. Escalation is strength. Silent struggle is risk.
- Communicate on the incident channel: what you see, what you are trying, next update time. Even "no change, next update in 15 m" calms people.
Warning
Do not "quick fix" production with unreviewed one-off commands you cannot reverse. Prefer known runbook steps and feature flags. If you must improvise, write down exactly what you ran.
After mitigation: capture timeline notes while fresh. Drop them in the incident channel before you go back to sleep. Memory decays faster than you expect.
Walking through a night page
00:42 — Alert: export_failure_rate_high. You ack and open the dashboard linked from the runbook. Failure rate 18% for 12 minutes; latency fine; object-store 5xx elevated.
00:45 — Recent deploys: your team's export worker deployed at 00:10. Dependency status page: object-store regional degradation. The platform team already noted it in the incident channel.
00:48 — Mitigate: flip admin_csv_export off for external customers (internal dogfood stays on). Failure rate drops. Users see "temporarily unavailable" — better than corrupt partial files.
00:55 — Communicate: "Mitigated via flag. Root appears to be object-store 5xx, not our last diff. Next update 01:15."
01:20 — Store recovers. You canary the flag 10% → 50% → 100% over thirty minutes. Clean.
Notice the order: mitigate with the flag before debating whether your deploy caused it. Root-cause purity during user pain is a tax everyone else pays.
Primary → secondary (or outgoing → incoming) should include five lines: ongoing incidents, hot deploys, risky dependencies, follow-up links. If your team has no handoff habit, start one in the on-call channel at shift change.
Common mistakes
- Debugging root cause while users still burn. Mitigate first.
- Silent struggle. Escalate when blast radius or depth says so.
- Improvised production commands with no record and no reverse.
Check yourself
Failure rate is 18% and you suspect last night's deploy or a dependency. First action?
Mitigate — flag off or roll back — before you debate the root cause.
You are past your depth and the blast radius is growing. What is the strong move?
Escalate. Silent struggle is the risk.
Key takeaways
- Ack, assess, mitigate, escalate, communicate. Diagnose after users are safe.
- Prefer runbook steps and flags over unrehearsed commands.
- Write the timeline before you sleep.
- Handoff five lines at shift change.
Further learning
Next: runbooks and toil.
Related
Owning your code in production
On-call, alerts, runbooks, and the loop that makes you trustworthy — and better at building systems that survive users.
Shipping with observability
A feature is not done when the happy path works locally. Done means someone half-asleep can tell whether it is healthy.
Incident response
Run an incident with clear roles, severity and communication. Mitigate first, debug second, then hand off to the postmortem. Start here for the map of incident pages.