Guide

Observability and reliability

Writing the summary and timeline

Start with user impact, then build the timeline from machine data. Record the silent window between failure and detection, because it is often the most important line.

120 of 162 · 2 min read

A postmortem is a written record of what happened, how bad it was, and what will change. Write it within a few days, while memory is fresh. Start with the parts a busy reader needs: the summary and the impact. Then build the timeline from machine data, not from memory.

Concept

Before you start. Read blameless culture. The timeline is where blameless thinking shows up: it records what the system and the responders knew, not who should have known better.

The template

SectionWhat goes here
SummaryTwo to four sentences an executive can read in a few seconds
ImpactUser-facing effect, duration, severity, and error budget burned
TimelineUTC timestamps for detection, diagnosis, mitigation and resolution
DetectionHow you found out, and how long the silent window was
Contributing factorsSeveral causes, not one villain
What went wellDetection, communication and lucky breaks
What went poorlyGaps in tooling, process and knowledge
Action itemsSpecific, owned, prioritized, with due dates

Summary and impact

Lead with user pain, not internal drama. This is a good summary:

Checkout returned 500s for about 18 minutes, affecting roughly 12% of orders in us-east-1. That is an estimated 4,200 failed checkouts. The incident used about 35% of the monthly error budget.

Give precise numbers when you have them. When you do not, use honest orders of magnitude. Do not invent precision.

Walking through a timeline

A good timeline is plain and dense. Every line has a time, an actor (a person, a system or an alert), and an observable fact.

13:02 UTC  Canary error rate rises on payments-v2 (no page: threshold too high)
13:11 UTC  Full rollout completes to 100% of us-east-1
13:14 UTC  PagerDuty: payments success rate below 99% for 5 minutes
13:16 UTC  Incident commander opens the channel; secondary paged
13:22 UTC  Rollback of payments-v2 starts
13:27 UTC  Success rate back above SLO; monitoring continues
13:45 UTC  Mitigated; root-cause work and cleanup remain

The gap from 13:02 to 13:14 is 12 minutes of failure that nobody was paged for. That silent window is often the most important line in the document. Detection lag is a failure mode of its own, and you can fix it with better alerts.

Show what responders believed at each minute, not what you know in hindsight. A timeline written with hindsight hides the gaps in what people could see. Those gaps are where you find the fixes.

Common mistakes

  • Writing the timeline from memory. Pull timestamps from pager history, deploy logs and dashboards.
  • Mixing "the system failed" with "we noticed". Keep them as separate times.
  • Skipping the impact numbers. Leadership cannot prioritize reliability work without them.

Check yourself

  1. Why separate the time the system failed from the time humans noticed? Answer: the gap is detection lag. It is a separate problem, and better alerts can shrink it.
  2. What makes a summary good? Answer: user impact in plain numbers, readable by an executive in seconds.

Key takeaways

  • Lead with impact, in user terms.
  • Build the timeline from machine data.
  • Record the silent window between failure and detection.
  • Show what responders knew at each step.
  • Next, contributing factors explains how to list the causes.

Further learning

Example postmortem (Google SRE book)This practical site reliability incident postmortem example outlines blameless analysis, action items and a detailed timeline for effective outage resolution.sre.google Postmortem templates (Atlassian)Incident response is a structured process to document incidents, track follow-ups, and drive continuous improvement.Atlassian Details of the Cloudflare outage on July 2, 2019Almost nine years ago, Cloudflare was a tiny company and I was a customer not an employee. Cloudflare had launched a month earlier and one day alerting told me that my little site, jgc.org, didn’t seem to have working…Cloudflare Blog

Blameless postmortems

Learning from failure with timelines, contributing factors and action items that actually get done — without turning people into the root cause.

Incident response

Run an incident with clear roles, severity and communication. Mitigate first, debug second, then hand off to the postmortem. Start here for the map of incident pages.

Blameless culture

Blameless postmortems ask which conditions made a failure possible, not who to blame. Accountability for fixes stays in place.