Alert fatigue and cleanup

Alert fatigue happens when pages are so frequent or so useless that responders mute the channel, auto-acknowledge, or leave the rotation. It is a reliability problem in the alerting system, not a flaw in the people using it. The fix is to measure the alerts and prune them on a schedule, the same way you would handle a noisy service.

The big idea

A smoke alarm that goes off every time you make toast gets unplugged. When a real fire comes, nobody notices. The trust in the alarm is the thing you are protecting, and every false alarm spends some of it.

Walking through it

Track a few simple numbers for the alerting system:

• Pages per on-call shift, and per service.
• The share of pages that led to a real incident, compared with pages that were acknowledged and closed.
• Median time to acknowledge and time to mitigate.
• The oldest flapping alert, the one that fires and resolves over and over.

Then run a monthly cleanup loop:

1. List the noisiest alerts by how often they fired.
2. For each one, do one of four things: fix the system behind it, widen the for window, raise the threshold, or turn it into a ticket. Delete it if nothing needs it.
3. Require a runbook link on every page-level alert. No runbook, no page.
4. After each incident, ask whether the alert fired too early, too late, or not at all. File one alert-quality action item.

An example of the loop

One team averaged nine pages a night. An audit found four were replica lag during a daily batch job, two were disk warnings at 70% on log volumes that never filled, and one was a dependency blip that cleared itself in 90 seconds. They moved the replica lag to a ticket, raised the disk page to 90% with a weekly capacity ticket at 75%, and required five minutes of sustained burn before paging on the dependency. Pages fell to about 1.5 per night. Real checkout failures still woke someone within minutes.

The work was alert design, not caring more.

Common mistakes

• Adding alerts after every incident and never removing any. Each incident should also remove or improve an alert.
• Tracking page count only. Track the share of pages that were actionable too.
• Muting an alert "for now". Give every silence an owner and an end time.

Check yourself

1. A rule fires 40 times a week, and each time it clears itself within two minutes. What do you do? Widen its for window, or convert it to a ticket, then check whether it still catches anything real.
2. Why track the share of pages that led to real incidents? It shows how much of the pager noise is signal, which tells you where to prune first.

Key takeaways

• Measure the alerting system: pages per shift, actionable share, time to acknowledge.
• Prune on a fixed schedule: fix, widen, raise, ticket, or delete.
• Every page needs a runbook, and every rule change goes through code review.

Further learning

The on-call side of this is in designing on-call. For what deserves a page in the first place, see alerting on symptoms.

My Philosophy on AlertingMy Philosophy on Alerting based my observations while I was a Site Reliability Engineer at Google Author: Rob Ewaschuk <rob@infinitepigeons.org> Introduction Vernacular Monitor for your users Cause-based alerts are ba…Google Docs