Paging policy

A page is a demand for immediate human attention, at any hour. Treat it as expensive, because every page costs sleep and focus. A paging policy decides what is worth that cost. It is the rules for which signals page, which go to a ticket, how fast someone must respond, and who gets the page next if the first person does not answer.

The big idea

Think of a hospital emergency room. Triage sends the critical cases to the doctor now, and the minor cases to a clinic in the morning. If every sprained wrist called the surgeon at 3 a.m., the surgeon would miss the heart attack. A paging policy is the triage rule for your service.

Walking through it

Google SRE gives a useful target: no more than about two incidents per 12-hour shift. It is a health constraint, not a vanity metric. If a primary handles six pages a night, the service is too fragile, the alerts are wrong, or the team is too small for the risk.

The ingredients of a sound policy:

• Page on symptoms that hurt users, such as an error rate or latency SLO burn. See SLOs.
• Send a ticket or a chat message for non-urgent signals, such as disk at 70% or a certificate that expires in 20 days.
• Set an acknowledgment target, for example 5 to 15 minutes, with automatic escalation to the secondary.
• Write the escalation tree down and test it every quarter.
• Set maintenance windows and planned silences with an owner and an end time. Never silence forever.

A real example from one team shows the pattern. It averaged nine pages a night. An audit found four were replica lag during a daily batch job, two were disk warnings at 70% on log volumes that never filled, and one was a dependency blip that cleared in 90 seconds. The team moved the replica lag to a ticket. It raised the disk page to 90%, with a weekly capacity ticket at 75%. It required five minutes of sustained burn before paging on the dependency. Pages fell to about 1.5 per night. Real checkout failures still woke someone within minutes.

Every paging alert should link to a runbook section. An alert with no next action is a toil generator, which the runbooks page covers in detail.

Common mistakes

• Paging on causes. CPU at 91% is a cause. Users feel the error rate.
• Muting the pager without changing the rule. The next shift gets the same noise.
• Treating all pages as equal. A page for a single failed scrape should not wake anyone.

Check yourself

1. A disk alert fires at 70%, and the disk has been at that level for weeks. Should it page? No. It is a capacity trend, so it belongs in a ticket. A page is for a user-facing problem happening now.
2. What does a target of two incidents per shift measure? The health of the service and the alerts. If it is much higher, the system or the team is under strain.

Key takeaways

• A page is an interruption. Reserve it for user pain happening now.
• Use a ticket for trends and capacity risk, and no action for expected noise.
• Track the pages per shift. If it is high, fix the service or the alerts.

Further learning

The alert design side is in alerting on symptoms. The pruning side is in alert fatigue and cleanup.

On-Call (Google SRE Workbook)How Google Site Reliability Engineering teams structure on-call rotations, site operations, structure approaches to handle incidents and production issues.sre.google