Alerting on symptoms

An alert is a rule that asks for a human. Each one costs sleep, so each one should be justified. The standard rule, from the SRE community, is to page on symptoms, not causes. Users care that checkout fails, not that CPU is at 91%. A cause-based alert can stay on a dashboard or in a ticket queue, where it is still useful.

The big idea

Think of a smoke alarm. It should go off for smoke, which is the thing that hurts people, and not for every time you toast bread. A CPU alarm goes off for toast. An error-ratio alarm goes off for smoke.

Walking through it

Each signal goes to one of three places:

Here is a symptom alert in Prometheus. It fires when the 5xx ratio stays above 1% for five minutes:

- alert: CheckoutHighErrorRate
  expr: |
    (
      sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
      /
      sum(rate(http_requests_total{job="checkout"}[5m]))
    ) > 0.01
  for: 5m
  labels:
    severity: page
  annotations:
    summary: "Checkout 5xx ratio above 1% for 5m"
    runbook: "https://runbooks.example.com/checkout-errors"

The for: 5m clause stops a single bad scrape from waking someone. The runbook annotation gives the responder a first step.

Two more habits:

• Compute the alert and the graph from the same expression. Record the ratio with a recording rule, and graph that same series. If the alert and the dashboard use slightly different queries, the dashboard will look fine while the pager fires.
• Make every page actionable. The alert names the service, the symptom, a link to the dashboard or runbook, and a first step. "CPU high on node-17" with no owner trains people to ignore the pager.

Common mistakes

• Paging on CPU or memory alone. Autoscaling and brief spikes are normal. Users did not ask you to watch a gauge.
• Silencing an alert without changing the rule. The next shift gets the same alert. Fix or delete the rule in code review.
• Dashboards as a substitute for alerts. Nobody watches graphs all day. If it matters at 3 a.m., it needs a page with a runbook.

Check yourself

1. Should a CPU alert on a horizontally scaled service wake someone? Usually not. CPU is a cause, and autoscaling handles brief spikes. Send it to a ticket unless it has been shown to mean user pain.
2. Why put for: 5m on the error alert? So one bad scrape or a short blip does not page someone. The condition must last five minutes.

Key takeaways

• Page on user symptoms, such as error ratio or burn rate. Ticket the trends. Ignore expected noise.
• Every page needs a runbook link and a first step.
• Use the same expression for the alert and the dashboard.

Further learning

The rest of the alerting story is in alert fatigue and cleanup. The alert triggers themselves come from SLOs.

My Philosophy on AlertingMy Philosophy on Alerting based my observations while I was a Site Reliability Engineer at Google Author: Rob Ewaschuk <rob@infinitepigeons.org> Introduction Vernacular Monitor for your users Cause-based alerts are ba…Google Docs