Observability and reliability
Metrics, dashboards and alerting
Golden signals, RED and USE, Prometheus histograms and alerts that page you only when users are hurting.
Logs and traces tell stories about single requests. Metrics compress millions of those requests into numbers you can graph, alert on and compare across deploys. They are the cheapest way to know whether a service is healthy right now. Used badly, they become a wall of graphs nobody trusts and a pager nobody answers.
Good metrics work comes down to three choices: pick the right shape for each number, read it in a way that is correct across many instances, and page only when users are hurting.
Walking through the topic
Read the pages in this order. The first two cover the basics. The last four cover dashboards and alerts.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Metric types and rate | Counter, gauge, histogram or summary, and why counters become rates? |
| 2 | Labels and cardinality | How many series does one metric create, and which labels are too many? |
| 3 | Histograms and fleet percentiles | How do I get a correct p99 across many pods? |
| 4 | RED, USE and dashboards | What goes on a service dashboard, and in what order? |
| 5 | Alerting on symptoms | Which signals should wake a human, and which should be a ticket? |
| 6 | Alert fatigue and cleanup | How do I measure and prune noisy alerts? |
The frameworks in one place
Three frameworks cover most services and hosts:
- Golden signals (Google SRE): latency, traffic, errors and saturation. If you can graph only four things for a user-facing service, graph these.
- RED for request-driven services: rate, errors and duration.
- USE for resources: utilization, saturation and errors. It comes from Linux performance analysis.
RED is the top of the dashboard, and USE is the drill-down. Page on RED and SLO symptoms. Use USE to diagnose.
Common mistakes
- Averaging percentiles across pods. The busiest pod should carry the most weight. Aggregate histogram buckets first.
- Alerting on CPU or memory for a scaled service. Users did not page you to watch a gauge.
- Adding labels for convenience. Each label multiplies the series count.
- Treating dashboards as alerts. If it matters at 3 a.m., it needs a page with a runbook.
Key takeaways
- Use counters for events, gauges for levels, and histograms for latency. Read counters as rates.
- Never average percentiles. Sum the buckets, then take the quantile.
- Keep label values bounded. Put user and request IDs in logs and traces.
- Page on user symptoms with a runbook. Ticket everything else, and prune noisy alerts monthly.
Further learning
Google's SRE book chapter on monitoring defines the golden signals and the symptom-versus-cause split. Rob Ewaschuk's short doc is still the best guide to what deserves a page.
Monitoring Distributed Systems (Google SRE book)SRE management techniques | How Google scales team collaborations and management to ensure operational excellence across distributed teams.sre.google Prometheus: OverviewPrometheus project documentation for Overviewprometheus.io
My Philosophy on AlertingMy Philosophy on Alerting based my observations while I was a Site Reliability Engineer at Google Author: Rob Ewaschuk <rob@infinitepigeons.org> Introduction Vernacular Monitor for your users Cause-based alerts are ba…Google Docs
Brian Brazil's short intro to scraping a machine with Prometheus and node_exporter is useful if you have never seen a live /metrics endpoint.
Related
Observability: logs and traces
Structured logs, distributed tracing and OpenTelemetry, so you can ask new questions of a running system instead of guessing which dashboard to open.
SLIs, SLOs and error budgets
Define reliability in numbers users care about, then use error budgets and burn-rate alerts to balance shipping speed against stability.
Linux performance analysis
The USE method and the 60-second checklist for finding why a box is slow, covering CPU, memory, disk, network and the container limits that look like application bugs.