RED, USE and dashboards

You rarely need a hundred metrics on day one. Three frameworks cover nearly every service and host. The four golden signals are latency, traffic, errors and saturation. RED is the request-driven version: rate, errors and duration. USE covers resources: utilization, saturation and errors. A good dashboard puts RED at the top and USE underneath, so you can see both the symptom and the likely cause.

The big idea

RED tells you whether users are hurting. USE tells you where the pressure is. They answer different questions, and each one alone can mislead. RED without saturation misses a service that is about to fall over. USE without RED pages on busy CPUs that still serve users well.

Walking through it

The service row answers "is this service okay?" with four panels, from the same histogram:

1. Rate: requests per second, stacked or split by status class.
2. Errors: the error ratio, or the 5xx rate.
3. Duration: p50, p95 and p99 from one histogram. See histograms and percentiles.
4. Saturation: thread-pool wait, database connections in use, or queue depth.

Drill-downs sit below, in their own rows or linked boards. They include RED per endpoint for the top routes, latency for each dependency, and USE for the pods and nodes behind the service.

Two habits make a dashboard trustworthy:

• Use explicit units. Mixing duration_ms and duration_seconds on adjacent panels causes misreadings. Prefer seconds for Prometheus latency.
• Overlay deploy markers. The fastest latency question is still "what changed?" A marker on every graph answers it without archaeology.

Avoid fifty panels of "maybe useful someday", mixing unrelated services on one board, and duplicate panels whose queries differ slightly.

Common mistakes

• Building the dashboard with CPU first. Start with RED. Users are the symptom.
• Two panels for the same thing with different PromQL. They drift, and people stop trusting either.
• Boards no one opens. Delete them after a quarter without views.

Check yourself

1. Rate and error ratio are normal, but p99 jumped. Which framework tells you that users are slow? RED. The duration panel shows the users are slow even though nothing is failing.
2. Your DB pool wait is high after the same jump. What does that point to? Saturation in a resource behind the service. Use USE on the pool to find where the wait comes from.

Key takeaways

• RED for the service, USE for the resources. Put RED on top.
• A service row of four panels answers "is it okay?" in one screen.
• Annotate deploys and use explicit units on every panel.

Further learning

Resource checks in more depth are in Linux performance analysis. Alerts built on the same signals are in alerting on symptoms.

Monitoring Distributed Systems (Google SRE book)SRE management techniques | How Google scales team collaborations and management to ensure operational excellence across distributed teams.sre.google