Observability and reliability
Monitoring versus observability
Monitoring checks the questions you wrote down in advance. Observability lets you ask new questions of a running system, by slicing high-cardinality events after the fact.
Monitoring asks questions you chose in advance: is the error rate above 1%? Is disk above 80%? It works well when failures look like the ones you have already seen. Observability is the ability to answer questions you did not plan for, by exploring the system's output until the cause is clear. The difference matters most when a failure is rare, specific to one customer, or the result of a combination of changes.
Concept
Cardinality is how many distinct values a field can take. status has a few values, so it is low cardinality. user_id or merchant_id can have millions, so it is high cardinality. Metrics systems struggle with high cardinality. Log and trace stores are built for it.
The big idea
Monitoring is a set of dials on the dashboard. Each dial was chosen because you knew what might go wrong. Observability is a way to search the whole recording of what happened. When a failure is new, the dials stay green, and only the recording shows it.
The average of a million healthy requests hides the 47 that failed for one merchant. Averages are the wrong tool for that question.
Walking through it
Compare the three approaches:
| Approach | Strength | Blind spot |
|---|---|---|
| Metrics dashboards | Cheap aggregates, good for trends and pages | You must pick labels in advance. Averages hide outliers. |
| Unstructured logs | Familiar, works everywhere | Grep does not scale. There is no causal graph across services. |
| Structured logs and traces | Causality across services, and arbitrary filters | More instrumentation and storage cost |
Here is the question observability answers well: "show me POST /checkout where merchant_id is acme and the duration is over 2 seconds." Nobody built a dashboard for that. A high-cardinality event store can answer it in seconds.
Trade-off
Keep metrics for paging and capacity, and use logs and traces for why. Putting every dimension into Prometheus labels will overload the metrics store. Debugging a multi-service latency spike with grep alone wastes the night.
Common mistakes
- Treating observability as a tool purchase. It is a property of the data you emit. A vendor cannot add fields you never logged.
- Replacing alerts with exploration. Someone still has to be paged. Use metrics to page and observability to explain.
- Averaging away the outliers. When a question is about one customer, check the distribution and the individual events.
Check yourself
- Your dashboards are green, but one merchant's checkouts fail. Why? The failure is rare for the total, so averages hide it. The dashboards were built for questions you wrote down in advance.
- Why are metrics poor at answering "which merchant"?
merchant_idhas high cardinality. Each value would create a new series in the metrics store.
Key takeaways
- Monitoring answers questions you planned. Observability answers new ones.
- High-cardinality events make per-customer questions possible.
- Use metrics to page, and logs and traces to explain.
Further learning
The fields to put on each event are in structured logs. The wider approach is in wide events.
Charity Majors, Liz Fong-Jones and George Miranda's book is the clearest statement of this thesis.
Observability Engineering (O'Reilly)oreilly.comRelated
Observability: logs and traces
Structured logs, distributed tracing and OpenTelemetry, so you can ask new questions of a running system instead of guessing which dashboard to open.
Metrics, dashboards and alerting
Golden signals, RED and USE, Prometheus histograms and alerts that page you only when users are hurting.
Labels and cardinality
Every unique label combination in Prometheus is a separate time series. Unbounded labels such as user IDs can crash the metrics store, so keep those values in logs and traces.