Labels and cardinality

In Prometheus, a sample is stored under a metric name plus a set of labels, such as method="POST" and code="200". Each unique combination of label values is a separate time series. Cardinality is the number of those series. Labels are cheap while each one has a few values. They become expensive the moment a label takes millions of values.

The big idea

Think of a filing cabinet. Each drawer is a label combination. A few drawers are easy to search. But if you make a drawer for every customer, you now have millions of drawers, and the cabinet no longer fits in the room. The cabinet is the metrics server, and the room is its memory.

Walking through it

Say http_requests_total has three labels:

• method: 4 values (GET, POST, PUT, DELETE)
• code: 5 classes (2xx, 3xx, 4xx, 5xx, other)
• route: 30 route templates

That is 4 × 5 × 30 = 600 series. Fine.

Now add user_id, which can take millions of values. Multiply the 600 series by every user, and the count reaches the billions. Prometheus runs out of memory, scrapes time out, and Grafana panels never load.

Histograms multiply too. Each bucket is one more series, because of the le label. A histogram with 10 buckets and the 600 combinations above is 6,000 series. So choose the buckets carefully: default buckets that jump from 10 ms to 100 ms are no use if your target is a 30 ms p99.

Rules of thumb:

• Bound every label to a small, known set. Use route templates like /users/:id, not raw paths.
• Keep status classes if you only alert on errors. Use exact codes only if you need them.
• Put high-cardinality IDs in logs and traces, where storage is built for them. See observability.
• Review new labels in code review, the same way you review an API change.

Common mistakes

• Adding a label because it might be useful someday. Every new label multiplies the series count.
• Putting raw URLs in labels. /users/48291 and /users/48292 become different series.
• Keeping default histogram buckets. They may miss the range you care about.

Check yourself

1. A metric has labels with 4, 5 and 30 values. How many series does it have? 4 × 5 × 30 = 600 series.
2. A team wants to add request_id as a label to debug one failing request. Is that a good idea? No. Each request creates a new series. Put request_id in logs or traces instead.

Key takeaways

• Each unique label combination is a separate time series.
• Multiply the label values to estimate the series count before adding a label.
• Put user, request and other unbounded IDs in logs and traces, not metrics.

Further learning

The trade-off between metrics and logs is in observability. For the rest of the metrics picture, return to the metrics, dashboards and alerting hub.

Prometheus: OverviewPrometheus project documentation for Overviewprometheus.io