Metric types and rate

A metric is a number recorded over time, such as requests served or current queue depth. Before you pick what to measure, pick the right shape for it. Prometheus, and most modern metrics systems, give you four. The shape decides what you can ask of the number later, and the most important rule is that a counter is almost never read on its own.

The four shapes

A counter tells you how much has happened in total, which is rarely useful on its own. A counter at 184,221 says nothing about whether the service is busy right now. A gauge says the current level, and a histogram says how the individual observations were spread.

Walking through it: counters become rates

Take a counter that records requests. It rises from 184,221 to 184,821 over one minute. The useful number is the rate, which is the change divided by the time:

rate(http_requests_total[5m])

This is requests per second, averaged over the last five minutes. Five minutes of data means at least two scrapes in the window. Use a longer window for sparse traffic, and a shorter one when you need to react quickly. increase gives the total change over the window when you want a count, rather than a rate.

The rule is strict. Never alert on a raw counter. A counter that stops rising is interesting. Its absolute value is not. A counter that was reset at a deploy would look like a fall if you compared raw values.

Common mistakes

• Graphing a counter's raw value. The line only goes up, and it tells you nothing about current traffic.
• Using a gauge for something that only increases. Use a counter. Gauges can go down, and rate calculations assume they don't.
• Choosing a window shorter than the scrape interval. rate needs at least two samples inside the window.

Check yourself

1. A counter goes from 1,000 to 1,300 in five minutes. What is the average rate? 300 ÷ 300 seconds = 1 request per second.
2. Why does a counter reset not look like an error to rate? rate handles a counter reset at a restart, so it does not treat the drop to zero as a huge negative rate.

Key takeaways

• Choose the shape first: counter for events, gauge for levels, histogram for latency.
• Counters only go up, and you read them as rates.
• Never alert on a raw counter.

Further learning

The next idea is how labels multiply series, in labels and cardinality. The fleet-level latency percentile is covered in histograms and percentiles.

Prometheus: OverviewPrometheus project documentation for Overviewprometheus.io