Prometheus metrics in /metrics and when to use each one

Anatomy of /metrics

First, it’s worth looking at what actually comes out of the endpoint. The format is OpenMetrics (an evolution of Prometheus’s original exposition format), and each metric has 3 parts: # HELP (description), # TYPE (type) and the series themselves.

# HELP http_requests_total Total HTTP requests received
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1547
http_requests_total{method="POST",status="201"} 89
http_requests_total{method="GET",status="500"} 3

# HELP process_resident_memory_bytes Resident memory in bytes
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 4.2598400e+07

# HELP http_request_duration_seconds Request latency
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.1"} 1430
http_request_duration_seconds_bucket{le="0.5"} 1520
http_request_duration_seconds_bucket{le="1.0"} 1540
http_request_duration_seconds_bucket{le="+Inf"} 1547
http_request_duration_seconds_sum 187.3
http_request_duration_seconds_count 1547

Each name + labels combination is an independent time series. The Prometheus scrape takes this snapshot every 15-30s and stores it with a timestamp.

Important point: /metrics always shows the current value (cumulative, in the case of counters). Prometheus computes rates and differences at query time, not at collection time.

The 4 metric types

flowchart LR
  Metric["Metric"] --> Counter["Counter<br/>only goes up"]
  Metric --> Gauge["Gauge<br/>goes up and down"]
  Metric --> Histogram["Histogram<br/>distribution<br/>(server-side buckets)"]
  Metric --> Summary["Summary<br/>distribution<br/>(client-side quantiles)"]

Which one to pick depends on what you want to answer: counting events, measuring an instantaneous value, or understanding the distribution of a measurement.

Counter

A counter is a metric that only goes up (or resets to zero when the process restarts). It’s the most common type and the one most used to count events.

# TYPE http_requests_total counter
http_requests_total{status="200"} 1547

The rule is simple: if the question is “how many times did X happen?”, it’s a counter. By convention, the name ends in _total.

# Requests per second over the last 5 minutes
rate(http_requests_total[5m])

# Errors per second, broken down by endpoint
sum by (endpoint) (rate(http_requests_total{status=~"5.."}[5m]))

# Error rate (errors / total)
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

Gauge

A gauge is a value that can go up and down freely. It’s a snapshot of “right now”.

# TYPE goroutines_count gauge
goroutines_count 42

# TYPE queue_depth gauge
queue_depth{queue="emails"} 17

# Change over the last 5 minutes (positive = going up, negative = going down)
delta(queue_depth[5m])

# Rate of change per second
deriv(queue_depth[5m])

# Maximum value over the last 1h, grouped by queue
max_over_time(queue_depth[1h])

Histogram

This is where things get interesting. A histogram measures the distribution of observed values, grouping them into predefined buckets. It’s the right type for latency, payload size and any measurement where you want percentiles.

# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.05"} 24054
http_request_duration_seconds_bucket{le="0.1"}  33444
http_request_duration_seconds_bucket{le="0.25"} 100392
http_request_duration_seconds_bucket{le="0.5"}  129389
http_request_duration_seconds_bucket{le="1.0"}  133988
http_request_duration_seconds_bucket{le="+Inf"} 144320
http_request_duration_seconds_sum 53423.1
http_request_duration_seconds_count 144320

Notice that the histogram produces 3 different series:

1. _bucket{le="..."}: a cumulative counter of observations at or below each bound (le = “less than or equal”).
2. _sum: the sum of all observed values.
3. _count: the total number of observations (equal to the +Inf bucket).

# p95 latency over the last 5 minutes
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

# p99 latency broken down by endpoint
histogram_quantile(0.99,
  sum by (le, endpoint) (rate(http_request_duration_seconds_bucket[5m]))
)

# Average latency (not a percentile, but useful)
rate(http_request_duration_seconds_sum[5m])
/
rate(http_request_duration_seconds_count[5m])

Summary

A summary also measures distribution, but it computes the quantiles inside the application and exports them directly:

# TYPE rpc_duration_seconds summary
rpc_duration_seconds{quantile="0.5"}  0.052
rpc_duration_seconds{quantile="0.9"}  0.198
rpc_duration_seconds{quantile="0.99"} 0.812
rpc_duration_seconds_sum 8421.3
rpc_duration_seconds_count 144320

At first glance it looks more convenient: you already have the p99 ready to go, with no need for histogram_quantile(). In practice, summary has a serious limitation: quantiles can’t be aggregated.

You can’t add up the p99 of two instances to get the p99 of the whole set. No math does that from the individual quantiles: you need the original data or the buckets.

That means if you run 5 replicas of your API, each one exports its own p99, and the best you can get is the average or the maximum of those p99s, and neither one is the true p99 of the set.

flowchart TB
  Q["Do I need percentiles?"] -->|Yes| Agg{"Aggregating across<br/>multiple instances?"}
  Q -->|No, just counting| Counter["Counter"]
  Q -->|No, instantaneous value| Gauge["Gauge"]
  Agg -->|Yes| Histogram["Histogram<br/>(prefer native)"]
  Agg -->|No, just 1 instance and<br/>I need exact p99.9| Summary["Summary"]

Cardinality: the brake pedal

Every unique name + labels combination is a new time series. And that costs memory, disk and money.

# BAD: exploding cardinality
http_requests_total{user_id="123", request_id="abc-def-...", path="/api/users/123"} 1

With user_id and request_id as labels, you potentially have millions of series. Each series uses memory in Prometheus and grows indefinitely.

http_requests_total{
  method,    # 5 values (GET, POST, ...)
  endpoint,  # 50 values
  status     # 10 values
}
# = 5 * 50 * 10 = 2500 series

Naming conventions

Prometheus has strong conventions that are worth following:

• Snake_case: http_requests_total, not httpRequestsTotal.
• Counters end in _total: errors_total, not errors.
• Units in seconds and bytes: prefer _seconds and _bytes in the name, not _milliseconds or _kilobytes. Grafana knows how to format them.
• Unit suffix in the name: request_duration_seconds, payload_size_bytes.
• Component prefix: http_*, db_*, kafka_*.

# Good
http_request_duration_seconds_bucket{...}
db_connections_open
kafka_consumer_lag_messages

# Bad
HTTPLatencyMs{...}
db_conns
lag

Summary

Conclusion

Prometheus’s 4 types look similar but solve different problems. In practice, counter and histogram cover the vast majority of cases: counters for events, histograms for distributions. Gauges come in when you need an instantaneous value, and summary is rare enough to be treated as a special case.

The biggest source of trouble with metrics isn’t picking the wrong type, it’s uncontrolled cardinality. Metrics answer “what’s happening on average”. If you want to answer “what happened in this specific request from customer X”, that’s a job for logs and traces, not metrics.

If you’re starting a new stack, consider:

• Standardizing instrumentation on OpenTelemetry (which already follows Prometheus conventions).
• Turning on native histograms from day one.
• Defining a standard set of allowed labels for the team, with cardinality review in the PR.

The difference between a metrics stack that scales and one that turns into a cost nightmare usually comes down to these early choices.

References

• Prometheus: Metric types
• OpenMetrics specification
• Prometheus naming conventions
• Histogram vs Summary
• Native histograms