Prometheus cardinality and retention checklist

A self-hosted Prometheus-compatible stack has no per-series invoice, but the series count still sets the cost. RAM follows the number of active series. Disk follows samples per second multiplied by retention. Query CPU follows how many series each dashboard and alert touches. When one of those grows, you pay for it in bigger nodes and bigger volumes instead of a vendor line item.

This is the checklist I work through, in this order, when a metrics stack gets expensive. I run the same discipline on RESTIA, the restaurant software platform I designed and operate, where VictoriaMetrics, VictoriaLogs and VictoriaTraces sit behind an OpenTelemetry collector.

The snippets below were validated on 24 October 2026 with promtool 3.15.1 (check config --syntax-only, check rules), VictoriaMetrics v1.154.1 (-dryRun) and OpenTelemetry Collector v0.163.2 (k8s distribution, started with the config). Where the text says what a config does, the figures come from a validation run that sent test data through it and counted the output.

1. Measure before you cut

Start with three lists: the metric names with the most series, the labels with the most distinct values, and the jobs that create new series fastest. Churn matters as much as size, because every new series costs index space even if it only lives for an hour.

In Prometheus, the Status → TSDB Status page shows the first two. The same data is in the TSDB stats API, which takes a limit parameter (10 by default, up to 10,000):

# Prometheus: top 20 metric names by series count
curl -s 'http://prometheus:9090/api/v1/status/tsdb?limit=20' | jq '.data.seriesCountByMetricName'

# VictoriaMetrics: top 20 values of the pod label
curl -s 'http://victoriametrics:8428/api/v1/status/tsdb?topN=20&focusLabel=pod' | jq '.data.seriesCountByFocusLabelValue'

# VictoriaMetrics: metric names that no query has read since tracking started
curl -s 'http://victoriametrics:8428/prometheus/api/v1/status/metric_names_stats?le=0&limit=100'

These PromQL queries cover what the status pages do not. The last two use series Prometheus generates for every scrape, and vmagent generates the same ones:

# Metric names with the most series. Expensive on large setups, so run it once, not on a dashboard.
topk(20, count by (__name__) ({__name__=~".+"}))

# Jobs that expose the most samples after relabeling
topk(20, sum by (job) (scrape_samples_post_metric_relabeling))

# Series churn: new series per job over the last hour
topk(20, sum by (job) (sum_over_time(scrape_series_added[1h])))

VictoriaMetrics has an Explore cardinality tab in vmui built on the same API, with a date picker and a series selector, described in its cardinality explorer docs. Since v1.113.0 it also tracks how often each metric name is queried. The metric_names_stats call above with le=0 returns names that were never read, which is the shortest route to a drop list. Alerting and recording rules query through the same API, so a metric used only by an alert does not show up as unused.

2. Drop the metrics nobody queries

metric_relabel_configs run after the scrape and before anything is stored, so a dropped metric costs a little scrape CPU and nothing else. Check the drop list against dashboards and alert rules first. Then drop by name:

# prometheus.yml (Prometheus 3.x)
global:
  scrape_interval: 30s

scrape_configs:
  - job_name: api
    # The whole scrape fails instead of ingesting a cardinality explosion.
    sample_limit: 20000
    label_limit: 30
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: api
        action: keep
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      # Copied pod labels that add nothing a query needs.
      - action: labeldrop
        regex: pod_template_hash|controller_revision_hash
    metric_relabel_configs:
      # Runtime metrics nobody on this team reads.
      - source_labels: [__name__]
        regex: go_gc_.*|go_memstats_(frees|lookups|mallocs)_total
        action: drop
      # Keep 6 of the 11 default latency buckets. Never drop +Inf.
      - source_labels: [__name__, le]
        regex: http_request_duration_seconds_bucket;(0\.005|0\.01|0\.025|0\.25|2\.5)
        action: drop

  - job_name: kubernetes-cadvisor
    scheme: https
    metrics_path: /metrics/cadvisor
    kubernetes_sd_configs:
      - role: node
    authorization:
      credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: container_spec.*|container_(file_descriptors|tasks_state|threads_max)
        action: drop
      # cgroup series that belong to no pod
      - source_labels: [id, pod]
        regex: .+;
        action: drop

The cAdvisor rules are a subset of the defaults that ship in the kube-prometheus-stack chart values (version 92.11.0 on 24 October 2026). If you run that chart, they are already on. If you wrote your own scrape config for the kubelet, cAdvisor is usually the biggest single source of series, and these rules are a good starting point.

The full list of relabel actions is in the Prometheus configuration docs.

3. Drop labels only when the rest still identifies the series

labeldrop is safe for labels that are constant per target, like the pod labels that labelmap copies onto every series above. The instance label still tells the series apart.

It is not safe for labels that distinguish series within one target. In a validation run with Prometheus 3.15.1, a target exposed app_requests_total{user_id="1"} 5 and app_requests_total{user_id="2"} 7, and a labeldrop removed user_id. Prometheus stored one series with the value 5. The 7 was gone, nothing was logged, and the duplicate-sample counter stayed at zero. The right answer was 12. Dropping a label like this gives you wrong data, silently.

For labels such as user IDs, request IDs or raw URL paths, fix the instrumentation: record the route template (/orders/{id}) instead of the path, and put per-user detail in logs or traces, where it costs bytes rather than series. If you cannot change the code soon, aggregate the label away (step 5) instead of dropping it.

4. Cut histogram buckets

A classic histogram is one series per bucket, plus _sum and _count, for every label combination. With the 11 default buckets of most client libraries plus +Inf, that is 14 series per combination before any label is added. The second metric_relabel_configs rule in step 2 keeps 6 of the 11 buckets plus +Inf. histogram_quantile() still works on what is left, with coarser resolution between the buckets you keep, so keep the ones around your SLO thresholds.

Two rules. Never drop +Inf, because it equals _count and quantile maths depends on it. And drop the same buckets on every target of a metric, or aggregations across targets mix bucket layouts.

If you run Prometheus 3 and your client libraries support them, native histograms store a whole distribution in one series. Check that every component that reads or forwards your metrics supports them before you switch.

5. Aggregate before you store

Most dashboards want per-service numbers, not per-pod ones. Per-pod series are useful during an incident for a few hours, and they cost the most.

In Prometheus, a recording rule adds series rather than removing them. The saving comes when long-term storage receives only the aggregates. Recording rule names contain colons by convention, which makes them easy to select:

# rules.yml
groups:
  - name: http-aggregates
    interval: 1m
    rules:
      - record: service_route:http_request_duration_seconds_bucket:rate5m
        expr: sum without (instance, pod) (rate(http_request_duration_seconds_bucket[5m]))
      - record: service_code:http_requests:rate5m
        expr: sum by (service, code) (rate(http_requests_total[5m]))

# prometheus.yml: send only recording rule output to long-term storage
remote_write:
  - url: https://long-term-storage.example.internal/api/v1/write
    write_relabel_configs:
      - source_labels: [__name__]
        regex: .+:.+
        action: keep

Local Prometheus then keeps raw data for a short retention, and long-term storage keeps aggregates for months.

vmagent can do the aggregation in the ingestion path with stream aggregation, so per-pod series never reach the database:

# stream-aggr.yml, passed to vmagent with -streamAggr.config=stream-aggr.yml
- match: 'http_request_duration_seconds_bucket'
  interval: 1m
  without: [instance, pod]
  outputs: [rate_sum]
- match: 'http_requests_total'
  interval: 1m
  without: [instance, pod]
  outputs: [total]

By default, raw samples that match a rule are dropped after aggregation; -streamAggr.keepInput keeps them. In a validation run with three pods’ worth of a three-bucket histogram and a counter, storage received three bucket series instead of nine, named http_request_duration_seconds_bucket:1m_without_instance_pod_rate_sum, and one counter instead of three, http_requests_total:1m_without_instance_pod_total. Dashboards have to query the new names, for example histogram_quantile(0.99, sum by (le, service) (http_request_duration_seconds_bucket:1m_without_instance_pod_rate_sum)). The docs recommend an interval of at least twice the scrape interval.

6. Choose the scrape interval on purpose

Samples per second equal active series divided by the scrape interval. Moving a job from 15 to 30 seconds halves its samples, which roughly halves its disk use and ingestion CPU. RAM barely changes, because the number of active series stays the same.

The cost is resolution. Grafana’s $__rate_interval is defined as max($__interval + scrape_interval, 4 * scrape_interval) (Grafana docs), so at a 60-second interval the shortest sensible rate() window is four minutes, and a 30-second spike disappears into it. A reasonable split is 15 to 30 seconds for services you alert on and 60 seconds for slow-moving infrastructure metrics, set per job with scrape_interval.

7. Put limits in place

Every fix above can be undone by one deploy that adds a label. Limits turn that into a visible failure:

sample_limit and label_limit per job, as in step 2. If a scrape goes over sample_limit after relabeling, Prometheus rejects the whole scrape, so up goes to 0 and prometheus_target_scrapes_exceeded_sample_limit_total increases. Alert on both.
In VictoriaMetrics, -storage.maxHourlySeries and -storage.maxDailySeries cap new series per hour and per day. Samples for new series over the limit are dropped and counted in vm_hourly_series_limit_rows_dropped_total and vm_daily_series_limit_rows_dropped_total (cardinality limiter docs).

Set the limits about 50% above today’s numbers, so they catch an explosion without blocking normal growth.

8. Size retention from bytes per sample

The Prometheus storage docs give the formula:

needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample

They also say Prometheus stores “an average of only 1-2 bytes per sample”. Measure yours instead of assuming it:

# Prometheus: bytes per sample written by compaction over the last day
rate(prometheus_tsdb_compaction_chunk_size_bytes_sum[1d]) / rate(prometheus_tsdb_compaction_chunk_samples_sum[1d])

# VictoriaMetrics single-node: bytes per stored sample
sum(vm_data_size_bytes{type=~"storage/.*"}) / sum(vm_rows{type=~"storage/.*"})

RESTIA’s production VictoriaMetrics measured 0.26 bytes per sample at about 2,300 samples per second, from about 68,000 active series. Its 13-month retention is sized from that: 2,300 × 86,400 × 395 days × 0.26 bytes is about 20 to 21 GB of samples, plus about 1 GB of index, so about 22 GB. That is a projection, since 13 months of data do not exist yet. The metrics database runs within 2 GiB of RAM and 1 CPU. Your bytes per sample will differ: it depends on how compressible your values are and on churn, so measure it on your own data.

Then add headroom. VictoriaMetrics asks for at least 20% free space on the data volume for background merges (capacity planning docs), and growth needs room on top. The retention settings themselves are one flag:

# Prometheus: keep 13 months, capped at 40 GiB (Prometheus reads GB as GiB), whichever limit hits first
prometheus --storage.tsdb.retention.time=395d --storage.tsdb.retention.size=40GB

# VictoriaMetrics: a bare number is months, so this keeps 13 months
victoria-metrics -retentionPeriod=13

Open-source VictoriaMetrics applies one retention per instance; per-series retention filters are an Enterprise feature. If raw data only needs weeks and aggregates need a year, run two stores, or use the remote-write pattern from step 5.

9. Logs: drop by level, sample the noise

Log cost follows bytes ingested and retained. A collector in front of the log store is the cheapest place to cut, because it decides before anything is written. This OpenTelemetry Collector config drops debug logs and health checks, keeps every warning and error, and keeps 20% of the remaining logs:

# otelcol.yaml (OpenTelemetry Collector v0.163, contrib or k8s distribution)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  filter/drop-debug:
    error_mode: ignore
    logs:
      log_record:
        - log.severity_number != SEVERITY_NUMBER_UNSPECIFIED and log.severity_number < SEVERITY_NUMBER_INFO
        - IsMatch(log.body, "GET /(healthz|readyz|livez)")
  transform/keep-warnings:
    error_mode: ignore
    log_statements:
      - set(log.attributes["sampling.priority"], 100) where log.severity_number >= SEVERITY_NUMBER_WARN
  probabilistic_sampler/logs:
    sampling_percentage: 20
    sampling_priority: sampling.priority
    attribute_source: record
    from_attribute: request.id
    fail_closed: false
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 2000
      - name: sample-the-rest
        type: probabilistic
        probabilistic:
          sampling_percentage: 10
  batch: {}

exporters:
  otlp_http/logs:
    endpoint: http://logs.example.internal:4318
  otlp_http/traces:
    endpoint: http://traces.example.internal:4318

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, filter/drop-debug, transform/keep-warnings, probabilistic_sampler/logs, batch]
      exporters: [otlp_http/logs]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp_http/traces]

The probabilistic sampler reads sampling.priority as a percentage for each log record (processor README), so the transform step sets it to 100 on warnings and errors. In a validation run with 700 records through the logs pipeline, 500 info lines with a request.id were cut to 106, all 70 warnings and errors passed, and the debug and health-check lines were dropped.

One gotcha: the sampler takes its randomness from the trace ID or from from_attribute, and fail_closed is true by default. In the validation run, with that default, all 20 records without a request.id were dropped; with fail_closed: false they passed. Set it deliberately.

Collector v0.163 names the HTTP exporter otlp_http; older releases call it otlphttp. Retention for logs works like metrics: 30 days covers most incident timelines, and anything kept for audit can go to object storage.

10. Traces: keep the interesting ones

Tail sampling decides after a trace is complete, so it can keep every trace with an error and every slow one, plus a share of the rest. The tail_sampling block in the config above does that: errors, traces over two seconds, and 10% of everything else. RESTIA uses the same idea, every error trace plus 10% of healthy ones. In a validation run, 1,000 fast healthy traces became 103, and all 25 error traces and all 15 slow ones were kept.

Two operational points from the tail sampling processor README. All spans of a trace must reach the same collector instance, so with more than one replica you need a first tier that routes by trace ID, usually the load-balancing exporter from the contrib distribution. And the processor holds spans in memory for decision_wait, so num_traces and the memory limiter need sizing for your traffic. The OpenTelemetry sampling concepts page explains head versus tail sampling if you are choosing between them.

The checklist

List the top metric names, top labels and churn per job.
Drop metrics nobody queries, checked against dashboards and alerts.
Drop only labels that are constant per target; fix ID labels in the code.
Cut histogram buckets, keeping +Inf and the buckets around your SLOs.
Store per-service aggregates long term and per-pod data short term.
Set scrape intervals per job, with rate() windows at four times the interval.
Add sample_limit, label_limit and series limits, and alert on them.
Measure bytes per sample and size retention and volumes from it.
Drop debug and health-check logs at the collector; sample noisy info logs.
Tail-sample traces: all errors, slow traces and a share of the rest.

If you are still on a SaaS vendor, the same series count shows up as an invoice line instead; why your Datadog bill grows faster than your product shows how. The observability cost calculator compares your current bill with a self-hosted stack, and the Observability Cost Review applies this checklist to your setup in five working days. Here is what the review covers.