Percentiles and Cardinality: The Two Numbers That Run Your Bill

Latency data is brutally skewed. Most requests are fast, a few are slow, and a handful are catastrophically slow. Averaging that shape produces a number that describes nobody.

Take ten requests: nine complete in 100 ms, one takes 4 seconds. The average is 490 ms, a latency no actual request experienced. Meanwhile the summary "average 490 ms" both understates the disaster, someone waited 4 seconds, and slanders the healthy majority, who got 100 ms.

Percentiles describe the distribution instead of collapsing it. The p50, the median, is the experience of the typical request. The p95 and p99 are the experience of the unluckiest 5 and 1 percent. "p50 of 100 ms, p99 of 4 s" tells the true story the average buried.

Key idea: the tail is not an edge case; it is your best customers. The users who hit p99 most often are the ones making the most requests, with the fullest carts and the biggest accounts. High-traffic pages compose many backend calls, so one slow dependency in a hundred touches almost every page load. Amazon and Google engineers have written for years about tail latency for exactly this reason.

The most common percentile mistake is not measurement but arithmetic done afterwards.

The correct machinery is the histogram: count observations into latency buckets, which can be summed across services, hosts and time windows, because counts add even though percentiles do not. Then compute the percentile once, at query time, from the merged buckets. This is precisely why Prometheus-style systems ship histograms and compute quantiles in the query language rather than letting services report their own p99s upward.

To understand metric cost you need one definition exactly right.

Definition: a time series is one uniquely-labelled stream of values: the metric name plus one specific combination of label values. http_requests_total{route="/checkout", method="POST", status="500"} is one series. Change any label value and you have a different series.

The storage engine keeps every series it has ever seen in the retention window: an index entry, an in-memory presence, compressed chunks on disk. The number of concurrent series, the cardinality, is the real capacity dimension of a metrics system, the thing vendor pricing tiers and self-hosted memory limits are actually measuring.

The multiplication is where intuition fails. Cardinality is the product of each label's distinct values:

A metric with 30 routes, 5 methods, 8 status codes and 50 pods is not 93 of anything: it is up to 30 times 5 times 8 times 50, which is 60,000 series, for one metric name. Add one more label and you multiply again. This is why cardinality problems arrive suddenly: growth is multiplicative, and each individual label looked innocent.

Every observability team has a version of this story, so it is worth running the arithmetic before living it.

The rule that falls out: label values must come from small, closed sets you can enumerate. Routes, methods, status classes, regions, yes. Anything unbounded belongs in logs and traces, which are built to carry per-event identity, and the correlation the engineer wanted is exactly what exemplar links from histogram buckets to traces exist for.

Cardinality is manageable the way any resource is: with a budget, an audit, and controls at the border.

Two of these deserve emphasis. Route templating is non-negotiable and easy to get wrong in one forgotten handler, one 404 logger recording raw paths has melted many a metrics cluster. And the audit matters because cardinality regressions ship silently: a deploy adds a label, nothing breaks that day, and the series count compounds with every new deployment target until the bill arrives. Treat series count as a metric about your metrics, alert on its growth rate, and the explosion becomes a code review comment instead of an incident.

The two halves of this lesson collide inside the histogram, because buckets are labels too.

A classic Prometheus histogram stores one series per bucket boundary, per label combination. Give a latency histogram 12 buckets and attach it to the 60,000-series metric from earlier and you are storing 720,000 series before anyone asks a question. Choosing bucket boundaries is therefore a real design act: too few buckets and your p99 estimate is coarse, since classic histograms interpolate within buckets; too many and you multiplied your whole cardinality by the bucket count.

Practical guidance that has aged well:

• Spend buckets where decisions live: near your SLO thresholds. An SLO at 300 ms wants boundaries like 250, 300, 350 ms, not a smooth geometric ladder that skips the region.
• Keep bucket sets consistent across services, or cross-service aggregation quietly degrades.
• Prefer native or exponential histograms where the stack supports them: they encode the distribution in a sparse, scale-invariant form that removes the manual boundary-picking and most of the bucket-cardinality tax.

The theme of the whole lesson in one line: every piece of measurement precision is bought with series, and good telemetry design is deciding where precision is worth it.

Put the mechanics together and some professional reading habits fall out.

• Always plot p50, p95 and p99 together. The gaps between them are diagnostic: a rising p99 with a flat p50 means a tail problem, retries, one slow shard, a garbage-collection pause, lock contention. Everything rising together means a systemic slowdown, saturation or a downstream dependency.
• Distrust improvements during incidents. If timeouts are killing the slowest requests, they never complete, never get measured, and every percentile improves. Latency charts must be read next to error rates, always.
• Check the sample count. A p99 computed from forty requests is four requests' worth of information; at low traffic, percentile charts are noise wearing a suit.
• Window with care. A five-minute p99 of 2 seconds that never appears in the one-hour p99 is not a contradiction; it is averaging over time doing to your incident what averaging over requests did to your users.

None of this requires tooling: it requires knowing what the numbers are made of. That, plus a cardinality budget, is most of what separates teams whose telemetry answers questions from teams whose telemetry generates invoices.