A team adds a label for user_id to a Prometheus-style metric so they can slice request latency per user, and their metrics backend falls over within a day. Using the concept of cardinality, explain what went wrong, and contrast how structured logging would have handled the same per-user breakdown safely.

## What cardinality is Cardinality, in the context of metrics, is the number of distinct combinations of label values that occur for a given metric name. A metric like `http_request_duration_seconds` with labels method, roughly 5 values, and status_code, roughly 10 to 20 common values, has a cardinality in the dozens: each combination like method equals post, status_code equals 500 is a distinct time series that the metrics backend, such as Prometheus, stores, indexes, and continuously appends new data points to. This is a deliberate design point of time-series databases like Prometheus: they are built assuming the total number of series is bounded and reasonably stable over time, and they pre-allocate memory structures, in Prometheus's case an in-memory index of every active series kept for fast lookup, proportional to that number. ## The explosion The failure mode is what happens when a label takes on a large, effectively unbounded number of distinct values, most commonly - a user ID, - an email address, - a session token, - a request ID, - or a raw IP address. Adding `user_id` as a label on a metric with, say, 2 million distinct users active over a retention window doesn't add 2 million data points, it adds 2 million entirely new, independently indexed time series, each with its own memory overhead for its label set and its own chunk of storage for its samples over time, even for users who only ever generated one data point. This is the cardinality explosion: the total number of series a metrics backend must track can go from a few thousand to tens of millions almost overnight, which - exhausts memory, since Prometheus keeps its full series index in RAM, - slows down every query across the whole system, not just queries touching the offending metric, since the index itself becomes the bottleneck, - and in the worst case causes out-of-memory crashes or ingestion backpressure that drops data for every metric, not just the misconfigured one. ## Why metrics are built to break this way The reason metrics systems are built this way, rather than being able to handle arbitrary-cardinality labels cheaply, is a direct trade-off for what makes metrics fast: pre-aggregated, indexed time series support extremely fast range queries and dashboards, sub-second queries over months of data, precisely because the storage engine can assume a bounded, well-indexed set of series and doesn't need to scan raw events at query time. That assumption is what breaks when cardinality explodes. ## How structured logging handles the same need Structured logging solves the same underlying need, per-user latency breakdown, through a fundamentally different storage model, which is why it doesn't share this failure mode. A structured log line is a self-contained JSON, or similarly structured, event: fields like `timestamp`, `service`, `user_id`, `latency_ms`, `status_code`. A log backend like Elasticsearch, Loki, or Splunk stores each event as its own document and builds inverted indexes over the field values, but it does not need to pre-create a persistent aggregated series for every distinct `user_id` value the way a metrics time-series database does; it can efficiently search, filter, and aggregate over fields at query time instead of pre-aggregating at write time. Asking what the latency distribution is for a specific user over the last day is answered by filtering and aggregating matching log events on demand, which log-oriented storage is built to handle even when the field being filtered on has millions of distinct values, at the cost of that per-query aggregation being slower and more resource-intensive at query time than a pre-computed metric would be. ## Where the work happens | | Metrics | Logs | |---|---|---| | Aggregation | at write time | at query time | | Query | cheap, but only along axes decided in advance and only safely for low-cardinality dimensions | flexible for any field including high-cardinality ones, but comparatively more expensive at scale | The trade-off, then, is not that logs are simply better at this, but that logs and metrics sit at different points on the write-cost versus query-cost spectrum: metrics do the aggregation work at write time, cheap to query later but only along axes decided in advance and only safely for low-cardinality dimensions, while logs defer aggregation to query time, flexible for any field including high-cardinality ones, but comparatively more expensive to query at scale and not well suited to the kind of always-on alerting dashboards metrics are built for. Distributed tracing spans occupy a related middle ground, carrying high-cardinality attributes per span without exploding a series count, because a trace store also indexes rather than pre-aggregates. ## What it looks like in production Common failure signals in production: - a Prometheus instance whose memory usage grows unboundedly and correlates with new users or new request IDs appearing, rather than with traffic volume; - queries that used to return in milliseconds suddenly taking seconds because the label index has bloated; - and, notoriously, any label sourced from something inherently unbounded, like a raw URL path with embedded IDs instead of a templated route used as a label, which is one of the most common real-world causes of this exact incident. The fix is almost always the same: keep genuinely high-cardinality identifiers, like `user_id`, out of metric labels entirely, and push them into structured logs or trace span attributes instead, where they belong, reserving metric labels for a small, bounded, known set of dimensions.