skip to content

What are KeyValues on an Observation, and what is the difference between low-cardinality and high-cardinality KeyValues?

level: seniorimportance: must knowfreq 58%

answer

  1. KeyValue = tag/label pair
  2. low = metric tag + span tag
  3. high = span only, never metrics
  4. IDs are high-cardinality → cardinality explosion
  5. convention supplies both buckets from Context

basics

~20 s

KeyValues are key/value tags you attach to an Observation. Low-cardinality ones (few distinct values, like status) become metric tags and span tags. High-cardinality ones (many values, like user ID) go only on the span, not on metrics.

solid answer

~40 s

`KeyValues` (a collection of `KeyValue` key/value pairs) enrich an Observation with dimensions. There are two buckets. **Low-cardinality** KeyValues — few possible values (HTTP method, outcome, status class) — are added as **both metric tags and span tags**. **High-cardinality** KeyValues — potentially unbounded values (user ID, order ID, full URI) — are attached **only to the trace span**, never to metrics. The split exists because every distinct combination of metric tags creates a separate time series; unbounded tag values cause a **cardinality explosion** that can OOM your metrics backend. Traces store each span individually, so high-cardinality context there is cheap and valuable for debugging a specific request. You add them via `lowCardinalityKeyValue(...)` / `highCardinalityKeyValue(...)` on the Observation, or declaratively through an `ObservationConvention`. Keys should be stable and documented, ideally via an `ObservationDocumentation` enum.

code

java · 26 lines
java
import io.micrometer.common.KeyValues;
import io.micrometer.observation.Observation;
import io.micrometer.observation.ObservationConvention;

// A reusable convention keeps IDs off metrics and on spans.
class OrderObservationConvention
        implements ObservationConvention<OrderContext> {

    @Override
    public KeyValues getLowCardinalityKeyValues(OrderContext ctx) {
        // bounded values -> metric tags AND span tags
        return KeyValues.of("order.type", ctx.getType(),
                            "order.outcome", ctx.getOutcome());
    }

    @Override
    public KeyValues getHighCardinalityKeyValues(OrderContext ctx) {
        // unbounded identifier -> span only, excluded from metrics
        return KeyValues.of("order.id", ctx.getOrderId());
    }

    @Override
    public boolean supportsContext(Observation.Context context) {
        return context instanceof OrderContext;
    }
}

go deeper

for a junior

Knows KeyValues are tags and IDs shouldn't be metric tags.

for a middle

States which bucket lands on metrics vs spans.

for a senior

Explains cardinality explosion and drives the low/high decision from aggregate-vs-search intent.

for a principal

Encapsulates the split in an ObservationConvention + ObservationDocumentation and sets org-wide tagging standards.

## KeyValue and KeyValues A **`KeyValue`** (Micrometer's `io.micrometer.common.KeyValue`) is a single immutable **key/value string pair** — Micrometer's word for a *tag/label/attribute*. **`KeyValues`** is an ordered, de-duplicated collection of them. On an Observation, KeyValues are the **dimensions** that let you slice a metric or filter a span. ## Two cardinality buckets — the central distinction **Cardinality** = the number of distinct values a key can take. 1. **Low-cardinality KeyValues** — bounded, small set of values. Examples: `method=GET`, `status=200`, `outcome=SUCCESS`, `exception=none`. Added with `observation.lowCardinalityKeyValue("status", "200")`. - Used as **metric tags** *and* **span tags**. 2. **High-cardinality KeyValues** — potentially unbounded values. Examples: `userId=...`, `orderId=...`, `uri=/orders/12345`, `sessionId=...`. Added with `observation.highCardinalityKeyValue("orderId", id)`. - Attached **only to the trace span**, **excluded from metrics**. ## Why the split matters (the gotcha that bites people) Dimensional metrics create **one time series per unique tag-value combination**. If you tag a Timer with `userId`, a system with a million users produces a million time series → memory blowup in Micrometer and your TSDB (Prometheus), slow scrapes, huge cardinality bills. This is the classic **cardinality explosion**. Traces don't have that problem: each request's span is stored **individually**, so putting `orderId` on the span is exactly what you want when debugging *one* slow request. Hence: **identifiers and unbounded values → high cardinality (span only); bounded classifiers → low cardinality (metric + span).** Getting this wrong is the most common Observation mistake: never put an ID in a low-cardinality KeyValue. ## How to add them - Fluent on the Observation: ```java Observation.createNotStarted("order.process", registry) .lowCardinalityKeyValue("type", order.type()) // metric + span .highCardinalityKeyValue("order.id", order.id()) // span only .observe(() -> process(order)); ``` - Declaratively via an **`ObservationConvention`** (recommended for reusable instrumentation), whose `getLowCardinalityKeyValues(Context)` and `getHighCardinalityKeyValues(Context)` read data off the typed `Observation.Context`. - Documented via an **`ObservationDocumentation`** enum listing the expected key names for both buckets — this is how Spring documents its built-in observations. ## Naming conventions - Keys are **dot-separated, stable strings** (`http.method`, `order.type`). Don't compute keys dynamically. - Micrometer may transform key style per backend (e.g., dots → underscores for Prometheus). ## Edge cases / gotchas - A KeyValue value must be **non-null**; use a sentinel like `"none"`/`"unknown"` rather than null. - Low-cardinality keys should be a **fixed, known set** — same keys on every Observation of that name (Prometheus dislikes inconsistent label sets). - Adding the *same* key twice: KeyValues de-duplicate by key (last write wins in the builder semantics). - High-cardinality data still costs storage in your tracing backend; be reasonable. ## When to use which - Bounded classifier you'll **group/aggregate** on a dashboard → **low**. - A specific identifier you'll **search a single trace by** → **high**.

  • A teammate tagged a Timer with userId and Prometheus started OOMing. Explain and fix.
    userId is high-cardinality; each distinct user became a separate time series, exploding memory. Move it to a high-cardinality KeyValue so it lands on the span only, and keep the metric tagged with bounded classifiers like outcome/status.
  • Where do low- and high-cardinality KeyValues each actually end up?
    Low-cardinality KeyValues become both metric tags and span tags; high-cardinality KeyValues are added to the trace span only and are deliberately excluded from metrics.

saying these in an interview costs you the question

  • Putting user/order IDs into low-cardinality KeyValues (or plain metric tags).
  • Thinking high-cardinality KeyValues show up on metrics.
  • Using null KeyValue values instead of a 'none'/'unknown' sentinel.
  • Computing KeyValue keys dynamically per request, creating inconsistent label sets.

context

resolving links…