skip to content

In Prometheus, how does a relabel_configs list transform a discovered target, and what determines the address finally scraped?

level: middleimportance: must knowfreq 64%

answer

  1. an ordered pipeline, not a filter set
  2. the regex is anchored at both ends
  3. three actions read values, three read names
  4. keep and drop are mirror images
  5. whatever __address__ holds at the end wins

basics

~20 s

Prometheus applies relabel_configs rules in order, each seeing the label set as the previous rule left it. Rules keep or drop the target, rewrite labels, or copy metadata into real labels. Whatever address holds at the end is the address scraped.

solid answer

~40 s

A rule joins its `source_labels` values with `separator` (`;` by default), matches the result against `regex` (fully anchored, `(.*)` by default), and then acts. `keep` discards the target unless the regex matches; `drop` discards it when the regex matches; `replace` — the default action — writes `replacement`, with `$1`-style capture-group expansion, into `target_label`; `labelmap` matches label **names** and copies their values to new names; `labeldrop` and `labelkeep` remove or retain labels by name; `hashmod` writes a hash of the source values modulo `modulus`, which is how target sets are sharded across servers. The list is a pipeline over an accumulating label set, so order is load-bearing. When it finishes, `__address__`, `__scheme__` and `__metrics_path__` define the request, every remaining `__`-prefixed label is discarded, and `instance` falls back to `__address__`.

code

yaml · 12 lines
yaml
relabel_configs:
  - source_labels: [__meta_kubernetes_namespace]
    action: keep
    regex: touring-logistics
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_pod_name]
    target_label: pod

go deeper

for a junior

Recall that these rules decide which discovered targets are actually scraped and what labels they end up with. Know that keep and drop are opposites and that the rules run in the order written.

for a middle

Explain a rule field by field — source labels joined by a separator, matched by an anchored regex, then acted on — and explain which actions read label values and which read label names.

for a senior

Show you can debug one: read the label set at each stage, recognise an ordering bug, and understand that rewriting the scrape address or the instance label silently renames every series from that target.

for a principal

Decide how much relabeling belongs in a central configuration at all. Weigh a small reviewable pipeline everyone understands against clever rules that only their author can change, and set the conventions that keep target identity stable over years.

Relabeling is where discovery output becomes a scrape configuration. It is the most powerful part of a Prometheus configuration and the part where most configuration bugs live, because it is an **ordered, mutating pipeline** and almost everyone reads it as a set of independent filters. ## The shape of one rule Every rule in `relabel_configs` is built from the same fields, and most of them have defaults that make short rules readable: | Field | Meaning | Default | |---|---|---| | `source_labels` | Labels whose values are read, in order | none | | `separator` | String joining those values before matching | `;` | | `regex` | Pattern matched against the joined value, anchored at both ends | `(.*)` | | `target_label` | Label the result is written to | none | | `replacement` | Value written, with capture-group expansion | `$1` | | `modulus` | Divisor used by the hashing action | none | | `action` | What the rule does | `replace` | Two defaults cause most surprises. The regex is **fully anchored**, so `regex: freight` does not match `tour-freight-01`; to match a substring you write the wildcards yourself. And when a `replace` rule's regex does not match, the rule is simply a no-op — nothing is written and no error is raised, which is why a broken rule so often looks like nothing happening at all. ## What each action does | Action | Operates on | Effect | |---|---|---| | `keep` | Joined source values | Discards the target unless the regex matches | | `drop` | Joined source values | Discards the target when the regex matches | | `replace` | Joined source values | Writes the expanded replacement into `target_label` | | `labelmap` | Label **names** | Copies matching labels' values to newly named labels | | `labeldrop` | Label **names** | Removes every label whose name matches | | `labelkeep` | Label **names** | Removes every label whose name does not match | | `hashmod` | Joined source values | Writes hash modulo `modulus` into `target_label` | The split in the middle column is the one to have straight in an interview: three actions read label *values*, three read label *names*. `hashmod` is the odd one out and has a single real use — write the hash into a temporary label, then `keep` only the shard number that this server owns, and a fleet of identically configured servers partitions the same discovered target set deterministically. ## The pipeline accumulates Rules run top to bottom over one label set that each rule may change, so: - A `keep` placed after a `labeldrop` that removed the label it tests will match nothing and delete every target. - A `replace` that overwrites `__address__` early makes the original address unavailable to any later rule that wanted it. - Two rules writing the same `target_label` mean the last one wins, silently. - A rule that drops a target ends processing for that target; later rules never see it. This is also why a configuration is not reviewable rule by rule. You have to read it as a program. ## What is actually scraped When the list finishes, the request is assembled from what remains: `__scheme__` and `__address__` and `__metrics_path__`, plus any `__param_<name>` labels as query parameters. Rewriting `__address__` is normal and expected — the canonical example takes a host from discovery and a port from an annotation and composes a new address. After that, every label still beginning with `__` is discarded, `instance` defaults to whatever `__address__` ended up being, and `job` defaults to the job's `job_name`. Everything else that survived is written onto every series from that target. ## Where the bugs are, and how to see them 1. **Read the discovered labels, not the ones you assume.** The `/service-discovery` page shows each target's labels before and after relabeling and lists the targets that were dropped. Nearly every relabeling bug is visible there in seconds. 2. **Check anchoring first.** An unexpectedly empty target list is an anchored regex more often than anything else. 3. **Validate, but do not trust validation.** `promtool check config` catches malformed rules; it cannot know that your third rule deleted the label your fourth rule needed. 4. **Watch for identity changes.** Rewriting `__address__` or setting `instance` renames every series from that target. Old series stop at that instant and new ones begin, and a graph spanning the change shows a gap rather than an error. 5. **Order the list deliberately.** Filter first, then compose the address, then map metadata into real labels, then remove what you do not want kept. Written that way, the pipeline reads in the order it executes.

  • You added a keep rule at the end of the list and every target disappeared. What is the likely cause?
    The label it tests no longer exists by the time the rule runs. Rules see the accumulated label set, not the discovery output, so an earlier `labeldrop`, `labelkeep` or overwriting `replace` can remove or change it — and a `keep` on a label that is absent matches nothing, so it deletes everything. The `/service-discovery` page shows the label set at the end of the pipeline, which settles it immediately.
  • What can hashmod do that no other relabeling action can?
    Partition one discovered target set across several servers without coordination. Each server runs the identical configuration, hashes a stable label such as the address into a temporary label modulo the shard count, then keeps only its own shard number. Every target lands on exactly one server, the split is deterministic, and adding a shard reshuffles targets predictably rather than arbitrarily.
  • Is a relabeling regex matched as a substring or as a whole value?
    As a whole value — it is anchored at both ends, so it must match everything the source labels joined to. To match a fragment you add the wildcards explicitly on both sides. This is the single most common cause of a rule that appears to do nothing, and of a `keep` rule that unexpectedly empties a job.

saying these in an interview costs you the question

  • Treats the rules as independent filters rather than a sequence
  • Believes the regex matches a substring by default
  • Confuses keep with drop semantics
  • Thinks the discovered address cannot be rewritten before scraping
  • Cannot say where the instance label's value comes from
  • Expects a non-matching replace rule to raise an error