How does Prometheus find its scrape targets, and what does a service-discovery mechanism hand it besides a list of addresses?

Prometheus scrapes exactly what its configuration tells it to scrape. In an estate where processes are replaced hourly, nobody can maintain that list by hand, so the configuration describes **how to find** targets rather than naming them. Service discovery is the part of the server that turns that description into a live set of things to poll, and it re-runs continuously rather than once at startup. ## What a scrape job actually contains Each entry in `scrape_configs` has a `job_name`, options that govern the HTTP scrape itself, one or more **discovery mechanisms**, and usually a `relabel_configs` list. The discovery mechanisms are the plural blocks: - `static_configs` — a literal list of addresses written in the config file. - `file_sd_configs` — a set of file globs; the files hold target lists and are re-read when they change and on a `refresh_interval`. - `kubernetes_sd_configs`, `ec2_sd_configs`, `gce_sd_configs`, `consul_sd_configs`, `dns_sd_configs` and others — mechanisms that ask a platform what currently exists. A single job may list several mechanisms, and their results are combined into one target set for that job. ## A target is a label set, not an address This is the point most candidates miss. Discovery hands Prometheus a **label set per target**, and the address is simply one of the labels. A handful of reserved names control the scrape: | Label | What it controls | |---|---| | `__address__` | The host and port the scrape request goes to | | `__scheme__` | `http` or `https` | | `__metrics_path__` | The path requested, `/metrics` by default | | `__param_<name>` | A URL query parameter added to the request | | `__meta_*` | Read-only metadata supplied by the discovery mechanism | The `__meta_*` labels are where the mechanism describes what it found: `__meta_kubernetes_namespace` and `__meta_kubernetes_pod_name` from a cluster, `__meta_ec2_instance_id` and `__meta_ec2_tag_<key>` from a cloud API, `__meta_filepath` from a file-based mechanism. They exist so that relabeling has something to decide on. Two consequences follow. First, every label whose name still begins with `__` is removed once target relabeling has finished, so metadata is invisible on the stored series unless a rule copies it into an ordinary label. Second, if you never set `instance` yourself, it defaults to the value of `__address__`, and `job` defaults to the job's `job_name`. That is why two targets discovered at the same host and port collapse into the same identity unless you distinguish them. ## File-based discovery versus an API-backed mechanism Both produce the same kind of output; they differ entirely in operations. | | File-based | API-backed | |---|---|---| | Where truth lives | Files a generator writes | The platform's own API | | How change is noticed | File change plus a refresh interval | A watch or a poll against the API | | Credentials needed | None beyond filesystem access | Read permissions on the platform | | Failure mode | Stale file, silently old target list | Discovery stalls when the API is unreachable | | Who can change it | Whatever writes the files | Anyone who can create objects in the platform | | Reviewability | The file can be committed and diffed | Emergent, and only observable after the fact | File-based discovery is not a beginner's fallback. It is the standard adaptor for anything the server cannot query directly: hardware, appliances, a partner's endpoints, or an inventory that already exists in a configuration-management system. You point a job at a glob, have a generator write the files, and the two systems stay decoupled. ## Operating it 1. **Look at what discovery produced, not at what you expected.** The server's `/service-discovery` page lists discovered targets with their labels before and after relabeling, including the ones that were dropped; `/targets` shows only what survived. 2. **Know what needs a reload.** Editing the configuration file needs a `SIGHUP` or a POST to `/-/reload` when the server was started with `--web.enable-lifecycle`. Changing the *contents* of a file-based target file needs nothing — that is the whole point of the mechanism. 3. **Write target files atomically.** Write to a temporary name and rename it into place, so a half-written file is never read mid-update. 4. **Count your watches.** Every job with its own API-backed mechanism is another consumer of that API, and a large configuration can put real load on it. ## Where the boundary sits Keep three concerns separate when you answer. Discovery answers *what exists right now*. Relabeling answers *which of those do we scrape, and what do we call them*. The scrape itself — how often, how long it may take, what the endpoint returns — is a third concern that is configured on the same job but is not part of discovery at all.