When writing a Prometheus exporter for a system you cannot instrument, what work belongs on the scrape path?

seniornice to knowfreq 34%

answer

Do it when asked, while that stays cheap
The cost scales with readers, not users
All or nothing at the worst moment
A cache has to declare its own age
up means the translator answered, nothing more

basics

~20 s

Fetch on the scrape path while it stays cheap, so every scrape returns current state. If the fetch is expensive, refresh it on the exporter's own background timer, serve the cached snapshot, and expose whether the last refresh succeeded and when.

solid answer

~50 s

A Prometheus exporter is a separate HTTP server that translates another system's state into metrics when scraped. The default is to fetch on the request: the data is genuinely current and the exporter holds no state. Abandon that as soon as the fetch is expensive, because the scrape path is all or nothing — breach scrape_timeout and Prometheus keeps nothing from that scrape, exactly when the monitored system is struggling — and because the cost multiplies by the number of readers, not by user traffic: two servers, a replica pair and one ad-hoc request are four independent queries. The alternative is a background refresh on the exporter's own timer serving a cached snapshot, which then has to declare its own health: a gauge for whether the last fetch reached the backend, a last-success timestamp, fetch duration and an error counter. up only ever means the exporter answered.

A Prometheus exporter is a separate HTTP server that translates another system's state into metrics when scraped. The default is to fetch on the request: the data is genuinely current and the exporter holds no state. Abandon that as soon as the fetch is expensive, because the scrape path is all or nothing — breach `scrape_timeout` and Prometheus keeps nothing from that scrape, exactly when the monitored system is struggling — and because the cost multiplies by the number of readers, not by user traffic: two servers, a replica pair and one ad-hoc request are four independent queries. The alternative is a background refresh on the exporter's own timer serving a cached snapshot, which then has to declare its own health: a gauge for whether the last fetch reached the backend, a last-success timestamp, fetch duration and an error counter. `up` only ever means the exporter answered.

code

pseudocode · 17 lines

pseudocode

# background: owns its own cadence, never triggered by a scrape
every 60s:
    start = now()
    try:
        snapshot = backend.fetch_fleet_status()   # ~6.8s
        cache.replace(snapshot)
        backend_up.set(1)
        last_success_timestamp.set(now())
    except Error as e:
        backend_up.set(0)
        fetch_errors.increment(kind = classify(e))
    fetch_duration.observe(now() - start)

# scrape path: reads memory only, always fast
on GET /metrics:
    render(cache.current(), backend_up, last_success_timestamp,
           fetch_duration, fetch_errors)

go deeper

for a junior

Know what an exporter is for: a separate process that turns something Prometheus cannot scrape into something it can. Being able to name a couple of off-the-shelf ones and say why you would not write your own first is enough.

for a middle

Explain the two shapes — fetch on request versus background refresh with a cache — and what each costs. Be ready to say why an expensive query on the scrape path risks losing the whole scrape rather than part of it.

for a senior

Demonstrate the operational judgement: cost multiplied by readers, the feedback loop where a slow backend erases its own monitoring, and the health signals a cached exporter owes its consumers beyond the up series.

for a principal

Own the conventions across teams: when a bespoke exporter is justified at all, the health signals every one must expose, and how you stop a fleet of hand-written translators from becoming an unowned tier nobody can debug.

What a Prometheus exporter is, exactly

A Prometheus exporter is a small HTTP server that Prometheus scrapes, whose job is to translate some other system's current state into metrics. It is the opposite direction from the exporter component inside an instrumented application that ships telemetry out to a backend: this one sits still and waits to be read.

You write one when the system you care about cannot be instrumented — a database, a message broker, a network appliance, a vendor product, a legacy service nobody will recompile. Its whole contract is: answer an HTTP request with the current numbers, correctly labelled.

Fetch on the scrape path, until you cannot

The default shape is to do the work when asked. On each request the exporter queries the underlying system, converts what it gets into metrics, and writes the response. That is preferable when it is affordable, because:

Every scrape returns genuinely current state, with no cache to reason about.
There is exactly one notion of freshness — the scrape timestamp — so nobody has to ask how old the numbers are.
The exporter holds no state between requests, which makes it trivially restartable and horizontally boring.

The reason to abandon it is cost. If the underlying query is expensive — an enumeration API call, a heavy aggregate over a large table, an appliance that takes seconds to answer — the scrape path becomes the wrong place for it.

Why an expensive scrape path is a trap

It is all or nothing, at the worst moment. Exceed the scrape timeout and Prometheus keeps nothing from that scrape. You lose the data precisely when the monitored system is slow, which is the only time you wanted it.
It multiplies by readers, not by traffic. Two Prometheus servers, a high-availability pair, a staging instance somebody pointed at production, and an engineer running an ad-hoc request are five independent readers. A ten-second query on a fifteen-second interval is not one query per fifteen seconds.
It builds a feedback loop. The monitored system slows down, the exporter slows down with it, scrapes start failing, and the series you need to explain the slowdown stop existing. Monitoring should degrade more gracefully than the thing it monitors.
Requests can stack. Without a concurrency guard, overlapping scrapes mean overlapping expensive queries, which is how an exporter turns a slow backend into an unavailable one.

The wind-farm maintenance planner has exactly this shape: its site historian answers a fleet-status query in about 6.8 seconds. On a fifteen-second interval with two Prometheus replicas, putting that query on the scrape path means the historian answers it roughly every seven and a half seconds forever, and every one of those scrapes is a coin flip against a ten-second timeout.

The background-refresh shape

When the fetch is expensive, decouple it:

Refresh on the exporter's own timer, at a period you choose, independent of any scraper's interval.
Serve the last good snapshot from memory on every scrape, which is then fast and predictable.
Guard the refresh so only one runs at a time, and never start one because a scrape arrived.
For anything still done inline, honour the timeout the scraper advertises in its request header, and return what you have rather than blocking past it.

The tradeoff you have accepted is that the numbers are now up to one refresh period old, and Prometheus cannot possibly know that. So you have to tell it.

What the exporter must expose about itself

Signal	Shape	What it is for
Backend reachability	a gauge, 1 or 0, conventionally named like `<system>_up`	separates "the exporter answered" from "the exporter could reach the system"
Last successful fetch	a gauge holding a Unix timestamp	lets a query compute true staleness at any moment, rather than trusting an age computed at render time
Fetch duration	a gauge or histogram of the backend call	the leading indicator, exactly as scrape duration is for scrapes
Fetch errors	a counter, ideally split by kind	distinguishes a credential problem from a timeout without reading logs

The critical distinction is the first row. The up series Prometheus writes says only that the exporter answered an HTTP request. An exporter serving a three-hour-old cache from a backend it has not reached since breakfast is up at 1, and every dashboard built on it looks fine. Alerting on up alone is the single most common mistake with cached exporters.

Client libraries also expose the exporter process's own runtime metrics for free — memory, CPU, request counts on the metrics handler — and they are worth keeping, because they answer "is the translator itself sick".

Rules that survive contact with production

Expose raw counters, never rates. A rate computed inside the exporter cannot be re-aggregated, cannot be windowed by the query, and is wrong across a restart. Give the query engine the cumulative number.
Do not aggregate across monitored instances. One exporter reporting a cluster sum hides exactly the outlier you are looking for.
Label the series with the monitored system, not the exporter's host. Otherwise every alert names the machine running the translator rather than the thing that is broken.
Do not attach your own timestamps to cached values. Expose a last-success timestamp as a metric instead and let queries do the arithmetic.

## What a Prometheus exporter is, exactly A Prometheus exporter is a small HTTP server that Prometheus scrapes, whose job is to translate some other system's current state into metrics. It is the opposite direction from the exporter component inside an instrumented application that ships telemetry out to a backend: this one sits still and waits to be read. You write one when the system you care about cannot be instrumented — a database, a message broker, a network appliance, a vendor product, a legacy service nobody will recompile. Its whole contract is: answer an HTTP request with the current numbers, correctly labelled. ## Fetch on the scrape path, until you cannot The default shape is to do the work when asked. On each request the exporter queries the underlying system, converts what it gets into metrics, and writes the response. That is preferable when it is affordable, because: - Every scrape returns genuinely current state, with no cache to reason about. - There is exactly one notion of freshness — the scrape timestamp — so nobody has to ask how old the numbers are. - The exporter holds no state between requests, which makes it trivially restartable and horizontally boring. The reason to abandon it is cost. If the underlying query is expensive — an enumeration API call, a heavy aggregate over a large table, an appliance that takes seconds to answer — the scrape path becomes the wrong place for it. ## Why an expensive scrape path is a trap 1. **It is all or nothing, at the worst moment.** Exceed the scrape timeout and Prometheus keeps nothing from that scrape. You lose the data precisely when the monitored system is slow, which is the only time you wanted it. 2. **It multiplies by readers, not by traffic.** Two Prometheus servers, a high-availability pair, a staging instance somebody pointed at production, and an engineer running an ad-hoc request are five independent readers. A ten-second query on a fifteen-second interval is not one query per fifteen seconds. 3. **It builds a feedback loop.** The monitored system slows down, the exporter slows down with it, scrapes start failing, and the series you need to explain the slowdown stop existing. Monitoring should degrade more gracefully than the thing it monitors. 4. **Requests can stack.** Without a concurrency guard, overlapping scrapes mean overlapping expensive queries, which is how an exporter turns a slow backend into an unavailable one. The wind-farm maintenance planner has exactly this shape: its site historian answers a fleet-status query in about 6.8 seconds. On a fifteen-second interval with two Prometheus replicas, putting that query on the scrape path means the historian answers it roughly every seven and a half seconds forever, and every one of those scrapes is a coin flip against a ten-second timeout. ## The background-refresh shape When the fetch is expensive, decouple it: 1. Refresh on the exporter's **own** timer, at a period you choose, independent of any scraper's interval. 2. Serve the last good snapshot from memory on every scrape, which is then fast and predictable. 3. Guard the refresh so only one runs at a time, and never start one because a scrape arrived. 4. For anything still done inline, honour the timeout the scraper advertises in its request header, and return what you have rather than blocking past it. The tradeoff you have accepted is that the numbers are now up to one refresh period old, and Prometheus cannot possibly know that. So you have to tell it. ## What the exporter must expose about itself | Signal | Shape | What it is for | |---|---|---| | Backend reachability | a gauge, 1 or 0, conventionally named like `<system>_up` | separates "the exporter answered" from "the exporter could reach the system" | | Last successful fetch | a gauge holding a Unix timestamp | lets a query compute true staleness at any moment, rather than trusting an age computed at render time | | Fetch duration | a gauge or histogram of the backend call | the leading indicator, exactly as scrape duration is for scrapes | | Fetch errors | a counter, ideally split by kind | distinguishes a credential problem from a timeout without reading logs | The critical distinction is the first row. The `up` series Prometheus writes says only that the exporter answered an HTTP request. An exporter serving a three-hour-old cache from a backend it has not reached since breakfast is `up` at 1, and every dashboard built on it looks fine. Alerting on `up` alone is the single most common mistake with cached exporters. Client libraries also expose the exporter process's own runtime metrics for free — memory, CPU, request counts on the metrics handler — and they are worth keeping, because they answer "is the translator itself sick". ## Rules that survive contact with production - **Expose raw counters, never rates.** A rate computed inside the exporter cannot be re-aggregated, cannot be windowed by the query, and is wrong across a restart. Give the query engine the cumulative number. - **Do not aggregate across monitored instances.** One exporter reporting a cluster sum hides exactly the outlier you are looking for. - **Label the series with the monitored system, not the exporter's host.** Otherwise every alert names the machine running the translator rather than the thing that is broken. - **Do not attach your own timestamps to cached values.** Expose a last-success timestamp as a metric instead and let queries do the arithmetic.

Why expose a last-success timestamp rather than an age in seconds?
Because an age computed inside the exporter is only true at the instant it is rendered, and it is meaningless in historical data — every stored sample says how stale things were then, in a form nothing can recompute. A timestamp is a fact that does not decay: a query subtracts it from the evaluation time and gets the real staleness at any point in the past.
The exporter is running but cannot reach the system it translates. What should Prometheus see?
The `up` series at 1, because the exporter answered the HTTP request perfectly well, and the exporter's own backend-reachability gauge at 0. That pair is the whole point of the second signal. An alerting setup that watches only `up` will show a green estate while the exporter serves an increasingly ancient cache, which is the classic silent failure of this design.
Should one exporter serve many monitored instances, or should you run one each?
Prefer one per monitored instance where deployment allows, so a failure costs you visibility of one thing and the identifying labels fall out naturally. When one process must cover many, the discipline is that each series carries labels identifying the monitored system rather than the exporter's own host, and that you accept the exporter as a shared failure domain and alert on its health explicitly.

A translator standing beside a machine that speaks no common language: fast questions he answers live, slow ones he researches in advance — and then he must say out loud when he last spoke to the machine.

saying these in an interview costs you the question

Runs an expensive backend query inline on every scrape
Assumes an up value of 1 means the monitored system is reachable
Computes rates or percentages inside the exporter
Caches results without exposing whether the cache is fresh
Labels series with the exporter's host instead of the monitored system
Thinks an exporter pushes its data to Prometheus

When writing a Prometheus exporter for a system you cannot instrument, what work belongs on the scrape path?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~5 min

What a Prometheus exporter is, exactly

Fetch on the scrape path, until you cannot

Why an expensive scrape path is a trap

The background-refresh shape

What the exporter must expose about itself

Rules that survive contact with production

▾common follow-ups3 questions

▾analogy~20 s

▾red flags6 flags

What a Prometheus exporter is, exactly

Fetch on the scrape path, until you cannot

Why an expensive scrape path is a trap

The background-refresh shape

What the exporter must expose about itself

Rules that survive contact with production

answer

go deeper