How does Prometheus remote_write behave when the receiver is slow or down, and how does federation differ?

A single Prometheus is deliberately a single node with local storage. There are two supported ways to get its data somewhere else, and they behave nothing alike under failure. ## remote_write: a push driven from the log Configure a `remote_write` block and the Prometheus process starts a queue manager for that endpoint. That manager reads the local write-ahead log — the same log used for crash recovery — and streams what it finds to the configured `url` as compressed batches over HTTP. The mechanics that matter under load: - **Sharding.** Samples are distributed across parallel senders. The manager scales the shard count between `min_shards` and `max_shards` based on how far behind it is, so a receiver that is merely slow gets more concurrency thrown at it. - **Batching.** Each shard buffers up to `capacity` samples and sends up to `max_samples_per_send` at a time, flushing early when `batch_send_deadline` elapses. - **Retry with backoff.** Failed sends are retried with a delay growing between `min_backoff` and `max_backoff`. - **A write-time filter.** The block has its own filter stage so you can decline to ship a subset of series without touching what is stored locally. The crucial property is that **remote_write is not in the ingest path**. If the receiver is slow or unreachable, scraping continues, the local TSDB continues, queries against the local server continue to be correct. What degrades is the remote copy's freshness and the sending process's memory, because pending samples and additional shards are held in RAM. That leads directly to the durability bound. The queue reads from the WAL, and the WAL is truncated when blocks are cut and checkpoints are written. If the receiver stays down long enough that truncation passes the point the queue had reached, those samples are gone — they exist in local blocks, but they will never be shipped. The window is therefore roughly "how much WAL you keep", not "forever". The signals to watch are the ones the sending server exposes about itself: - `prometheus_remote_storage_highest_timestamp_in_seconds` minus `prometheus_remote_storage_queue_highest_sent_timestamp_seconds` is the lag in seconds — the single most useful number. - `prometheus_remote_storage_samples_pending` shows the backlog held in memory. - `prometheus_remote_storage_shards` against the configured maximum shows whether the manager has run out of concurrency to throw at the problem. - `prometheus_remote_storage_samples_failed_total` and the dropped counter separate "retrying" from "given up". ## Federation: a pull of the latest values Federation reuses the scrape machinery. A higher-level Prometheus is configured with an ordinary job whose `metrics_path` is `/federate` and whose `params` carry one or more `match[]` selectors; `honor_labels: true` keeps the labels the federated server returns rather than overwriting them with the scrape job's own. ```yaml scrape_configs: - job_name: federate-cellar-sites honor_labels: true metrics_path: /federate params: 'match[]': - '{__name__=~"site:.*"}' static_configs: - targets: ['prom-cellar-eu:9090', 'prom-cellar-us:9090'] ``` The response is one exposition-format payload containing the **most recent sample** of every series matching the selectors, evaluated at the moment of the request. Three consequences follow, and they are what the question is really testing: 1. **Resolution is lost.** Whatever the leaf server stored at fifteen-second granularity arrives at the parent's scrape interval. The intervening samples do not exist upstream and never will. 2. **A missed scrape is a permanent hole.** There is no queue, no retry of past data, no catch-up. If the parent could not reach the leaf for ten minutes, the parent simply has no data for those ten minutes. 3. **It is bounded by one HTTP request.** Matching a large series set means one enormous response the parent must parse inside a scrape timeout, and failing that timeout produces nothing at all rather than a partial result. ## Choosing between them, and the thing federation must never do | | remote_write | federation | remote_read | |---|---|---|---| | Direction | push, out of the sender | pull, into the parent | pull, on query | | Granularity | every sample | latest value per scrape | every sample | | Behaviour when the far end fails | queues, retries, then loses | permanent gap | query fails | | Suited to | long-term storage, a global store | a small aggregated roll-up | occasional access to older data | **Federation must never be used to copy an entire Prometheus's data to a global server.** That misuse is the single most common architectural mistake on this subject: it downsamples silently, it drops data whenever a scrape is missed, it collapses under the series volume, and it leaves the parent one timeout away from having nothing. The legitimate use is small and deliberate — pulling a handful of already-aggregated, cross-service series up to a higher-level view. Everything else, including long retention and a genuine global view, belongs on `remote_write`. `remote_read` completes the picture: it lets the local Prometheus fetch raw series from a remote store at query time and evaluate locally. It can be told to skip data the local server already holds and to fire only for queries carrying particular label matchers, because pulling wide ranges of raw series across the network at query time is exactly as expensive as it sounds.