Your Prometheus pairs keep six weeks locally, but a regulator wants six months and one global view. How would you choose between a sidecar-plus-object-store design and a receive-and-ingest design, and how does each deduplicate the replica pair?

Prometheus is single-node by design, and both long retention and a global view are things it deliberately does not do. Two architectures fill that gap, and choosing between them is a judgement call about where you hold state. The precondition for either is that every Prometheus carries a distinct `external_labels` set in its `global` configuration: those labels identify which server produced a block or a sample, and every deduplication downstream is built on them. ## What each design moves where | | Sidecar plus object store | Receive and ingest | |---|---|---| | How data leaves Prometheus | a process beside it uploads cut blocks | Prometheus remote-writes samples | | Network direction | outbound to a bucket, plus inbound queries | outbound writes into a shared cluster | | Where recent data is queried from | still the Prometheus itself | the ingestion tier's memory | | Where historic data is queried from | a gateway reading the bucket | a gateway reading the bucket | | State you now operate | a bucket, a gateway, a querier, a compactor | all of that plus a replicated write path | | If the long-term tier is down | monitoring and alerting are unaffected | senders queue, then lose, and recent data is gone | | Multi-tenancy | bolted on, per-bucket at best | first-class, because writes are addressed | In the sidecar shape, the sidecar uploads each two-hour block as Prometheus cuts it, a **store gateway** makes the bucket's blocks queryable, a **querier** fans out to the live servers and the gateway and merges the results, and a **compactor** runs against the bucket, doing the merging and downsampling the local servers no longer do. Local compaction must be disabled on any Prometheus whose blocks are uploaded, so the uploader always sees stable, uniform blocks rather than ones rewritten underneath it. In the receive shape, a distributor accepts remote writes, hashes each series onto a set of **ingesters**, and each ingester builds blocks in memory and flushes them to the same object storage. You gain a genuine write API, tenant isolation, and the ability to accept data from networks you cannot reach inbound — and you acquire a replicated, stateful cluster in the path of every sample, with a replication factor to choose, a hash ring to operate, and rollouts that must not lose unflushed data. ## Deduplicating a highly-available pair Two Prometheus servers scraping the same targets produce two near-identical copies with slightly different timestamps. Neither design tolerates that by accident. - **Sidecar design — dedupe on read.** Each server's `external_labels` include a replica identifier as well as the labels that say which cluster it belongs to. Both replicas upload; the querier is told which label distinguishes replicas, and at query time it merges series that are identical apart from that label, preferring one replica's samples and filling from the other where there are gaps. The upside is that a replica outage is invisible — the other replica's blocks cover the hole. The cost is storing two copies, and keeping the bucket compactor from merging two replicas' overlapping blocks in ways you did not intend. - **Receive design — dedupe on write.** Both replicas remote-write, and the ingestion tier runs an availability tracker that elects one replica per cluster as leader and discards the other's samples while the leader keeps sending; when the leader goes quiet it fails over. Only one copy is stored, so the storage bill is halved, but a failover leaves a small seam, and the mechanism depends on both servers carrying agreed cluster and replica labels. ## How to actually decide 1. **Can you operate a stateful distributed system on the ingest path?** If the answer is honestly no, the sidecar design is the one that fails safely: a bucket outage costs you history, not alerting. 2. **How many tenants?** One platform team with a handful of clusters does not need addressed writes; dozens of teams needing isolation and per-tenant limits is what the receive design was built for. 3. **Which way does the network go?** Clusters you cannot reach inbound — a customer site, a restricted environment — can push, but cannot be queried by a central querier reaching into them. That constraint often decides it. 4. **What blast radius will you accept?** Sidecar keeps failure in the read path; receive puts it in the write path, where an outage means data that was never stored anywhere. 5. **What does it cost?** Object storage is cheap; ingester memory and the replication factor multiplying it are not. ## What six months of evidence actually demands Long retention for a regulator is not the same problem as long retention for a dashboard. Beyond keeping the blocks, you owe three things. **Downsampling**, because nobody queries six months at raw resolution — the compactor writes reduced-resolution copies alongside the raw blocks and the querier picks the right one for the query's step. **A retention policy expressed per resolution**, so raw data can age out while the coarse copies survive the full window. And **a story about the bucket itself**: versioning or object-lock so the evidence cannot be quietly deleted, and a restore you have actually rehearsed rather than assumed. The failure mode to name out loud is the tempting one: bigger local disks. That gives no global view, does not survive the server, and moves restart and compaction cost in exactly the wrong direction.