Storage and Operations

Prometheus stores metrics in its own embedded time-series database rooted at the directory given by `--storage.tsdb.path` (`data/` by default). There is no external database and no clustering: one Prometheus process owns that directory exclusively, and almost everything about operating Prometheus at size follows from that fact. ## Two destinations for every sample When a scrape completes, each sample — a metric name plus its label set, one timestamp and one value — goes two places at once. 1. **The write-ahead log.** The sample is appended to a segment file under `wal/`. The WAL is append-only and sequential, so writing to it is cheap. It exists only so that an unclean shutdown does not lose acknowledged data; **no query ever reads the WAL**. 2. **The head.** The same sample is appended to the head, the mutable, most-recent part of the database that lives in memory. Every query over recent time is served from the head. Inside the head, samples for one series accumulate into a chunk. A chunk is closed when it reaches its sample limit or when its time range elapses; closed head chunks are written out and memory-mapped from `chunks_head/`, so the process keeps chunk metadata resident but not every raw value. This is the reason head memory tracks the count of **active series** far more strongly than it tracks the scrape interval: each series carries fixed per-series overhead whether it is sampled every fifteen seconds or every minute. ## Cutting a block, then compacting it The head only covers the most recent window. Roughly every two hours Prometheus cuts the oldest completed range of the head into a **persistent block** — a directory named with a ULID containing: | Entry | What it holds | |---|---| | `chunks/` | the compressed sample data, in numbered segment files | | `index` | the inverted index from label pairs to series, and from series to chunks | | `meta.json` | the block's time range, series and sample counts, and its compaction level | | `tombstones` | ranges marked for deletion but not yet physically removed | A block is **immutable**. Nothing is written into it after it is cut. That is what keeps the design simple: readers never take a lock, and a block can safely be copied or uploaded elsewhere while Prometheus is running. Once a block is durable on disk, the WAL no longer needs the records that produced it. Prometheus writes a **checkpoint** beside the segments — a compacted directory holding the series records and any samples still required — and deletes the segments it supersedes. That truncation is the only thing that stops the WAL growing without bound, and it is why WAL size is roughly a function of ingest rate over the last couple of hours rather than of retention. A background **compactor** then merges adjacent blocks into wider ones. Compaction is not tidying for its own sake; it pays for itself three ways: - One index over a wide range is far cheaper to search than many small indexes covering the same range. - Series that were deleted, or that simply stopped being scraped, drop out of the merged index instead of being carried forward in every small block. - Retention can then be enforced by deleting whole blocks, which is a directory removal rather than a rewrite. The compactor writes the merged block first and removes the sources afterwards, so a volume sized to exactly the steady-state data will eventually wedge with no room to compact. ## What a restart actually costs On start-up Prometheus reloads the memory-mapped head chunks and replays the remaining WAL segments to rebuild the head exactly as it was. Only when that finishes does the server begin scraping and answering queries. Replay time is driven by how much data the head holds, which in turn is driven by active series count — so the servers that most need to be restarted quickly are the ones slowest to come back. Practical consequences engineers are expected to draw from this: - **Restarts, not writes, are the expensive operation.** A config reload is cheap; a process restart on a multi-million-series server is minutes of blindness. - **Run replicas, and never restart them together.** Two identically configured servers scraping the same targets let you take one down without a gap. - **Give the volume headroom.** WAL, memory-mapped head chunks, blocks and compaction output all share one filesystem. - **`--storage.tsdb.wal-compression` trades a little CPU for smaller WAL segments**, which means fewer bytes to read back on replay. - **The head is where cardinality hurts first.** A sudden flood of new series inflates head memory immediately, and inflates replay time on the next restart. Because a block is a self-contained, immutable directory, it is also the unit that every long-term storage design for Prometheus builds on: uploading blocks to object storage is possible precisely because nothing will ever rewrite them in place.