In Prometheus, what happens to a time series when its target stops exposing it, and why does it vanish from an instant query instead of reading zero?

## What a sample is, and what identifies the series it belongs to A Prometheus **sample** is exactly two things: a `float64` value and an `int64` timestamp in milliseconds since the Unix epoch. It carries no labels of its own. The labels belong to the **series**, and a series' identity is its complete label set including the reserved `__name__` label that holds the metric name. `seedorder_dispatch_queue_depth{region="eu-west-3"}` is one series; change any label value and you have a different series with its own independent history. This is why there is no such thing as "updating" a series or "renaming" one. A target that changes a label value stops writing to the old series and starts writing to a new one, and the old series keeps whatever samples it already had until retention removes them. ## What happens when a line stops appearing Suppose a target that has been exposing a line simply stops exposing it — the metric was removed, the code path is gone, or the whole target disappeared from service discovery. Prometheus does not leave the series dangling with its last value. At the scrape where the line is missing, the server appends a **stale marker**: a special value written on that series at that timestamp, which query evaluation understands as "this series has no value from here on". That marker is what turns a disappearance into a clean end rather than a value that lingers. ## Instant queries and the lookback window An instant query asks for the value of every matching series *at one timestamp*. Samples do not usually land exactly on that timestamp, so evaluation looks **backwards** from it for the most recent sample on each series, up to a bounded window — five minutes by default, set by `--query.lookback-delta`. Three outcomes: - there is a real sample inside the window, and its value answers the query; - the most recent sample is older than the window, and the series is simply not in the result; - a stale marker lies inside the window, and the series is not in the result either — the marker suppresses the older real samples behind it. Without stale markers, the second rule alone would carry a dead series' last value forward for the whole lookback window before it faded out. The marker collapses that to the next scrape. | Situation | What an instant query returns | |---|---| | a real sample within the lookback window | that sample's value | | the newest sample is older than the window | the series is absent from the result | | the target stopped exposing the line | absent from the stale marker's timestamp onward | | the target was removed from discovery | its series go stale, and so does the synthetic `up` series for it | | the series never existed | absent | ## Why absent rather than zero Because Prometheus stores no default value and knows nothing about what a series *should* contain. Filling in a zero would be inventing data, and for most metrics it would be a lie in the most dangerous direction: - A missing cumulative counter is not "nothing happened". It is "nobody told us", and those two look identical on a graph if you substitute zero. - A missing gauge such as a queue depth is not an empty queue. Zero would be a positive claim of emptiness that no measurement supports. - Substituting zero would make a broken exporter, a crashed process, a firewall rule and a genuinely idle service indistinguishable — which is precisely the distinction an operator most needs. The consequence you feel is that expressions over a vanished series produce **no series at all**, not a series reading zero. A comparison filters nothing because there is nothing to filter, and anything downstream that expected a value has to be written to treat absence explicitly rather than assuming a number will always arrive. ## The case where this bites hardest The seed-catalogue ordering service is rolled out to a new region and the pods come up without their metrics port exposed. Nothing in the metrics stack complains loudly: no series carrying `region="eu-west-3"` ever exists, so every dashboard that groups by region simply shows one fewer line and every threshold expression silently matches nothing. The only telemetry actually arriving from the region is an 18 GB-per-day log stream that nobody is watching, and the gap survives until somebody notices the missing line on a panel. That failure mode — *never instrumented* rather than *instrumented and broken* — is invisible to anything built on the series themselves, because you cannot alert on the absence of a series you have never seen by looking at that series. Catching it needs a check written against what you expect to exist: a comparison against the inventory of what should be reporting, or a rule that asserts a series is present rather than one that asserts its value is high. ## The short version to say out loud A sample is a value and a millisecond timestamp on a series identified by its full label set. When a target stops exposing a series, Prometheus writes a stale marker and evaluation stops resolving it. Absence is a first-class state, distinct from zero, and every query and alert you write has to decide what it means.