Why does recording a latency distribution create a family of time series while a counter creates one?

## One instrument is not one time series An **instrument** is the thing you add to the code: one call site that records something. A **time series** is what the store keeps: one stream of timestamped values under one identity. Engineers estimating the cost of instrumentation almost always count instruments, and the store bills them for series. For a counter or a gauge the two numbers coincide — one instrument, one series, before any dimensional label is attached — so the habit survives right up to the moment somebody records a distribution, at which point it is off by an order of magnitude. ## Where the extra series come from A distribution has to answer questions about the *shape* of many observations, and a single number per timestamp cannot carry a shape. To make the recorded shape combinable across processes, the client publishes a **family** of series rather than one: - one **cumulative count series per configured upper boundary**, each carrying how many observations so far fell at or below that boundary; - one series carrying the **total number of observations**; - one series carrying the **running sum of all observed values**. So an instrument configured with twelve boundaries is stored as fourteen series. The count and the sum are not overhead — the count gives you throughput and the pair gives you a mean — but the boundary series are where the multiplier lives. A representation that publishes pre-computed quantiles instead behaves the same way structurally: one series per published quantile, plus a count and a sum. | Instrument as written in code | Time series stored (before any label) | |---|---| | Counter | 1 | | Gauge | 1 | | Distribution with B upper boundaries | B + 2 | | Pre-computed quantile set of size Q | Q + 2 | The last column is the multiplier that applies **before** dimensions are involved. Every dimension you attach afterwards multiplies the whole family rather than a single series, so a distribution and a counter do not merely differ in cost, they diverge as the instrumentation grows. ## What series count costs a store Series count, not metric-name count and not sample volume alone, is the number a time-series store scales with: 1. **Index.** Every distinct series needs an entry in whatever structure maps an identity to its data, and that structure is typically held in memory for the recent window so that queries can resolve fast. 2. **Write path.** Each series has its own append buffer or head chunk. Fourteen series means fourteen small in-memory structures churning instead of one, and small chunks compress worse than full ones. 3. **Read path.** A query that spans the family touches every boundary series it needs, so the cost of reading a distribution scales with the family too, not only the cost of writing it. 4. **Retention.** Every series is retained independently, so the multiplier is paid again for every day the data is kept. ## A worked example A seed-catalogue ordering service instrumented forty processes. The checkout path had one counter for orders placed and one distribution for checkout latency, the latter configured with twelve boundaries. Counting instruments, that reads as two things. Counting series, it is `(1 + 14) x 40 = 600` — the identity of the publishing process is itself a dimension, so each process contributes its own complete family. When the team then split checkout latency by payment method with six values, they expected to "add one label". They added `14 x 6 x 40 = 3,360` series where the counter beside it grew to 240, and the ingest tier's memory footprint moved accordingly. Nothing had gone wrong, and nobody had done anything foolish; they had simply budgeted using the wrong unit, and during the incident that followed a 5,400-request-per-second peak the store was already running close to its memory ceiling. ## When the family earns its keep None of this is an argument against distributions — it is an argument for knowing what you are buying. - If the only question you will ever ask is *how often*, a counter answers it at one-fourteenth of the storage, and a distribution adds nothing. - If you need a mean, a distribution's count and sum give it to you, but so do two plain counters, and two counters are far cheaper than a full family. - If you need to know what the slow end of the distribution looks like, there is no cheaper option that is honest: the shape has to be recorded at the time, and one number per timestamp cannot carry it. The practical discipline is to estimate before instrumenting. Multiply the family size by the number of publishing processes, then by every dimension's value count, and see whether the resulting series figure is one you are willing to pay for over the retention window. That arithmetic takes a minute and routinely changes the design — most often by reserving distributions for the handful of operations whose shape anyone will actually look at, and counting everything else.