How do you pick a metric's type at instrumentation time, and what can you never recover afterwards?

## Choose from the question, not from the value The instinct at instrumentation time is to look at what is in front of you — a variable, a queue object — and publish it. That is backwards. Instrumentation is a **recording**, not a query: months later you will ask questions of what you recorded, and never of what you did not. So the decision starts from the question, and the type follows from it. - *How often, or how fast?* — an event happened, so **count** it. A counter's rate answers "per second" at any window you choose later. - *How much, right now?* — a level, so publish a **gauge**, and accept that nothing between two observations is preserved. - *What does the slow end look like?* — you need the shape of many observations, so record a **distribution**. Nothing else can answer it after the fact. - *What is the typical value?* — a mean is enough, and a mean is two counters: a running sum and a count. Far cheaper than a distribution, and legitimate when nobody will look past the typical case. | The question you will ask later | Type to record | What it costs | |---|---|---| | How many, how often, how fast | Counter | One series; cheapest thing you can publish | | How much right now, how deep, how full | Gauge | One series; blind between observations | | What the slow end looks like | Distribution | A family of series per publishing process | | What the typical value is | Two counters (sum and count) | Two series; no shape | ## What is reversible, and what is not This is the part that makes the decision worth a minute of thought rather than a reflex: 1. **The dimensions you attach are reversible.** You can add or remove them, and while that costs storage and produces a discontinuity in the affected series, the underlying measurement survives the change. 2. **Names, dashboards and alert thresholds are reversible.** They sit on top of the recording and can be rewritten at any time. 3. **What you never recorded is gone.** A gauge observed every twenty seconds has already discarded everything that happened in between; no query, no backfill and no reprocessing recovers it. If the level spiked and recovered inside one interval, that spike never existed as far as the store is concerned. 4. **Data already collected keeps the shape it was recorded with.** Changing an instrument improves data from the change forward, never behind it, so today's decision is frozen for the length of your retention window. That asymmetry is why experienced engineers over-instrument slightly at the start: the marginal cost of one extra counter is small and knowable, and the cost of discovering mid-incident that the question cannot be answered is not. ## Changing a type on a live metric name Sometimes you were simply wrong. Changing the type **in place, under the same name** is the worst option available: - Every query written against the old type keeps running and silently returns different arithmetic — a rate over something that no longer accumulates, or a raw reading of something that now climbs forever. - During a rolling deploy both shapes are published under one name at once, so the series is internally inconsistent for as long as the rollout lasts. - The history becomes uninterpretable, and alerts built on it either stop firing or start flapping. The safe migration is to publish the new instrument under a **new name**, run both long enough to cover the retention window that matters, move dashboards and alerts across deliberately, then retire the old one. It is more work, and it is the only version that leaves the history readable. ## A worked example A seed-catalogue ordering service published its pending-order backlog as a gauge, observed every twenty seconds. During a 5,400-request-per-second peak the service degraded, and nobody could explain the incident afterwards from the existing dashboards: the backlog panel showed a maximum of 812 across the whole event, well under the alert threshold of 2,000, yet customers reported minutes of failed checkouts. The gauge was not lying — it was answering a different question from the one being asked. Every observation had genuinely found the backlog under 812; what nobody could see was whether it had climbed far higher and drained in between, because a gauge records nothing between samples. Had the same code also published enqueued and dequeued counters, their rates would have shown the backlog's true growth at any resolution, and the peak would have been visible to a query written after the incident. That would have cost two series per process. ## The cheap insurance - **Count the events, always.** A counter beside a gauge is one extra series and it survives missed observations, which makes it the most retrospectively useful thing on the list. - **Record a distribution only where the shape will really be read**, and count the rest — the family multiplies per publishing process, so this is where a budget goes. - **Write down what question each instrument answers**, next to it in the code. It is the only artefact that makes the choice reviewable by the next person. - **Never publish a value you have already divided** if you will want it aggregated later; publish the parts and divide at query time.