OpenTelemetry metrics offer counters, up-down counters, gauges and histograms, each in a synchronous and an asynchronous (observable) form. How do you choose between them, and what does "asynchronous" actually change?

## Two independent choices Choosing an instrument is two decisions, not one: **what kind of quantity is this** (counter / up-down counter / histogram / gauge), and **how is the value produced** (synchronously at the event, or asynchronously by a callback). Candidates who conflate them end up with an observable histogram-shaped idea that does not exist, or a synchronous gauge where a counter belongs. ## The kinds **Counter** — monotonically increasing. You call `add(n)` with a non-negative `n`. Requests served, errors, bytes written, messages published. You never record the running total yourself; the SDK accumulates. Backends turn counters into rates, which is what you almost always want to graph. Recording a total that can decrease into a counter corrupts rate calculations, because consumers interpret a decrease as a restart. **UpDownCounter** — same additive idea, but `add(n)` accepts negatives. Queue depth, in-flight requests, open connections, cache entries. The distinguishing test is whether the quantity can go down for reasons other than a process restart. **Histogram** — you call `record(value)` and the SDK aggregates values into a distribution. Use it when the interesting questions are "what does the tail look like" — request duration, response size, batch size. A counter of total duration divided by a counter of requests gives you a mean, and means hide exactly the behaviour you are looking for. **Gauge** — a value that is *sampled*, not accumulated: the last observation wins, and there is no meaningful sum across time. Current heap bytes, temperature, a configured queue capacity. A gauge's data point tells you the value at an instant; it says nothing about what happened between observations, which is why a gauge is the wrong instrument for anything you might want to rate or sum. ## Synchronous versus asynchronous **Synchronous** instruments (Counter, UpDownCounter, Histogram, and a synchronous Gauge in recent specification versions) are called from application code at the moment the thing happens. That gives them two properties nothing else has: they can attach attributes derived from the current operation (route, status code, tenant), and they can produce **exemplars** — a recorded measurement can capture the trace and span id active at that moment, letting a dashboard jump from a slow bucket straight into a matching trace. **Asynchronous** (observable) instruments — ObservableCounter, ObservableUpDownCounter, ObservableGauge — are registered with a callback. The SDK invokes it once per collection cycle, and the callback **observes the absolute current value**, not a delta. This is the point people get wrong: an ObservableCounter callback reports the cumulative total so far (say, total bytes read since start), and the SDK derives deltas if the exporter needs them. Reporting an increment from an async callback double-counts or under-counts depending on temporality. When to prefer async: - The value already exists to be read cheaply and continuously changes (process CPU time, resident memory, garbage-collection totals, file-descriptor count). - Instrumenting every mutation would be invasive or hot-path expensive. - The value lives outside your code — a library's internal counter, an OS statistic. When to prefer sync: - The measurement is tied to an event you already handle (a request completing). - You need per-event attributes or exemplars. - Missing changes between collections would be misleading. Asynchronous callbacks have rules of their own: they must be fast and must not block, because they run on the collection path; they should report each distinct attribute set at most once per collection, since duplicate observations for the same attributes in one cycle are undefined; and they must not have side effects, because how often they run depends on collection configuration, not on your code. ## Data points, the shipped form Whatever instrument you choose, what leaves the process is a **data point**: an attribute set, a time window (a start timestamp and an observation timestamp), and a value. Sums (from counters and up-down counters) carry `is_monotonic` and an aggregation temporality; gauges carry neither, since a current value has no accumulation window. Histograms carry a count, a sum, and either explicit bucket boundaries with per-bucket counts or the exponential-histogram form. Exemplars may attach to any of them. ## Attributes are where the cost lives Every distinct attribute combination is a separate time series that the SDK holds in memory and the backend stores forever. A counter with a `user_id` attribute is not a metric, it is a very expensive log. Keep metric attributes to bounded sets — route template, status class, region, tenant if tenants are countable — and push high-cardinality identifiers to spans and logs, which are built for them. ## A worked selection HTTP server instrumentation: a **histogram** of request duration attributed by route template and status class; the count and sum come free with the histogram, so a separate request counter is redundant. In-flight requests: an **UpDownCounter**, incremented on entry and decremented on exit. Process memory: an **ObservableGauge** read from the runtime once per collection. Total bytes written by a library that already tracks them: an **ObservableCounter** reporting that library's cumulative total.