In k6, why does a Trend metric reach Prometheus as one `p99` series, and how do you change that?

## Where the single `p99` comes from Prometheus has no metric type that corresponds to a k6 Trend, so in k6 v2 the `experimental-prometheus-rw` output has to reduce each Trend into ordinary series before it can push anything. What it pushes is governed by one setting, `K6_PROMETHEUS_RW_TREND_STATS`, and that setting **defaults to `p(99)` alone**. So a run with no extra configuration produces exactly one series per Trend metric, and that series carries the 99th percentile — nothing else. A team that goes looking for the average or the maximum of `http_req_duration` in Prometheus after a long run finds neither, and there is no way to recover them from what was pushed. ## `K6_PROMETHEUS_RW_TREND_STATS` The variable takes a comma-separated list of stat names. The accepted tokens are: - `count` - `sum` - `min` - `max` - `avg` - `med` - `p(x)`, where `x` is a number from 0 to 100 Each token in the list becomes **its own Prometheus series** for every Trend metric in the run. Three tokens means three times as many series per Trend as the default; the setting is global to the output, so it applies to every Trend the run emits, built-in and custom alike. ## How a stat becomes a series name Series names are the k6 metric name with a `k6_` prefix and the stat appended as a suffix. Percentile tokens lose their punctuation on the way: | `K6_PROMETHEUS_RW_TREND_STATS` token | suffix | series for `http_req_duration` | |---|---|---| | `p(99)` (the default) | `p99` | `k6_http_req_duration_p99` | | `p(95)` | `p95` | `k6_http_req_duration_p95` | | `p(0.95)` | `p095` | `k6_http_req_duration_p095` | | `max` | `max` | `k6_http_req_duration_max` | | `avg` | `avg` | `k6_http_req_duration_avg` | The `p(0.95)` row is the one that bites: the dot is stripped rather than interpreted, so a mistyped percentile lands in a differently-named series instead of failing, and the query written for `_p95` simply finds nothing. Each point on one of those series is that stat computed by k6 over **one push interval's** worth of samples. `K6_PROMETHEUS_RW_PUSH_INTERVAL` sets that window and defaults to `5s`, so a long run pushes one point per stat every five seconds for as long as it runs. ## The output ignores its `-o` argument This is the trap that costs people an afternoon. `experimental-prometheus-rw` reads its configuration from `K6_PROMETHEUS_RW_*` environment variables and the k6 config file, and **nothing else**. The text after the `=` on the `-o` flag is accepted by the command line and then discarded by the output: ```bash # the URL here is silently ignored; k6 pushes to the default endpoint k6 run -o experimental-prometheus-rw=url=http://prom:9090/api/v1/write script.js # this is how you actually point it K6_PROMETHEUS_RW_SERVER_URL=http://prom:9090/api/v1/write \ K6_PROMETHEUS_RW_TREND_STATS='p(95),p(99),max' \ k6 run -o experimental-prometheus-rw script.js ``` The default server URL is `http://localhost:9090/api/v1/write`, so the failure mode of the first command is a run that appears to work while pushing at a local endpoint that is not there. The same rule holds for the `opentelemetry` output, which takes `K6_OTEL_*` variables and likewise never reads its `-o` argument. The two sets are independent: `K6_OTEL_EXPORT_INTERVAL` does nothing for Prometheus remote write, and `K6_PROMETHEUS_RW_TREND_STATS` does nothing for OpenTelemetry. ## Two remote outputs, two variable sets Attaching both remote targets to one run is perfectly legitimate, and each is configured entirely on its own terms: - **`experimental-prometheus-rw`** reads `K6_PROMETHEUS_RW_SERVER_URL`, `K6_PROMETHEUS_RW_PUSH_INTERVAL`, `K6_PROMETHEUS_RW_TREND_STATS` and the rest of that prefix. - **`opentelemetry`** reads `K6_OTEL_EXPORTER_PROTOCOL` (`grpc` by default, or `http/protobuf`), `K6_OTEL_GRPC_EXPORTER_ENDPOINT` (default `localhost:4317`), `K6_OTEL_EXPORT_INTERVAL` (default `10s`) and the rest of the `K6_OTEL_` prefix. - **Neither inspects the other's prefix.** There is no shared endpoint, no shared push interval and no shared stat list, so a run that streams to both is a run configured twice. ## When nothing arrives The two failure modes look identical from the outside — no series where you expect them — so separate them in order: 1. **Check the endpoint actually used.** If it was passed after `=` on the `-o` flag it was discarded, and k6 pushed at `http://localhost:9090/api/v1/write` instead. 2. **Check the series name you are querying.** The pushed name is `k6_` plus the metric plus the stat suffix, so a query for `k6_http_req_duration` with no suffix matches nothing. 3. **Check the stat is in the list.** Querying `_avg` when `K6_PROMETHEUS_RW_TREND_STATS` was left at its default means querying a series k6 never created. ## What a bad value does An unparseable stat token — a bare `p95` without the parentheses, or `p(150)` — is rejected while k6 is constructing the output, before any virtual user starts. That is worth knowing because it means the setting is safe to change: a typo costs a restart, never a run that streams half a metric set for an hour and then turns out to be missing a series. ## The alternative mapping k6 can instead map Trends onto Prometheus native histograms, which sends a distribution rather than a fixed set of pre-computed stats. Turning that on makes `K6_PROMETHEUS_RW_TREND_STATS` irrelevant, because the output is no longer choosing stats at all. It is worth naming in an interview as the other option, but the stat list remains the default behaviour and the one you are asked about.