How do you read a worker's expvar /debug/vars output to tell whether it is still making progress?

## What expvar gives you, and what it does not A request to `/debug/vars` returns the current value of every published variable as one JSON object. That is the entire contract. There is no rate, no timestamp on any value, no ring buffer of past readings, no reset, and no query language. Values published as counters only ever go up, from process start to process exit. This matters during an incident because a large number tells you nothing about the present. `"worker.jobs_done": 18422` is equally consistent with a worker finishing its 18,422nd job a moment ago and with one that finished it before lunch and has been wedged ever since. ## The technique: two reads, diffed The standard move is to fetch the endpoint twice, a known number of seconds apart, and compare the documents key by key. ``` $ curl -s localhost:8080/debug/vars # t = 0s $ sleep 10 $ curl -s localhost:8080/debug/vars # t = 10s ``` What the diff tells you: - **A counter with a non-zero delta** is a stage that did work. Divide by the interval and you have the rate the endpoint refused to give you. - **A counter with a zero delta** is the interesting one: that stage did nothing at all in the window. If `jobs_received` moved and `jobs_done` did not, the worker is taking work in and not completing it — a stall, not slowness. - **Two counters compared** answer the shape question. If a depth gauge grew while a completion counter also grew, the worker is keeping up badly; if depth grew while completions were flat, it is not consuming at all. Taking a third read at a different interval separates "slow" from "stopped" — slow still moves, just less. ## Three things that will mislead you **The snapshot is not atomic.** The handler walks the registry and renders each variable in turn. Each individual read is safe for concurrent use, but no lock is held across the whole document, so a variable near the start of the response can be from a slightly earlier instant than one near the end. For trends this is irrelevant. For exact accounting — "received minus done should equal depth" — it is not: expect the arithmetic to be off by the work of a few microseconds. **It is one process.** expvar has no idea that other replicas of the service exist. If your two reads went through a load balancer they may have hit two different instances, and the "delta" you computed is the difference between two unrelated processes' lifetime totals — which can be negative, and is meaningless either way. Address the instance directly, or at minimum publish something that identifies the process so you can tell when you have been bounced. **A restart resets everything.** Counters live in memory. A process that crash-looped between your two reads shows small numbers, not large ones, and the delta may be negative. Negative delta means restart, not underflow. ## Make one read enough The fix for the whole class of problem is to publish a value that is meaningful on its own. A gauge computed on demand — seconds since the last successful job, derived from a timestamp the worker stores on each success — turns "is it stuck" from a two-read comparison into a single number an on-call engineer can read at a glance. A large value is a stall regardless of how impressive the cumulative counters look. The same trick applies to depth: a current queue depth is self-describing where a cumulative enqueue count is not. This is worth doing before the incident, because during one the person reading the endpoint is often not the person who wrote it, and they may only get one shot at a process that is about to be restarted. ## The cost of scraping Rendering the document is not entirely free. Any variable published as a computed value runs its function on every request, and the `memstats` variable that expvar publishes by itself asks the runtime for a fresh memory reading each time rather than serving a cached one. A tight polling loop pays for that on every iteration. If a scrape is noticeably slower than the rest of the service, suspect a computed variable doing real work rather than the JSON encoding. ## What to remember One read of `/debug/vars` is a position, not a velocity. Two timed reads of the same process give you the velocity. Nothing gives you history, cross-variable consistency, or a view across replicas — and the durable fix is to publish a gauge that means something on its own.