skip to content

A dependency's init put expvar's /debug/vars on your public listener — what fleet-wide policy do you set?

level: principalimportance: nice to knowfreq 20%

answer

  1. an import decided, not a person
  2. who chose what gets published
  3. a nil handler means the default mux
  4. the argument list ships in the payload
  5. a global registry with no undo

basics

~20 s

Rule one: no service serves http.DefaultServeMux on a listener that takes untrusted traffic, so no import can add a route nobody reviewed. Then decide deliberately which services mount expvar.Handler, what they publish, and who may read it.

solid answer

~50 s

The defect is not that `expvar` exists; it is that the exposure was decided by an import rather than by a person. What went public is `cmdline` — the process's whole argument list, which routinely names internal hosts and paths and occasionally carries a secret — plus `memstats` and every counter anyone in the binary registered. So the policy I set has two halves. First, structural: every service builds its own `*http.ServeMux` and passes it to `http.Server`, never `nil`; `expvar.Handler()` is then mounted explicitly, on an internal surface, in one grep-able line a reviewer can see. That makes publishing an owned decision with a name attached. Second, the registry itself: it is process-global, has no namespaces, cannot be unpublished, and panics on a duplicate name, so once shared libraries publish, names are a fleet-wide contract. Either we fix a prefix convention or libraries hand values to the application instead of publishing them.

code

go · 12 lines
go
func newPublicMux() *http.ServeMux {
	mux := http.NewServeMux()
	mux.HandleFunc("/healthz", handleHealthz)
	// Nothing an import registered can reach this mux.
	return mux
}

func newAdminMux() *http.ServeMux {
	mux := http.NewServeMux()
	mux.Handle("/debug/vars", expvar.Handler()) // deliberate, reviewable, one line
	return mux
}

go deeper

for a junior

Know that passing a nil handler to http.ListenAndServe serves http.DefaultServeMux, which is exactly where an expvar import registers its route.

for a middle

Explain how an import's init reaches a public port, and how constructing your own mux and mounting expvar.Handler yourself takes that decision back.

for a senior

Show the review instinct: read what a new dependency registers, and check which mux each listener actually serves rather than trusting the routing file.

for a principal

Own the standing rule and its cost. Argue what the fleet publishes, who may read it, how names stay collision-free in a registry with no unpublish, and when a service should publish nothing.

## Separate the bug from the tool `expvar` is genuinely useful: zero dependencies, always linked, always available, and answerable with a single HTTP request when a process is misbehaving and you have nothing else. The problem in front of you is not that it was used. It is that **nobody decided to publish**. A blank import somewhere in the dependency graph ran an `init` that registered `/debug/vars` on `http.DefaultServeMux`, and the service happened to serve that mux on the port that faces the world. The service's routing file does not mention the route. No reviewer approved it. No threat model covered it. A policy that says "be careful with expvar" does not fix that, because the mechanism does not go through anyone's care. ## What is actually exposed Before arguing about policy, be precise about the payload, because this is what makes the difference between an interesting endpoint and an incident: - **`cmdline`** — the process's `os.Args`. In practice that means flag values: internal hostnames, bucket names, file paths, feature switches, and, in the shops where someone passed a token on the command line, a credential. This is the item that turns a debug endpoint into a finding. - **`memstats`** — a full runtime memory reading, recomputed on every request. Low sensitivity on its own, but it describes heap shape and workload size to an outsider, and it costs something to produce on each scrape. - **Whatever anyone published.** Every counter and gauge registered by your code *and* by every library in the binary appears in the same document, under names that were never coordinated. And the endpoint has no authentication, no rate limit, and no notion of who is asking. ## The structural rule The rule that actually prevents recurrence is one line long: **a listener that takes untrusted traffic must serve a mux you constructed.** ```go mux := http.NewServeMux() mux.HandleFunc("/healthz", handleHealthz) srv := &http.Server{Addr: ":8080", Handler: mux} ``` A mux built in your code contains exactly the routes written into it. No package's `init` can reach it, today or after the next dependency upgrade. Compare that with the `nil` handler idiom, which opts the service into whatever the entire import graph decided to register. The rule is enforceable — it is a grep for a `nil` handler and for `http.Handle`/`http.HandleFunc` at package scope — and it fixes the whole class, not the one instance you found. The second half is to mount the variables where you do want them: ```go admin.Handle("/debug/vars", expvar.Handler()) ``` That line is the decision. It is visible in review, attributable to a person, and easy to move or remove. An invisible side effect becomes an owned choice; the security reviewer's objection changes from "why is this public" to "is this the right surface", which is a conversation you can actually have. ## The part that is an organisational call The registry is process-global. It has no namespaces. There is no way to unpublish. Publishing a name that is already taken panics, and since publishing happens in `init`, that panic kills the binary at startup rather than at first request. At one service that is a curiosity. Across a fleet with shared internal libraries it is a coordination problem with three possible answers, and picking one is the lead's job: 1. **A naming convention.** Every published name carries the publishing package's prefix. Cheap, works, needs a linter or a review habit to hold, and does nothing about the fact that a library's variables ride into every binary that imports it. 2. **Libraries do not publish.** They expose a value — a method, a struct of numbers — and the application decides whether to publish it and under what name. More code, but the decision lands with the person who owns the process, which is where it belongs. This is usually the right default for anything shared. 3. **The fleet does not use expvar.** Legitimate if the numbers already arrive through another path and nobody's runbook says "curl this". An endpoint with no dashboard and no runbook step is surface area with no reader; the counters will drift out of date with the code and mislead the next person. ## What you are trading Be honest about the cost of the strict answer. `expvar`'s value is precisely that it is free and always there: during an incident, on a process you cannot rebuild, with a monitoring pipeline that may itself be broken, one HTTP request gets you the numbers. Every restriction you add — an internal-only surface, an auth check in front, a rule that libraries may not publish — chips at that availability. The judgment is not "secure or convenient"; it is deciding which processes are worth reaching in that state, keeping the endpoint genuinely reachable *there*, and closing it everywhere else. ## How I would land it Fix this service by building its mux. Ship the same change everywhere as a lint-backed rule about `nil` handlers and package-scope registration. Publish a short standard for what a service should expose and under what prefix. Then say plainly which surface the variables live on and who can reach it — because the failure being fixed is not a missing control, it is a missing owner.

  • What in the default /debug/vars payload would a security reviewer object to first?
    `cmdline` — the process's full `os.Args`, which in practice carries internal hostnames, paths, bucket names and sometimes a credential passed as a flag. `memstats` is less sensitive but still describes heap shape and workload size. Neither is authenticated: anything that can reach the port gets both.
  • Why does a process-global registry become an organisational problem at fleet scale?
    Names are process-wide, there is no namespace, there is no unpublish, and a duplicate name panics during `init`. Once several shared libraries publish, two of them can collide and take a binary down before it serves a request. That forces a decision: a mandated per-package prefix, or a rule that libraries expose values and only applications publish them.
  • When would you decide a service should publish no expvar variables at all?
    When nobody is going to read them. An endpoint with no dashboard and no runbook step is surface area with no reader, and its counters quietly drift out of date with the code until they mislead someone. I keep it where an on-call engineer has an actual instruction to fetch it, and drop it where the same numbers already arrive another way.

saying these in an interview costs you the question

  • Calls /debug/vars harmless because it is read-only
  • Fixes the one service instead of the default-mux rule
  • Assumes an internal network makes the endpoint safe
  • Forgets cmdline publishes the full argument list
  • Believes a published variable can later be unpublished