skip to content

Process Self-Reporting

The numbers and facts a Go process publishes about itself besides log records: runtime counters, expvar vars, timings you measure in middleware, and the build the binary came from.

part ofGo (Golang)overview, primer and where to startread it →
on this pageshow

explore

questions

21

How do you time the wrapped http.Handler inside a Go middleware function?

level: juniorimportance: must knowfreq 68%

answer

  1. bracket the call, do not replace it
  2. one clock read before, one after
  3. defer, so a panic still counts
  4. the closure, not the argument
  5. time.Since(start) evaluated at return

basics

~10 s

Read start := time.Now() before calling next.ServeHTTP, then record time.Since(start) inside a deferred closure. Deferring means the duration is still recorded when the handler panics or returns early.

solid answer

~40 s

Go middleware is `func(next http.Handler) http.Handler`, so it can bracket the call. I take `start := time.Now()` at the top, register `defer func() { record(time.Since(start)) }()`, then call `next.ServeHTTP(w, r)`. The closure matters: `defer record(time.Since(start))` evaluates its argument at the `defer` statement, not when the deferred call runs, so it would record roughly zero every time. Deferring rather than recording on the line after `ServeHTTP` also means a panicking handler still produces a data point — otherwise the slowest and most broken requests are exactly the ones missing from the latency numbers. `time.Since` subtracts monotonic clock readings, so the value survives clock corrections. The span it covers is handler work: it starts when my middleware runs and ends when `ServeHTTP` returns, not when the last byte reaches the client.

code

go · 9 lines
go
func timing(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		start := time.Now()
		defer func() {
			record(r.Method, time.Since(start))
		}()
		next.ServeHTTP(w, r)
	})
}

go deeper

for a junior

Be ready to write the six lines from memory: capture time.Now, defer a closure that records, then call next.ServeHTTP. Know that deferred functions run on the way out, including while a panic unwinds.

for a middle

Explain when a deferred call's arguments are evaluated and why that turns a one-line timer into a metric that always reports zero. Be able to say exactly which span of the request the number covers.

for a senior

Show what you do with the duration: which requests must still be counted (panics, early returns, cancelled contexts), and how you capture the status alongside it without breaking streaming handlers underneath.

for a principal

Own the convention. One timing middleware in a shared internal package rather than a copy per service, and a written definition of the span it measures, so latency numbers from different teams mean the same thing.

## What the middleware has to work with A Go HTTP middleware is a function that takes a handler and returns a handler: ```go func(next http.Handler) http.Handler ``` The returned handler can run code before and after `next.ServeHTTP(w, r)`. That call is the only hook you get, and it is deliberately bare: `ServeHTTP` returns nothing at all — no duration, no status code, no error. Anything you want to report about the request you must capture yourself around that one call. ## The shape of a request timer ```go func timing(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { start := time.Now() defer func() { record(r.Method, time.Since(start)) }() next.ServeHTTP(w, r) }) } ``` `record` is your own sink — a histogram, a counter, a log line. Three things in those lines are worth being able to defend. ## Why the closure, not `defer record(time.Since(start))` In Go, the arguments of a deferred call are evaluated **when the `defer` statement executes**, not when the deferred call finally runs. `defer record(time.Since(start))` therefore computes `time.Since(start)` immediately — microseconds after `start` was taken — and schedules `record` with that frozen value. The metric compiles, runs, reports numbers, and every one of them is near zero. Wrapping the work in `func() { ... }()` moves the evaluation inside the function body, which runs at return time. ## Why defer at all You could write `next.ServeHTTP(w, r)` and then the recording line, and for the happy path it behaves identically. It diverges on the paths you most want to see. If the handler panics, the following statement never executes, but deferred functions still run as the goroutine unwinds — so the deferred version records the request and the un-deferred version silently drops it. `net/http` recovers a handler panic per connection so the process survives, which means the failure shows up in your metrics as *nothing at all*: a latency histogram that quietly omits the failing route. The same argument covers any early `return` a future edit adds inside the wrapped code. ## What the measured span actually is The timer starts when your middleware body runs. By then `net/http` has already accepted the connection, read and parsed the request line and headers, and run every middleware outside yours. It stops when `ServeHTTP` returns, at which point the handler's writes have gone into the connection's buffered writer; `net/http` flushes the remainder and finishes the response after the handler returns. So the number is *handler work*, not *time on the wire*, and it excludes both inbound header parsing and the final flush. That is a perfectly good definition — it just has to be the same definition everywhere, or latency numbers from two services are not comparable. ## Why `time.Since` rather than arithmetic on wall-clock fields `time.Now()` returns a `time.Time` that carries a monotonic clock reading alongside the wall clock. `time.Since(start)` is `time.Now().Sub(start)`, and `Sub` uses the monotonic readings when both values have them, so an NTP correction or a manual clock change during the request cannot produce a negative or wildly inflated duration. Formatting the two times and subtracting seconds throws that away. ## Getting the status code alongside the duration A duration on its own cannot separate a fast 500 from a fast 200. Because `ServeHTTP` returns nothing, the usual approach is to pass the handler a wrapper around the `http.ResponseWriter` that remembers the code given to `WriteHeader`, defaulting to 200 for a handler that only calls `Write`. That wrapper has consequences of its own for streaming handlers, and those are worth understanding before you ship one.

  • Why does defer record(time.Since(start)) report almost zero?
    Because a deferred call's arguments are evaluated when the `defer` statement executes, not when the call finally runs. `time.Since(start)` is computed immediately after `start` is taken and that frozen `Duration` is what gets passed at return. Putting the call inside `defer func() { ... }()` moves the evaluation to return time.
  • The middleware has the duration, but how does it get the status code?
    `ServeHTTP` returns nothing, so you pass the handler a wrapper around the `http.ResponseWriter` that records the code handed to `WriteHeader`, defaulting to 200 because a handler that only calls `Write` never calls `WriteHeader`. If the handler panics before writing anything the wrapper still holds its zero value, so decide explicitly what you record in that case.
  • Does the recorded duration include the response reaching the client?
    No. It ends when `ServeHTTP` returns, and at that point the handler's writes have only reached the connection's buffered writer; `net/http` finishes the response afterwards. It also excludes the header parsing `net/http` did before your middleware ran. It measures handler work, which is a useful number as long as everyone reads it that way.

saying these in an interview costs you the question

  • Recording only after next.ServeHTTP returns, so panicking requests never appear
  • Writing defer record(time.Since(start)) and reporting near-zero durations
  • Keeping the start time in a package-level variable shared across requests
  • Assuming ServeHTTP returns the status code the handler wrote
  • Claiming the duration covers bytes arriving at the client
open as a page

Why is r.URL.Path a poor metric label in a Go HTTP service, and what does http.Request.Pattern give you instead?

level: juniorimportance: must knowfreq 50%

basics

~20 s

r.URL.Path holds the concrete requested path, so /items/1 and /items/2 become separate label values and the number of counters grows with traffic. http.Request.Pattern holds the ServeMux pattern that matched, a fixed string that changes only when you add a route.

open as a page

Why does runtime.ReadMemStats stop the world when runtime/metrics.Read does not?

level: middleimportance: must knowfreq 50%

basics

~20 s

runtime.ReadMemStats stops the world for the whole call so that every field of the MemStats struct is one mutually consistent snapshot. runtime/metrics.Read instead aggregates the runtime's per-P statistics under an internal lock, so it adds no stop-the-world pause.

open as a page

What does runtime/debug.ReadBuildInfo() tell a running Go binary about itself?

level: juniorimportance: should knowfreq 42%

basics

~20 s

ReadBuildInfo returns the build record the linker embedded in the binary: the Go toolchain version, the main module's path and version, the dependency modules linked in, and build settings such as flags and the VCS revision. It reports false when no record is present.

open as a page

What does a blank import of Go's expvar package register, and what does /debug/vars serve?

level: juniorimportance: should knowfreq 40%

basics

~10 s

Importing expvar runs its init, which registers a handler for /debug/vars on http.DefaultServeMux and publishes two variables of its own, cmdline and memstats. The endpoint returns every published variable as one JSON object.

open as a page

How do you read a value from Go's runtime/metrics package, such as the live goroutine count?

level: juniorimportance: should knowfreq 40%

basics

~10 s

Build a []metrics.Sample with each element's Name set to a metric name such as /sched/goroutines:goroutines, call metrics.Read on that slice, then switch on each Value's Kind to read a Uint64, Float64 or Float64Histogram.

open as a page

Which vcs build settings does the go command stamp into a binary, and when are they missing?

level: middleimportance: should knowfreq 34%

basics

~20 s

The go command stamps four build settings: vcs (the system, such as git), vcs.revision (the commit), vcs.time (that commit's timestamp) and vcs.modified (true when tracked files were edited). They are absent when the build is not from a supported working tree or -buildvcs=false was passed.

open as a page

When should you publish an expvar.Func instead of using expvar.NewInt or expvar.NewMap?

level: middleimportance: should knowfreq 32%

basics

~20 s

Use expvar.NewInt or expvar.NewMap when your code already knows the number and can add to it as work happens. Use expvar.Func when the value is better computed on demand: it stores nothing and runs on every read of the endpoint.

open as a page

Why is a latency measured with time.Since immune to wall-clock jumps, and what strips that protection?

level: middleimportance: should knowfreq 42%

basics

~20 s

time.Now attaches a monotonic clock reading to the time.Time it returns, and time.Since subtracts those readings, so an NTP step cannot distort the result. Calls such as UTC, Local, Round and any serialization drop the reading, leaving wall-clock arithmetic.

open as a page

A wrapping http.RoundTripper times RoundTrip — what span of the outbound call does that measure?

level: middleimportance: should knowfreq 38%

basics

~20 s

It measures from entering the transport to the response headers being available: connection acquisition, sending the request, and waiting for the first response bytes. It excludes reading the response body, which the caller does after RoundTrip returns.

open as a page

A middleware wrapping http.ServeMux reads r.Pattern before calling next.ServeHTTP and always gets an empty string. Why?

level: middleimportance: should knowfreq 40%

basics

~10 s

Nothing has routed the request yet. http.ServeMux fills in the Pattern field while it matches the request inside its own ServeHTTP, so an outer middleware sees an empty string until that call returns.

open as a page

A shipped Go binary's version output shows no commit hash. How do you establish what it was built from?

level: seniorimportance: should knowfreq 30%

basics

~20 s

Run go version -m against the artifact itself. It prints the embedded record straight from the file: toolchain, main module, linked modules and build settings. An absent vcs.revision means the build was never stamped, not that the data was stripped later.

open as a page

Your timing middleware wraps http.ResponseWriter to record the status, and a streaming endpoint under it stops flushing. Why?

level: seniorimportance: should knowfreq 40%

basics

~10 s

The handler probes its writer with a type assertion to http.Flusher, and the wrapper does not satisfy it, so flushing is silently skipped. Give the wrapper an Unwrap method and flush through http.NewResponseController.

open as a page

After switching a service's request counters from r.URL.Path to r.Pattern, the label map still grows without bound. How do you find and fix what is left?

level: seniorimportance: should knowfreq 34%

basics

~10 s

Unmatched requests carry an empty r.Pattern, and a fallback to the raw path there is still caller-controlled. Confirm with a heap profile, then emit only patterns you registered and one fixed bucket otherwise.

open as a page

Your reporting goroutine samples runtime/metrics every 10 ms and p99 latency rose. How do you decide what to sample and how often?

level: seniorimportance: should knowfreq 32%

basics

~20 s

Sample no faster than something reads the numbers, usually the scrape interval in seconds rather than milliseconds, batch every name into a single metrics.Read call from one goroutine, and keep runtime.ReadMemStats off any request path because it stops the world.

open as a page

How do you turn /gc/pauses:seconds from runtime/metrics into a per-interval distribution?

level: middleimportance: nice to knowfreq 25%

basics

~10 s

Call Value.Float64Histogram, copy its Counts, and subtract the previous tick's Counts elementwise, because /gc/pauses:seconds is cumulative since process start. Buckets holds len(Counts)+1 boundaries, so Counts[n] covers the range from Buckets[n] to Buckets[n+1].

open as a page

How do you read a worker's expvar /debug/vars output to tell whether it is still making progress?

level: seniorimportance: nice to knowfreq 26%

basics

~20 s

Fetch the endpoint twice a known number of seconds apart, from the same process, and diff the two documents. expvar publishes raw monotonic counters with no rates or history, so only the delta shows what is still moving.

open as a page

How do you use httptrace.ClientTrace to split one outbound call into DNS, connect, TLS and first-byte phases?

level: seniorimportance: nice to knowfreq 26%

basics

~10 s

Build a httptrace.ClientTrace with hook functions that stamp time.Now, attach it to the request's context with httptrace.WithClientTrace and req.WithContext, then send the request normally. The hooks fire as the transport reaches each phase.

open as a page

How do you set a Go release policy that can prove, months later, which commit a binary was built from?

level: principalimportance: nice to knowfreq 22%

basics

~20 s

Split it by who can know the fact. Let the toolchain record the source through its vcs settings, inject only what Go cannot know, such as a build time or pipeline identifier, and gate the release on a present revision and a clean tree rather than trusting the pipeline's own claims.

open as a page

A dependency's init put expvar's /debug/vars on your public listener — what fleet-wide policy do you set?

level: principalimportance: nice to knowfreq 20%

basics

~20 s

Rule one: no service serves http.DefaultServeMux on a listener that takes untrusted traffic, so no import can add a route nobody reviewed. Then decide deliberately which services mount expvar.Handler, what they publish, and who may read it.

open as a page

You maintain the Go HTTP metrics middleware every service imports. Should it expose a caller-supplied label hook or derive labels only from registered ServeMux patterns?

level: principalimportance: nice to knowfreq 26%

basics

~10 s

Derive labels inside the library from the registered ServeMux patterns, method and status class. An exported hook taking the request lets any importing team return a user id, and you cannot withdraw it later.

open as a page