skip to content

How do you flip a running Go service's slog verbosity to debug without restarting it or rebuilding the logger?

level: seniorimportance: should knowfreq 45%

answer

  1. the options field takes an interface
  2. the handler asks again per record
  3. one pointer, shared by derived loggers
  4. no mutex required around the change
  5. a mutable Leveler ships in the package

basics

~20 s

Give the handler a *slog.LevelVar as its HandlerOptions.Level instead of a plain constant. The handler reads it for every record, so calling Set on that LevelVar from a signal handler, an admin endpoint or a config reload changes verbosity immediately.

solid answer

~50 s

`HandlerOptions.Level` takes a `slog.Leveler`, so instead of pinning it with `slog.LevelInfo` you pass a `*slog.LevelVar` that you keep a reference to. The handler calls `Level()` on it for every record, which means `lvl.Set(slog.LevelDebug)` takes effect on the next log line — for every logger built on that handler, including ones already derived with `With`. `LevelVar` is safe for concurrent use, so the goroutine handling a SIGHUP, an admin HTTP request or a config-reload loop can call `Set` while request goroutines are logging; no mutex is needed. Operationally, treat it as a blast-radius decision: debug is process-wide, not per-request, so the volume and cost land on everything. I schedule the revert at the same time I enable it — `time.AfterFunc` restoring the previous level — so a 3am debugging session cannot leave the service verbose forever.

code

go · 5 lines
go
var lvl slog.LevelVar // zero value is LevelInfo
logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: &lvl}))

// later, from the reconcile loop after re-reading config, or from an admin request:
lvl.Set(slog.LevelDebug) // the next record, and every record after it, is checked against Debug

go deeper

for a junior

Recall that the handler's level field accepts more than a constant, and that log/slog ships a mutable level type you can hold on to and change later.

for a middle

Explain the mechanism: the field is an interface, the handler calls Level() per record, and *slog.LevelVar implements it over an atomically updated value.

for a senior

Demonstrate the operating judgment — one instance rather than the fleet, an automatic revert, an authenticated trigger, and awareness of what a tenfold jump in log volume costs.

for a principal

Own the policy: who is allowed to raise verbosity in production, through which control plane, with what audit trail and what bounded window, and whether the money and risk favour targeted sampling instead.

## The problem The floor a `slog` handler applies is fixed when you write `Level: slog.LevelInfo`, because a `slog.Level` is a value and the handler holds a copy of it. Restarting a process to see debug output is exactly what you cannot afford when the bug reproduces only under production traffic, and it is worse in a service whose state takes time to rebuild — a controller reconciling desired state loses its caches and its place in the loop when it restarts. ## The mechanism `HandlerOptions.Level` is typed as the interface `slog.Leveler`, whose only method is `Level() slog.Level`, and the built-in handlers call it **once per record** rather than caching the result. `*slog.LevelVar` implements that interface over a mutable, atomically updated value: ```go func (v *LevelVar) Level() Level func (v *LevelVar) Set(l Level) func (v *LevelVar) String() string ``` So the pattern is: declare the `LevelVar` where you can reach it later, hand its address to the options, and keep the variable. ```go var lvl slog.LevelVar // the zero value is LevelInfo h := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: &lvl}) ``` After that, `lvl.Set(slog.LevelDebug)` lowers the floor for the next record and every record after it. ## Three properties worth stating explicitly **It is concurrency-safe.** `LevelVar` stores its value atomically and is documented as safe for use by multiple goroutines. A signal handler calling `Set` while hundreds of request goroutines call `Level()` is fine, and wrapping it in a mutex signals that you did not read the contract. **It reaches loggers that already exist.** `Logger.With` derives a new logger over a derived handler, but that derived handler holds the same `*LevelVar` pointer. A logger created at startup and passed down through a request's `context.Context` will see the change, which is what makes the technique useful at all — you are not chasing down every logger you handed out. **Its zero value is Info.** `var lvl slog.LevelVar` needs no initialisation to be usable, and a `LevelVar` you forgot to set is not accidentally silent or accidentally maximal. ## Wiring it to an operator The `LevelVar` is the mechanism; you still need a way to move it. The common shapes are: - **A config reload.** A controller that re-reads its own configuration on a timer or a file-watch decodes the level field and calls `Set`. This makes verbosity a normal configuration value rather than a special case, and the change is auditable in whatever holds the config. - **A signal.** `SIGHUP` triggers a reload, or `SIGUSR1`/`SIGUSR2` toggle debug directly. Cheap, but invisible to anyone who was not watching. - **An authenticated admin endpoint.** A small handler that parses a level and calls `Set`. Convenient at 3am; it must be authenticated, because turning debug on in a hot service is a denial-of-service vector and debug lines often carry data that Info deliberately omits. ## The judgment part A `LevelVar` moves the floor for the **whole process**. It is not per-request, per-tenant or per-package sampling; every goroutine gets louder at once. On a service handling meaningful traffic that can multiply log volume by one or two orders of magnitude, which costs CPU in formatting, bandwidth to the collector, money in the log store, and sometimes latency if the writer backs up. Three habits keep that from turning an investigation into an incident: 1. **Schedule the revert when you enable it.** `time.AfterFunc(10*time.Minute, func() { lvl.Set(prev) })` means the worst case is ten minutes of noise, even if the person who turned it on is paged elsewhere. 2. **Enable it on one instance.** If the change goes through config that fans out to a fleet, you have made a global change to debug a local problem. 3. **Log the change itself at a level that is always visible**, so the record of who made the process verbose survives in the same stream as the consequences. ## Alternatives you should be able to compare If what you actually need is deep detail for *one* request or *one* tenant rather than the whole process, a global floor is the wrong instrument: you want per-request sampling, or a second handler that is fed only records tagged with the tenant. Reach for `LevelVar` when the question is "turn the whole service up for a few minutes", which is the common on-call case. ## Common mistakes - Passing `slog.LevelInfo` and then trying to change it later. There is nothing to change; the handler copied the value. - Building a whole new logger and handler on every change, then discovering that goroutines holding the old logger never noticed. - Guarding `Set` with a mutex, which is unnecessary. - Turning debug on and going back to bed.

  • Does a logger derived earlier with Logger.With see a later LevelVar change?
    Yes. `With` produces a new logger over a handler derived with `WithAttrs`, and that derived handler carries the same `*slog.LevelVar` pointer, not a copy of the level. A logger created at startup and threaded through requests picks up the new floor on its next record, which is precisely why the technique scales past the one logger you happen to be holding.
  • Do you need a mutex around LevelVar.Set?
    No. `slog.LevelVar` stores its value atomically and is documented as safe for concurrent use, so one goroutine may call `Set` while others call `Level` through the handler. Adding a mutex costs contention on the hottest path in the logger and buys nothing.
  • What are the risks of exposing an endpoint that sets the level, and how do you contain them?
    It is unauthenticated remote log amplification if you leave it open: an attacker can multiply log volume, burn CPU and fill the log store, and debug lines often carry payload detail that Info withholds. Require authentication and authorisation, restrict it to one instance rather than a fleet-wide config change, auto-revert after a bounded window, and record who changed it at a level that is always emitted.
  • When is a process-wide LevelVar the wrong instrument?
    When you need detail for one request, one tenant or one code path rather than everything. Lowering the global floor makes every goroutine louder at once, which can cost more than the bug does. For targeted detail, sample per request or route tagged records to a second handler, and keep the global floor where it is.

A plain level constant is a floor poured in concrete; a LevelVar is a dial the handler reads each time it is about to speak.

saying these in an interview costs you the question

  • Rebuilds the whole logger to change verbosity
  • Guards LevelVar.Set with a mutex, thinking it is unsafe
  • Assumes a plain slog.LevelDebug constant can be changed later
  • Thinks the change affects only loggers created afterwards
  • Turns debug on globally with no plan to turn it off
  • Exposes an unauthenticated endpoint that sets the level