One logger is emitting thousands of lines a second in production. How do you cut that volume without hiding the failure?

## What the volume actually costs A logger emitting thousands of lines a second is usually described as a cost problem, and the invoice is the least interesting part of it. On a cheese-ageing inventory platform, one humidity-reconciliation service logged a WARN per sensor read across 47 vaults; at 384 sensors each and a two-second reconcile loop, that came to roughly 18,400 lines a second. Three things happened long before anyone looked at a bill: - **The request path got slower.** Each line costs an argument-formatting pass and a handful of allocations. When the appender's bounded queue filled, the implementation blocked the calling thread rather than dropping, and p99 latency moved from 41 ms to 610 ms. - **The collection tier lost data.** That service's share of ingest jumped from 4.6 GB/day to 63 GB/day, outran what the shared collection tier would accept, and lines were shed — including lines belonging to the other forty services using it. - **Nobody could read anything.** The incident nobody could explain from the existing dashboards was in the logs the whole time, three thousand identical WARN lines away from the one that mattered. The ordering matters because it decides what you are optimising. Cutting volume is not primarily an economy measure; it is how you keep logs usable and stop logging from becoming the failure itself. ## Suppress, but report the suppression One rule separates a fix from a cover-up: **a drop has to leave a trace.** A logger that quietly stops emitting is indistinguishable, on an incident timeline, from a system that stopped doing the thing — and that ambiguity is most expensive at exactly the worst moment. What a suppression mechanism should preserve: 1. **The first occurrence, in full.** The first line carries the detail; the thousandth carries none. 2. **A count of what was dropped, and over what window**, emitted as a line of its own when the window closes: "suppressed 17,326 further lines from this logger in the last 10s". 3. **The identity of what was collapsed** — the logger name and, better, the message template or an exception fingerprint — so the count is attributable to something. 4. **A counter in the metrics stream**, labelled by logger or template, so suppression is visible on a dashboard rather than only inside the logs it is thinning. A spike in suppressions is itself worth noticing. 5. **Independent budgets per template.** If the cap is keyed on the message template, a genuinely different failure is a different template and is not silenced by the noisy one's allowance. ## Choosing the axis you cut on | Approach | Keeps | Loses | Use when | |---|---|---|---| | Cap per message template per window | First occurrences of every distinct message | Repeat detail inside a burst | One or a few loggers are loud | | Deduplicate by exception fingerprint | One example per distinct failure, with a count | Per-occurrence context | The same stack repeats endlessly | | Sample by request identity | Complete narratives for a known share of traffic | Everything about unselected requests | Volume is spread evenly across the service | | Independent per-line coin flip | A fixed fraction of lines | Coherence: every request becomes fragments | Almost never | The last row is the mistake worth naming out loud. Deciding per line looks equivalent to deciding per request and is not: you keep the third line of one request and the seventh of another and can reconstruct neither. Hashing a request or trace identifier and keeping every line for the selected fraction costs the same and leaves you with stories rather than fragments — and it lines up with whatever share of traces was retained, so the two correlate instead of missing each other. Two things suppression must not become. It is not a substitute for fixing severity: a WARN emitted on a path that is normal and expected is a defect in the code, and the fix is to lower or delete the line rather than to build machinery that tolerates it. And a cap keyed on severity alone is dangerous, because the first real ERROR then arrives inside the same budget as the noise it is competing with. ## The honest argument about verbose levels in production **For.** Some incidents are not reproducible. Detail already being captured is the only detail you get, and turning DEBUG on afterwards captures the next occurrence rather than the one under investigation. For a small deterministic slice — one replica, or a hashed fraction of requests — the cost is bounded and known in advance, and the payoff on the rare unexplainable incident is large. **Against.** That cost is paid on every request, by every instance, permanently, for a benefit that materialises a handful of times a year. It concentrates risk in the logging path, where a full queue converts a degradation into an outage. And the volume erodes the very thing it was bought for: a reader who has to skim fifty times as much material is slower, not better informed. The defensible position is neither extreme. Run INFO by default, make verbose reachable per logger at runtime, and keep a small number of deliberately-chosen high-value lines permanently on the paths that history says you always end up wanting. Choose that list from real incidents rather than from whatever was easiest to add.