When is leaving a sampling profiler always on in production worth its cost, versus profiling on demand?

## What continuous collection actually costs The overhead is real but usually modest, and it comes from four places rather than one. - **CPU.** Cost tracks the interrupt rate and the depth of the stack being walked, not the amount of work the service does. Sampling tens of times a second rather than thousands is what makes always-on collection affordable in the first place. - **Tail latency.** Interrupts land unevenly, and a request unlucky enough to absorb several stack walks pays for them. Judging the overhead by average CPU alone is the classic mistake; watch the high percentiles. - **Storage and symbolisation.** Aggregated profiles merge identical stacks, so they compress well, but the bill is fleet size times window count times retention. Worse, the symbol data that maps addresses to function names must be kept for as long as the profiles it explains, or old profiles become unreadable. - **Operations.** Something extra runs on every node or inside every process, and it must be deployed, upgraded and reasoned about when it misbehaves. None of those numbers is knowable from a document. The only honest answer to 'what does it cost' is that you measured it on your workload. ## What it buys that on-demand cannot 1. **Retrospective incidents.** This is the argument that decides it. A process that has been restarted, rescheduled or scaled away cannot be profiled after the fact. Continuous collection is the only way to have data from a window nobody was watching. 2. **Comparison over time and across builds.** 'Was this code path this expensive last week?' is unanswerable from a single capture, and it is the question that separates a regression from a long-standing cost. 3. **Fleet-wide attribution.** The total cost of a library across every instance is a different quantity from its cost on the one host somebody profiled, and it is the quantity that justifies engineering effort. 4. **Steady-state waste.** Continuous data surfaces permanent inefficiency — a serializer costing a fifth of the fleet's CPU — that nobody would ever open an incident about. ## Where on-demand still wins - **Reproducible workloads.** A batch job, a load test or a staging repro can simply be run again with a profiler attached, at zero standing cost. - **Expensive profile kinds.** Periodic stack sampling for CPU is cheap. Profilers that hook every allocation or every lock acquisition are a different order of cost and are better enabled on demand, or continuously at a heavily reduced rate. - **Constrained runtimes and environments.** Where stack walking is unreliable, where symbols cannot be shipped, or where the platform forbids the required privileges, continuous collection buys unreadable data. - **Small estates.** With a handful of long-lived services and someone always on hand, the retrospective argument weakens considerably. ## A worked decision A cheese-ageing inventory platform brings up a new region, and the 7-node cluster there goes live before the observability rollout reaches it. At 02:14 the nightly maturation-schedule recompute drives p99 from 340 ms to 1.9 s for 26 minutes on three of the seven nodes, then recovers. By the time anyone looks in the morning the pods have been rescheduled twice. With metrics alone the team knows *that* it happened and roughly *where*. With continuous profiles they would open the 02:14 window, filter to the three affected instances, and read which code path took the CPU — a question that no longer has an answer once the processes are gone. Without them, the only plan is to wait for a recurrence with a capture standing by, which is a plan that costs a second incident. That asymmetry is the whole case. The cost of always-on is paid continuously and is measurable; the cost of not having it is paid in incidents that stay unexplained. ## How to decide responsibly - **Measure, do not assume.** Run representative load with and without the profiler on a subset — two of those seven nodes — and compare throughput and tail latency, not just mean CPU. - **Enable per profile kind.** CPU sampling continuously is a different decision from allocation or contention profiling; make them separately. - **Set the rate deliberately.** Lower rates cost less and need longer windows to reach a confident conclusion. Choose the tradeoff rather than inheriting a default. - **Plan symbol retention with profile retention.** Profiles you cannot symbolise are storage without value. - **Roll out progressively** and be willing to turn it off for a service where the measurement says the tail cost is unacceptable. ## What interviewers listen for - Leads with the retrospective argument rather than with overhead numbers. - Knows overhead scales with sampling rate and stack depth, not with throughput. - Mentions symbolisation and retention, which almost everybody forgets. - Says how they would measure the cost instead of quoting a figure.