What does running with tracemalloc enabled cost, and how do you keep that cost bounded?

### Tracing is instrumentation, and instrumentation is not free `tracemalloc` works by intercepting CPython's allocator calls. Every allocation while tracing is active does additional work — capture the current frames, hash and store an entry keyed by the block's address — and every free must find and remove that entry. The result is a diagnostic mode: fine to switch on deliberately, wrong to leave on by default. Code that allocates heavily in a tight loop feels it most, and code that spends its time in I/O or in a compiled routine barely notices, so a benchmark on your own workload beats any general multiplier you might be quoted. ### The two costs **Time.** Proportional to allocation rate, not to memory size. A request handler that creates a few large buffers pays little; one that churns millions of small objects pays a lot. This is why enabling tracing sometimes changes the very behaviour you are chasing: under a latency budget, a traced process may start shedding work, and the memory profile of a struggling process is not the profile you wanted to measure. **Memory.** Proportional to the number of *live traced blocks*, because each one stores its traceback. `tracemalloc.get_tracemalloc_memory()` returns the bytes tracing itself is using, and it is worth logging next to `tracemalloc.get_traced_memory()` — in a process with tens of millions of small live objects the traces can become a significant fraction of the heap, and an investigation that pushes the container over its limit tells you nothing. ### nframe is the main dial `tracemalloc.start(nframe=1)` records one frame per trace: the line that allocated. That is the cheapest setting and it is enough whenever the allocating line is itself the answer. Raise it when the allocating line is a shared helper — a deserialiser, a factory, a caching decorator — called from everywhere, so that a `'traceback'` grouping in `Snapshot.statistics()` can distinguish callers. But the cost of both dials rises with `nframe`: more frames captured per allocation, more bytes stored per live block. Ten frames is a common working compromise; twenty-five is what the interpreter's own suggestion uses when it wants a full picture; a hundred is almost always waste. ### Bounding the rest * **Filter.** `Snapshot.filter_traces()` takes `tracemalloc.Filter` objects and returns a reduced snapshot — dropping the standard library and installed packages typically shrinks a snapshot dramatically and makes the remainder readable. Filtering happens at snapshot-analysis time, so it reduces what you carry around rather than what tracing costs at runtime. * **Dump and analyse elsewhere.** `Snapshot.dump()` writes a snapshot to a file and the `Snapshot.load()` classmethod reads it back. The service does the cheap part; the expensive comparison happens on another machine. * **Clear between phases.** `tracemalloc.clear_traces()` discards accumulated traces without stopping tracing, which is useful when you want a clean baseline for the next phase of work. * **Stop when done.** `tracemalloc.stop()` disables tracing and releases the traces; `tracemalloc.is_tracing()` tells you the current state, which matters in code that conditionally profiles. ### Enabling it in a real deployment Editing the program is often the wrong lever. `PYTHONTRACEMALLOC=25`, or the equivalent `-X tracemalloc=25` interpreter option, starts tracing before the first import, which is the only way to attribute allocations that happen at import time — and it means an operator can turn tracing on by restarting one instance with an extra environment variable, no deploy required. The pattern that works in production is a canary: route a small share of traffic to one instance started with tracing enabled, let it dump snapshots on a schedule or on a signal, and diff them offline. The fleet keeps its normal performance, the sample is representative because it serves real traffic, and if the traced instance degrades you have lost one instance rather than the service. Where the code is under your control, a feature flag that calls `tracemalloc.start()` and arranges periodic dumps gives the same effect without a restart, at the price of missing import-time allocations. The judgement to articulate is that a memory investigation has a cost, that cost is paid in latency and in memory on the instance you are watching, and the job is to spend it where it buys evidence — one instance, a bounded frame count, and snapshots small enough to keep.