sys.settrace and setprofile

## Why there are two callbacks, not one A naive tracing API would call one function on every event in the process. That would make every traced program crawl, since a per-line callback in code you do not care about is pure waste. CPython instead splits the job: * the **global trace function**, installed by `sys.settrace(func)`, is called once per Python frame entered - the `'call'` event, and effectively only that; * the **local trace function** is whatever the global one *returned* for that frame, and it receives the frame's fine-grained events: `'line'` before each executed line, `'return'` as the frame finishes, `'exception'` when an exception is raised or propagates through it. The return value is the whole control mechanism. Return `None` from the global handler and the frame executes with no further instrumentation, at close to normal speed. Return a callable and you have opted that one frame in. ## The dispatch, concretely ```python def global_trace(frame, event, arg): if frame.f_code.co_filename.startswith(MY_PACKAGE_ROOT): return local_trace # instrument this frame return None # skip everything else ``` That single test is how a coverage or debugging tool avoids paying per-line cost inside the standard library. The information it needs is on the frame's code object: `co_filename`, `co_name`, `co_firstlineno`. Note that the decision is made once per frame, not once per line, which is what makes it cheap. ## What each event carries The callback signature is always `(frame, event, arg)`: | event | when | `arg` | |---|---|---| | `'call'` | a Python frame is entered | `None` | | `'line'` | a new source line is about to execute | `None` | | `'return'` | the frame is about to return | the value being returned | | `'exception'` | an exception is raised in the frame | `(type, value, traceback)` | The current line is read from the frame, and locals and globals are reachable from it too - which is exactly how a debugger can show you a variable at a breakpoint. ## Returning from the local function The documented contract is that a local trace function returns a reference to itself, or to another callable, to continue tracing that scope. In practice most tracers simply `return local_trace`. The subtle part, and a favourite follow-up: **returning `None` from a local trace function does not turn tracing off for that frame.** CPython only replaces the frame's stored local callback when the return value is not `None`; a `None` return leaves the existing one installed, so the events keep coming. This is genuinely surprising the first time you meet it, and it is why the right place to exclude a frame is the `'call'` event in the global handler, not a late `None` from inside. ## Errors in the callback If the trace function raises, the exception propagates at the traced location and the interpreter *unsets* tracing entirely, exactly as if `sys.settrace(None)` had run. There is no error channel and no warning. A tracer that occasionally trips over an unusual frame therefore does not produce a partial report - it produces a report that silently stops, and if the traced code swallows exceptions broadly you will never see why. Defensive tracers wrap their body in `try`/`except`, log to a side channel, and return themselves regardless. ## Cost, stated honestly Each delivered event is a genuine Python-level call made from the evaluation loop, so the per-line mode is proportional to *lines executed*, not to time spent. A tight numeric loop is where it hurts most: several times slower is normal, and instrumented code cannot use the specialized fast paths the adaptive interpreter would otherwise pick. The two-level design is the only real lever you have, and using it well - a narrow filter in the global handler - is the difference between a tracer you can run on a whole test suite and one you cannot. ## Where the pattern shows up `bdb.Bdb`, the debugger base class in the standard library, is this protocol wrapped in a class: it installs a dispatcher, decodes each event, and calls `user_line`, `user_call`, `user_return` and `user_exception` for the events that survive its breakpoint checks. Reading it is the fastest way to see the protocol used properly. ## Frames that were already running The global handler is called on frame *entry*, which means arming the hook does nothing for frames already on the stack. Call `sys.settrace(tracer)` in the middle of a function and that function produces no `'line'` events for its remaining statements; instrumentation begins with the next call it makes. Removal is not symmetric with arming: clearing with `sys.settrace(None)` takes effect immediately, so even the frame that made the call goes quiet from the next statement onwards. Tracers that need to cover the caller too usually arrange to be armed one frame earlier, in a wrapper, rather than trying to retro-fit the frame they are standing in. ## A note on ordering Within a frame, the `'line'` event fires *before* the line executes, and the `'return'` event fires before the value leaves the frame - which is why a debugger can change a local, or inspect the value about to be returned, at exactly the moment it matters. Getting this the wrong way round is the most common source of off-by-one confusion when reading a hand-written tracer's output.