Why can operator.itemgetter cross a process boundary when an equivalent lambda cannot?

Take a subscription-billing run that fans invoice lines out to worker processes to keep up with a 1,200-invoice-per-minute peak. The parent sorts and aggregates chunks; the callables that do the sorting and the summing have to reach the workers. Written with anonymous inline functions, the submission fails outright; written with `operator` callables or module-level functions, it works. Understanding why is a genuinely senior piece of Python knowledge, because it explains a whole family of 'it worked in threads, it explodes in processes' incidents. ## How pickle treats callables Pickle does not serialize code. For a function or a class it writes a **reference**: the `__module__` and `__qualname__`, with the expectation that the receiving interpreter can import that module and find that name. Reconstruction is a lookup, not a rebuild. - This is why a module-level `def add_line(a, b)` pickles fine — its qualified name is importable — - and why an anonymous function does not: its `__qualname__` is `<lambda>`, there is no such name to import, and pickle raises rather than guessing. - A function defined *inside* another function fails for the same reason, its qualified name pointing at a local scope nobody can reach. Note the corollary that trips people up: pickling a function by name means the **child must be able to import the defining module**, and it means the child gets whatever version of that function it currently has on disk, not a snapshot of the parent's. ## Why the operator objects are different `operator.itemgetter(2)` does not produce a function. It produces an *instance* of the `itemgetter` type, which lives in the `operator` module and is importable everywhere. Instances are pickled by value, and this type defines `__reduce__`, the hook that lets an object dictate its own reconstruction: it returns, in effect, 'call `itemgetter` with the arguments `(2,)`'. The child imports the type by name, calls it with the stored indices, and gets an equivalent object. - `operator.attrgetter` and `operator.methodcaller` do the same with their stored names and arguments — with the proviso that a `methodcaller`'s captured arguments must themselves be picklable, since they travel too. - And `operator.add` needs no special mechanism at all: it is a module-level function in an importable module, so the plain by-name path works. That is the concrete reason `functools.reduce(operator.add, chunk, 0)` inside a worker is safe while an inline two-argument adder is not. ## Why the process pool cares under every start method A common wrong answer is that `fork` avoids the problem because the child inherits the parent's memory. It does not. `concurrent.futures.ProcessPoolExecutor` and `multiprocessing.Pool` deliver each work item — the callable, its arguments, and later its result — through a **queue**, and everything on that queue is pickled regardless of how the worker process was created. Inheritance only covers what existed at fork time; a callable you submit afterwards still has to be serialized. So the failure is start-method-independent, and it is also **deterministic**: it happens on the very first submission, every run, in every environment. That determinism is itself a diagnostic. If the billing run's symptom is an *intermittent* timeout that appears only near peak, pickling is not the culprit and you should be looking at: - queue back-pressure, - a worker that died mid-item, - or a blocking call inside the task — not at how the key function was written. ## The version detail worth naming Python 3.14 changed the default `multiprocessing` start method to `forkserver` on Unix platforms other than macOS; macOS and Windows continue to default to `spawn`, and `fork` must now be requested explicitly. That change makes the *inheritance* assumption even less safe than before — under `spawn` and `forkserver` the worker does not inherit the parent's objects at all — but it does not change the rule above, which held under `fork` too. ## What to do about it in practice Three options, in order of preference. 1. Use the `operator` factories where the operation is a fetch or a named-method call: they are declarative, picklable, and implemented in C so a call does not push a Python frame — a modest but real saving across hundreds of thousands of rows. 2. Otherwise define a **module-level function** and submit that; it pickles by name and is the plainest thing a reviewer can follow. 3. Only if neither fits should you reach for heavier machinery such as an initializer that builds the callable inside each worker. The same picklability constraint reappears wherever callables are stored rather than merely called — a work item persisted to a queue, a key function held in a cached configuration object, a task handed to a distributed runner — so it is worth treating 'is this callable picklable?' as a property of any callable that leaves the process, not as a quirk of one library. ## The limits of the trick - Picklability is not magic: an `attrgetter` pickles, but the objects it will be applied to must also cross the boundary, and if a captured argument or a result is unpicklable you have simply moved the failure. - And picklable does not imply cheap — every item crossing the boundary is serialized, so a fan-out whose per-item work is small can easily spend more time on serialization than on the computation.