Why must an exception class raised in a child process be importable in the parent?

middlemust knowfreq 45%

answer

The class travels as a reference
Module name plus qualified name
Unpickling imports, then looks up
<locals> is not an import path
Renamed class means AttributeError on receive

basics

~20 s

Pickle does not serialize a class body. It stores the class by reference -- module name plus qualified name -- and the receiving process re-imports that module and looks the name up. If the lookup fails there, your exception never reconstructs.

solid answer

~40 s

When an exception crosses a process boundary it is pickled, and pickle stores its class as a reference: __module__ plus __qualname__. Reconstructing it in the parent means importing that module and getting that attribute, so the class must live in an ordinary importable module both sides can reach. A class defined inside a function or a closure fails at send time with pickle.PicklingError: Can't pickle local object; a class in a module the parent cannot import fails at receive time with ModuleNotFoundError; a class that was renamed or moved fails with AttributeError. The practical rule is to declare worker exceptions in a small shared module that both parent and child import by the same name, and never define them inline in the function that raises them.

When an exception crosses a process boundary it is pickled, and pickle stores its *class* as a reference: `__module__` plus `__qualname__`. Reconstructing it in the parent means importing that module and getting that attribute, so the class must live in an ordinary importable module both sides can reach. A class defined inside a function or a closure fails at send time with `pickle.PicklingError: Can't pickle local object`; a class in a module the parent cannot import fails at receive time with `ModuleNotFoundError`; a class that was renamed or moved fails with `AttributeError`. The practical rule is to declare worker exceptions in a small shared module that both parent and child import by the same name, and never define them inline in the function that raises them.

code

python · 14 lines

python

import pickle


def build():
    class ScrapeFailed(Exception):  # defined inside a function
        pass

    return ScrapeFailed("metrics-7 unreachable")


try:
    pickle.dumps(build())
except pickle.PicklingError as exc:
    print(exc)  # Can't pickle local object ...<locals>.ScrapeFailed

go deeper

for a junior

Remember the headline: the exception's class is sent as a name, not as code, so the receiving process must be able to import it. Define such classes at module level, never inside a function.

for a middle

Explain the mechanism concretely -- __module__ plus __qualname__, an import and an attribute lookup on the receiving side -- and name the distinct symptoms: PicklingError when sending, ModuleNotFoundError or AttributeError when receiving.

for a senior

Diagnose from the error alone: which side failed, and whether the cause is a path difference or code skew between parent and child. Structure worker code so exception classes live in one shared, stable module.

for a principal

Treat cross-process exception classes as a versioned interface between deployed components: renaming one is a compatibility event during a rolling deploy, and a fleet-wide policy on where such classes live is cheaper than debugging skew later.

Pickle serializes instances, not classes. This is the fact the whole question turns on. When pickle encounters an object, it writes out the object's state -- but for the type it writes only a reference: the value of the class's __module__ and __qualname__. Unpickling therefore performs, in effect, an import of that module followed by an attribute lookup along that dotted qualified name. The class body, its methods and its base classes are never transmitted. Two processes agree on what ScrapeFailed means only because both can independently import the same module and find the same object there.

The three failure shapes. They fail at different moments and with different error types, which is what makes them diagnosable:

Not addressable at all. A class defined inside a function, a closure, a comprehension or a class body has a __qualname__ containing <locals>, which is not a path anyone can import. Pickling raises immediately, in the sending process: pickle.PicklingError: Can't pickle local object. Dynamically created classes built by calling type(...) at runtime fail the same way unless you register a reduction for them.
Addressable but not importable here. The child pickles a class from a module the parent's sys.path does not reach -- a worker-only package, a script directory, a differently installed environment. The bytes are fine; the parent raises ModuleNotFoundError while unpickling.
Importable but not there any more. The module imports, but the attribute is gone: the class was renamed, moved to another module, or nested differently. The parent raises AttributeError: module 'x' has no attribute 'Y'. This is the version-skew case -- parent and child running slightly different code -- and it is why rolling deploys of a worker fleet can produce a burst of unpickling errors that have nothing to do with the actual failure being reported.

Where __main__ bites. A class defined in your entry-point script has __module__ == "__main__". Under fork the child inherits the parent's __main__ and this happens to work. Under spawn and forkserver -- which since 3.14 means macOS, Windows, and now Unix by default via forkserver -- the child re-imports the entry module under a synthetic name and runs a bootstrap, so the identity of __main__ is no longer something you should rely on. Exception classes belong in a real module, not in the script.

The failure is doubly confusing because it replaces your error. The exception you actually care about -- "scrape target 7 timed out" -- is gone, and what surfaces is a pickling or import error from the plumbing. Worse, the failure happens on the sending side for case 1: the child cannot even serialize the object it was trying to report, so the machinery may substitute a generic error, or the send may fail and leave the parent waiting on a result that will never arrive. Reading such a stack trace, the instinct to debug pickle is exactly backwards; the fix is always to make the class addressable.

What to do. Declare exceptions for cross-process work in a dedicated, boring module -- something like an errors module in the worker package -- imported the same way by both sides. Keep them at module level, never nested in a function. Keep parent and child on the same code version, and treat an AttributeError during unpickling as a deploy-skew signal rather than an application bug. If you genuinely need a dynamic exception type, do not send it: send a stable class whose payload carries the varying detail. And remember that importability is only the first gate -- once the class resolves, unpickling still has to call it with the stored arguments, which is a separate failure with its own symptoms.

Checking addressability cheaply. You do not need two processes to test any of this. pickle.loads(pickle.dumps(exc)) in a plain unit test exercises exactly the same lookup path and fails on exactly the same classes, and reading type(exc).__qualname__ tells you immediately whether the name contains <locals>. The one case that a single-process test cannot catch is the environment difference: a module the child imports but the parent cannot. That one is caught by keeping shared exception classes in a package both sides declare as a dependency, rather than in worker-only code. Beware too of exception classes a library manufactures at runtime -- some drivers synthesize a type per error condition -- since those are addressable only if the library also registers a way to rebuild them; catching such an error in the child and re-raising your own stable type is the reliable answer.

**Pickle serializes instances, not classes.** This is the fact the whole question turns on. When pickle encounters an object, it writes out the object's state -- but for the *type* it writes only a reference: the value of the class's `__module__` and `__qualname__`. Unpickling therefore performs, in effect, an import of that module followed by an attribute lookup along that dotted qualified name. The class body, its methods and its base classes are never transmitted. Two processes agree on what `ScrapeFailed` means only because both can independently import the same module and find the same object there. **The three failure shapes.** They fail at different moments and with different error types, which is what makes them diagnosable: 1. *Not addressable at all.* A class defined inside a function, a closure, a comprehension or a class body has a `__qualname__` containing `<locals>`, which is not a path anyone can import. Pickling raises immediately, in the sending process: `pickle.PicklingError: Can't pickle local object`. Dynamically created classes built by calling `type(...)` at runtime fail the same way unless you register a reduction for them. 2. *Addressable but not importable here.* The child pickles a class from a module the parent's `sys.path` does not reach -- a worker-only package, a script directory, a differently installed environment. The bytes are fine; the parent raises `ModuleNotFoundError` while unpickling. 3. *Importable but not there any more.* The module imports, but the attribute is gone: the class was renamed, moved to another module, or nested differently. The parent raises `AttributeError: module 'x' has no attribute 'Y'`. This is the version-skew case -- parent and child running slightly different code -- and it is why rolling deploys of a worker fleet can produce a burst of unpickling errors that have nothing to do with the actual failure being reported. **Where `__main__` bites.** A class defined in your entry-point script has `__module__ == "__main__"`. Under `fork` the child inherits the parent's `__main__` and this happens to work. Under `spawn` and `forkserver` -- which since 3.14 means macOS, Windows, and now Unix by default via `forkserver` -- the child re-imports the entry module under a synthetic name and runs a bootstrap, so the identity of `__main__` is no longer something you should rely on. Exception classes belong in a real module, not in the script. **The failure is doubly confusing because it replaces your error.** The exception you actually care about -- "scrape target 7 timed out" -- is gone, and what surfaces is a pickling or import error from the plumbing. Worse, the failure happens on the *sending* side for case 1: the child cannot even serialize the object it was trying to report, so the machinery may substitute a generic error, or the send may fail and leave the parent waiting on a result that will never arrive. Reading such a stack trace, the instinct to debug pickle is exactly backwards; the fix is always to make the class addressable. **What to do.** Declare exceptions for cross-process work in a dedicated, boring module -- something like an `errors` module in the worker package -- imported the same way by both sides. Keep them at module level, never nested in a function. Keep parent and child on the same code version, and treat an `AttributeError` during unpickling as a deploy-skew signal rather than an application bug. If you genuinely need a dynamic exception type, do not send it: send a stable class whose payload carries the varying detail. And remember that importability is only the first gate -- once the class resolves, unpickling still has to *call* it with the stored arguments, which is a separate failure with its own symptoms. **Checking addressability cheaply.** You do not need two processes to test any of this. `pickle.loads(pickle.dumps(exc))` in a plain unit test exercises exactly the same lookup path and fails on exactly the same classes, and reading `type(exc).__qualname__` tells you immediately whether the name contains `<locals>`. The one case that a single-process test cannot catch is the environment difference: a module the child imports but the parent cannot. That one is caught by keeping shared exception classes in a package both sides declare as a dependency, rather than in worker-only code. Beware too of exception classes a library manufactures at runtime -- some drivers synthesize a type per error condition -- since those are addressable only if the library also registers a way to rebuild them; catching such an error in the child and re-raising your own stable type is the reliable answer.

How do the error messages differ between an unimportable module and a renamed class?
A missing module raises `ModuleNotFoundError` during unpickling -- the parent cannot even reach the file. A class that was moved or renamed raises `AttributeError: module 'x' has no attribute 'Y'` -- the module imported fine but the name is gone. The first usually means a path or packaging difference between parent and child; the second almost always means the two processes are running different code versions.
A class defined inside a function fails to pickle. Where does that error surface?
In the *sending* process, at serialization time, as `pickle.PicklingError: Can't pickle local object`, because the class's `__qualname__` contains `<locals>` and is not addressable. The child cannot report the real failure at all, so the parent may see a plumbing error instead, or nothing until it gives up waiting. Move the class to module level in a shared module.
Why is defining worker exceptions in the entry-point script risky?
Their `__module__` is `"__main__"`, which is not a stable import target. Under `fork` the child inherits the parent's already-initialised `__main__` and it appears to work; under `spawn` and `forkserver` the child re-imports the entry module through a bootstrap, so `__main__` no longer identifies the same thing. Put the classes in an ordinary module both sides import by name.

saying these in an interview costs you the question

Thinks pickle sends the class definition itself
Defines exception classes inside the function that raises them
Blames pickle instead of the unaddressable class
Assumes parent and child always run identical code
Confuses a class it cannot import with a corrupt payload
Creates exception types dynamically and sends them across

Why must an exception class raised in a child process be importable in the parent?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

▾common follow-ups3 questions

▾red flags6 flags

§ context

▾related questions3 questions

▾related topics3 topics

answer

go deeper

context