Why can a custom exception raised in a multiprocessing worker fail to reach the parent?

**Exceptions are just objects, and they travel the same road.** When a worker process raises, the pool catches the exception, pickles it, and sends it back through the result pipe so the parent can re-raise it. Nothing about being an exception exempts it from pickling rules, which means an exception can fail to travel for exactly the same reasons an argument can — and the failure is more confusing, because it replaces a meaningful domain error with a serialization error. **The default reduction.** `BaseException` defines a reduction that yields the exception's class and its `args` tuple, plus the instance `__dict__` as state when it is non-empty. Unpickling therefore *calls the class* with those args — not the allocate-and-restore path a plain object gets — and only afterwards updates the new instance's `__dict__`. Two facts fall out of that. First, attributes you set in `__init__` do survive, because they live in `__dict__` and are restored as state. Second, and fatally, the constructor must accept `args` as its positional arguments. **The common break.** The idiomatic-looking custom exception is the one that fails: ```python class UpdateFailed(Exception): def __init__(self, record_id, reason): super().__init__(f"{record_id}: {reason}") self.record_id = record_id ``` Because `super().__init__` was handed one formatted string, `args` has length one, while `__init__` demands two positional parameters. Round-tripping raises `TypeError: UpdateFailed.__init__() missing 1 required positional argument: 'reason'`. In a pool, the parent sees that `TypeError` — raised while unpickling the result — rather than the update failure the worker actually hit. Engineers chase a phantom bug in the parent for hours before noticing the exception class is the culprit. **Two fixes.** The simplest is to keep the constructor compatible with `args`: forward every parameter to `super().__init__(record_id, reason)` and derive the message from them in `__str__`, so `MyError(*exc.args)` reconstructs cleanly. The more explicit fix is `__reduce__`, returning `(UpdateFailed, (self.record_id, self.reason))`, which states the rebuild recipe directly and leaves the constructor free to look however you like. Either way the guard is a one-line unit test: `pickle.loads(pickle.dumps(exc))` for each exception type your workers can raise. That test is worth writing precisely because the failure only shows up across a process boundary. **Other ways an exception refuses.** Even with a compatible constructor, an attribute stored on the exception can sink it — an open file object, a connection, a socket, a lock, a lambda captured for a retry callback. The `__dict__` is pickled as state, so anything you attached to the exception for debugging must itself be picklable. Attaching the offending payload to the error, which feels helpful locally, is the exact move that makes it unreportable from a worker. **The traceback never travels.** A traceback object holds live frames and cannot be pickled at all. Plain `multiprocessing` re-raises the exception in the parent with the parent's own stack, so the worker's frames are simply gone. `concurrent.futures.ProcessPoolExecutor` mitigates this by formatting the worker traceback into a string and attaching it to what it re-raises, so the parent's traceback shows the remote one as text — useful, and worth knowing as a reason to prefer that executor when diagnosability matters. If you need the frames themselves, format and log them inside the worker at the point of failure, where they still exist. **The class has to be importable too.** An exception is rebuilt by looking its class up in the receiving process, exactly the way a function reference is resolved. An exception class defined inside a function, or generated dynamically at run time, cannot be found by that lookup and fails to unpickle even with a perfectly compatible constructor. The same is true of a class whose module the parent cannot import — a worker-only helper module, for instance. Keeping worker exceptions in a shared, importable module is therefore not just tidiness; it is what makes them reportable at all. **Design consequence.** Exception classes that cross a process boundary are part of your serialization contract, not just your error vocabulary. Keep them plain: constructor parameters that map onto `args`, attributes that are simple data, no live resources, no callbacks. That is enough to make worker failures arrive intact, which is the difference between an actionable error and a mysterious `TypeError` from deep inside the result-handling machinery.