How do you make an object holding a threading.Lock picklable for a worker process?

## How a plain instance pickles With no hooks defined, pickling an instance produces two things: - a **reference to its class**, recorded as `__module__` plus `__qualname__` the same way a function is; - and the **instance's state**, which for an ordinary object is its `__dict__`. Unpickling imports the class in the target process, allocates an instance without calling `__init__`, and then updates its `__dict__` with the restored state. Two consequences follow immediately: the class must be importable in the worker, and every *value* in `__dict__` must itself be picklable. ## Why some attributes refuse A `threading.Lock` is a handle on a synchronization primitive owned by one process's kernel or runtime. A `socket.socket` wraps a file descriptor whose number means nothing in another process. An open file object, a database connection, a generator paused mid-iteration, a running coroutine, an inner class or a lambda — all of these either cannot be reconstructed from bytes or would be actively wrong if they were. They therefore raise rather than pretend, and the traceback names the whole object graph you tried to send, not just the leaf that failed. ## The `__getstate__` / `__setstate__` contract - **`__getstate__`** is called during pickling and returns the state to store; returning a shallow copy of `__dict__` with the unpicklable keys deleted is the idiom. - **`__setstate__`** is called during unpickling with exactly that value, and its job is to put the object back into a usable condition — typically `self.__dict__.update(state)` followed by recreating whatever was dropped, such as a brand-new lock. Note that `__init__` is *not* called on the unpickling side, so anything `__init__` would have set up must be re-established in `__setstate__` or lazily on first use. If you define `__getstate__` but forget `__setstate__`, the default restore simply applies the reduced state and the attribute is silently absent until the first access blows up with `AttributeError` inside a worker — a nasty place to discover it. ## The alternatives - **`__reduce__`** gives you full control: it returns a callable and the arguments to call it with, plus optional state, so you can specify "rebuild me by calling this factory". - **`copyreg.pickle`** registers a reducer for a type you do not own, which is the escape hatch when the unpicklable attribute belongs to someone else's class. Both are heavier than the state hooks and are worth it mainly when reconstruction is more than "drop and recreate". ## The semantic trap Getting an object with a lock across the boundary **does not give you a shared lock**. The child gets a *different* lock object that guards nothing the parent can see, because the two processes have separate memories. If you actually need mutual exclusion across processes you need a cross-process primitive, not a pickled one; the state hooks are for objects that merely *carry* a lock as an implementation detail while the meaningful payload is data. The same is true for a socket or a connection: silently reconstructing one per unpickle can quietly open a connection per task, which is a load bug hiding inside a serialization fix. ## When not to ship the object at all Often the honest answer in an interview is that the object should not cross. If a worker needs a connection, let each worker process open its own once — at worker start-up, or lazily on first use keyed by `os.getpid()` — and send only the small, plain data each task needs: identifiers, ranges, configuration. That: - keeps payloads small, - avoids reconstructing expensive resources per task, - and sidesteps the whole class of pickling errors instead of patching them one class at a time. A useful rule of thumb: **if `__getstate__` is dropping more than it keeps, you are sending the wrong object.** ## Two details that bite 1. First, the class itself must be importable in the receiving process under the same `__module__` and `__qualname__` it had in the sender, because the class travels as a reference just like a function does; a class defined inside a function, or one whose module cannot be imported in a worker, fails no matter how clean its state is. 2. Second, `__slots__` changes the shape of the state: an object without a `__dict__` reduces to a two-element tuple of dictionary state and slot state, so a hand-written `__setstate__` must be prepared to receive that shape rather than a plain dictionary. Both surface only across a real boundary, which is why the round-trip test below is worth more than it looks. ## Testing it The cheapest guard is a unit test that does `pickle.loads(pickle.dumps(obj))` and asserts the rebuilt object is usable. Pickling failures otherwise surface only under a real pool, often on a rarely taken branch, and the error arrives from a worker with little context about which attribute was to blame.