__reduce__ and copyreg

### What the reduce protocol actually is Pickling never copies an object's memory. It writes a small program of opcodes that, when replayed, *reconstructs* an equivalent object. The reduce protocol is how an object dictates that program: it hands back a callable plus the arguments needed to call it. Think of it as posting a recipe rather than a photograph. The entry point is `__reduce_ex__(protocol)`. The pickler calls that, never `__reduce__` directly. `object.__reduce_ex__` looks at whether the class overrides `__reduce__`; if it does, it calls it and uses the result, and if it does not, it builds a default reduction itself using helpers from `copyreg`. So writing `__reduce__` is enough for almost every case; you override `__reduce_ex__` only when the recipe genuinely has to differ per protocol version. ### The tuple The return value is either a string or a tuple of two to six items: 1. **A callable.** Called at load time to produce the object. 2. **A tuple of arguments** for that callable. It may be empty, but it must be a tuple. 3. *(optional)* **State**, handed to the rebuilt object's restore step. 4. *(optional)* **An iterator of items** appended to the rebuilt object, for list-like types. 5. *(optional)* **An iterator of key/value pairs** set on the rebuilt object, for dict-like types. 6. *(optional)* **A callable** used instead of the default way of applying the state. Most hand-written reductions use only the first two. The list and dict iterator slots exist so a huge container can be streamed in pieces instead of materialised as one giant argument tuple. ### How the callable is stored This is where most interview follow-ups go. The callable is itself pickled, and functions and classes are pickled **by reference**: `pickle` writes the callable's module name and qualified name and looks that pair up again on load. Two consequences fall straight out of that. First, the callable has to be reachable at module level in an importable module — a `lambda`, a function defined inside another function, or a class built at runtime all fail with `pickle.PicklingError` because the lookup cannot resolve them. Second, the arguments in the tuple are pickled recursively, so they must be picklable too, and they are the wrong place to put an open file, a socket or a lock. ### The string form If `__reduce__` returns a plain string, `pickle` treats it as the name of a global to look up in the object's module at load time, and stores no state at all. That is the idiomatic way to make a sentinel survive a round trip with its identity intact: ```python class _Missing: def __reduce__(self): return "MISSING" MISSING = _Missing() ``` After `pickle.loads(pickle.dumps(MISSING))`, the result **is** `MISSING`, so `is` comparisons in the loading process still work. The pickler verifies that the name really resolves back to the object being pickled and raises `pickle.PicklingError` if it does not, which catches the case where the global was renamed. ### Who else uses it The reduce protocol is not only for files on disk. Anything that moves objects between processes goes through it: sending arguments to a worker in a process pool serialises them the same way, so a "this cannot be pickled" error in a pool is a reduce-protocol error. Because of that, `__reduce__` is worth reaching for whenever an object is *reconstructible from a few parameters* — a connection described by a URL, a compiled pattern described by its source, a handle described by the identifier it was opened with. ### Practical cautions Keep the reduce callable boring: a module-level factory function or the class itself. Keep the arguments small and immutable — they are the payload, and anything huge in them is written into every pickle. Remember that the callable runs at load time with whatever arguments the payload contains, so a reduce recipe is executable content; that is the reason loading data you did not produce is dangerous, which is a separate discussion from writing the hook. Finally, if you want to see what your hook actually produced, disassemble the bytes with `pickletools.dis` rather than guessing from the repr. ### Versions The protocol itself has been stable for years, but the default protocol has not: `pickle.DEFAULT_PROTOCOL` is 5 on Python 3.14, and was 4 from 3.8 through 3.13. Protocol only affects the *encoding* and which default reduction path is used, not the shape of the tuple you return.