How does `copyreg.pickle` make a type you do not own picklable, when an ETL export must ship it through a process pool?

seniorshould knowfreq 30%

answer

You cannot edit the class
A registry keyed by type
Same reduce tuple, written from outside
Exact type only, no subclasses
Must run at import in every process

basics

~10 s

copyreg.pickle(SomeType, reduce_func) registers a reduction function for a type whose source you cannot change. It stores the function in copyreg.dispatch_table, and the pickler consults that table for objects of exactly that type.

solid answer

~40 s

copyreg.pickle(SomeType, reduce_func) registers reduce_func in copyreg.dispatch_table under that type. reduce_func takes the instance and returns the same thing __reduce__ would — a callable plus its argument tuple — so you write the rebuild recipe from outside the class. For an export job that hands work to worker processes, that recipe is where you drop the live resource: return a module-level factory plus the parameters needed to reopen, so the payload never carries an open handle and the worker never inherits one it will leave unclosed. Two operational catches: the table is keyed on the exact type, so subclasses are not covered, and registration is process-global state, so it must run at import time in every process that pickles or unpickles, not inside the function that happens to call pickle.dumps.

`copyreg.pickle(SomeType, reduce_func)` registers `reduce_func` in `copyreg.dispatch_table` under that type. `reduce_func` takes the instance and returns the same thing `__reduce__` would — a callable plus its argument tuple — so you write the rebuild recipe from outside the class. For an export job that hands work to worker processes, that recipe is where you drop the live resource: return a module-level factory plus the parameters needed to reopen, so the payload never carries an open handle and the worker never inherits one it will leave unclosed. Two operational catches: the table is keyed on the **exact** type, so subclasses are not covered, and registration is process-global state, so it must run at import time in every process that pickles or unpickles, not inside the function that happens to call `pickle.dumps`.

code

python · 22 lines

python

import copyreg
import pickle
import threading

class ExportCursor:                    # a type from a library you cannot edit
    def __init__(self, table):
        self.table = table
        self.lock = threading.Lock()   # a live resource, not picklable

try:
    pickle.dumps(ExportCursor("orders"))
except TypeError as exc:
    print("before:", exc)              # cannot pickle a lock object

def rebuild_cursor(table):
    return ExportCursor(table)

def reduce_cursor(cursor):
    return rebuild_cursor, (cursor.table,)   # rebuild parameters only

copyreg.pickle(ExportCursor, reduce_cursor)
print("after:", pickle.loads(pickle.dumps(ExportCursor("orders"))).table)

go deeper

for a junior

Know that a type you did not write can still be made picklable from outside, by registering a function that says how to rebuild its instances, rather than by editing the library.

for a middle

Explain the registration call, the dispatch table it fills, and the shape of the reduction function's return value. Be able to write one for a small class in an interview.

for a senior

Show the operational reasoning: exclude live handles from the payload so the rebuilding process owns what it opens, place the registration at import time for worker processes, and know that subclasses are missed.

for a principal

Weigh a process-global registry against a wrapper type your own code controls, and set the team norm for what may cross a process boundary at all rather than fixing each failure as it appears.

The problem `copyreg` exists for

You own the code that pickles the object; you do not own the class. A library type holds a lock, a socket or an open cursor, so pickle.dumps fails with a TypeError saying the object cannot be pickled, and you cannot add __reduce__ to a class you did not write. copyreg is the standard library's answer: a registry of reduction functions, keyed by type, that the pickler consults for types that do not carry their own hook.

copyreg.pickle(ob_type, pickle_function) is the registration call. It validates that the function is callable and stores it in copyreg.dispatch_table, a plain dict mapping type to function. The function takes one argument — the instance — and returns exactly what __reduce__ would: a callable plus a tuple of arguments, optionally followed by state and iterator items. Everything you know about reduce tuples applies unchanged; only where the recipe lives has moved.

The ETL case, concretely

A four-person team runs a nightly export into a warehouse. The job builds cursor objects from a vendor library and fans batches out to worker processes, which serialises every argument. Two bad outcomes are possible without a reduction. Either the dump fails outright because the cursor holds an unpicklable handle, or — worse, when the handle happens to be picklable — each worker deserialises something that looks like a live cursor, opens or half-inherits a connection nobody owns, and leaves it unclosed at the end of the batch, so the run slowly exhausts the warehouse's connection budget.

The reduction fixes both by drawing an explicit line: the payload carries only the parameters a cursor is built from — the table name, the credentials reference, the batch bounds — and a module-level factory that opens a fresh handle in the process that will actually use it. Rebuilding is then the worker's responsibility, and the handle it opens is one it can close.

The catches that separate a senior answer

Exact-type keying. The lookup is a dict get on type(obj). A subclass of the registered type is a different key and gets no reduction, so it fails exactly as before. Register each concrete type you actually pickle, or register the subclass too.

Where registration runs. copyreg.dispatch_table lives in one process's memory. If the registration sits inside the function that submits work, it never runs in a worker — and any process that loads the payload needs the factory importable, though not the registration itself. The reliable placement is at import time in a module that every participant imports.

It is global state. Registering changes serialisation for that type everywhere in the process, including for library code you did not write and did not consider. Where you can, prefer a small wrapper type you own with its own __reduce__ and pass that around; reach for copyreg when the foreign type is already deep inside data structures you do not control.

Types the pickler handles internally are unaffected. Built-in types such as int, str and list are dispatched by the pickler before the table is consulted, so registering a reduction for them changes nothing. The table is for classes and extension types.

The recipe must be resolvable by name. The callable you return is pickled by module plus qualified name like any other function reference, so it has to be a module-level function, not a closure built next to the registration.

The neighbouring mechanisms

The same copyreg.dispatch_table is consulted by the copy module, so a registration also changes copying behaviour for that type — usually what you want, occasionally a surprise. And a pickle.Pickler instance can carry its own private dispatch table, which is the scoped alternative when one call site needs a special reduction and the rest of the process should be left alone.

Versions

copyreg and its dispatch table have been stable for many releases; nothing about the registration API changed in Python 3.14. What did change is the context this most often bites in: multiprocessing's default start method on Unix other than macOS became forkserver in 3.14, where macOS and Windows already used spawn. Under both, workers re-import your modules rather than inheriting the parent's memory, so import-time registration is not optional hygiene — it is the only placement that works.

### The problem `copyreg` exists for You own the code that pickles the object; you do not own the class. A library type holds a lock, a socket or an open cursor, so `pickle.dumps` fails with a `TypeError` saying the object cannot be pickled, and you cannot add `__reduce__` to a class you did not write. `copyreg` is the standard library's answer: a registry of reduction functions, keyed by type, that the pickler consults for types that do not carry their own hook. `copyreg.pickle(ob_type, pickle_function)` is the registration call. It validates that the function is callable and stores it in `copyreg.dispatch_table`, a plain dict mapping type to function. The function takes one argument — the instance — and returns exactly what `__reduce__` would: a callable plus a tuple of arguments, optionally followed by state and iterator items. Everything you know about reduce tuples applies unchanged; only *where the recipe lives* has moved. ### The ETL case, concretely A four-person team runs a nightly export into a warehouse. The job builds cursor objects from a vendor library and fans batches out to worker processes, which serialises every argument. Two bad outcomes are possible without a reduction. Either the dump fails outright because the cursor holds an unpicklable handle, or — worse, when the handle happens to be picklable — each worker deserialises something that looks like a live cursor, opens or half-inherits a connection nobody owns, and leaves it unclosed at the end of the batch, so the run slowly exhausts the warehouse's connection budget. The reduction fixes both by drawing an explicit line: the payload carries only the *parameters* a cursor is built from — the table name, the credentials reference, the batch bounds — and a module-level factory that opens a fresh handle in the process that will actually use it. Rebuilding is then the worker's responsibility, and the handle it opens is one it can close. ### The catches that separate a senior answer **Exact-type keying.** The lookup is a dict `get` on `type(obj)`. A subclass of the registered type is a different key and gets no reduction, so it fails exactly as before. Register each concrete type you actually pickle, or register the subclass too. **Where registration runs.** `copyreg.dispatch_table` lives in one process's memory. If the registration sits inside the function that submits work, it never runs in a worker — and any process that *loads* the payload needs the factory importable, though not the registration itself. The reliable placement is at import time in a module that every participant imports. **It is global state.** Registering changes serialisation for that type everywhere in the process, including for library code you did not write and did not consider. Where you can, prefer a small wrapper type you own with its own `__reduce__` and pass that around; reach for `copyreg` when the foreign type is already deep inside data structures you do not control. **Types the pickler handles internally are unaffected.** Built-in types such as `int`, `str` and `list` are dispatched by the pickler before the table is consulted, so registering a reduction for them changes nothing. The table is for classes and extension types. **The recipe must be resolvable by name.** The callable you return is pickled by module plus qualified name like any other function reference, so it has to be a module-level function, not a closure built next to the registration. ### The neighbouring mechanisms The same `copyreg.dispatch_table` is consulted by the `copy` module, so a registration also changes copying behaviour for that type — usually what you want, occasionally a surprise. And a `pickle.Pickler` instance can carry its own private dispatch table, which is the scoped alternative when one call site needs a special reduction and the rest of the process should be left alone. ### Versions `copyreg` and its dispatch table have been stable for many releases; nothing about the registration API changed in Python 3.14. What did change is the context this most often bites in: `multiprocessing`'s default start method on Unix other than macOS became `forkserver` in 3.14, where macOS and Windows already used `spawn`. Under both, workers re-import your modules rather than inheriting the parent's memory, so import-time registration is not optional hygiene — it is the only placement that works.

Where must the `copyreg.pickle` call run for a process pool to work, and why?
At import time, in a module every participating process imports. The dispatch table is per-process memory, so a registration inside the submitting function never reaches a worker. Under the `spawn` and `forkserver` start methods children re-import rather than inheriting the parent's state, and since Python 3.14 `forkserver` is the Unix default outside macOS, so the inherited-memory shortcut is no longer available.
Does a reduction registered for a base class also apply to its subclasses?
No. The pickler looks the exact `type(obj)` up in the table, so a subclass is a different key and falls back to the default reduction, failing exactly as it did before. Register every concrete type you pickle, or give the subclass its own `__reduce__` if you own it.
When would you wrap the foreign object in a type you own instead of registering a reduction?
Whenever you control the boundary the object crosses. A wrapper with its own `__reduce__` is local, explicit, discoverable by the next reader and testable in isolation, while `copyreg` mutates behaviour for the whole process, including for library code that never asked for it. Registration wins when the foreign objects are already buried inside structures you do not construct.

saying these in an interview costs you the question

Assumes the registration covers subclasses of the registered type
Registers inside the function that pickles, so workers never see it
Puts the live handle into the reduce arguments
Thinks `copyreg` makes an object picklable without a rebuild recipe
Ignores that the registry is process-global shared state
Returns a closure as the rebuild callable

How does `copyreg.pickle` make a type you do not own picklable, when an ETL export must ship it through a process pool?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

The problem copyreg exists for

The ETL case, concretely

The catches that separate a senior answer

The neighbouring mechanisms

Versions

▾common follow-ups3 questions

▾red flags6 flags

§ context

▾related questions5 questions

▾related topics3 topics

The problem copyreg exists for

The ETL case, concretely

The catches that separate a senior answer

The neighbouring mechanisms

Versions

answer

go deeper

The problem `copyreg` exists for

context

The problem `copyreg` exists for