skip to content

Why does a `tuple` subclass with a custom `__new__` fail to unpickle, and what fixes it?

level: middleimportance: nice to knowfreq 16%

answer

  1. Loading does not call `__init__`
  2. The allocator needs its arguments back
  3. Immutable values are fixed at construction
  4. The inherited hook matches the base signature
  5. One method whose return matches `__new__`

basics

~20 s

Loading rebuilds the instance by calling cls.new(cls, *args), where args come from getnewargs. A tuple subclass inherits one that returns the whole tuple, so a two-parameter new receives a single argument and raises TypeError. Define getnewargs to match new.

solid answer

~40 s

For protocol 2 and later, `object.__reduce_ex__` builds a default reduction that reconstructs the instance through `cls.__new__(cls, *args)` and never calls `__init__`. Those arguments come from `__getnewargs__` — or `__getnewargs_ex__`, which returns a positional tuple and a keyword dict — and default to nothing at all. That is fine for a mutable class, whose blank instance can have its attributes filled in afterwards, but an immutable type carries its value in construction, so the arguments must be reproduced. A `tuple` subclass inherits `tuple.__getnewargs__`, which returns a one-item tuple holding the whole tuple, so a `__new__` declared as `(cls, x, y)` gets a single argument and the load fails with a `TypeError` about a missing parameter. The fix is to define `__getnewargs__` returning exactly what your own `__new__` expects.

code

python · 19 lines
python
import pickle

class Bad(tuple):
    def __new__(cls, x, y):
        return super().__new__(cls, (x, y))

try:
    pickle.loads(pickle.dumps(Bad(3, 4)))
except TypeError as exc:
    print("without:", exc)    # Bad.__new__() missing 1 required positional argument: 'y'

class Point(tuple):
    def __new__(cls, x, y):
        return super().__new__(cls, (x, y))

    def __getnewargs__(self):
        return (self[0], self[1])

print("with:", pickle.loads(pickle.dumps(Point(3, 4))))

go deeper

for a junior

Know that rebuilding a pickled object does not run __init__, so a class whose constructor takes required arguments needs a way to hand those arguments back.

for a middle

Explain the default reconstruction through cls.__new__(cls, *args), where those arguments come from, and why an inherited hook describes the base class's signature rather than yours.

for a senior

Show that you round-trip every persisted type in tests, because this mismatch is silent at write time and only fails in the process that later reads the data.

for a principal

Decide whether value types that are persisted should rely on language hooks at all, or be defined by an explicit schema with construction rules that do not shift when a constructor signature changes.

### Where the arguments come from When a class defines no `__reduce__` of its own, `object.__reduce_ex__(protocol)` composes a default reduction. Under protocol 2 and later the callable it produces reconstructs the object by calling `cls.__new__(cls, *args)` — the class's own allocator, with `__init__` deliberately skipped, because rerunning an initialiser would re-do work and could have side effects. The `args` in that call are not invented: they come from `__getnewargs__` if the class has one, from `__getnewargs_ex__` if keyword arguments are needed, and otherwise from nothing at all, giving `cls.__new__(cls)`. For an ordinary mutable class this is invisible. `__new__` accepts no extra parameters, a blank instance appears, and the attribute state recorded in the reduction is applied afterwards. The object is whole and nobody thinks about the hook. ### Why immutable types are different An immutable object cannot be filled in after the fact. Its value is fixed by `__new__`, so a reduction that constructs a blank instance and then assigns attributes cannot reproduce it. This is exactly what `__getnewargs__` exists for: it lets the class say "to build me again, call `__new__` with these arguments". The built-in immutable types implement it — `"abc".__getnewargs__()` returns `('abc',)`, and `(3, 4).__getnewargs__()` returns `((3, 4),)`, one argument holding the whole tuple, which is exactly what `tuple.__new__` wants. ### The trap in a subclass Now subclass `tuple` with your own two-parameter constructor: ```python class Bad(tuple): def __new__(cls, x, y): return super().__new__(cls, (x, y)) ``` Dumping works. Loading fails, with `TypeError: Bad.__new__() missing 1 required positional argument: 'y'`. The reason is that `Bad` inherited `tuple.__getnewargs__`, which reports one argument — the whole tuple — while `Bad.__new__` demands two. The inherited hook describes the base class's constructor signature, not yours, and nothing warns you at dump time; the mismatch only surfaces in whatever program later reads the data. The fix is one method: ```python class Point(tuple): def __new__(cls, x, y): return super().__new__(cls, (x, y)) def __getnewargs__(self): return (self[0], self[1]) ``` The rule is simply that `__getnewargs__` must return a tuple that can be splatted into your own `__new__`. If your constructor needs keyword-only parameters, define `__getnewargs_ex__` instead, returning a two-item tuple of a positional argument tuple and a keyword dictionary; a class should define one or the other, not both. ### How this relates to writing `__reduce__` These hooks belong to the *default* reduction. The moment a class defines `__reduce__`, `object.__reduce_ex__` delegates to it wholesale and `__getnewargs__` is never consulted — your tuple is the recipe, start to finish. That is worth stating explicitly in an interview, because candidates often add both and then debug a hook that is not running. Choose one: `__getnewargs__` when the default machinery is nearly right and only the construction arguments are missing, a full `__reduce__` when you want to control the callable as well. ### Related sharp edges The same reasoning applies to any class whose `__new__` has required parameters, immutable or not — a class hierarchy that pushes required arguments into `__new__` will fail to round-trip until the newargs hook matches. Frozen value objects assembled by libraries usually generate the hook for you; hand-written ones frequently do not. And because the failure appears at load time in a different process or a different day's run, it is the sort of defect that reaches production data before anyone sees it, so a round-trip test of every persisted type is cheap insurance. ### Versions The newargs hooks only participate in protocol 2 and later; protocols 0 and 1 use an older reconstruction path. That is a non-issue in practice, since `pickle.DEFAULT_PROTOCOL` is 5 on Python 3.14 and was 4 from 3.8 through 3.13. `__getnewargs_ex__`, the keyword-capable form, is likewise a protocol 2-and-later mechanism. ### Reading the failure The diagnostic habit worth showing is to ask the object what it will produce *before* trusting a round trip. Calling `__getnewargs__` on an instance and mentally splatting the result into your own `__new__` answers the question in one line, and a round trip through `pickle.dumps` and `pickle.loads` in a unit test answers it for real. The error text is unusually helpful here: it names the class and the missing parameter, which points straight at a constructor signature rather than at the serialisation machinery, and that is why the cause is so often mislabelled as a pickle bug when it is really an arity mismatch the class itself declared. ### A note on what travels Only the newargs values are consumed by the allocator; anything else the object carries is handled by the state part of the default reduction and applied afterwards. So a subclass of an immutable built-in that also keeps extra attributes needs both halves to be right: the arguments must rebuild the underlying value, and the extra attributes ride along separately. Keeping the newargs tuple minimal — exactly the parameters `__new__` declares, nothing more — is what keeps the two halves from overlapping and writing the same data twice.

  • When do you need `__getnewargs_ex__` rather than `__getnewargs__`?
    When the constructor takes keyword arguments, including keyword-only parameters. `__getnewargs_ex__` returns a two-item tuple of a positional argument tuple and a keyword dictionary, which is then splatted into `cls.__new__`. A class defines one form or the other; defining both is a source of confusion because only one is consulted.
  • If a class defines `__reduce__`, is `__getnewargs__` still used?
    No. `object.__reduce_ex__` delegates entirely to a user-defined `__reduce__`, so the default reconstruction path — and with it the newargs hook — is bypassed. Your tuple names the callable and its arguments outright. Adding both and then wondering why the newargs hook never runs is a common debugging detour.
  • Why do mutable classes almost never need this hook?
    Their default reduction allocates a blank instance with no constructor arguments and restores attribute state afterwards, which is enough to reproduce the object. The hook is needed when a value cannot be assigned after allocation — immutable types — or when `__new__` declares required parameters that the blank-allocation path cannot satisfy.

saying these in an interview costs you the question

  • Thinks unpickling calls `__init__` on the rebuilt instance
  • Assumes subclassing an immutable built-in needs no pickle work
  • Returns the wrong number of items from `__getnewargs__`
  • Confuses construction arguments with restored attribute state
  • Defines both `__reduce__` and `__getnewargs__` and expects both to run
  • Expects the mismatch to be reported when dumping rather than loading

context