pickle and Untrusted Data

### A pickle is code, not a document The mental model that makes `pickle` dangerous is the one most people carry: "a pickle is a serialized object, like a JSON document with more types." It is not. A pickle stream is a **program** written for a small stack-based virtual machine that lives inside the `pickle` module. `pickle.loads` is an interpreter for that program. Opcodes push constants, build tuples and dicts, memoize intermediate results — and, crucially, import names and call things. Two opcodes carry the whole security story: * `GLOBAL` / `STACK_GLOBAL` — take a module name and an attribute name, import the module, and push `getattr(module, name)` onto the stack. Any importable name. * `REDUCE` — pop an argument tuple and a callable, call it, push the result. Put together, `<import something> <build args> REDUCE` is "call this function with these arguments", encoded in a few dozen bytes. ### Why the machinery exists This is not an oversight bolted on to a data format; it is the format's core extension point. When `pickle` meets an object it has no built-in encoding for, it asks the object how to rebuild itself by calling `__reduce_ex__`, which by default routes to `__reduce__`. The contract of `__reduce__` is: *return a callable and the arguments to call it with* (optionally plus state and iterators). The unpickler's job is to perform that call. So "unpickling calls a function chosen by the stream" is the feature, and a hostile stream simply chooses a different function. That is why the usual defences do not apply. There is no stage where the payload exists as inert parsed data that you could inspect before anything happens — the side effects *are* the parse. Checking `isinstance(result, MyClass)` after `pickle.loads` returns runs after the damage. Wrapping the call in `try`/`except pickle.UnpicklingError` does not help either: an exception can be raised long after a call has already succeeded. ```python import pickle class Payload: def __reduce__(self): return (print, ("side effect during loads",)) pickle.loads(pickle.dumps(Payload())) # prints before returning None ``` Swap `print` for anything importable that does real work and you have the classic remote-code-execution payload. Note also that the class `Payload` never has to exist on the loading side — the stream names `builtins.print`, not your class. ### What actually mitigates it Ranked honestly, for a Python service: 1. **Do not unpickle data that crossed a trust boundary.** If the bytes came from a client, a queue other teams write to, a downloaded artefact or a shared cache anyone can write, choose a data-only format. JSON, CSV and similar formats describe values only: a decoder for them constructs dicts, lists, strings and numbers and cannot be talked into calling your code. You still validate the decoded values — a data-only format prevents code execution, not bad data. 2. **Authenticate the bytes.** If a pickle must travel (a signed cache entry, a checkpoint file), compute an HMAC over the exact bytes with a key only your writer holds, and verify it with `hmac.compare_digest` *before* calling `pickle.loads`. This proves the bytes came from you; it does nothing if the writer itself is compromised, and it is not encryption. 3. **Restrict the reachable globals.** Subclass `pickle.Unpickler` and override `find_class(module, name)` to raise unless the pair is on a small allow-list. This is a real narrowing — it is what the standard library documentation recommends when you have no alternative — but it is a hardening measure, not a boundary: allowed classes still run their own `__setstate__`, and a purely "safe" allow-list can still be driven into memory exhaustion by a crafted stream. ### Where pickle is fine Inside a trust boundary, pickle is an excellent tool and there is no reason to avoid it: objects handed between worker processes on the same host, a local memo cache written and read by the same code, an interpreter session snapshot. The rule is about *provenance*, not about the module. The moment you cannot name the process that wrote the bytes and prove it was yours, the format is the wrong one.