skip to content

Which objects can cross a `concurrent.interpreters` queue, and which cannot?

level: middleimportance: should knowfreq 26%

answer

  1. Three tiers, not a yes/no answer
  2. Efficient, copied, or refused
  3. Equal on arrival, never identical
  4. memoryview is the real sharer
  5. prepare_main has no copy fallback

basics

~20 s

Immutable scalars and tuples of them cross efficiently, and a memoryview genuinely shares its buffer. Dicts, lists and ordinary instances are copied, so the receiver gets an equivalent new object. Locks, sockets, generators, modules and closures raise NotShareableError.

solid answer

~50 s

There are three tiers. `concurrent.interpreters.is_shareable()` returns `True` for the values that cross cheaply — `None`, `bool`, `int`, `float`, `str`, `bytes`, tuples whose items are themselves shareable, `memoryview`, and the cross-interpreter types such as a `Queue` or an `Interpreter`. Everything else that can be serialized still goes through `Queue.put()`, but as a **copy**: put a dict in, and `Queue.get()` on the far side hands back an equivalent new dict, never the same object, so mutating it afterwards changes nothing on the other side. The third tier is refused outright with `concurrent.interpreters.NotShareableError`: anything bound to interpreter or OS state, such as a `threading.Lock`, an open file, a socket, a generator, a module object or a closure. `memoryview` is the interesting exception — it really does share the underlying buffer, so writes are visible across interpreters. Note `Interpreter.prepare_main()` is stricter than the queue and accepts only shareable values.

code

python · 16 lines
python
import threading
from concurrent import interpreters

print(interpreters.is_shareable("EUR"), interpreters.is_shareable((1, 2)))
print(interpreters.is_shareable([1, 2]), interpreters.is_shareable({"amount": 1}))

q = interpreters.create_queue()
rows = [{"id": 1}, {"id": 2}]
q.put(rows)
received = q.get()
print("equal:", received == rows, "identical:", received is rows)

try:
    q.put(threading.Lock())
except interpreters.NotShareableError:
    print("a threading.Lock cannot cross")

go deeper

for a junior

Hold on to the headline: simple immutable values and tuples cross cheaply, containers are copied, and objects tied to the running interpreter such as locks or open files are rejected.

for a middle

Be able to name the three tiers, use concurrent.interpreters.is_shareable() to classify a value, and explain that a received dict is equal to but not identical with the one that was sent.

for a senior

Show that you design the worker protocol around the tiers: small immutable control messages, memoryview for bulk payloads, resources rebuilt on the far side from a path or a connection string rather than shipped across.

for a principal

Own the cost model. Argue where copy-per-message is acceptable and where a shared buffer or an out-of-band store is the right answer, and set the team convention for message shapes before the protocol calcifies.

### Three tiers, not two People reach for a binary — "shareable or not" — and get the model wrong. In Python 3.14 the cross-interpreter queue created by `concurrent.interpreters.create_queue()` sorts objects into three groups. **Tier 1 — actually shareable.** `concurrent.interpreters.is_shareable()` reports these: `None`, `bool`, `int`, `float`, `str`, `bytes`, a `tuple` whose items are themselves shareable, `memoryview`, and the cross-interpreter objects themselves (a `Queue`, an `Interpreter`). These have a cheap, purpose-built path across the boundary. Note that "shareable" does not mean "identical": put a `str` in and the object you `get()` on the other side is not the same object, it is an equivalent one produced cheaply. The one member of the tier that genuinely shares memory is `memoryview` — a view onto a `bytearray` handed across a queue lets the receiving interpreter write into the sender's buffer, and the sender sees the change. That makes it the tool for moving a large payload without copying it. **Tier 2 — copied.** A `dict`, a `list`, a set, an ordinary instance of your own class: `is_shareable()` says `False`, but `Queue.put()` accepts them anyway and falls back to serializing them. The receiver gets a fresh, equal object. This is the tier where a mental model matters most, because the code reads exactly like a thread-safe queue and behaves nothing like one: with `queue.Queue` two threads hold the *same* dict, so a mutation on one side is a mutation on both; with a cross-interpreter queue they hold two dicts that merely started out equal. ```python from concurrent import interpreters q = interpreters.create_queue() rows = [{"id": 1}] q.put(rows) received = q.get() print(received == rows, received is rows) # True False ``` **Tier 3 — refused.** Some objects have no meaning outside the interpreter or the moment that made them, and `Queue.put()` raises `concurrent.interpreters.NotShareableError` rather than pretending. A `threading.Lock` (a lock belongs to one interpreter's threading state), a generator (it holds a live frame), an open file object, a socket, a module object, and — the one that bites during a port — a **closure** over free variables. ### Why `NotShareableError` is a feature Every refusal in tier 3 is an object whose copy would be silently wrong. Copying a `threading.Lock` would give two locks that guard nothing in common; copying a socket object would give a wrapper around a file descriptor with no owner. Raising is the honest outcome. When you hit it, the fix is almost always to send the *data* and rebuild the *resource* on the far side: send a path and open the file there, send a connection string and connect there, send the loop-invariant configuration as an argument instead of capturing it in a closure. ### The queue is a cross-interpreter object itself A `Queue` returned by `create_queue()` is in tier 1, which is what makes the whole pattern work: you create the queue in the parent, hand it to the child with `Interpreter.prepare_main(q=queue)` or as an argument, and both sides then hold usable handles to the same underlying queue. `Queue.put()`, `Queue.get()`, `Queue.get_nowait()`, `Queue.qsize()` and `Queue.empty()` mirror the familiar `queue.Queue` API, and `get_nowait()` on an empty queue raises `concurrent.interpreters.QueueEmpty`. ### `prepare_main()` is stricter than the queue A detail worth knowing because it surprises people: `Interpreter.prepare_main()` has **no** copying fallback. Passing a list raises `NotShareableError` even though the same list would go through a queue fine. Seed the child with shareable values — a tuple, a string, a queue — and push structured data over the queue afterwards. ### Designing around the tiers The practical rule for a worker protocol: **make your messages tier 1 or cheap tier 2.** Prefer tuples and strings for control messages, `bytes` or a `memoryview` for bulk payloads, and small plain dicts for structured records. Avoid sending large nested structures repeatedly, because each `put()`/`get()` pair is a full copy; if the same reference table is needed by every worker, it is cheaper to build it once per interpreter than to ship it per task. And never design a protocol that depends on the receiver mutating an object the sender still holds — outside `memoryview`, that is exactly the thing this boundary does not do.

  • Why does `Interpreter.prepare_main()` reject a list that a queue accepts?
    `prepare_main()` binds names directly into the target interpreter's `__main__` and has no serialization fallback, so it accepts only genuinely shareable values — scalars, strings, bytes, tuples of shareable items, and cross-interpreter objects such as a queue. `Queue.put()` is more forgiving because it will copy anything it can serialize. The idiomatic pattern is to seed the child with a queue via `prepare_main()` and send everything structured over that queue.
  • You need to hand a worker interpreter a 200 MB payload. What do you send?
    A `memoryview` over the buffer, which is the one object that genuinely shares its underlying memory across interpreters rather than being copied — the receiving side reads, and can write, the same bytes. Sending a `bytes` object or a list of records would serialize and duplicate the payload per worker. If the data lives on disk, cheaper still is to send the path and have each interpreter open or map it itself.
  • How does this differ from putting an object on a `queue.Queue` between threads?
    `queue.Queue` moves references inside one interpreter: two threads end up holding the same object, so a mutation on one side is immediately visible on the other and shared state needs a lock. A cross-interpreter queue moves values: outside `memoryview`, the receiver gets an equal but distinct object, so there is no shared mutable state to guard and no way for one side to observe the other's later mutations.

Sending a document by post: some things go as a cheap postcard, most go as a photocopy that the recipient can scribble on without touching your original, and a few things — your house keys, your dog — the post office simply refuses.

saying these in an interview costs you the question

  • Thinking anything not shareable is automatically rejected
  • Expecting the received object to be the same object
  • Assuming a dict crosses by reference like on a thread queue
  • Saying nothing at all can be shared without copying
  • Trying to send a lock, socket or generator to a worker
  • Assuming prepare_main accepts whatever a queue accepts

context

resolving links…