How does multiprocessing.shared_memory.SharedMemory avoid pickling a large buffer between processes?
answer
- One block, many address spaces
- Only the name crosses the boundary
- The buffer is a memoryview, zero copy
- You supply the layout and the locking
- close everywhere, unlink exactly once
basics
~20 sIt allocates one named block of raw memory that the operating system maps into every process that attaches to it. Only the short name travels between processes; reads and writes through the block's memoryview hit the same physical bytes, with no serialization.
solid answer
~40 s`multiprocessing.shared_memory.SharedMemory(create=True, size=n)` asks the OS for a named block of raw bytes and maps it into the calling process. Any other process constructs `SharedMemory(name=...)` and gets the same physical pages mapped into its own address space, so the only thing that crosses the process boundary is a short name string. The block exposes `.buf`, a `memoryview` over the raw bytes, so slicing and assignment are zero-copy. What you do not get is structure or safety: the block is bytes, so you must lay out and interpret it yourself (or use `ShareableList` for a fixed set of simple values), and it provides no locking, so concurrent writers need a `multiprocessing.Lock`. Lifecycle is manual: every attached process calls `close()`, and exactly one process calls `unlink()` to destroy the block.
code
python · 11 linesfrom multiprocessing import shared_memory
block = shared_memory.SharedMemory(create=True, size=64)
block.buf[:6] = b"TICKET"
attached = shared_memory.SharedMemory(name=block.name)
print(block.name, bytes(attached.buf[:6]), attached.size)
attached.close()
block.close()
block.unlink()go deeper
Know that it exists and what it buys: a named block of bytes several processes can map, so only the name is sent instead of the data. Remember that it holds raw bytes, not Python objects.
Explain the mechanics: create with a size, attach by name, read and write through the memoryview, and manage the two-verb lifecycle where everyone closes but only one process unlinks. Note that you provide the layout yourself.
Show operational judgment: partition the block or lock it, plan for a worker dying mid-write since there is no rollback, avoid leaking segments, and know when a memory-mapped read-only file is the simpler and cheaper answer.
Frame it as a durability and ownership decision. Raw shared memory buys throughput and costs you consistency guarantees, so decide whether the workload can tolerate torn state, who owns cleanup on crash, and whether a slower but recoverable channel is worth the safety.
## The mechanism `multiprocessing.shared_memory.SharedMemory`, added in Python 3.8, is a thin wrapper over the platform's named shared-memory primitive — a POSIX shared-memory object on Unix, a named file mapping on Windows. Creating it with `SharedMemory(create=True, size=n)` allocates a block of at least `n` bytes and maps it into the creating process; the object's `.name` is a short string identifying that block system-wide. Any other process that constructs `SharedMemory(name=that_name)` gets the very same physical pages mapped into its own address space. Note "at least": the OS rounds the allocation up to a page multiple, so `.size` is frequently larger than what you asked for. Never treat `.size` as your logical payload length — store the length yourself, in a header inside the block or alongside the name. The block's `.buf` attribute is a `memoryview` over those bytes. Slicing it produces views, not copies, and assigning into a slice writes straight through to the shared pages. That is the whole win: a worker receives a name of a few dozen bytes, attaches, and reads a 2.4 GB working set without a single byte of pickling. ## What it deliberately does not give you **Structure.** A block is bytes. Anything with fields, lengths or variable-size records is a layout you design and encode yourself, typically with `struct` or by viewing the buffer as an `array.array`. `ShareableList` exists for the easy case — a fixed-length sequence of simple immutable values — but it cannot grow and its per-element sizes are fixed at creation. **Synchronization.** There is no implicit locking. Two processes writing overlapping regions, or one reading while another writes, race exactly as they would in any shared-memory system. Pair the block with a `multiprocessing.Lock`, or partition it so each worker owns a disjoint region and never touches anyone else's. **Atomicity.** This is where a triage bot that shares a 2.4 GB block of candidate classifications gets hurt. If a worker is killed halfway through updating a region, there is no transaction and no rollback: the block keeps whatever partial bytes landed, and every reader afterwards sees a torn record as if it were valid. Shared memory has no journal. The standard remedies are to make updates single-word and atomic-by-construction, to write into a scratch region and flip an index or epoch counter under a lock once the write is complete, or to keep a per-region status byte a reader checks before trusting the payload. Whichever you choose, the recovery discipline is yours to build. ## Lifecycle, the part that bites Two different verbs, and mixing them up is the most common production defect here: * `close()` unmaps the block from *this* process. Every process that attached should call it. * `unlink()` destroys the block system-wide. Exactly one process should call it, exactly once, after everyone has finished. On Windows `unlink()` has no effect — the block disappears when the last handle closes. Forgetting `unlink()` on Unix leaks a segment that survives your program and keeps occupying memory until reboot or manual cleanup. To catch that, `multiprocessing` runs a `resource_tracker` process that knows about created blocks and cleans up leaked ones at exit, printing a warning that names them — which is also why you sometimes see "leaked shared_memory objects" warnings from a program that carefully unlinked in a child, because the tracker in the parent had also registered the block. Python 3.13 added a `track` parameter to opt a block out of that tracker when you are managing its lifetime yourself. One more trap: you cannot `close()` a block while any `memoryview` derived from `.buf` is still alive. Release your views first, or the close raises. ## When a memory-mapped file is the better tool For a large **read-only** dataset that already exists on disk, `mmap.mmap(fileno, 0, access=mmap.ACCESS_READ)` is usually a better fit than a shared block. Every process maps the same file, the kernel's page cache backs it so pages are shared and evictable under pressure, nothing needs to be loaded up front, and there is no lifecycle to manage beyond closing the mapping. Reach for `SharedMemory` when the data is produced at runtime rather than read from a file, or when workers must write into it.
- A worker is killed midway through writing into the shared block. What does the next reader see?Whatever partial bytes landed, indistinguishable from a complete record — there is no transaction and no rollback. Make updates safe by construction: write into a scratch region and flip an index or epoch under a lock once complete, or keep a per-region status flag readers check before trusting the payload. The block itself will never help you detect the tear.
- Who should call unlink, and what happens if nobody does?Exactly one process, once, after every attached process has closed. If nobody does, the named block outlives your program on Unix and keeps occupying memory until it is removed manually or the machine reboots. The `multiprocessing.resource_tracker` mitigates this by unlinking blocks it believes leaked and warning about them at exit.
- When would you choose a memory-mapped file over a shared memory block?When the dataset is large, read-only and already on disk. Every process maps the same file, the kernel page cache backs it so the pages are shared and evictable under memory pressure, there is no upfront load, and there is no cross-process lifecycle to get wrong. A shared block is the right choice when the data is produced at runtime or when workers must write into it.
It is a whiteboard in a shared room rather than photocopies handed to each person: everyone sees the same surface, nobody gets a copy, and nothing stops two people writing over each other.
saying these in an interview costs you the question
- Assumes the block synchronizes concurrent writers for you
- Calls unlink in every process instead of exactly one
- Treats the block's size attribute as the payload length
- Expects Python objects, not raw bytes, to live in the block
- Thinks a killed writer leaves the block consistent
- Never closes, then wonders why the memoryview blocks cleanup