skip to content

Why does a live memoryview make bytearray.append raise BufferError, and how do you scope the view in a long-running ingest worker?

level: seniorimportance: should knowfreq 30%

answer

  1. The view holds a raw address
  2. Growing may move the storage elsewhere
  3. The exporting object counts its exports
  4. Length changes blocked, in-place writes fine
  5. memoryview is a context manager for a reason

basics

~10 s

A memoryview holds an export of the bytearray's storage, and a bytearray refuses to reallocate while any export is outstanding. So append, extend and clear raise BufferError; release the view before resizing.

solid answer

~50 s

A `memoryview` records a raw pointer into the `bytearray`'s storage. Any operation that changes the object's length may reallocate that storage, which would leave the view dangling, so `bytearray` counts outstanding exports and refuses to resize while any exist — `append`, `extend`, `+=`, `del`, `clear` and length-changing slice assignment all raise `BufferError`. Same-length writes are unaffected. The fix is to scope the export, not to catch the error: use `with memoryview(buf) as view:` so `__exit__` calls `release()` at the end of the block, or call `release()` explicitly, or copy the field out with `tobytes()` if it must outlive the block. In a log-ingest worker the failure is nastier than it looks: a view stashed on a parsed record survives into the next batch, so only *some* batches fail, and if the framework retries the failed batch the symptom operators see is an intermittent ingest timeout rather than an obvious crash.

code

python · 11 lines
python
buf = bytearray(b"abc")
view = memoryview(buf)
try:
    buf.append(0x64)
except BufferError as exc:
    print(exc)   # Existing exports of data: object cannot be re-sized

buf[0] = 122     # same-length writes are still allowed
view.release()
buf.append(0x64)
print(buf)       # bytearray(b'zbcd')

go deeper

for a junior

Recall that a memoryview keeps the bytearray from changing size and that BufferError is the message you get. Knowing release() and the with block exists is the bar at this level.

for a middle

Explain the mechanism: the view holds a raw pointer, growing may reallocate and move the storage, so the exporter counts exports and refuses length changes while any are outstanding. Same-length writes still work.

for a senior

Show how this presents in production — a retained field view making only some batches fail, retries turning it into an intermittent stall — and give a concrete lifetime fix: with-block scoping, tobytes() at the boundary, or a preallocated buffer filled through recv_into.

for a principal

Own the design rule: zero-copy parsing couples buffer lifetime to record lifetime, so decide once at the API boundary whether records carry views or copies, and prefer never-resized buffers over discipline that every future contributor must remember.

## The invariant being protected When you create a `memoryview` over a `bytearray`, the `bytearray` hands out a raw pointer to its internal storage and increments an export counter. That pointer is not managed by Python's object machinery; it is an address. If the `bytearray` were then allowed to grow past its allocated capacity, it would allocate a larger block, copy the contents across and free the old one — and the view would be left pointing at freed memory. In a language whose whole promise is that you cannot get a dangling pointer, that is unacceptable, so `bytearray` refuses: ```python buf = bytearray(b"abc") view = memoryview(buf) try: buf.append(0x64) except BufferError as exc: print(exc) # Existing exports of data: object cannot be re-sized view.release() buf.append(0x64) print(buf) # bytearray(b'abcd') ``` The check is on *resizing*, not on writing. Everything that can change the length is blocked while an export is outstanding — `append`, `extend`, `+=`, `del buf[i]`, `clear()`, and a slice assignment whose replacement has a different length. Everything that keeps the length fixed still works: `buf[0] = 122` and `buf[0:2] = b'XY'` succeed, and the change is immediately visible through the view, because there is one copy of the bytes. It is worth stressing that the counter is on exports, not on the *number of names*. Slicing a view creates another export, and so does `cast()`. Releasing one of them does not release the others; the `bytearray` stays frozen until the last export is gone. ## Why this shows up intermittently The interesting failure is not the two-line reproduction, it is the version that reaches production. A log-ingest worker keeps a `bytearray` as its accumulation buffer, appends each newly read chunk, and parses complete records out of it through a `memoryview` so that field extraction costs nothing. If a parsed record keeps one of those field views — a timestamp slice, say, stored on the record object and handed downstream — the export outlives the parse. That gives you a failure that depends on data, not on code paths. Batches whose records were fully consumed and dropped before the next read succeed. Batches whose records are still referenced anywhere — a retry queue, a batching accumulator, a traceback holding a frame alive — leave an export outstanding, and the next `append` raises `BufferError`. If the worker's framework treats an exception as a transient batch failure and requeues it, nobody ever sees the `BufferError`: the visible symptom is an ingest stage that intermittently stalls and times out on a fraction of batches, with throughput that degrades as more records are retained downstream. The diagnosis, once you suspect it, is quick: `BufferError` has essentially one cause in Python code, and the message names it. The harder part is finding *which* view is still alive, which is a reference-holder hunt like any other leak. ## Scoping the export The fix is lifetime discipline, and there are three shapes of it. **Bound the view to a block.** `memoryview` is a context manager, and `__exit__` calls `release()`: ```python buf = bytearray(b"abcd") with memoryview(buf) as view: header = bytes(view[:2]) buf.extend(b"efg") # fine, the export is gone ``` This is the default choice. Do not rely on the view simply going out of scope: CPython's refcounting usually releases it promptly, but a reference captured in a traceback, a closure, a logging call or a cycle awaiting collection defeats that, and "usually prompt" is precisely the property that turns into an intermittent failure. **Copy at the boundary.** If a parsed field must outlive the buffer, `tobytes()` it. You pay one small copy per retained field and you get an object with no lifetime coupling at all. For a 20-byte timestamp out of a 4 MB batch this is obviously the right trade; the zero-copy win was never in the small fields. **Do not resize at all.** The most robust design for an ingest loop is a pre-allocated fixed-size `bytearray` filled through `socket.socket.recv_into` or a stream's `readinto` with a view of the free region as the destination, tracking a write offset yourself. Nothing ever appends, so nothing can raise `BufferError`, and you also stop paying for repeated reallocation and copying as the buffer grows. When a record spans reads, compact by moving the tail down with a same-length slice assignment rather than by slicing off the head, since the latter would resize. ## Ready answers for the follow-ups `release()` is idempotent and safe to call twice, but any access to a released view raises `ValueError: operation forbidden on released memoryview object`. A released view cannot be revived — take a new one. And `view.obj` keeps the `bytearray` alive for as long as the view exists, so a stray view is both a resize block and a memory retainer; the two symptoms usually show up together, which is a useful confirmation when you are diagnosing one of them. The short version: the export counter exists so a view can never dangle; resizing is what it blocks; and the cure is to scope the view with a `with` block, copy small fields out, or design a buffer that never needs to resize.

  • Which bytearray operations still work while a memoryview of it is alive?
    Anything that keeps the length fixed. `buf[0] = 122` and an equal-length slice assignment such as `buf[0:2] = b'XY'` both succeed, and the change is immediately visible through the view since there is a single copy of the bytes. Blocked are `append`, `extend`, `+=`, `del buf[i]`, `clear()` and any slice assignment whose replacement differs in length — all of them may reallocate.
  • How would you redesign an ingest loop so BufferError cannot happen at all?
    Pre-allocate a fixed-size `bytearray` and never resize it. Fill it through `socket.socket.recv_into` or a stream's `readinto` using a view of the free region as the destination, and track the write offset yourself. Compact leftover partial records by moving the tail down with a same-length slice assignment rather than slicing off the head. You also stop paying for repeated reallocation as the buffer grows.
  • Can you use a memoryview again after calling release()?
    No. `release()` is idempotent, so calling it twice is harmless, but any subsequent access — indexing, slicing, `tobytes()`, even `len()` — raises `ValueError: operation forbidden on released memoryview object`. There is no way to revive it; create a fresh view over the exporter. That finality is the point: release is what lets the exporter know it is free to reallocate.
  • Why not just wrap the append in a try/except BufferError and retry?
    Retrying cannot help — nothing about the export changes between attempts, so the retry raises again, and in a worker with a batch-level retry policy it turns a deterministic bug into an intermittent stall. `BufferError` here is a design signal, not a transient fault: it says a view is outliving the region it describes. Fix the lifetime with a `with` block or a `tobytes()` copy.

It is a scaffolding permit: while anyone has scaffolding bolted to the wall, the building cannot be extended, because extending it means demolishing and rebuilding the wall the scaffolding is bolted to.

  • Thinks BufferError means the bytearray is locked by a thread
  • Retries the append instead of releasing the view
  • Believes deleting the variable reliably releases the export
  • Says all writes to the bytearray are blocked, not just resizes
  • Assumes release() can be undone or the view reused
  • Suggests catching BufferError as a transient error

This behaviour is unchanged through Python 3.14. memoryview has been usable as a context manager, with release(), since 3.2.

context