Why does a live memoryview make bytearray.append raise BufferError, and how do you scope the view in a long-running ingest worker?

## The invariant being protected When you create a `memoryview` over a `bytearray`, the `bytearray` hands out a raw pointer to its internal storage and increments an export counter. That pointer is not managed by Python's object machinery; it is an address. If the `bytearray` were then allowed to grow past its allocated capacity, it would allocate a larger block, copy the contents across and free the old one — and the view would be left pointing at freed memory. In a language whose whole promise is that you cannot get a dangling pointer, that is unacceptable, so `bytearray` refuses: ```python buf = bytearray(b"abc") view = memoryview(buf) try: buf.append(0x64) except BufferError as exc: print(exc) # Existing exports of data: object cannot be re-sized view.release() buf.append(0x64) print(buf) # bytearray(b'abcd') ``` The check is on *resizing*, not on writing. Everything that can change the length is blocked while an export is outstanding — `append`, `extend`, `+=`, `del buf[i]`, `clear()`, and a slice assignment whose replacement has a different length. Everything that keeps the length fixed still works: `buf[0] = 122` and `buf[0:2] = b'XY'` succeed, and the change is immediately visible through the view, because there is one copy of the bytes. It is worth stressing that the counter is on exports, not on the *number of names*. Slicing a view creates another export, and so does `cast()`. Releasing one of them does not release the others; the `bytearray` stays frozen until the last export is gone. ## Why this shows up intermittently The interesting failure is not the two-line reproduction, it is the version that reaches production. A log-ingest worker keeps a `bytearray` as its accumulation buffer, appends each newly read chunk, and parses complete records out of it through a `memoryview` so that field extraction costs nothing. If a parsed record keeps one of those field views — a timestamp slice, say, stored on the record object and handed downstream — the export outlives the parse. That gives you a failure that depends on data, not on code paths. Batches whose records were fully consumed and dropped before the next read succeed. Batches whose records are still referenced anywhere — a retry queue, a batching accumulator, a traceback holding a frame alive — leave an export outstanding, and the next `append` raises `BufferError`. If the worker's framework treats an exception as a transient batch failure and requeues it, nobody ever sees the `BufferError`: the visible symptom is an ingest stage that intermittently stalls and times out on a fraction of batches, with throughput that degrades as more records are retained downstream. The diagnosis, once you suspect it, is quick: `BufferError` has essentially one cause in Python code, and the message names it. The harder part is finding *which* view is still alive, which is a reference-holder hunt like any other leak. ## Scoping the export The fix is lifetime discipline, and there are three shapes of it. **Bound the view to a block.** `memoryview` is a context manager, and `__exit__` calls `release()`: ```python buf = bytearray(b"abcd") with memoryview(buf) as view: header = bytes(view[:2]) buf.extend(b"efg") # fine, the export is gone ``` This is the default choice. Do not rely on the view simply going out of scope: CPython's refcounting usually releases it promptly, but a reference captured in a traceback, a closure, a logging call or a cycle awaiting collection defeats that, and "usually prompt" is precisely the property that turns into an intermittent failure. **Copy at the boundary.** If a parsed field must outlive the buffer, `tobytes()` it. You pay one small copy per retained field and you get an object with no lifetime coupling at all. For a 20-byte timestamp out of a 4 MB batch this is obviously the right trade; the zero-copy win was never in the small fields. **Do not resize at all.** The most robust design for an ingest loop is a pre-allocated fixed-size `bytearray` filled through `socket.socket.recv_into` or a stream's `readinto` with a view of the free region as the destination, tracking a write offset yourself. Nothing ever appends, so nothing can raise `BufferError`, and you also stop paying for repeated reallocation and copying as the buffer grows. When a record spans reads, compact by moving the tail down with a same-length slice assignment rather than by slicing off the head, since the latter would resize. ## Ready answers for the follow-ups `release()` is idempotent and safe to call twice, but any access to a released view raises `ValueError: operation forbidden on released memoryview object`. A released view cannot be revived — take a new one. And `view.obj` keeps the `bytearray` alive for as long as the view exists, so a stray view is both a resize block and a memory retainer; the two symptoms usually show up together, which is a useful confirmation when you are diagnosing one of them. The short version: the export counter exists so a view can never dangle; resizing is what it blocks; and the cure is to scope the view with a `with` block, copy small fields out, or design a buffer that never needs to resize.