skip to content

struct.unpack per record dominates a 45-second index cold start — how do you cut it?

level: seniorimportance: should knowfreq 20%

answer

  1. Measure before optimising the cold start
  2. Hoist the compiled layout out of the loop
  3. Stop slicing a record at a time
  4. Let a C-level iterator drive the walk
  5. Blocks must be a whole number of records

basics

~20 s

Compile the layout once as a module-level struct.Struct and stream the buffer through Struct.iter_unpack instead of slicing and calling struct.unpack per record. The C-level iterator removes the per-record slice, the lookup and most of the Python loop overhead.

solid answer

~40 s

Profile first, then attack the loop shape rather than the format. Build `rec = struct.Struct('<IIfH')` once at module scope and iterate with `rec.iter_unpack(buffer)`, which yields one tuple per record straight out of a C loop — no `blob[i:i+size]` slice, no per-call format handling. Read the file in blocks that are an exact multiple of `rec.size`, or memory-map it, so a record never straddles a boundary; `iter_unpack` raises `struct.error` on a length that is not a whole multiple. Be honest about the ceiling: the module-level functions already cache compiled formats, so the `Struct` object alone is a modest win — the savings are the slices you stop allocating. Once per-record Python work dominates, the answer is bulk decoding in a C extension, not more `struct` tuning. Keep the byte-order prefix pinned throughout.

code

python · 18 lines
python
import struct
import time

rec = struct.Struct("<IIfH")
blob = b"".join(rec.pack(i, i * 2, float(i), i % 3) for i in range(200_000))

start = time.perf_counter()
total_slow = 0
for i in range(0, len(blob), rec.size):
    doc_id, term_id, score, flags = struct.unpack("<IIfH", blob[i:i + rec.size])
    total_slow += doc_id
slow = time.perf_counter() - start

start = time.perf_counter()
total_fast = sum(doc_id for doc_id, _, _, _ in rec.iter_unpack(blob))
fast = time.perf_counter() - start

print(total_slow == total_fast, round(slow / fast, 1))

go deeper

for a junior

Know that struct.Struct compiles a format once and exposes size, and that iter_unpack walks a buffer of fixed-width records. Recognise the per-record slice as the thing being removed.

for a middle

Explain the three costs in the naive loop — the slice allocation, the per-call lookup, the interpreter loop — and which of them iter_unpack actually removes. Know its whole-multiple buffer requirement and how blocked reads satisfy it.

for a senior

Show the measurement first, then the loop rewrite, then the honest ceiling: name where per-record Python work takes over, keep the byte order pinned, and refuse the speed-for-portability trades that make the shared index unreadable.

for a principal

Own the cold-start budget end to end: what a stale served value costs, whether the fix is a faster parse, a smaller on-disk layout, incremental rather than full rebuilds, or a warm standby — and what you are willing to spend in complexity for each second saved.

### Frame the problem before touching the code A rebuilder that takes 45 seconds to come up is 45 seconds of serving a stale cached value, so the cold start is a correctness-adjacent number, not just a performance one. That framing matters in the interview: it tells you the target is the *whole* cold start, and the first move is to measure which part of it is `struct` at all. A run under `cProfile`, or a few `time.perf_counter()` fences around the read loop, will tell you whether you are spending the time in `struct`, in the I/O, or in whatever you build out of the tuples afterwards. Optimising an unpack loop that costs 4 of the 45 seconds is a wasted afternoon. Assume the profile does point at the record loop. The naive shape is: ```python size = struct.calcsize("<IIfH") for i in range(0, len(blob), size): doc_id, term_id, score, flags = struct.unpack("<IIfH", blob[i:i + size]) ``` Three costs hide in those two lines: a fresh `bytes` object allocated, filled and freed **per record**; a module-level call that must find and validate the compiled format **per record**; and a Python-level loop doing the offset arithmetic **per record**. ### The compiled `Struct` object `struct.Struct(fmt)` compiles the format once into an object with `pack`, `unpack`, `pack_into`, `unpack_from`, `iter_unpack`, a `size` attribute and the original `format`. Hoisting it to module scope is good practice for reasons beyond speed: `rec.size` is now the single source of truth for offsets and block sizes, and the format string appears exactly once in the file. Be accurate about the speed claim, though — this is where candidates overreach. The module-level functions keep an internal cache of recently compiled formats, so `struct.unpack(fmt, ...)` is **not** re-parsing the format string on every call. The `Struct` object saves the cache lookup and a little argument handling; it does not save a compile. Claiming a 10x win from that alone is a red flag. ### `iter_unpack` is where the win is ```python for doc_id, term_id, score, flags in rec.iter_unpack(blob): ... ``` `Struct.iter_unpack` returns an iterator that walks the whole buffer in C, yielding one tuple per record. The slice disappears entirely — no per-record allocation — and the offset arithmetic moves out of the interpreter. On a stream of millions of fixed-width records this is typically the difference that shows up in the wall clock, and the code is shorter than what it replaces. Its one contract: the buffer length must be an exact multiple of `rec.size`, and the size must be non-zero, or you get a `struct.error`. That is a feature for a well-formed index file and a trap for a blocked reader, which leads to the next point. ### Blocking, memory maps and boundaries Reading the whole index into one `bytes` object costs its full size in RSS. Two better shapes: * **Blocked reads sized to a whole number of records.** Choose `block = rec.size * N` and `fh.read(block)`; every chunk then holds complete records and feeds `iter_unpack` directly. If the block size is *not* a multiple, you must carry the remainder into the next chunk yourself — the usual bug is discarding it and silently losing one record per block. * **`mmap`**, so the pages are faulted in by the OS and never copied into a Python object. Wrap it in a `memoryview` if you need to hand sub-ranges elsewhere without copying. ### Everything downstream of the tuple Once `iter_unpack` is doing the decoding, the remaining cost is usually what you *do* with each tuple. Appending to a list of small objects, building a dict per record, or constructing a class instance per record will dwarf the unpack itself. Options in order of effort: keep the per-record work to primitive operations; accumulate into an `array.array` or a preallocated `bytearray`; or accept the honest ceiling — when the per-record Python work is the bottleneck, no amount of `struct` tuning helps, and the answer is to move bulk decoding into a C extension or an array library that reads the whole block in one call and releases the GIL while it does. Knowing where that line is, and saying so, is the senior part of this answer. ### Do not trade correctness for speed Two temptations to name and refuse. Switching the format from `'<'` to `'@'` because native alignment "matches the machine" makes the file unreadable on a host with different padding or endianness — and the index is shared. Skipping the length and magic-number validation on the header to save a few microseconds turns a truncated or half-written file into a plausible-looking parse. Pin the byte order, validate the header, and take the speed from the loop shape, which is where it actually is.

  • What breaks if your read block is not a multiple of the Struct's size?
    `iter_unpack` raises `struct.error`, because it requires a buffer whose length is a multiple of the record size. The fix is to size the block as `rec.size * N`, or to keep the leftover bytes and prepend them to the next chunk. The dangerous non-fix is catching the error and dropping the tail, which silently loses a record per block.
  • Does struct.unpack recompile the format string on every call?
    No. The module-level functions keep an internal cache of compiled formats, so repeated calls with the same string reuse the compiled layout. Hoisting a `struct.Struct` out of the loop still helps a little — it skips the lookup and some argument handling — but claiming it avoids a per-call compile overstates it. The real per-record costs are the slice and the Python loop.
  • How do you know when struct has stopped being the bottleneck?
    When a profile shows the time inside what you build from each tuple rather than inside the unpack — object construction, dict building, per-record function calls. At that point more `struct` tuning buys nothing; you either reduce the per-record Python work to primitives and preallocated buffers, or move bulk decoding into a C extension or array library that consumes a whole block in one call.
  • Would you switch the format to native mode to speed the loop up?
    No. Native mode changes nothing meaningful about decode speed, and it bakes the writer's endianness and alignment into a file other hosts must read. The index is shared, so the byte-order prefix stays pinned; the speed comes from the loop shape, the block sizing and the work done per tuple.

saying these in an interview costs you the question

  • Claims struct.unpack recompiles the format every call
  • Slices a fresh bytes object per record in the loop
  • Reads blocks that are not a multiple of the record size
  • Optimises the loop before profiling the cold start
  • Switches to native byte order for speed, breaking portability
  • Assumes the experimental JIT is on and will fix it

context

resolving links…