skip to content

In Node.js, how do you parse a length-prefixed binary frame with Buffer accessors such as readUInt32BE, and what goes wrong when a frame arrives split across chunks?

level: middleimportance: should knowfreq 40%

answer

  1. width, signedness, byte order
  2. offset checked, RangeError thrown
  3. chunk boundaries are not frame boundaries
  4. cap the length before waiting

basics

~20 s

Read fixed-width fields with accessors such as buf.readUInt32BE(offset) and buf.readUInt8(offset), matching the protocol's width, signedness and byte order. Chunks do not align with frames, so accumulate bytes, wait until header and payload are complete, then subarray the frame out.

solid answer

~40 s

Each accessor names its width, signedness and byte order: `readUInt32BE(0)` reads four bytes as an unsigned big-endian integer, while `readInt32BE` would turn lengths above 2^31−1 negative and `readUInt32LE` would read the bytes in the wrong order. Offsets are validated: reading past the end throws a `RangeError` (`ERR_OUT_OF_RANGE`, or `ERR_BUFFER_OUT_OF_BOUNDS` when the buffer is shorter than the field) instead of returning garbage. Chunks from a socket or a file have no relation to frame boundaries, so keep the pending bytes, append each chunk, and loop: stop if the header is incomplete, read the length, reject it above a maximum, stop if the payload is incomplete, otherwise take it with `subarray` and keep the remainder. Without the cap, one peer can make you buffer up to 4 GiB.

code

javascript · 27 lines
javascript
import { Buffer } from 'node:buffer';

const HEADER = 5; // 4-byte big-endian length + 1-byte type
const MAX_PAYLOAD = 1024 * 1024;

function createFrameParser(onFrame) {
  let pending = Buffer.alloc(0);
  return function push(chunk) {
    pending = pending.length === 0 ? chunk : Buffer.concat([pending, chunk]);
    while (pending.length >= HEADER) {
      const len = pending.readUInt32BE(0);
      if (len > MAX_PAYLOAD) throw new RangeError(`frame too large: ${len}`);
      if (pending.length < HEADER + len) break; // wait for more bytes
      const type = pending.readUInt8(4);
      onFrame(type, pending.subarray(HEADER, HEADER + len));
      pending = pending.subarray(HEADER + len);
    }
  };
}

const push = createFrameParser((type, payload) => console.log(type, payload.toString()));
const frame = Buffer.alloc(HEADER + 5);
frame.writeUInt32BE(5, 0);
frame.writeUInt8(1, 4);
frame.write('hello', HEADER);
push(frame.subarray(0, 3)); // header incomplete: nothing printed
push(frame.subarray(3));    // prints: 1 hello

go deeper

for a junior

Recall that accessor names state width, signedness and byte order, and that readUInt32BE reads four bytes as an unsigned big-endian number.

for a middle

Explain the accumulate-and-loop parser: header check, length read, payload check, subarray, remainder kept, and why chunk boundaries never match frames. Know which RangeError codes an early read throws.

for a senior

Show the production guards: cap the length before buffering, avoid re-concatenating large frames on every chunk, and copy payload views before retaining them so they do not pin or alias the source chunk.

for a principal

Treat framing as a shared library with one tested parser, explicit size limits per protocol and fuzz tests for split and oversized frames, rather than each service hand-rolling offsets.

## The setting: a length-prefixed frame Many binary protocols put a fixed-size **header** in front of each message. Take a frame whose first four bytes hold the payload length as an unsigned big-endian 32-bit integer, followed by one byte of message type, followed by the payload. Node's `Buffer` reads such headers with **fixed-width accessors**: methods whose names state the width, the signedness and the byte order they read. ## The accessor family | Field in the protocol spec | Accessor | Returns | |---|---|---| | unsigned 8-bit | `buf.readUInt8(offset)` | 0 to 255 | | unsigned 16-bit big-endian | `buf.readUInt16BE(offset)` | 0 to 65535 | | unsigned 32-bit big-endian | `buf.readUInt32BE(offset)` | 0 to 2^32−1 | | signed 32-bit little-endian | `buf.readInt32LE(offset)` | −2^31 to 2^31−1 | | unsigned 64-bit big-endian | `buf.readBigUInt64BE(offset)` | a `BigInt` | | unsigned, 1 to 6 bytes | `buf.readUIntBE(offset, byteLength)` | up to 48 bits | | float or double | `buf.readFloatBE`, `buf.readDoubleLE` | a Number | Each name encodes three decisions: - **Width** decides how many bytes are consumed. - **`U` or not** decides whether the top bit is a sign bit. `readInt32BE` on `ff ff ff fe` gives `-2`; `readUInt32BE` on the same bytes gives `4294967294`. - **`BE` or `LE`** decides the byte order. The protocol fixes it, not the CPU running Node. The unsigned accessors also exist under lower-case aliases such as `readUint32BE`, and the mirror-image writers (`writeUInt32BE(value, offset)`) return the offset just past the bytes they wrote, which makes building a header a chain of offsets. ## Offsets are checked `readUInt32BE(offset)` requires `0 <= offset <= buf.length - 4`. Outside that range Node throws a `RangeError`: - `ERR_OUT_OF_RANGE` when the buffer is long enough for the field but the offset is past the last legal start, for example `readUInt32BE(4)` on a 6-byte buffer; - `ERR_BUFFER_OUT_OF_BOUNDS` when the buffer is shorter than the field itself, for example `readUInt32BE(0)` on 3 bytes. That is a feature: a parser that reads before enough bytes have arrived fails loudly instead of inventing a length. Hand-assembled arithmetic such as `buf[0] << 24 | buf[1] << 16 | ...` has no such guard. A missing byte reads as `undefined`, which the bitwise operators treat as `0`, and those operators work on signed 32-bit integers, so a first byte of `0x80` or more makes the result negative. ## Chunks are not frames Bytes reach a parser in chunks, from a socket's `'data'` events or a file read, and chunk boundaries have nothing to do with frame boundaries. One chunk can hold half a header; another can hold three frames and part of a fourth. A correct parser therefore: 1. appends each chunk to the pending bytes; 2. stops if fewer than 5 bytes are pending, because the header is incomplete; 3. reads the length with `readUInt32BE(0)` and **rejects it if it exceeds the protocol's maximum**, before waiting for that many bytes; 4. stops if fewer than `5 + length` bytes are pending; 5. takes the payload with `pending.subarray(5, 5 + length)`, hands it on, and keeps `pending.subarray(5 + length)` as the new pending bytes; 6. loops, because one chunk may complete several frames. Step 3 is the production one: a length prefix is input from the peer, and an unchecked `readUInt32BE` value lets a single connection make the process hold up to 4 GiB while it waits. ## Cost and aliasing - `Buffer.concat()` allocates a new Buffer and copies every input on each call. Rebuilding a 1 MiB frame from 64 KiB chunks by concatenating after every chunk copies the growing prefix again and again; for large frames, keep an array of chunks plus a running byte count and concatenate once the frame is complete. - `subarray()` copies nothing: the payload shares memory with the pending bytes, which may share memory with the original chunk. Copy it with `Buffer.from(payload)` before storing it beyond the current call. - Small results of `Buffer.concat()` can be carved from Node's shared allocation pool, so their bytes sit inside a larger block next to unrelated data. ## Writing the header on the way out Building a frame mirrors parsing it: `Buffer.alloc(5 + payload.length)`, then `writeUInt32BE(payload.length, 0)`, `writeUInt8(type, 4)` and `payload.copy(frame, 5)`. Node's documentation calls a value outside the unsigned 32-bit range **undefined behaviour** for `writeUInt32BE`, so check the payload size against the protocol's limit before writing it rather than relying on the writer to catch it.

  • Why not compose the length yourself with buf[0] << 24 | buf[1] << 16 | buf[2] << 8 | buf[3]?
    JavaScript's bitwise operators work on signed 32-bit integers, so a first byte of `0x80` or more yields a negative number unless you add `>>> 0`, and a missing byte silently becomes `0`. `readUInt32BE` returns an unsigned value from 0 to 2^32−1, states the byte order in its name and throws a `RangeError` when the bytes are not there.
  • How do you read a 64-bit length field in Node?
    `buf.readBigUInt64BE(offset)` returns a `BigInt`, because a Number cannot represent every 64-bit integer exactly. Compare it against your maximum while it is still a `BigInt`, then convert with `Number(n)` once it is known to be small. For odd widths up to 6 bytes, `readUIntBE(offset, byteLength)` returns a Number with up to 48 bits of accuracy.

  • readInt32BE and readUInt32BE return the same value for any length field
  • Each socket chunk carries exactly one complete frame
  • readUInt32BE past the end of a Buffer quietly returns NaN or 0
  • Trust the length prefix and buffer that many bytes without a cap
  • BE and LE accessors differ only in speed, not in the value read