skip to content

Why does a Node.js net.Socket 'data' handler sometimes receive half a device message or two messages at once, and how should you frame them?

level: middleimportance: must knowfreq 55%

answer

  1. a byte stream has no boundaries
  2. chunks follow reads, not writes
  3. keep the leftover tail
  4. delimiter or length prefix
  5. setEncoding guards split characters

basics

~20 s

A net.Socket delivers a TCP byte stream, and each 'data' chunk is whatever one read returned, not one write. Frame messages yourself: buffer bytes, split on a delimiter or length prefix, and keep the incomplete tail.

solid answer

~50 s

TCP carries an ordered stream of bytes with no message boundaries, and Node emits `'data'` with whatever the last read from the kernel returned, so one device write can arrive in two chunks and two writes can arrive in one. The fix is a framer per connection: append each chunk to a buffer, extract every complete message, and keep the remainder for the next chunk. For a line protocol, call `socket.setEncoding('utf8')` so a multi-byte character split across chunks still decodes correctly, then split on `'\n'`, or let `readline.createInterface({ input: socket })` emit `'line'` events. For a binary protocol, prefix each message with its length, read it with an accessor such as `buf.readUInt32BE(0)`, and wait until the whole body has arrived. Cap the buffered tail so a client that never sends a delimiter cannot exhaust memory.

code

javascript · 31 lines
javascript
import net from 'node:net';

const MAX_LINE = 4096;

net.createServer((socket) => {
  socket.setEncoding('utf8');
  let pending = '';

  socket.on('data', (chunk) => {
    pending += chunk;
    let newline;
    while ((newline = pending.indexOf('\n')) !== -1) {
      handleReading(pending.slice(0, newline));
      pending = pending.slice(newline + 1);
    }
    if (pending.length > MAX_LINE) {
      socket.destroy(new Error('line too long'));
    }
  });

  socket.on('end', () => {
    if (pending.length > 0) console.warn(`dropped partial line: ${pending.length} chars`);
  });
  socket.on('error', (err) => {
    console.error(`device socket error: ${err.message}`);
  });
}).listen(7070);

function handleReading(line) {
  console.log(`reading: ${line}`);
}

go deeper

for a junior

Recall that a TCP socket is a byte stream: 'data' chunks follow reads, not writes, so one message can be split or merged with the next.

for a middle

Walk through a framer: per-connection buffer, extract complete messages, keep the tail, and explain why setEncoding('utf8') matters for characters split across chunks.

for a senior

Show the hardening: a cap on the buffered tail, validating length headers before buffering, and what an unterminated message at 'end' should mean for the protocol.

for a principal

Argue for one shared, tested framing helper for every raw-socket service, since hand-rolled framers repeat the same unbounded-buffer and split-character bugs.

## Why chunks do not match messages A `net.Socket` from `node:net` is a **Duplex stream over a TCP connection**, and TCP delivers an ordered stream of bytes, not a sequence of messages. When bytes arrive, libuv reads whatever the kernel has buffered for that socket and Node emits one `'data'` event with that `Buffer`. The size of each chunk depends on how the bytes happened to cross the network and how much had accumulated before the read, not on how the sender called `write()`. So an IoT device that sends `temp=21\n` and then `temp=22\n` can produce, on the server: - one `'data'` event holding both lines; - two events, one line each; - `temp=2` in one event and `1\ntemp=22\n` in the next. All three are correct behaviour. Code that treats one chunk as one message works on a quiet local network and fails under load, over slow links, or as soon as messages grow past a single packet. Turning off Nagle's algorithm with `socket.setNoDelay()` changes when the sender transmits, not what the receiver's reads return, so it does not remove the need for framing. ## Two framing schemes | Scheme | How a message ends | Good for | Watch out for | |---|---|---|---| | **Delimiter** | A byte sequence such as `\n` | Text protocols, simple device feeds | The delimiter must never appear inside a message; cap the line length | | **Length prefix** | A fixed-size header carries the body length | Binary payloads, arbitrary bytes | Validate the length before buffering; agree on byte order | Either way, the receiver keeps a per-connection buffer of bytes it has received but not yet consumed. ## A line framer for an IoT server For each connection: 1. Call `socket.setEncoding('utf8')` once, so `'data'` hands you strings and a character split across two chunks is decoded correctly. 2. On each `'data'`, append the chunk to a `pending` string. 3. While `pending` contains a `\n`, cut off everything before it as one message and keep the rest. 4. If `pending` grows past a protocol maximum with no newline, destroy the socket with an error. 5. On `'end'`, decide what an unterminated tail means for your protocol; usually it is a truncated message to log and drop. `node:readline` can do steps 2 and 3 for you: `readline.createInterface({ input: socket })` emits a `'line'` event per line, and the interface can be consumed with `for await...of`. It still leaves the length cap to you. ## Strings, Buffers and split characters Calling `chunk.toString()` on every `Buffer` looks harmless and is a classic bug. UTF-8 encodes many characters in two to four bytes, and a chunk boundary can fall in the middle of one; decoding each half separately produces replacement characters. `readable.setEncoding()` (which `socket.setEncoding()` delegates to) keeps the incomplete bytes and joins them with the next chunk before decoding. For binary framing, stay with `Buffer` throughout: accumulate with `Buffer.concat()`, slice complete messages with `subarray()`, and decode only a complete body. How those `Buffer` methods copy or share memory is covered with binary data handling; for framing it is enough that `subarray()` shares memory and is cheap. ## Hardening the framer - **Cap the tail.** Without a maximum, a client that streams bytes and never sends a delimiter, or announces a huge length, makes the server buffer without limit. - **Validate length headers first.** Reject an announced length above the protocol maximum before accepting any of the body. - **One framer per socket.** A module-level buffer shared across connections mixes clients' bytes together. - **Handle the last message on `'end'`.** A peer can end the stream right after a message with no trailing delimiter, depending on the protocol's rules. - **Do not parse in `'close'`.** By then the socket is gone; process complete messages as they appear. - **Respect slow handlers.** If handling a message is slow, `socket.pause()` and `socket.resume()` stop reading while you catch up; the stream mechanics behind that belong to the streams material on backpressure. The same reasoning applies to the client side: a reply from the server can also arrive in pieces, so a client that waits for one `'data'` event and treats it as the whole response has the same bug.

  • How would you frame a binary protocol with a 4-byte length prefix in Node?
    Keep a `Buffer` of pending bytes and append each chunk with `Buffer.concat()`. While at least 4 bytes are buffered, read the length with `readUInt32BE(0)`; if `4 + length` bytes are present, take the body with `subarray(4, 4 + length)`, handle it and drop those bytes, otherwise wait for more data. Reject a length above the protocol maximum before buffering the body.
  • Does calling socket.setNoDelay() stop messages from arriving merged?
    No. `setNoDelay()` disables Nagle's algorithm so small writes leave sooner, which helps latency, but the receiver still reads a byte stream and still sees merged or split chunks. Framing is required either way. Note that a raw `net` socket keeps Nagle enabled unless you call `setNoDelay()` or pass `noDelay: true`, whereas `node:http` servers default `noDelay` to `true`.

  • Each socket.write() on the sender arrives as exactly one 'data' event.
  • Calling socket.setNoDelay(true) makes message boundaries reliable.
  • Calling chunk.toString() on every Buffer is safe for UTF-8 text.
  • Small messages always arrive whole, so only large ones need buffering.
  • One shared module-level buffer is fine for every connection's partial data.