skip to content

In Node.js, how would you write a stream.Transform that turns newline-delimited JSON bytes into parsed objects, handling records split across chunks?

level: middleimportance: should knowfreq 30%

answer

  1. bytes in, objects out
  2. keep the tail between calls
  3. split on the newline byte
  4. last line lives in _flush
  5. push(null) means end

basics

~20 s

Create the Transform with readableObjectMode: true, keep the bytes after the last newline between _transform calls, push one parsed object per complete line, parse the remainder in _flush, and report bad JSON with callback(err) exactly once.

solid answer

~40 s

I construct the `Transform` with `readableObjectMode: true`, so the input side keeps taking bytes and `push()` accepts plain objects. In `_transform` I join the saved remainder with the chunk, find each newline byte, parse the complete lines and keep the tail in `this.rest`; splitting on byte `0x0a` is safe for UTF-8. Parsing happens inside `try`, the `catch` calls `callback(err)` and returns, and the pushes and `callback()` come after it, so there is exactly one callback per chunk. `_flush` parses the final line, which matters when the file has no trailing newline. I also reject records that are not objects, because `push(null)` would end the stream.

code

javascript · 50 lines
javascript
import { Transform } from 'node:stream';

const NEWLINE = 0x0a;

export class NdjsonParse extends Transform {
  constructor() {
    super({ readableObjectMode: true });
    this.rest = Buffer.alloc(0);
  }

  #parse(bytes) {
    const text = bytes.toString('utf8').trim();
    if (text === '') return undefined;
    const value = JSON.parse(text);
    if (typeof value !== 'object' || value === null) {
      throw new TypeError('NDJSON record is not an object');
    }
    return value;
  }

  _transform(chunk, encoding, callback) {
    let buf = this.rest.length > 0 ? Buffer.concat([this.rest, chunk]) : chunk;
    const records = [];
    try {
      let nl;
      while ((nl = buf.indexOf(NEWLINE)) !== -1) {
        const record = this.#parse(buf.subarray(0, nl));
        if (record !== undefined) records.push(record);
        buf = buf.subarray(nl + 1);
      }
    } catch (err) {
      callback(err);
      return;
    }
    this.rest = buf;
    for (const record of records) this.push(record);
    callback();
  }

  _flush(callback) {
    let record;
    try {
      record = this.#parse(this.rest);
    } catch (err) {
      callback(err);
      return;
    }
    callback(null, record);
  }
}

go deeper

for a junior

Remember the three pieces: a remainder field, a loop over newlines in _transform, and _flush for the last line.

for a middle

Explain why readableObjectMode rather than objectMode, why you split on the newline byte, and how the try block guarantees one callback.

for a senior

Cover the failure edges: an unbounded remainder, a JSON null ending the stream, and a malformed line mid-file producing one error and nothing after it.

for a principal

Set the team's rule for parsing untrusted record streams: a maximum record size, a policy on skipping or failing bad lines, and tests that split input at every offset.

## The task Newline-delimited JSON (**NDJSON**) puts one JSON value per line. A file or socket stream delivers it as `Buffer` chunks of arbitrary size, so a chunk boundary can fall in the middle of a record. The interview task is a `stream.Transform` from `node:stream` that accepts those bytes and emits **one parsed object per record**. Four things decide whether the answer is correct: 1. **Mixed modes**: bytes in, objects out. 2. **Carry-over**: keep the partial line between `_transform` calls. 3. **End of input**: parse the last line even when the file has no trailing newline. 4. **Errors**: a bad line fails the stream through the callback, exactly once. ## Bytes in, objects out A stream is in **object mode** when it carries arbitrary JavaScript values instead of strings and buffers. A Transform has two sides, and `node:stream` lets you set the mode per side: | Option | Affects | Use here | |---|---|---| | `readableObjectMode: true` | Output side | Yes: lets `push()` accept plain objects | | `writableObjectMode: true` | Input side | No: the input is bytes | | `objectMode: true` | Both sides | Works, but counts the input buffer in chunks rather than bytes | If the readable side stays in byte mode, pushing a plain object fails with `ERR_INVALID_ARG_TYPE` and the stream is destroyed. With `readableObjectMode: true`, the readable side's buffer is measured in objects (default `highWaterMark` 16) while the writable side keeps counting bytes. ## Carrying the partial line Each call joins the saved remainder with the new chunk, then repeatedly finds the next newline byte (`0x0a`), parses the bytes before it, and moves past it. Whatever follows the last newline is saved in `this.rest` for the next call. - Splitting on the **newline byte** is safe for UTF-8: `0x0a` never occurs inside a multi-byte character, so each complete line decodes cleanly. Decoding each raw chunk to a string first would risk cutting a character in half (decoding across chunk boundaries is covered with Node's Buffer API). - **Blank lines** are skipped, not treated as errors. - `subarray()` returns a view, not a copy, so the saved remainder keeps the underlying chunk alive until the next call replaces it. That is fine for line-sized remainders. ## The last line and `_flush` A file whose final record has no trailing newline leaves that record in `this.rest` when input ends. Node calls **`_flush(callback)`** after `end()` and after every chunk has been transformed, before `'end'`. It parses the remainder and passes it as `callback(null, record)`. When the remainder is empty, `record` is `undefined`, and `callback(null, undefined)` pushes nothing, because the Transform only pushes a second argument that is not `null` or `undefined`. ## Errors, and exactly one callback `JSON.parse` throws on a malformed line. The docs define `callback(err)` as how `_transform` reports a failure, and they call throwing from `_write()`, which is what runs `_transform`, undefined behaviour. So the parse sits in `try`, the `catch` calls `callback(err)` and **returns**, and the pushes and the success callback happen **after** the `try`. That shape guarantees one callback per chunk: if `push()` or the success callback threw, it would not land in a `catch` that calls back a second time (`ERR_MULTIPLE_CALLBACK`). `callback(err)` destroys the Transform and emits `'error'`. In a chain built with `stream.pipeline`, that error tears the rest of the chain down and reaches pipeline's callback or promise (composing and cleanup are covered with backpressure and pipelines). ## The `null` trap in object mode `push(null)` means **end of stream** in every mode. A record that is the JSON literal `null` must never be pushed as-is: it would end the readable side, and the next record's `push()` would fail with `ERR_STREAM_PUSH_AFTER_EOF`. Records are normally objects, so the parser rejects any non-object value. A format that must carry bare `null` has to wrap each value, for example `{ value }`. ## Checking your own solution - Feed the same input split at every byte offset; the output must not change. - Test an input with and without a trailing newline. - Test a malformed line in the middle: exactly one `'error'`, no further records. In this design, valid records earlier in the same chunk are dropped too, because the whole chunk fails; push them before calling back with the error if your format needs them. - Test a chunk boundary exactly on the newline byte, and one that falls inside a multi-byte character.

  • What should the parser do with a single line that grows without a newline, say 500 MB?
    Nothing in `node:stream` caps `this.rest`, so the remainder grows until memory runs out. A production parser sets a maximum record size and calls `callback(err)` once the remainder exceeds it, which fails the stream instead of the process.
  • Why not decode each chunk with chunk.toString() and split the string on newlines?
    A chunk boundary can fall inside a multi-byte UTF-8 character, and `toString()` on each half produces replacement characters. Splitting on the newline byte first avoids that, because `0x0a` never appears inside a multi-byte sequence, so each complete line decodes correctly.

  • Each chunk from a file stream holds whole lines, so no carry-over is needed.
  • writableObjectMode: true is what lets a Transform push parsed objects.
  • The last line is emitted automatically when the input ends.
  • Throwing from _transform is how a bad line should be reported.
  • Pushing a parsed JSON null is fine because object mode carries any value.