skip to content

Implementing Custom Transforms

Writing your own Transform, such as a line splitter or chunker, via _transform and _flush, calling back exactly once. A common live-coding task: one missed callback stalls the pipeline silently.

on this pageshow

explore

questions

5

In Node.js, what must a custom stream.Transform implement, and what do its _transform and _flush methods each do?

level: juniorimportance: should knowfreq 35%

answer

  1. one method required, one optional
  2. zero or more pushes per chunk
  3. one callback per input chunk
  4. leftovers emitted before 'end'

basics

~10 s

A custom stream.Transform must implement _transform(chunk, encoding, callback), which pushes zero or more output chunks and calls callback once per input chunk; the optional _flush(callback) emits leftover output after the last input, before 'end'.

solid answer

~30 s

A `Transform` is a Duplex stream whose output is computed from its input, and Node already provides its `_write()` and `_read()`. A subclass must implement `_transform(chunk, encoding, callback)`: it may call `this.push()` zero, one or many times for the chunk, then must call `callback()` exactly once, or `callback(err)` on failure; `callback(null, data)` is shorthand for push-then-callback. Node will not deliver the next chunk until that callback runs. The optional `_flush(callback)` runs after `end()` once every chunk has been transformed, and before `'end'` is emitted, so it is where a chunker or parser pushes the partial data it was still holding.

code

javascript · 29 lines
javascript
import { Transform } from 'node:stream';

class Chunker extends Transform {
  constructor(size) {
    super();
    this.size = size;
    this.pending = Buffer.alloc(0);
  }

  _transform(chunk, encoding, callback) {
    let buf = Buffer.concat([this.pending, chunk]);
    while (buf.length >= this.size) {
      this.push(buf.subarray(0, this.size));
      buf = buf.subarray(this.size);
    }
    this.pending = buf;
    callback();
  }

  _flush(callback) {
    if (this.pending.length > 0) this.push(this.pending);
    callback();
  }
}

const chunker = new Chunker(4);
chunker.on('data', (block) => console.log(block.toString()));
chunker.write('abcdef');
chunker.end('gh');

go deeper

for a junior

Name the one required method and the optional one, and say what callback means: this chunk is done, send the next.

for a middle

Explain push zero-or-many times versus callback(null, data), and why a chunker or parser needs _flush for the bytes it is still holding.

for a senior

Show you know the callback is also the flow-control signal: no callback means no next chunk, two callbacks destroy the stream.

for a principal

Weigh when a hand-written Transform earns its keep against reusing a core stream or a simpler composition, and how the team tests its end-of-input path.

## What a Transform is A **`stream.Transform`** from `node:stream` is a **Duplex** stream whose output is computed from its input: you `write()` data into its writable side and read the result from its readable side. Core examples are the `zlib` compression streams and `crypto` hashes. Node supplies the plumbing (two internal buffers, backpressure, the `'finish'` and `'end'` events), and a custom subclass supplies only the computation. There is **no requirement** that output matches input one to one. A chunk may produce no output, one chunk, or many, and output may appear later than the input that caused it. A hash produces a single chunk only when the input ends. ## The two methods you write | Method | Required? | Called when | Your job | |---|---|---|---| | `_transform(chunk, encoding, callback)` | Yes | Once per written chunk, never in parallel | Push zero or more output chunks, then call `callback` exactly once | | `_flush(callback)` | No | After `end()` and after every chunk has been transformed, before `'end'` | Push any output still held, then call `callback` | `Transform` already implements `_write()` and `_read()` for you, so a subclass does not write those. The default `_transform` throws `ERR_METHOD_NOT_IMPLEMENTED`, which is why it is the one method you must provide. ## Inside `_transform` - **`chunk`** is a `Buffer` by default. A string passed to `write()` is converted to a `Buffer` first (the `decodeStrings` option defaults to `true`), and `encoding` is then the special value `'buffer'`, which you ignore. - **`this.push(data)`** hands a chunk to the readable side. Call it as many times as the input warrants. - **`callback(err)`** reports failure; **`callback()`** reports that this chunk is fully consumed. **`callback(null, data)`** is shorthand for `this.push(data); callback();`. - The **next chunk is not delivered until you call back**, synchronously or asynchronously. That is the queue Node maintains for you. ## A fixed-size chunker Re-blocking a byte stream into fixed-size pieces (for example, fixed-length frames for a block-oriented device or cipher) is a common live-coding exercise because it needs both methods: 1. Join the bytes left over from last time with the new chunk. 2. While at least `size` bytes are available, push a `size`-byte slice. 3. Keep the remainder for the next call and call `callback()`. 4. In `_flush`, push the final short block if one is left, then call `callback()`. Without step 4 the last partial block would be lost silently: Node does not know your private `pending` field exists, so nothing emits it for you. ## The events around your methods A Transform has two ends, and each reports completion with its own event: - **`'finish'`** belongs to the writable side. It fires after the producer calls `end()` and every chunk has been through `_transform`. - **`'end'`** belongs to the readable side. It fires after the `_flush` callback has run and the consumer has read all the output. - Node pushes the end-of-stream marker for you once `_flush` calls back, so the implementation never calls `this.push(null)` in the normal case. - If nothing reads the output, `'end'` never fires and, as the docs warn, the writable side pauses once the readable buffer is full. A Transform used only for its side effects still needs a consumer. In an error case neither event should be expected: the stream is destroyed and emits `'error'` and then `'close'`. ## Rules the stream machinery relies on - Call `callback` **once per chunk**, on every code path. Never calling it stalls the stream; calling it twice destroys the stream with `ERR_MULTIPLE_CALLBACK`. - Report errors through `callback(err)`. The docs define that path for `_transform`, and they call throwing from `_write()` (which is what runs your `_transform`), or emitting `'error'` by hand, undefined behaviour. - Never call the consumer-facing methods (`write()`, `end()`, `read()`) or `emit('data')` from inside the implementation; use `push()` and the callback. - `_flush` is for end-of-input work only. It does not run after each chunk. ## Class or options object The same Transform can be written without a class by passing `transform` and `flush` functions to the constructor: `new Transform({ transform(chunk, encoding, callback) { ... }, flush(callback) { ... } })`. Node installs them as `_transform` and `_flush`, so use ordinary `function` or method syntax: an arrow function does not receive the stream as `this`, and `this.push` then fails. A class is clearer when the transform holds state such as the chunker's `pending` buffer and takes constructor arguments.

  • Why does the chunker need _flush when _transform already pushes every full block?
    Because the bytes left in `pending` after the last write are private state that only the subclass knows about. When the input ends Node calls `_flush`, and if it is missing or does not push, those trailing bytes are dropped silently with no error. `_flush` is the only hook that runs after the final chunk and before `'end'`.
  • May _transform call its callback asynchronously, for example after a lookup finishes?
    Yes. The docs allow the callback synchronously or asynchronously; `_transform` is never called in parallel, so the next chunk simply waits until you call back. What is not allowed is calling it zero times or more than once for a chunk.

A Transform is a station on an assembly line: for each part that arrives it may hand on nothing, one piece or several, but the conveyor brings the next part only after the worker presses the done button, and at the end of the shift the worker clears whatever is left on the bench.

  • A Transform must emit exactly one output chunk for every input chunk.
  • _flush is called after every chunk to push that chunk's output.
  • Leftover partial data is emitted automatically when the input ends.
  • A custom Transform also has to implement _read and _write.
  • Calling this.emit('data') inside _transform is a fine way to output data.
open as a page

In Node.js, how would you write a stream.Transform that turns newline-delimited JSON bytes into parsed objects, handling records split across chunks?

level: middleimportance: should knowfreq 30%

basics

~20 s

Create the Transform with readableObjectMode: true, keep the bytes after the last newline between _transform calls, push one parsed object per complete line, parse the remainder in _flush, and report bad JSON with callback(err) exactly once.

open as a page

In a custom Node.js stream.Writable that sends records to a database, what do _write, _writev and _final each do, and how do you report a failed write?

level: middleimportance: should knowfreq 22%

basics

~20 s

_write handles one chunk and calls back when it is stored; _writev receives all buffered chunks at once for a batch insert; _final runs after end() and delays 'finish'. A failed write is reported with callback(err), never by throwing.

open as a page

In Node.js 24, how do objectMode and highWaterMark govern buffering in a custom Readable or Transform, and what should the implementation do when push() returns false?

level: seniorimportance: should knowfreq 25%

basics

~20 s

highWaterMark is a buffering threshold: bytes in byte mode (64 KiB default in Node 24, 16 KiB on Windows), objects in object mode (default 16). When push() returns false, a Readable stops pushing until Node calls _read() again.

open as a page

A Node.js import job feeds lines through a custom Transform and sometimes stops midway with no error and no 'finish'; what in _transform causes this, and how do you fix it?

level: seniorimportance: should knowfreq 30%

basics

~10 s

Some path in _transform never calls its callback, so the Transform waits forever for that chunk, buffers the rest and never emits 'finish'. Make every path call back exactly once, reporting failures with callback(err).

open as a page