In Node.js, what must a custom stream.Transform implement, and what do its _transform and _flush methods each do?
answer
- one method required, one optional
- zero or more pushes per chunk
- one callback per input chunk
- leftovers emitted before 'end'
basics
~10 sA custom stream.Transform must implement _transform(chunk, encoding, callback), which pushes zero or more output chunks and calls callback once per input chunk; the optional _flush(callback) emits leftover output after the last input, before 'end'.
solid answer
~30 sA `Transform` is a Duplex stream whose output is computed from its input, and Node already provides its `_write()` and `_read()`. A subclass must implement `_transform(chunk, encoding, callback)`: it may call `this.push()` zero, one or many times for the chunk, then must call `callback()` exactly once, or `callback(err)` on failure; `callback(null, data)` is shorthand for push-then-callback. Node will not deliver the next chunk until that callback runs. The optional `_flush(callback)` runs after `end()` once every chunk has been transformed, and before `'end'` is emitted, so it is where a chunker or parser pushes the partial data it was still holding.
code
javascript · 29 linesimport { Transform } from 'node:stream';
class Chunker extends Transform {
constructor(size) {
super();
this.size = size;
this.pending = Buffer.alloc(0);
}
_transform(chunk, encoding, callback) {
let buf = Buffer.concat([this.pending, chunk]);
while (buf.length >= this.size) {
this.push(buf.subarray(0, this.size));
buf = buf.subarray(this.size);
}
this.pending = buf;
callback();
}
_flush(callback) {
if (this.pending.length > 0) this.push(this.pending);
callback();
}
}
const chunker = new Chunker(4);
chunker.on('data', (block) => console.log(block.toString()));
chunker.write('abcdef');
chunker.end('gh');go deeper
Name the one required method and the optional one, and say what callback means: this chunk is done, send the next.
Explain push zero-or-many times versus callback(null, data), and why a chunker or parser needs _flush for the bytes it is still holding.
Show you know the callback is also the flow-control signal: no callback means no next chunk, two callbacks destroy the stream.
Weigh when a hand-written Transform earns its keep against reusing a core stream or a simpler composition, and how the team tests its end-of-input path.
## What a Transform is A **`stream.Transform`** from `node:stream` is a **Duplex** stream whose output is computed from its input: you `write()` data into its writable side and read the result from its readable side. Core examples are the `zlib` compression streams and `crypto` hashes. Node supplies the plumbing (two internal buffers, backpressure, the `'finish'` and `'end'` events), and a custom subclass supplies only the computation. There is **no requirement** that output matches input one to one. A chunk may produce no output, one chunk, or many, and output may appear later than the input that caused it. A hash produces a single chunk only when the input ends. ## The two methods you write | Method | Required? | Called when | Your job | |---|---|---|---| | `_transform(chunk, encoding, callback)` | Yes | Once per written chunk, never in parallel | Push zero or more output chunks, then call `callback` exactly once | | `_flush(callback)` | No | After `end()` and after every chunk has been transformed, before `'end'` | Push any output still held, then call `callback` | `Transform` already implements `_write()` and `_read()` for you, so a subclass does not write those. The default `_transform` throws `ERR_METHOD_NOT_IMPLEMENTED`, which is why it is the one method you must provide. ## Inside `_transform` - **`chunk`** is a `Buffer` by default. A string passed to `write()` is converted to a `Buffer` first (the `decodeStrings` option defaults to `true`), and `encoding` is then the special value `'buffer'`, which you ignore. - **`this.push(data)`** hands a chunk to the readable side. Call it as many times as the input warrants. - **`callback(err)`** reports failure; **`callback()`** reports that this chunk is fully consumed. **`callback(null, data)`** is shorthand for `this.push(data); callback();`. - The **next chunk is not delivered until you call back**, synchronously or asynchronously. That is the queue Node maintains for you. ## A fixed-size chunker Re-blocking a byte stream into fixed-size pieces (for example, fixed-length frames for a block-oriented device or cipher) is a common live-coding exercise because it needs both methods: 1. Join the bytes left over from last time with the new chunk. 2. While at least `size` bytes are available, push a `size`-byte slice. 3. Keep the remainder for the next call and call `callback()`. 4. In `_flush`, push the final short block if one is left, then call `callback()`. Without step 4 the last partial block would be lost silently: Node does not know your private `pending` field exists, so nothing emits it for you. ## The events around your methods A Transform has two ends, and each reports completion with its own event: - **`'finish'`** belongs to the writable side. It fires after the producer calls `end()` and every chunk has been through `_transform`. - **`'end'`** belongs to the readable side. It fires after the `_flush` callback has run and the consumer has read all the output. - Node pushes the end-of-stream marker for you once `_flush` calls back, so the implementation never calls `this.push(null)` in the normal case. - If nothing reads the output, `'end'` never fires and, as the docs warn, the writable side pauses once the readable buffer is full. A Transform used only for its side effects still needs a consumer. In an error case neither event should be expected: the stream is destroyed and emits `'error'` and then `'close'`. ## Rules the stream machinery relies on - Call `callback` **once per chunk**, on every code path. Never calling it stalls the stream; calling it twice destroys the stream with `ERR_MULTIPLE_CALLBACK`. - Report errors through `callback(err)`. The docs define that path for `_transform`, and they call throwing from `_write()` (which is what runs your `_transform`), or emitting `'error'` by hand, undefined behaviour. - Never call the consumer-facing methods (`write()`, `end()`, `read()`) or `emit('data')` from inside the implementation; use `push()` and the callback. - `_flush` is for end-of-input work only. It does not run after each chunk. ## Class or options object The same Transform can be written without a class by passing `transform` and `flush` functions to the constructor: `new Transform({ transform(chunk, encoding, callback) { ... }, flush(callback) { ... } })`. Node installs them as `_transform` and `_flush`, so use ordinary `function` or method syntax: an arrow function does not receive the stream as `this`, and `this.push` then fails. A class is clearer when the transform holds state such as the chunker's `pending` buffer and takes constructor arguments.
- Why does the chunker need _flush when _transform already pushes every full block?Because the bytes left in `pending` after the last write are private state that only the subclass knows about. When the input ends Node calls `_flush`, and if it is missing or does not push, those trailing bytes are dropped silently with no error. `_flush` is the only hook that runs after the final chunk and before `'end'`.
- May _transform call its callback asynchronously, for example after a lookup finishes?Yes. The docs allow the callback synchronously or asynchronously; `_transform` is never called in parallel, so the next chunk simply waits until you call back. What is not allowed is calling it zero times or more than once for a chunk.
A Transform is a station on an assembly line: for each part that arrives it may hand on nothing, one piece or several, but the conveyor brings the next part only after the worker presses the done button, and at the end of the shift the worker clears whatever is left on the bench.
- A Transform must emit exactly one output chunk for every input chunk.
- _flush is called after every chunk to push that chunk's output.
- Leftover partial data is emitted automatically when the input ends.
- A custom Transform also has to implement _read and _write.
- Calling this.emit('data') inside _transform is a fine way to output data.