skip to content

A Node.js import job feeds lines through a custom Transform and sometimes stops midway with no error and no 'finish'; what in _transform causes this, and how do you fix it?

level: seniorimportance: should knowfreq 30%

answer

  1. one chunk in flight at a time
  2. a path that never calls back
  3. async method, ignored promise
  4. twice is an error, never is silence

basics

~10 s

Some path in _transform never calls its callback, so the Transform waits forever for that chunk, buffers the rest and never emits 'finish'. Make every path call back exactly once, reporting failures with callback(err).

solid answer

~40 s

A Transform processes one chunk at a time: until `_transform` calls its `callback`, Node buffers later writes and never calls `_transform` again, and nothing times out. So an early `return` for a blank line, a filter branch, or an `async transform` whose awaited work rejects (Node ignores the returned promise) leaves the stream waiting with no error. The fix is one exit path per chunk: parse in `try`, `callback(err)` and return in `catch`, success `callback` outside the `try`, and bridge promises with `.then((out) => callback(null, out), callback)`. The opposite mistake, calling back twice, destroys the stream with `ERR_MULTIPLE_CALLBACK`, and throwing is not a supported way to fail it.

code

javascript · 33 lines
javascript
import { Transform } from 'node:stream';

const enrich = async (row) => ({ ...row, seenAt: Date.now() });

// Buggy: a blank line returns without calling back, so the stream stalls,
// and a JSON.parse failure rejects a promise that the stream ignores.
export const buggy = new Transform({
  objectMode: true,
  async transform(line, encoding, callback) {
    if (line.length === 0) return;
    const row = await enrich(JSON.parse(line));
    callback(null, row);
  },
});

// Fixed: every path calls back exactly once, and errors go through callback(err).
export const fixed = new Transform({
  objectMode: true,
  transform(line, encoding, callback) {
    if (line.length === 0) {
      callback();
      return;
    }
    let row;
    try {
      row = JSON.parse(line);
    } catch (err) {
      callback(err);
      return;
    }
    enrich(row).then((out) => callback(null, out), callback);
  },
});

go deeper

for a junior

Remember that the callback means this chunk is done: missing it stops the stream, and calling it twice is an error.

for a middle

Explain the one-chunk-at-a-time queue, why an async transform's rejection never reaches the stream, and the try/catch shape that calls back once.

for a senior

Diagnose the silent stall from symptoms, point at writableFinished and per-chunk counters, and show a fix that keeps every branch to a single callback.

for a principal

Decide what the team enforces for custom streams: lint or review rules against async transform methods, mandatory deadlines on long jobs, and tests for every skip and error branch.

## The symptom An import job reads a newline-delimited JSON file, splits it into lines, and passes each line through a custom `stream.Transform` from `node:stream` that parses and enriches the record. Usually it finishes. Sometimes it stops partway: no `'error'`, no `'finish'`, no exception, and no log line. Either the process sits idle forever (a server keeps the event loop alive) or a one-off script simply exits early because nothing else is pending. That signature points at the **callback contract** of `_transform(chunk, encoding, callback)`. ## Why a missed callback is silent A Transform's writable side handles **one chunk at a time**. Node calls `_transform` for a chunk and marks the stream as writing; until that chunk's callback runs, every later `write()` is buffered and `_transform` is not called again. The docs state it directly: `_transform()` is never called in parallel, and to receive the next chunk, `callback` must be called. So a code path that returns without calling back leaves the stream waiting for good. Nothing times out, because streams have no timer of their own. Upstream, `write()` starts returning `false` once the buffer reaches `highWaterMark`, so a well-behaved source pauses too, and the whole chain goes quiet. `'finish'` needs every chunk processed, so it never fires; a `stream.pipeline` around it never settles. In an ES module that awaits that pipeline at top level with nothing else keeping the loop alive, Node exits with code **13**, *Unsettled Top-Level Await*. ## The usual culprits 1. **An early return.** `if (line.length === 0) return;` skips the callback for blank lines. 2. **A filter branch.** Dropping a record without `callback()`. 3. **An async `transform`.** Node ignores the promise an `async` method returns. If the awaited work rejects, the rejection never reaches the stream: it becomes an unhandled rejection, and in Node 24's default `--unhandled-rejections=throw` mode, with no `unhandledRejection` listener, it is raised as an uncaught exception, which crashes the process unless an `uncaughtException` handler catches it. If a listener or a broad `catch` swallows it instead, the callback is never called and the job stalls. 4. **A callback buried in an event listener** that never fires, for example waiting for a reply that never comes. ## The opposite mistakes | Mistake | What Node does | |---|---| | Never calling `callback` | Nothing: the stream waits forever | | Calling it twice for one chunk | Destroys the stream with `ERR_MULTIPLE_CALLBACK` | | Throwing from `_transform` | Not a supported path: the docs call throwing from `_write()`, which runs `_transform`, undefined behaviour | | Calling `this.emit('error', err)` | Not supported: the docs advise against emitting `'error'` yourself | | `callback(err)` once | The documented way: the stream is destroyed and emits `'error'` | A double callback typically comes from `try { ...; callback(null, out); } catch (err) { callback(err); }`, where a throw raised during the success callback lands in the `catch` and calls back again. ## The fix: one exit path per chunk - Make every branch end in exactly one `callback(...)` and `return` immediately after it. - Do synchronous parsing in `try`, call back with the error in `catch`, and call back with success **outside** the `try`. - Drop `async` from the method and bridge the promise explicitly: `work(row).then((out) => callback(null, out), callback)`. The rejection handler only covers `work`, so a throw from the success callback is not converted into a second call. - Turn validation failures into `callback(err)` deliberately, or skip the record with `callback()`; never fall off the end of the function. - Keep the callback out of event listeners that may never fire; if the work depends on a reply, give that wait its own timeout that ends in `callback(err)`. ## How to find it in production - **Count per chunk.** Log or meter chunks entering `_transform` and callbacks invoked; a growing gap names the stuck stream. - **Inspect state.** On a stuck instance, the Transform's `writableFinished` stays `false` and `writableLength` holds the buffered input. - **Fail fast in tests.** Feed blank lines, malformed lines and filtered records through the Transform and assert that `'finish'` arrives, or that exactly one `'error'` does. - **Add a deadline where it belongs.** Wrap the job's pipeline in an `AbortSignal.timeout(...)` (composing with a signal is covered with pipelines) so a stall becomes an error rather than a hung job.

  • Why does the stall also stop the file being read upstream?
    Unprocessed chunks pile up in the Transform's writable buffer. Once that buffer reaches `highWaterMark`, `write()` returns `false`, and a well-behaved source waits for `'drain'`, which never comes because the buffer never empties. Backpressure turns one missed callback into a stalled chain.
  • Is an async _transform method ever acceptable?
    It can work if the body catches every error and calls the callback exactly once on every path, because Node ignores the returned promise. That is easy to break in a later edit, so most teams prefer a plain method that bridges the promise with `.then(onOk, callback)`.
  • How would you make a stall fail loudly instead of hanging the job?
    Give the job a deadline: pass an `AbortSignal.timeout(ms)` to the pipeline promise so the chain is destroyed with an `AbortError`. Add per-chunk counters in the Transform so the alert says which stage stopped calling back.

  • A stalled stream always emits an error eventually, so silence means it is still working.
  • Node awaits an async transform method and routes its rejection to 'error'.
  • Throwing inside _transform is the normal way to fail the stream.
  • Calling the callback twice is harmless; the second call is ignored.
  • Skipping a record is done by returning early from _transform.