skip to content

In a custom Node.js stream.Writable that sends records to a database, what do _write, _writev and _final each do, and how do you report a failed write?

level: middleimportance: should knowfreq 22%

answer

  1. one chunk, then the batch
  2. callback is flow control
  3. buffered chunks arrive together
  4. 'finish' waits on one hook
  5. error as first argument

basics

~20 s

_write handles one chunk and calls back when it is stored; _writev receives all buffered chunks at once for a batch insert; _final runs after end() and delays 'finish'. A failed write is reported with callback(err), never by throwing.

solid answer

~40 s

`_write(chunk, encoding, callback)` handles one chunk; Node buffers later writes until you call back, so the callback is the flow control as well as the result. If `_writev(chunks, callback)` exists and more than one chunk is buffered, Node passes them all as `{ chunk, encoding }` entries, which makes a natural batch insert; `cork()` and `uncork()` can encourage that. `_final(callback)` runs once after `end()` and after every write has completed, and `'finish'` waits for it, so it is where you commit or close. A failure goes to `callback(err)` exactly once; the stream is destroyed and emits `'error'`, while throwing is undefined behaviour.

code

javascript · 19 lines
javascript
import { Writable } from 'node:stream';

// Sends records to a store in batches; insertMany(rows) returns a promise.
export function batchWriter(insertMany) {
  return new Writable({
    objectMode: true,
    write(row, encoding, callback) {
      insertMany([row]).then(() => callback(), callback);
    },
    writev(entries, callback) {
      const rows = entries.map((entry) => entry.chunk);
      insertMany(rows).then(() => callback(), callback);
    },
    final(callback) {
      // Runs after end() and all writes; 'finish' waits for this callback.
      callback();
    },
  });
}

go deeper

for a junior

Recall which method handles one chunk, which handles many, and which runs at the end.

for a middle

Explain how buffering behind a pending callback produces the batches _writev sees, and why errors go through the callback.

for a senior

Show the production shape: callback only after the data is stored, _final to commit, _destroy to clean up, and highWaterMark sized for batches.

for a principal

Judge whether a hand-written sink should own batching and retries or leave them to the client library, and what delivery guarantee the team promises.

## The three methods on a custom Writable A custom **`stream.Writable`** from `node:stream` is how you turn "something that accepts data" into a stream that can sit at the end of a chain: a file, a socket, a database client, an upload. Node handles the queue, backpressure and events; the subclass implements up to three methods. | Method | Called | Contract | |---|---|---| | `_write(chunk, encoding, callback)` | For one chunk at a time | Call `callback()` when that chunk is durably handled, or `callback(err)` | | `_writev(chunks, callback)` | Instead of `_write` when more than one chunk is buffered | Handle the whole array; one `callback` for all of them | | `_final(callback)` | Once, after `end()` and after all writes completed | Finish up; `'finish'` waits for its `callback` | At least one of `_write` and `_writev` is required. If only `_writev` exists, Node's default `_write` wraps a single chunk in a one-item array and calls `_writev`. ## `_write`: one chunk, one callback Node calls `_write` with the next chunk and waits. While that chunk's callback is pending, further `write()` calls are **buffered**, and once the buffered amount reaches `highWaterMark`, `write()` returns `false` to the producer. When you call back, Node takes the next chunk, and may emit `'drain'` once the buffer has emptied. So the callback is both the result and the flow control: calling it before the data is really handled hides failures and lets the producer outrun the destination. Calling it twice destroys the stream with `ERR_MULTIPLE_CALLBACK`. ## `_writev`: batching for free Batching is where a database sink earns its keep: one round trip for many rows. If the destination is slow, chunks accumulate in the buffer while a write is in flight; when the callback runs and more than one chunk is waiting, Node calls **`_writev`** with all of them as an array of `{ chunk, encoding }` entries. In object mode `encoding` is irrelevant; for a byte stream it is `'buffer'` for Buffers. - The batch size is **whatever accumulated**, not a fixed number. For a producer that waits whenever `write()` returns `false`, it stays close to `highWaterMark`; a producer that ignores `false` can make it arbitrarily large. - A producer can encourage batching with `writable.cork()` before several writes and `uncork()` afterwards; the corked chunks go to `_writev` together. - One `callback` covers the whole batch. A partial failure must be reported as `callback(err)`, which fails every write in it, or handled inside the method. ## `_final`: the end-of-input hook After the producer calls `end()` and every pending write has called back, Node calls **`_final(callback)`** and delays `'finish'` until that callback runs. Use it to write a partial batch you were holding yourself, to commit, or to close a connection. Releasing resources on failure belongs in `_destroy(err, callback)`, which runs on `destroy()` and after errors. A related hook, **`_construct(callback)`**, runs in a tick after the constructor and delays `_write`, `_final` and `_destroy` until it calls back; it is where you open a file or connection asynchronously. ## Reporting a failed write The stream docs are explicit: errors from `_write`, `_writev` and `_final` must be passed as the callback's first argument. **Throwing** from these methods, or emitting `'error'` yourself, is undefined behaviour. 1. Call `callback(err)` exactly once. 2. Node marks the stream errored; with `autoDestroy` (default `true`) it is destroyed, `'error'` is emitted, and later writes fail. 3. A `Readable` piped into it is unpiped; a `stream.pipeline` destroys the other stages and reports the error (that composition is covered with backpressure and pipelines). ## Mistakes that show up in review - **Calling back too early.** `callback()` before the insert resolves acknowledges data that may never be stored, and lets the producer outrun the database. - **Losing a batch's error.** In `_writev` one callback covers every entry; reporting success after a partial failure silently drops rows. - **Overriding public methods.** Replacing `write()` or `end()`, or emitting `'finish'` yourself, breaks invariants that `stream.pipeline` and other utilities rely on; the docs advise against it. - **Forgetting object mode.** Without `objectMode: true`, `write({ id: 1 })` throws `ERR_INVALID_ARG_TYPE` because a byte stream accepts only strings, Buffers, TypedArrays and DataViews. - **Writing `null`.** `write(null)` throws `ERR_STREAM_NULL_VALUES` in every mode, object mode included. ## Object mode for records A sink for rows sets `objectMode: true` so `write()` accepts objects. `highWaterMark` then counts **rows**, defaulting to 16, which, for a producer that respects `write()` returning `false`, also keeps a batch that builds up behind one slow write near 16 rows.

  • How large will the batches passed to _writev be?
    As large as what accumulated while the previous write was in flight, so batches grow when the destination is slow and shrink to single rows when it is fast. The producer cannot write far past `highWaterMark` without ignoring `write()` returning `false`, so in object mode the default keeps a batch near 16 rows. Raise `highWaterMark` to allow bigger batches.
  • Where should the connection be opened and closed?
    Open it in `_construct(callback)`, which delays the first `_write` until it calls back. Close it on success in `_final`, and on failure or `destroy()` in `_destroy(err, callback)`, so the connection is released on both paths.

  • _writev is called on every write with a one-element array.
  • _final runs after 'finish' is emitted, so it cannot delay it.
  • Throwing from _write is the normal way to fail the stream.
  • Calling back before the insert completes is safe and makes the sink faster.
  • _flush is where a Writable writes its last partial batch.