skip to content

When a Node.js service wraps a queue consumer as Readable.from(messages(), { highWaterMark: 16 * 1024 }), why do thousands of unprocessed messages pile up in memory?

level: seniorimportance: should knowfreq 40%

answer

  1. units, not size
  2. Readable.from defaults objectMode
  3. count of values, not bytes
  4. prefetch until the threshold

basics

~20 s

Readable.from() creates an object-mode stream, where highWaterMark counts values, not bytes. A value of 16,384 tells Node to prefetch up to 16,384 messages from the generator before it stops pulling. Use a small count instead.

solid answer

~40 s

`Readable.from(iterable)` sets `objectMode: true` unless you opt out, and in object mode `highWaterMark` is a **number of objects**, not bytes. A value copied from byte-stream code, `16 * 1024`, therefore tells the stream to keep calling the generator until 16,384 messages are buffered, each one already pulled from the queue but not yet processed by your code. The defaults show the difference: `stream.getDefaultHighWaterMark(true)` is 16 objects, the byte default is 64 KiB on non-Windows platforms since Node 22, and in the Node 24 source `Readable.from()` passes `highWaterMark: 1` when you give none. Pick a small count sized to how many in-flight messages you can afford, and remember that destroying the stream calls the generator's `return()`, so its `finally` block can close the consumer.

code

javascript · 21 lines
javascript
import { Readable } from 'node:stream';

async function* messages() {
  let id = 0;
  try {
    while (true) yield { id: ++id, body: `event ${id}` }; // stands in for polling a queue
  } finally {
    console.log('consumer closed');
  }
}

const stream = Readable.from(messages(), { highWaterMark: 16 * 1024 });
console.log(stream.readableObjectMode, stream.readableHighWaterMark); // true 16384

stream.once('readable', () => {
  const msg = stream.read(); // the app takes one message
  setTimeout(() => {
    console.log(msg.id, stream.readableLength); // 1 16384
    stream.destroy(); // returns the generator, so its finally block runs
  }, 100);
});

go deeper

for a junior

Know that object mode lets a stream carry JavaScript objects and that Readable.from() makes object-mode streams.

for a middle

Explain that highWaterMark counts objects in object mode, give the defaults, and describe how read(size) and null behave there.

for a senior

Diagnose oversized prefetch from a byte-sized highWaterMark on an object-mode stream, and tie generator cleanup to stream destruction.

for a principal

Decide a team default for in-flight message counts on stream-wrapped consumers, weighing throughput against memory and redelivery exposure.

## Two axes: class and mode Every Node stream has a **class** (Readable, Writable, Duplex, Transform) and a **mode**. In the default byte mode, chunks are strings, `Buffer`, `TypedArray` or `DataView` values. In **object mode** (`objectMode: true`), a chunk can be any JavaScript value except `null`, which is reserved to mean end of stream. The mode is set when the stream is constructed; the docs call switching an existing stream into object mode unsafe. ## What highWaterMark measures `highWaterMark` is the threshold at which a Readable stops asking its source for more data. Its **unit depends on the mode**: | Stream | `highWaterMark` counts | Default in Node 24 | |---|---|---| | byte mode | bytes | 65,536 (64 KiB) on non-Windows, 16,384 (16 KiB) on Windows | | string chunks, not decoded from bytes | UTF-16 code units | same as bytes | | object mode | **objects** | 16 | | `Readable.from(iterable)` with no `highWaterMark` | objects | 1 in the Node 24 source (not stated in the API docs) | `stream.getDefaultHighWaterMark(objectMode)` returns the byte and object-mode defaults, and `stream.setDefaultHighWaterMark()` changes them process-wide. The byte default was raised in v22.0.0. The threshold is a trigger for refilling, not a hard memory cap; the backpressure lesson covers that side. ## What went wrong in the scenario The message consumer was exposed like this: ```js const stream = Readable.from(messages(), { highWaterMark: 16 * 1024 }); ``` 1. `Readable.from()` defaults to `objectMode: true`, so the stream counts messages. 2. `16 * 1024` was meant as "16 KiB", a byte-stream habit. In object mode it means **16,384 messages**. 3. When a consumer first reads, the stream calls the generator again and again while its buffer is below the threshold, so it **prefetches** up to 16,384 messages. 4. Each prefetched message is already out of the queue (and possibly leased or awaiting acknowledgement) but not yet handled by your code. Memory grows, redelivery timers can expire, and a crash abandons the whole buffered batch. The fix is to choose the count deliberately: the default of 1 from `Readable.from()`, or a small number such as 16 to 64 that matches how many in-flight messages the service can afford. ## Other object-mode behaviours worth knowing - `readable.read(size)` in object mode returns **a single item** regardless of `size`. - Pushing a plain object into a byte-mode Readable errors the stream with `ERR_INVALID_ARG_TYPE`; writing one to a byte-mode Writable throws it synchronously. - `null` cannot travel as data. In the Node 24 source, an iterable that yields `null` makes `Readable.from()` destroy the stream with `ERR_STREAM_NULL_VALUES`. - `Readable.from('a string')` and `Readable.from(buffer)` emit the whole string or Buffer as **one** chunk; they are not iterated character by character. - A `Duplex` or `Transform` can be in object mode on one side only, using `readableObjectMode` or `writableObjectMode`. - `readable.readableObjectMode`, `readable.readableHighWaterMark` and `readable.readableLength` let you inspect all of this at run time. ## Choosing the count There is no universal right value; derive it from the workload: - **Processing time per message**: a buffer only needs to cover the gap between the moment your code finishes one message and the moment the next would otherwise arrive. - **Memory per message**: multiply the count by the typical message size and by the number of streams in the process. - **Lease or acknowledgement windows**: a prefetched message waits in the buffer while its clock runs, so a large buffer can let messages expire before they are processed. - **Observability**: export `readable.readableLength` as a metric; if it sits at the threshold, the consumer is the bottleneck and a larger buffer will not help. ## Shutting the consumer down `Readable.from()` keeps a link to the generator. When the stream is destroyed (by `destroy()`, by an error, or by leaving a `for await` loop early), Node calls the generator's `return()`, which runs its `finally` block. Put the consumer's close or unsubscribe call there, and the stream's lifecycle and the queue consumer's lifecycle stay tied together. ## Summary - Object mode changes the **unit** of `highWaterMark` from bytes to values. - `Readable.from()` turns object mode on by default and, in the Node 24 source, keeps a buffer of one item. - A byte-sized number in an object-mode stream silently turns into a large prefetch of whole messages.

  • What highWaterMark does a stream created by new Readable({ objectMode: true }) get by default?
    16 objects, the value `stream.getDefaultHighWaterMark(true)` returns. Byte streams default to 64 KiB on non-Windows platforms and 16 KiB on Windows since v22.0.0. `Readable.from()` is the exception: in the Node 24 source it passes `highWaterMark: 1` unless you supply one.
  • How do you make sure the queue consumer is closed when the stream is abandoned?
    Put the close or unsubscribe call in the generator's `finally` block. When the stream is destroyed, whether by `destroy()`, an error or leaving a `for await` loop early, `Readable.from()` calls the generator's `return()`, and the `finally` block runs.
  • Can you turn object mode on for an existing byte stream?
    Not safely. Object mode is set with the `objectMode` option at construction, and the docs say switching an existing stream into it is not safe. Create a new object-mode stream, for example a Transform with `readableObjectMode: true` that parses bytes into objects.

  • highWaterMark is always a number of bytes.
  • Readable.from() produces a byte stream unless objectMode is set.
  • readable.read(1024) on an object-mode stream returns 1024 objects.
  • Readable.from('hello') emits one chunk per character.
  • You can set objectMode on a stream after it has been created.

Assumes Node.js 24: the default byte-stream highWaterMark was raised to 64 KiB on non-Windows platforms in v22.0.0, while the object-mode default of 16 did not change.