skip to content

Streams and Byte Buffers

Node's way of moving data you do not want in memory all at once, plus the Buffer type underneath. Interviewers ask about backpressure and pipeline because read-it-all-first dies on a large file.

on this pageshow

explore

questions

21

In Node.js, how do Buffer.from(string, encoding) and buf.toString(encoding) convert between text and bytes, and why can a string's length differ from Buffer.byteLength()?

level: juniorimportance: must knowfreq 60%

answer

  1. default encoding when none is given
  2. base64, base64url, hex: bytes as text
  3. UTF-16 code units vs encoded bytes
  4. what a Content-Length must count

basics

~20 s

Buffer.from(str, enc) encodes a string into bytes and buf.toString(enc) decodes bytes back, both defaulting to utf8; base64 and hex render bytes as text. A string's length counts UTF-16 code units, while Buffer.byteLength() counts encoded bytes.

solid answer

~40 s

`Buffer.from(str, encoding)` encodes a string into a new `Buffer` and `buf.toString(encoding)` decodes it back; both default to `'utf8'`. Node also speaks `'base64'`, `'base64url'`, `'hex'`, `'latin1'` and `'utf16le'`, so returning a thumbnail as text is `img.toString('base64')` and reading it back is `Buffer.from(b64, 'base64')`. A string's `length` counts UTF-16 code units, not bytes: `'½ + ¼ = ¾'` is 9 characters but 12 UTF-8 bytes, so anything that promises bytes (a `Content-Length`, a length prefix, a size limit) must use `Buffer.byteLength(str, 'utf8')`. Decoding is lenient rather than strict: invalid UTF-8 becomes `U+FFFD`, and hex decoding stops at the first non-hex character, so validate untrusted input yourself.

code

javascript · 10 lines
javascript
import { Buffer } from 'node:buffer';

const s = '½ + ¼ = ¾';
console.log(s.length, Buffer.byteLength(s, 'utf8')); // 9 12

const b64 = Buffer.from('hello world').toString('base64');
console.log(b64); // aGVsbG8gd29ybGQ=
console.log(Buffer.from(b64, 'base64').toString()); // hello world

console.log(Buffer.from('1ag123', 'hex')); // <Buffer 1a>

go deeper

for a junior

Recall the two calls, Buffer.from(str, enc) and buf.toString(enc), the utf8 default, and that base64 and hex turn bytes into printable text. Have the 9-characters-12-bytes example ready.

for a middle

Explain why length and Buffer.byteLength differ, where a byte count is the contract, and how Node decodes leniently: U+FFFD for bad UTF-8, hex stopping at a bad character, base64 ignoring whitespace.

for a senior

Show the production edge: binary routed through utf8 is corrupted for good, a Content-Length built from length breaks on non-ASCII, and untrusted hex or base64 must be validated because Node will not reject it.

for a principal

Frame it as a team convention: binary stays a Buffer until the boundary, every byte budget uses Buffer.byteLength, and inbound encoded values are validated explicitly rather than trusted to the decoder.

## What a Buffer is A **`Buffer`** is Node's fixed-length container for raw bytes. Since Node 3 it has been a subclass of `Uint8Array`, so it inherits the typed-array methods (with a few deliberate differences), and Node adds what servers need on top: conversion to and from strings in named **encodings**, fixed-width integer accessors and comparison helpers. It lives in `node:buffer` and is also a global, so `import { Buffer } from 'node:buffer'` is optional but makes the dependency explicit. Two words carry this question: - **Encoding** a string turns characters into bytes: `Buffer.from(str, encoding)`. - **Decoding** turns bytes back into characters: `buf.toString(encoding)`. For the binary-to-text encodings (`base64`, `base64url`, `hex`) Node's documentation flips the naming: `buf.toString('base64')` is called *encoding*, because it turns arbitrary bytes into printable text. ## The encodings Node accepts Both calls default to `'utf8'` when no encoding is passed, and encoding names are case-insensitive (`'UTF8'` works too). | Encoding | Typical direction | What to know | |---|---|---| | `'utf8'` / `'utf-8'` | text to bytes and back | The default; invalid input decodes to `U+FFFD` | | `'utf16le'` / `'ucs2'` | text to bytes and back | Little-endian only, 2 or 4 bytes per character | | `'latin1'` / `'binary'` | text to bytes and back | One byte per character, `U+0000` to `U+00FF` only | | `'base64'` | bytes to printable text | Decoding also accepts the URL-safe alphabet and ignores whitespace | | `'base64url'` | bytes to URL-safe text | Encoding omits `=` padding; decoding also accepts regular base64 | | `'hex'` | bytes to two hex digits each | Decoding stops at the first non-hex character | | `'ascii'` | legacy | Kept for compatibility; `'utf8'` or `'latin1'` is the better choice | The name `'binary'` misleads: it is only an alias for `'latin1'`, and every encoding in the table converts between strings and binary data. ## Length is not byte length A JavaScript string's `length` counts **UTF-16 code units**, not bytes. `Buffer.byteLength(str, encoding)` returns how many bytes the string becomes in a given encoding, `'utf8'` by default: ```js const s = '½ + ¼ = ¾'; console.log(s.length); // 9 console.log(Buffer.byteLength(s, 'utf8')); // 12 ``` ASCII text gives the same number both ways, which is why this bug survives tests written with English sample data and surfaces with the first accented name or emoji. Wherever a byte count is the contract, use `Buffer.byteLength` or the `length` of the Buffer you are about to send: 1. a `Content-Length` header on a response body you assemble yourself; 2. a length prefix in a binary protocol; 3. a size limit enforced before a write or an upload. For `'base64'`, `'base64url'` and `'hex'`, `Buffer.byteLength` assumes valid input; with whitespace or junk in the string it can report more bytes than `Buffer.from` actually produces. ## Node decodes leniently Node prefers producing a value over throwing, which is convenient and occasionally dangerous: - `buf.toString('utf8')` replaces invalid byte sequences with `U+FFFD`, the replacement character, instead of raising an error. - `Buffer.from('1ag123', 'hex')` returns `<Buffer 1a>`: decoding stops at the first non-hex character, and a dangling odd digit is dropped as well. - `Buffer.from(str, 'base64')` skips spaces, tabs and newlines and accepts `-` and `_` from the URL-safe alphabet. - An **unknown encoding name** is treated as a programming error: `Buffer.from('x', 'utf9')` throws a `TypeError` with code `ERR_UNKNOWN_ENCODING`. So when a silently shortened value would matter, such as a token, a key or a checksum arriving as hex or base64, check its length and alphabet yourself before trusting the decoded bytes. ## Worked case: returning a thumbnail as base64 An endpoint must return a small image inside a JSON payload: 1. Keep the image as a `Buffer` from the moment it is read; never route it through `toString('utf8')`, which turns every byte sequence that is not valid UTF-8 into `U+FFFD` and cannot be undone. 2. Encode once, at the edge: `const b64 = img.toString('base64')`, or `'base64url'` if the value travels inside a URL. 3. Budget for growth: base64 spends four characters on every three bytes, so the text is about a third larger than the image. 4. On the way back in, `Buffer.from(b64, 'base64')` restores the exact bytes, and `decoded.equals(img)` proves the round trip in a test. ## Mistakes that show up in review - Using `str.length` for a header or prefix that promises a byte count. - Calling `toString()` on binary data to log or store it, then trying to recover the bytes later. - Encoding a large binary to base64 only to decode it again inside the same process, instead of passing the `Buffer`. - Assuming invalid hex or base64 input throws, and skipping validation because of it.

  • What does 'base64url' change compared with 'base64' in Node?
    When encoding, `'base64url'` uses the URL- and filename-safe alphabet (`-` and `_` in place of `+` and `/`) and omits `=` padding, so the result can sit in a URL path or query without escaping. When decoding, Node is lenient both ways: `Buffer.from(str, 'base64url')` also accepts regular base64, and `'base64'` decoding also accepts the URL-safe alphabet and ignores whitespace.
  • Why is 'binary' a misleading encoding name in Node?
    `'binary'` is only an alias for `'latin1'`, which maps each character from `U+0000` to `U+00FF` to one byte and truncates anything above that range. Every encoding converts between strings and binary data, so the name says nothing useful. For text, `'utf8'` is almost always right; for raw bytes, the right move is to keep the `Buffer` rather than convert it to a string.
  • Does Buffer.byteLength() give the right answer for a base64 string?
    Only for clean input. For `'base64'`, `'base64url'` and `'hex'`, Node's `Buffer.byteLength` assumes the string is valid; whitespace or other non-alphabet characters make it report more bytes than `Buffer.from(str, 'base64')` actually produces. When the exact decoded size matters, decode first and read the resulting Buffer's `length`.

  • A string's length is the number of bytes it will put on the wire
  • Buffer.from(str) without an encoding defaults to ASCII or latin1
  • buf.toString('utf8') throws when the bytes are not valid UTF-8
  • Node throws on hex input containing a non-hex character
  • Base64 text is smaller than the bytes it encodes
open as a page

In a Node.js HTTP server, why stream a large CSV export to the response rather than build it in memory first?

level: juniorimportance: must knowfreq 65%

basics

~20 s

Building the whole export holds every row in memory for each request and sends nothing until the end. Streaming it through stream.pipeline sends rows as they are made and pauses the generator when the client reads slowly, so memory stays bounded.

open as a page

In Node.js's node:stream module, what are the four fundamental stream classes, and how does a Transform differ from a plain Duplex?

level: juniorimportance: must knowfreq 70%

basics

~20 s

Readable is a source, Writable a destination, Duplex is both with two independent sides (a TCP socket), and Transform is a Duplex whose output is computed from what was written to it (gzip, a cipher).

open as a page

Why did stream.pipeline replace readable.pipe() as the way to chain Node.js streams, such as a gzip stage feeding an HTTP response?

level: middleimportance: must knowfreq 70%

basics

~20 s

readable.pipe() handles backpressure but not failure: errors are not forwarded along the chain and the other streams are not closed, so they leak. stream.pipeline forwards the first error, destroys every stream, and reports completion once.

open as a page

In Node.js, what does writable.write() returning false mean, and what must the code writing to that stream do next?

level: middleimportance: must knowfreq 75%

basics

~20 s

false means the stream's internal buffer has reached its highWaterMark after accepting the chunk. Nothing is lost, but the writer should stop calling write() and resume on the 'drain' event, or memory grows without limit.

open as a page

In Node.js, what is the difference between flowing and paused mode on a stream.Readable, and which calls switch a stream between them?

level: middleimportance: must knowfreq 68%

basics

~20 s

In flowing mode a Readable pushes chunks to 'data' listeners as soon as they arrive; in paused mode you pull them with read(). A 'data' listener, resume() or pipe() starts the flow; pause() or unpiping every destination stops it.

open as a page

In Node.js 24, how do Buffer.alloc and Buffer.allocUnsafe differ, what is the shared Buffer pool, and when does allocUnsafe leak sensitive data?

level: seniorimportance: must knowfreq 50%

basics

~20 s

Buffer.alloc zero-fills and never uses the pool; Buffer.allocUnsafe skips initialisation and carves small sizes from a shared slab of Buffer.poolSize bytes, 64 KiB since Node 24.18.0. Any allocUnsafe byte sent before being overwritten can leak earlier data.

open as a page

In Node.js, what must a custom stream.Transform implement, and what do its _transform and _flush methods each do?

level: juniorimportance: should knowfreq 35%

basics

~10 s

A custom stream.Transform must implement _transform(chunk, encoding, callback), which pushes zero or more output chunks and calls callback once per input chunk; the optional _flush(callback) emits leftover output after the last input, before 'end'.

open as a page

In Node.js, how do you parse a length-prefixed binary frame with Buffer accessors such as readUInt32BE, and what goes wrong when a frame arrives split across chunks?

level: middleimportance: should knowfreq 40%

basics

~20 s

Read fixed-width fields with accessors such as buf.readUInt32BE(offset) and buf.readUInt8(offset), matching the protocol's width, signedness and byte order. Chunks do not align with frames, so accumulate bytes, wait until header and payload are complete, then subarray the frame out.

open as a page

In Node.js, why does calling toString('utf8') on each incoming Buffer chunk sometimes produce U+FFFD characters, and how do StringDecoder and TextDecoder fix it?

level: middleimportance: should knowfreq 35%

basics

~20 s

A chunk boundary can fall inside a multi-byte UTF-8 character, so decoding each chunk alone turns the split bytes into U+FFFD. StringDecoder.write() and TextDecoder.decode(chunk, { stream: true }) hold the incomplete trailing bytes until the next chunk completes them.

open as a page

In Node.js, when do you reach for stream.finished() and a stream's destroy() method instead of a 'finish' listener and end()?

level: middleimportance: should knowfreq 35%

basics

~20 s

'end' and 'finish' fire only when a stream completes gracefully, so code waiting on them hangs after an error or abort. stream.finished() settles on every outcome. end() flushes buffered data; destroy() abandons the stream at once and releases its resources.

open as a page

In Node.js, how would you write a stream.Transform that turns newline-delimited JSON bytes into parsed objects, handling records split across chunks?

level: middleimportance: should knowfreq 30%

basics

~20 s

Create the Transform with readableObjectMode: true, keep the bytes after the last newline between _transform calls, push one parsed object per complete line, parse the remainder in _flush, and report bad JSON with callback(err) exactly once.

open as a page

In a custom Node.js stream.Writable that sends records to a database, what do _write, _writev and _final each do, and how do you report a failed write?

level: middleimportance: should knowfreq 22%

basics

~20 s

_write handles one chunk and calls back when it is stored; _writev receives all buffered chunks at once for a batch insert; _final runs after end() and delays 'finish'. A failed write is reported with callback(err), never by throwing.

open as a page

In Node.js, how does for await...of over a stream.Readable differ from an async 'data' handler, and what does an early break do to the stream?

level: middleimportance: should knowfreq 55%

basics

~20 s

for await reads one chunk, waits for the loop body, then reads the next, and throws stream errors into the loop; a 'data' handler's promise is ignored. An early break destroys the stream unless you use readable.iterator({ destroyOnReturn: false }).

open as a page

In Node.js, buf.subarray() returns a view rather than a copy; what production bugs does that cause, and when should you copy with Buffer.from()?

level: seniorimportance: should knowfreq 38%

basics

~20 s

buf.subarray() shares memory with buf, so later writes to the source show through and a small view keeps the whole underlying ArrayBuffer alive. Copy with Buffer.from(view) or Buffer.copyBytesFrom() whenever the bytes outlive the call or the source buffer is reused.

open as a page

Why do open upstream connections pile up when a Node.js upload proxy forwards request bodies to object storage with req.pipe(upload) and clients disconnect mid-upload?

level: seniorimportance: should knowfreq 45%

basics

~20 s

When a client disconnects, Node destroys req, which closes without ever emitting 'end'. pipe() only ends its destination on 'end', so each upstream upload is left half-written and open. await pipeline(req, upload, { signal }) destroys both sides on any failure.

open as a page

In Node.js 24, how do objectMode and highWaterMark govern buffering in a custom Readable or Transform, and what should the implementation do when push() returns false?

level: seniorimportance: should knowfreq 25%

basics

~20 s

highWaterMark is a buffering threshold: bytes in byte mode (64 KiB default in Node 24, 16 KiB on Windows), objects in object mode (default 16). When push() returns false, a Readable stops pushing until Node calls _read() again.

open as a page

A Node.js import job feeds lines through a custom Transform and sometimes stops midway with no error and no 'finish'; what in _transform causes this, and how do you fix it?

level: seniorimportance: should knowfreq 30%

basics

~10 s

Some path in _transform never calls its callback, so the Transform waits forever for that chunk, buffers the rest and never emits 'finish'. Make every path call back exactly once, reporting failures with callback(err).

open as a page

When a Node.js service wraps a queue consumer as Readable.from(messages(), { highWaterMark: 16 * 1024 }), why do thousands of unprocessed messages pile up in memory?

level: seniorimportance: should knowfreq 40%

basics

~20 s

Readable.from() creates an object-mode stream, where highWaterMark counts values, not bytes. A value of 16,384 tells Node to prefetch up to 16,384 messages from the generator before it stops pulling. Use a small count instead.

open as a page

Why does a Node.js log-tailing service lose its early lines when it calls readline.createInterface() on fs.createReadStream(), awaits a queue connection, and only then iterates with for await?

level: seniorimportance: should knowfreq 35%

basics

~20 s

readline.createInterface() starts consuming its input immediately, switching the file stream to flowing and emitting 'line' events. The for await iterator only listens once the loop starts, so lines emitted during the await are lost. Create the interface after the await.

open as a page

Your team's shared Node.js library streams data to Node 24 services and to browser code: should its public API return a Node Readable or a web ReadableStream, and why?

level: principalimportance: nice to knowfreq 25%

basics

~20 s

If browsers or other runtimes consume it, return a web ReadableStream, the standard type fetch bodies use, and let Node callers convert with Readable.fromWeb(). If only Node code consumes it, a Node Readable fits its ecosystem better.

open as a page