skip to content

In Node.js, why does calling toString('utf8') on each incoming Buffer chunk sometimes produce U+FFFD characters, and how do StringDecoder and TextDecoder fix it?

level: middleimportance: should knowfreq 35%

answer

  1. a character spanning two chunks
  2. decoder that holds trailing bytes
  3. stream: true per decode call
  4. flush at the end

basics

~20 s

A chunk boundary can fall inside a multi-byte UTF-8 character, so decoding each chunk alone turns the split bytes into U+FFFD. StringDecoder.write() and TextDecoder.decode(chunk, { stream: true }) hold the incomplete trailing bytes until the next chunk completes them.

solid answer

~40 s

UTF-8 spends up to four bytes on a character, and chunks from a socket, a child process or a file read split wherever they split. `chunk.toString('utf8')` sees an incomplete sequence at the end of one chunk and stray continuation bytes at the start of the next, and Node substitutes `U+FFFD` instead of throwing. A stateful decoder fixes it: `new StringDecoder('utf8')` from `node:string_decoder` returns only complete characters from `write(buf)` and holds the rest, and `end()` flushes; the global `TextDecoder` does the same when each call passes `{ stream: true }`, with a final `decode()` to flush, and can throw instead of substituting when built with `{ fatal: true }`. For a small bounded body, collecting the Buffers and decoding once after `Buffer.concat` is just as correct.

code

javascript · 13 lines
javascript
import { Buffer } from 'node:buffer';
import { StringDecoder } from 'node:string_decoder';

const euro = Buffer.from('€', 'utf8'); // <Buffer e2 82 ac>
const parts = [euro.subarray(0, 2), euro.subarray(2)];

console.log(parts.map((p) => p.toString('utf8')).join('') === '€'); // false

const decoder = new StringDecoder('utf8');
let text = '';
for (const p of parts) text += decoder.write(p);
text += decoder.end();
console.log(text === '€'); // true

go deeper

for a junior

Recall that UTF-8 characters can be several bytes and that decoding chunks one by one can split them into U+FFFD characters.

for a middle

Explain how StringDecoder.write() and end() and TextDecoder with stream: true hold partial bytes, and why Node substitutes instead of throwing.

for a senior

Diagnose the intermittent garbled-text report: non-ASCII data plus chunk boundaries, toString or string concatenation per chunk, and fix it with a per-input decoder, a final flush and fatal mode where invalid input must be rejected.

for a principal

Set a team rule that text decoding happens in exactly one place per input, with a stateful decoder or a decode-once path behind a size limit, and pick TextDecoder where code is shared across runtimes.

## Why chunked text breaks UTF-8 encodes a character in one to four bytes; `€` is `E2 82 AC`. When bytes arrive in pieces, from a socket, a child process's stdout or a file read with a fixed buffer size, a chunk boundary can fall in the middle of such a sequence. Decoding each piece on its own with `chunk.toString('utf8')` sees an incomplete sequence at the end of one chunk and orphaned continuation bytes at the start of the next. Node's UTF-8 decoding does not throw on invalid input; it substitutes `U+FFFD`, the **replacement character**. The output carries `�` where a character was, and that text is then logged, stored or compared. The bug is intermittent by nature. It needs non-ASCII text **and** an unlucky boundary, so it passes tests written with English sample data and appears under real traffic, often only for some users. ## StringDecoder: Node's stateful decoder `node:string_decoder` exists for exactly this problem. A `StringDecoder` keeps the trailing bytes of an incomplete character in an internal buffer instead of decoding them: - `decoder.write(buf)` returns the decoded text **without** any incomplete multi-byte character at the end, and holds those bytes. - The next `write()` prepends the held bytes, so the split character comes out whole. - `decoder.end([buf])` performs a final `write()` if a buffer is given, then returns whatever is still held, with incomplete UTF-8 or UTF-16 sequences replaced by substitution characters. After `end()` the decoder can be reused for new input. The constructor takes the same encoding names as `Buffer` and defaults to `'utf8'`. The holding behaviour is what matters for UTF-8 and UTF-16, where one character spans several bytes. ## TextDecoder: the WHATWG equivalent `TextDecoder` is a global in Node, also exported from `node:util`, and implements the WHATWG Encoding Standard, so the same decoding code runs in browsers and other runtimes. Its streaming mode is opt-in per call: ```js const td = new TextDecoder('utf-8'); let text = ''; for (const chunk of chunks) text += td.decode(chunk, { stream: true }); text += td.decode(); // flush whatever is still held ``` - With `{ stream: true }`, incomplete trailing bytes are buffered and emitted after the next `decode()` call. - A final `decode()` without `stream` flushes, like `StringDecoder`'s `end()`. - `new TextDecoder('utf-8', { fatal: true })` throws a `TypeError` with code `ERR_ENCODING_INVALID_ENCODED_DATA` on invalid data instead of substituting, which suits input that must be valid. The `fatal` option is unavailable on Node builds without ICU. - `ignoreBOM` is a constructor option; by default a leading byte-order mark is stripped from the output. ## Choosing between them | | `StringDecoder` | `TextDecoder` | |---|---|---| | Where | `node:string_decoder` | global, also `node:util` | | Streaming | always: `write()` holds partial bytes | per call: `decode(chunk, { stream: true })` | | Flush | `end()` | `decode()` without `stream` | | Invalid input | substitutes | substitutes, or throws with `fatal: true` | | Accepts | string, Buffer, TypedArray, DataView | ArrayBuffer, TypedArray, DataView | | Portability | Node only | WHATWG standard | Both fix the split-character bug. `TextDecoder` suits code shared with other runtimes and inputs that should be rejected rather than repaired; `StringDecoder` is common in existing Node code and accepts Node's own encoding names. ## Other ways out 1. **Collect bytes, decode once.** For a bounded body, keep the chunks as Buffers, `Buffer.concat()` them, and call `toString()` once. It is simple and correct, at the cost of holding the whole body in memory, so pair it with a size limit. 2. **Let the stream decode.** `readable.setEncoding('utf8')` makes a Node `Readable` emit strings and handles split characters for you; how a stream delivers data is a subject of its own. 3. **Split on bytes before decoding.** A newline-delimited protocol can find each line with `buf.indexOf(0x0a)` and decode whole lines, because in UTF-8 the byte `0x0a` never occurs inside a multi-byte character. ## Review checklist - Any `chunk.toString()` inside a read loop or a `'data'` handler over text is suspect. - `text += chunk`, where `chunk` is a Buffer, is the same bug in disguise: string concatenation calls `toString()` on each chunk. - Skipping the final `end()` or flush silently drops a truncated trailing character instead of surfacing it as `U+FFFD`. - A decoder instance belongs to one input; sharing it across interleaved connections splices their bytes together.

  • Why does a decoder instance need to be per connection rather than shared?
    A `StringDecoder` or a streaming `TextDecoder` keeps the held bytes of an incomplete character as internal state. If two connections feed the same instance, the trailing bytes of one connection's chunk get prepended to the other's next chunk, producing wrong characters on both. Create one decoder per input and call `end()` or a final `decode()` when that input finishes.
  • When would you choose TextDecoder with fatal: true over StringDecoder?
    When invalid UTF-8 means the input is wrong and should be rejected, for example a field that is specified as UTF-8 text in a protocol. `new TextDecoder('utf-8', { fatal: true })` throws a `TypeError` with code `ERR_ENCODING_INVALID_ENCODED_DATA` instead of quietly inserting `U+FFFD`, and the same code also runs outside Node. `StringDecoder` has no strict mode; it always substitutes.

  • toString('utf8') on each chunk is safe because Node buffers partial characters automatically
  • Invalid UTF-8 in a Buffer makes toString() throw, so the bug would be noticed
  • StringDecoder.write() returns a replacement character for the held bytes
  • TextDecoder buffers incomplete bytes by default without any option
  • The problem only affects UTF-16, not UTF-8