skip to content

Binary Data Handling

Buffer is a Uint8Array subclass with encodings and fixed-width accessors for parsing binary formats. Asked about allocUnsafe returning uninitialised memory and about subarray sharing, not copying.

on this pageshow

explore

questions

5

In Node.js, how do Buffer.from(string, encoding) and buf.toString(encoding) convert between text and bytes, and why can a string's length differ from Buffer.byteLength()?

level: juniorimportance: must knowfreq 60%

answer

  1. default encoding when none is given
  2. base64, base64url, hex: bytes as text
  3. UTF-16 code units vs encoded bytes
  4. what a Content-Length must count

basics

~20 s

Buffer.from(str, enc) encodes a string into bytes and buf.toString(enc) decodes bytes back, both defaulting to utf8; base64 and hex render bytes as text. A string's length counts UTF-16 code units, while Buffer.byteLength() counts encoded bytes.

solid answer

~40 s

`Buffer.from(str, encoding)` encodes a string into a new `Buffer` and `buf.toString(encoding)` decodes it back; both default to `'utf8'`. Node also speaks `'base64'`, `'base64url'`, `'hex'`, `'latin1'` and `'utf16le'`, so returning a thumbnail as text is `img.toString('base64')` and reading it back is `Buffer.from(b64, 'base64')`. A string's `length` counts UTF-16 code units, not bytes: `'½ + ¼ = ¾'` is 9 characters but 12 UTF-8 bytes, so anything that promises bytes (a `Content-Length`, a length prefix, a size limit) must use `Buffer.byteLength(str, 'utf8')`. Decoding is lenient rather than strict: invalid UTF-8 becomes `U+FFFD`, and hex decoding stops at the first non-hex character, so validate untrusted input yourself.

code

javascript · 10 lines
javascript
import { Buffer } from 'node:buffer';

const s = '½ + ¼ = ¾';
console.log(s.length, Buffer.byteLength(s, 'utf8')); // 9 12

const b64 = Buffer.from('hello world').toString('base64');
console.log(b64); // aGVsbG8gd29ybGQ=
console.log(Buffer.from(b64, 'base64').toString()); // hello world

console.log(Buffer.from('1ag123', 'hex')); // <Buffer 1a>

go deeper

for a junior

Recall the two calls, Buffer.from(str, enc) and buf.toString(enc), the utf8 default, and that base64 and hex turn bytes into printable text. Have the 9-characters-12-bytes example ready.

for a middle

Explain why length and Buffer.byteLength differ, where a byte count is the contract, and how Node decodes leniently: U+FFFD for bad UTF-8, hex stopping at a bad character, base64 ignoring whitespace.

for a senior

Show the production edge: binary routed through utf8 is corrupted for good, a Content-Length built from length breaks on non-ASCII, and untrusted hex or base64 must be validated because Node will not reject it.

for a principal

Frame it as a team convention: binary stays a Buffer until the boundary, every byte budget uses Buffer.byteLength, and inbound encoded values are validated explicitly rather than trusted to the decoder.

## What a Buffer is A **`Buffer`** is Node's fixed-length container for raw bytes. Since Node 3 it has been a subclass of `Uint8Array`, so it inherits the typed-array methods (with a few deliberate differences), and Node adds what servers need on top: conversion to and from strings in named **encodings**, fixed-width integer accessors and comparison helpers. It lives in `node:buffer` and is also a global, so `import { Buffer } from 'node:buffer'` is optional but makes the dependency explicit. Two words carry this question: - **Encoding** a string turns characters into bytes: `Buffer.from(str, encoding)`. - **Decoding** turns bytes back into characters: `buf.toString(encoding)`. For the binary-to-text encodings (`base64`, `base64url`, `hex`) Node's documentation flips the naming: `buf.toString('base64')` is called *encoding*, because it turns arbitrary bytes into printable text. ## The encodings Node accepts Both calls default to `'utf8'` when no encoding is passed, and encoding names are case-insensitive (`'UTF8'` works too). | Encoding | Typical direction | What to know | |---|---|---| | `'utf8'` / `'utf-8'` | text to bytes and back | The default; invalid input decodes to `U+FFFD` | | `'utf16le'` / `'ucs2'` | text to bytes and back | Little-endian only, 2 or 4 bytes per character | | `'latin1'` / `'binary'` | text to bytes and back | One byte per character, `U+0000` to `U+00FF` only | | `'base64'` | bytes to printable text | Decoding also accepts the URL-safe alphabet and ignores whitespace | | `'base64url'` | bytes to URL-safe text | Encoding omits `=` padding; decoding also accepts regular base64 | | `'hex'` | bytes to two hex digits each | Decoding stops at the first non-hex character | | `'ascii'` | legacy | Kept for compatibility; `'utf8'` or `'latin1'` is the better choice | The name `'binary'` misleads: it is only an alias for `'latin1'`, and every encoding in the table converts between strings and binary data. ## Length is not byte length A JavaScript string's `length` counts **UTF-16 code units**, not bytes. `Buffer.byteLength(str, encoding)` returns how many bytes the string becomes in a given encoding, `'utf8'` by default: ```js const s = '½ + ¼ = ¾'; console.log(s.length); // 9 console.log(Buffer.byteLength(s, 'utf8')); // 12 ``` ASCII text gives the same number both ways, which is why this bug survives tests written with English sample data and surfaces with the first accented name or emoji. Wherever a byte count is the contract, use `Buffer.byteLength` or the `length` of the Buffer you are about to send: 1. a `Content-Length` header on a response body you assemble yourself; 2. a length prefix in a binary protocol; 3. a size limit enforced before a write or an upload. For `'base64'`, `'base64url'` and `'hex'`, `Buffer.byteLength` assumes valid input; with whitespace or junk in the string it can report more bytes than `Buffer.from` actually produces. ## Node decodes leniently Node prefers producing a value over throwing, which is convenient and occasionally dangerous: - `buf.toString('utf8')` replaces invalid byte sequences with `U+FFFD`, the replacement character, instead of raising an error. - `Buffer.from('1ag123', 'hex')` returns `<Buffer 1a>`: decoding stops at the first non-hex character, and a dangling odd digit is dropped as well. - `Buffer.from(str, 'base64')` skips spaces, tabs and newlines and accepts `-` and `_` from the URL-safe alphabet. - An **unknown encoding name** is treated as a programming error: `Buffer.from('x', 'utf9')` throws a `TypeError` with code `ERR_UNKNOWN_ENCODING`. So when a silently shortened value would matter, such as a token, a key or a checksum arriving as hex or base64, check its length and alphabet yourself before trusting the decoded bytes. ## Worked case: returning a thumbnail as base64 An endpoint must return a small image inside a JSON payload: 1. Keep the image as a `Buffer` from the moment it is read; never route it through `toString('utf8')`, which turns every byte sequence that is not valid UTF-8 into `U+FFFD` and cannot be undone. 2. Encode once, at the edge: `const b64 = img.toString('base64')`, or `'base64url'` if the value travels inside a URL. 3. Budget for growth: base64 spends four characters on every three bytes, so the text is about a third larger than the image. 4. On the way back in, `Buffer.from(b64, 'base64')` restores the exact bytes, and `decoded.equals(img)` proves the round trip in a test. ## Mistakes that show up in review - Using `str.length` for a header or prefix that promises a byte count. - Calling `toString()` on binary data to log or store it, then trying to recover the bytes later. - Encoding a large binary to base64 only to decode it again inside the same process, instead of passing the `Buffer`. - Assuming invalid hex or base64 input throws, and skipping validation because of it.

  • What does 'base64url' change compared with 'base64' in Node?
    When encoding, `'base64url'` uses the URL- and filename-safe alphabet (`-` and `_` in place of `+` and `/`) and omits `=` padding, so the result can sit in a URL path or query without escaping. When decoding, Node is lenient both ways: `Buffer.from(str, 'base64url')` also accepts regular base64, and `'base64'` decoding also accepts the URL-safe alphabet and ignores whitespace.
  • Why is 'binary' a misleading encoding name in Node?
    `'binary'` is only an alias for `'latin1'`, which maps each character from `U+0000` to `U+00FF` to one byte and truncates anything above that range. Every encoding converts between strings and binary data, so the name says nothing useful. For text, `'utf8'` is almost always right; for raw bytes, the right move is to keep the `Buffer` rather than convert it to a string.
  • Does Buffer.byteLength() give the right answer for a base64 string?
    Only for clean input. For `'base64'`, `'base64url'` and `'hex'`, Node's `Buffer.byteLength` assumes the string is valid; whitespace or other non-alphabet characters make it report more bytes than `Buffer.from(str, 'base64')` actually produces. When the exact decoded size matters, decode first and read the resulting Buffer's `length`.

  • A string's length is the number of bytes it will put on the wire
  • Buffer.from(str) without an encoding defaults to ASCII or latin1
  • buf.toString('utf8') throws when the bytes are not valid UTF-8
  • Node throws on hex input containing a non-hex character
  • Base64 text is smaller than the bytes it encodes
open as a page

In Node.js 24, how do Buffer.alloc and Buffer.allocUnsafe differ, what is the shared Buffer pool, and when does allocUnsafe leak sensitive data?

level: seniorimportance: must knowfreq 50%

basics

~20 s

Buffer.alloc zero-fills and never uses the pool; Buffer.allocUnsafe skips initialisation and carves small sizes from a shared slab of Buffer.poolSize bytes, 64 KiB since Node 24.18.0. Any allocUnsafe byte sent before being overwritten can leak earlier data.

open as a page

In Node.js, how do you parse a length-prefixed binary frame with Buffer accessors such as readUInt32BE, and what goes wrong when a frame arrives split across chunks?

level: middleimportance: should knowfreq 40%

basics

~20 s

Read fixed-width fields with accessors such as buf.readUInt32BE(offset) and buf.readUInt8(offset), matching the protocol's width, signedness and byte order. Chunks do not align with frames, so accumulate bytes, wait until header and payload are complete, then subarray the frame out.

open as a page

In Node.js, why does calling toString('utf8') on each incoming Buffer chunk sometimes produce U+FFFD characters, and how do StringDecoder and TextDecoder fix it?

level: middleimportance: should knowfreq 35%

basics

~20 s

A chunk boundary can fall inside a multi-byte UTF-8 character, so decoding each chunk alone turns the split bytes into U+FFFD. StringDecoder.write() and TextDecoder.decode(chunk, { stream: true }) hold the incomplete trailing bytes until the next chunk completes them.

open as a page

In Node.js, buf.subarray() returns a view rather than a copy; what production bugs does that cause, and when should you copy with Buffer.from()?

level: seniorimportance: should knowfreq 38%

basics

~20 s

buf.subarray() shares memory with buf, so later writes to the source show through and a small view keeps the whole underlying ArrayBuffer alive. Copy with Buffer.from(view) or Buffer.copyBytesFrom() whenever the bytes outlive the call or the source buffer is reused.

open as a page