skip to content

In Node.js, how do Buffer.from(string, encoding) and buf.toString(encoding) convert between text and bytes, and why can a string's length differ from Buffer.byteLength()?

level: juniorimportance: must knowfreq 60%

answer

  1. default encoding when none is given
  2. base64, base64url, hex: bytes as text
  3. UTF-16 code units vs encoded bytes
  4. what a Content-Length must count

basics

~20 s

Buffer.from(str, enc) encodes a string into bytes and buf.toString(enc) decodes bytes back, both defaulting to utf8; base64 and hex render bytes as text. A string's length counts UTF-16 code units, while Buffer.byteLength() counts encoded bytes.

solid answer

~40 s

`Buffer.from(str, encoding)` encodes a string into a new `Buffer` and `buf.toString(encoding)` decodes it back; both default to `'utf8'`. Node also speaks `'base64'`, `'base64url'`, `'hex'`, `'latin1'` and `'utf16le'`, so returning a thumbnail as text is `img.toString('base64')` and reading it back is `Buffer.from(b64, 'base64')`. A string's `length` counts UTF-16 code units, not bytes: `'½ + ¼ = ¾'` is 9 characters but 12 UTF-8 bytes, so anything that promises bytes (a `Content-Length`, a length prefix, a size limit) must use `Buffer.byteLength(str, 'utf8')`. Decoding is lenient rather than strict: invalid UTF-8 becomes `U+FFFD`, and hex decoding stops at the first non-hex character, so validate untrusted input yourself.

code

javascript · 10 lines
javascript
import { Buffer } from 'node:buffer';

const s = '½ + ¼ = ¾';
console.log(s.length, Buffer.byteLength(s, 'utf8')); // 9 12

const b64 = Buffer.from('hello world').toString('base64');
console.log(b64); // aGVsbG8gd29ybGQ=
console.log(Buffer.from(b64, 'base64').toString()); // hello world

console.log(Buffer.from('1ag123', 'hex')); // <Buffer 1a>

go deeper

for a junior

Recall the two calls, Buffer.from(str, enc) and buf.toString(enc), the utf8 default, and that base64 and hex turn bytes into printable text. Have the 9-characters-12-bytes example ready.

for a middle

Explain why length and Buffer.byteLength differ, where a byte count is the contract, and how Node decodes leniently: U+FFFD for bad UTF-8, hex stopping at a bad character, base64 ignoring whitespace.

for a senior

Show the production edge: binary routed through utf8 is corrupted for good, a Content-Length built from length breaks on non-ASCII, and untrusted hex or base64 must be validated because Node will not reject it.

for a principal

Frame it as a team convention: binary stays a Buffer until the boundary, every byte budget uses Buffer.byteLength, and inbound encoded values are validated explicitly rather than trusted to the decoder.

## What a Buffer is A **`Buffer`** is Node's fixed-length container for raw bytes. Since Node 3 it has been a subclass of `Uint8Array`, so it inherits the typed-array methods (with a few deliberate differences), and Node adds what servers need on top: conversion to and from strings in named **encodings**, fixed-width integer accessors and comparison helpers. It lives in `node:buffer` and is also a global, so `import { Buffer } from 'node:buffer'` is optional but makes the dependency explicit. Two words carry this question: - **Encoding** a string turns characters into bytes: `Buffer.from(str, encoding)`. - **Decoding** turns bytes back into characters: `buf.toString(encoding)`. For the binary-to-text encodings (`base64`, `base64url`, `hex`) Node's documentation flips the naming: `buf.toString('base64')` is called *encoding*, because it turns arbitrary bytes into printable text. ## The encodings Node accepts Both calls default to `'utf8'` when no encoding is passed, and encoding names are case-insensitive (`'UTF8'` works too). | Encoding | Typical direction | What to know | |---|---|---| | `'utf8'` / `'utf-8'` | text to bytes and back | The default; invalid input decodes to `U+FFFD` | | `'utf16le'` / `'ucs2'` | text to bytes and back | Little-endian only, 2 or 4 bytes per character | | `'latin1'` / `'binary'` | text to bytes and back | One byte per character, `U+0000` to `U+00FF` only | | `'base64'` | bytes to printable text | Decoding also accepts the URL-safe alphabet and ignores whitespace | | `'base64url'` | bytes to URL-safe text | Encoding omits `=` padding; decoding also accepts regular base64 | | `'hex'` | bytes to two hex digits each | Decoding stops at the first non-hex character | | `'ascii'` | legacy | Kept for compatibility; `'utf8'` or `'latin1'` is the better choice | The name `'binary'` misleads: it is only an alias for `'latin1'`, and every encoding in the table converts between strings and binary data. ## Length is not byte length A JavaScript string's `length` counts **UTF-16 code units**, not bytes. `Buffer.byteLength(str, encoding)` returns how many bytes the string becomes in a given encoding, `'utf8'` by default: ```js const s = '½ + ¼ = ¾'; console.log(s.length); // 9 console.log(Buffer.byteLength(s, 'utf8')); // 12 ``` ASCII text gives the same number both ways, which is why this bug survives tests written with English sample data and surfaces with the first accented name or emoji. Wherever a byte count is the contract, use `Buffer.byteLength` or the `length` of the Buffer you are about to send: 1. a `Content-Length` header on a response body you assemble yourself; 2. a length prefix in a binary protocol; 3. a size limit enforced before a write or an upload. For `'base64'`, `'base64url'` and `'hex'`, `Buffer.byteLength` assumes valid input; with whitespace or junk in the string it can report more bytes than `Buffer.from` actually produces. ## Node decodes leniently Node prefers producing a value over throwing, which is convenient and occasionally dangerous: - `buf.toString('utf8')` replaces invalid byte sequences with `U+FFFD`, the replacement character, instead of raising an error. - `Buffer.from('1ag123', 'hex')` returns `<Buffer 1a>`: decoding stops at the first non-hex character, and a dangling odd digit is dropped as well. - `Buffer.from(str, 'base64')` skips spaces, tabs and newlines and accepts `-` and `_` from the URL-safe alphabet. - An **unknown encoding name** is treated as a programming error: `Buffer.from('x', 'utf9')` throws a `TypeError` with code `ERR_UNKNOWN_ENCODING`. So when a silently shortened value would matter, such as a token, a key or a checksum arriving as hex or base64, check its length and alphabet yourself before trusting the decoded bytes. ## Worked case: returning a thumbnail as base64 An endpoint must return a small image inside a JSON payload: 1. Keep the image as a `Buffer` from the moment it is read; never route it through `toString('utf8')`, which turns every byte sequence that is not valid UTF-8 into `U+FFFD` and cannot be undone. 2. Encode once, at the edge: `const b64 = img.toString('base64')`, or `'base64url'` if the value travels inside a URL. 3. Budget for growth: base64 spends four characters on every three bytes, so the text is about a third larger than the image. 4. On the way back in, `Buffer.from(b64, 'base64')` restores the exact bytes, and `decoded.equals(img)` proves the round trip in a test. ## Mistakes that show up in review - Using `str.length` for a header or prefix that promises a byte count. - Calling `toString()` on binary data to log or store it, then trying to recover the bytes later. - Encoding a large binary to base64 only to decode it again inside the same process, instead of passing the `Buffer`. - Assuming invalid hex or base64 input throws, and skipping validation because of it.

  • What does 'base64url' change compared with 'base64' in Node?
    When encoding, `'base64url'` uses the URL- and filename-safe alphabet (`-` and `_` in place of `+` and `/`) and omits `=` padding, so the result can sit in a URL path or query without escaping. When decoding, Node is lenient both ways: `Buffer.from(str, 'base64url')` also accepts regular base64, and `'base64'` decoding also accepts the URL-safe alphabet and ignores whitespace.
  • Why is 'binary' a misleading encoding name in Node?
    `'binary'` is only an alias for `'latin1'`, which maps each character from `U+0000` to `U+00FF` to one byte and truncates anything above that range. Every encoding converts between strings and binary data, so the name says nothing useful. For text, `'utf8'` is almost always right; for raw bytes, the right move is to keep the `Buffer` rather than convert it to a string.
  • Does Buffer.byteLength() give the right answer for a base64 string?
    Only for clean input. For `'base64'`, `'base64url'` and `'hex'`, Node's `Buffer.byteLength` assumes the string is valid; whitespace or other non-alphabet characters make it report more bytes than `Buffer.from(str, 'base64')` actually produces. When the exact decoded size matters, decode first and read the resulting Buffer's `length`.

  • A string's length is the number of bytes it will put on the wire
  • Buffer.from(str) without an encoding defaults to ASCII or latin1
  • buf.toString('utf8') throws when the bytes are not valid UTF-8
  • Node throws on hex input containing a non-hex character
  • Base64 text is smaller than the bytes it encodes