skip to content

A Node.js service converting documents with child_process.execFile now fails with 'stdout maxBuffer length exceeded'; what did Node do, and how do you fix it?

level: middleimportance: should knowfreq 40%

answer

  1. buffered APIs carry a byte cap
  2. one mebibyte per stream
  3. the child is signalled too
  4. stream it or write a file

basics

~20 s

execFile buffers stdout and stderr up to maxBuffer bytes each (1024 * 1024 by default); past that Node truncates the output, signals the child with killSignal and fails with ERR_CHILD_PROCESS_STDIO_MAXBUFFER. Stream large output with spawn instead.

solid answer

~40 s

`exec` and `execFile` collect output in the parent's memory and enforce `maxBuffer`, counted in bytes separately for stdout and stderr, default `1024 * 1024`. When a chunk pushes either stream past it, Node keeps only the bytes up to the limit, sends the child `killSignal` (`'SIGTERM'` by default) and calls back with a `RangeError` whose `code` is `ERR_CHILD_PROCESS_STDIO_MAXBUFFER` and whose message is `stdout maxBuffer length exceeded`. So the conversion itself was cut short, not just its log. The fix is to stop buffering output whose size you do not control: have the converter write to an output file, or use `spawn` and pipe `subprocess.stdout` to its destination with `stream.pipeline`. Raising `maxBuffer` is right only when the output is known to be bounded, because every concurrent request holds its whole output in memory.

code

javascript · 7 lines
javascript
import { execFile } from 'node:child_process';

const script = 'process.stdout.write("x".repeat(2 * 1024 * 1024))';
execFile(process.execPath, ['-e', script], (err, stdout) => {
  console.log(err.code, '|', err.message, '|', stdout.length);
  // ERR_CHILD_PROCESS_STDIO_MAXBUFFER | stdout maxBuffer length exceeded | 1048576
});

go deeper

for a junior

Remember that exec and execFile collect output up to maxBuffer, 1024 * 1024 bytes per stream by default, while spawn hands you streams with no such cap.

for a middle

Walk through what Node does at the limit: truncate, signal the child with killSignal, and fail with ERR_CHILD_PROCESS_STDIO_MAXBUFFER, counted in bytes.

for a senior

Show you fix the design rather than the number: output files or spawn with stream.pipeline, and you also guard against unread pipes stalling the child.

for a principal

Frame buffer limits as memory budgets multiplied by concurrency, and make bounded output a property you design for in every subprocess integration.

## Why the buffered functions have a cap `child_process.exec()` and `child_process.execFile()` are conveniences: instead of handing you streams, they **collect everything** the child writes to stdout and stderr and pass it to your callback (or resolve the promisified version) when the child closes. Collecting means holding it in the parent's memory, so both functions enforce **`maxBuffer`**: - It is the largest amount of data, **in bytes**, allowed on stdout *or* stderr. Each stream is counted separately. - The default is **`1024 * 1024`** bytes (1 MiB) per stream. - `spawn()` and `fork()` have no such option, because they never accumulate output for you. The cap is a safety valve. Without it, one program that prints far more than expected would grow the parent's heap until the service fell over. ## What happens when the cap is crossed The logic in `lib/child_process.js` runs on every chunk the child writes: 1. The chunk's byte length is added to that stream's running total. 2. If the total now exceeds `maxBuffer`, Node keeps only the part of the chunk that fits and discards the rest, so the output you receive is **truncated** at the limit. 3. Node destroys the stdout and stderr streams and sends the child **`killSignal`**, which defaults to `'SIGTERM'`. 4. Your callback receives a **`RangeError`** with `code: 'ERR_CHILD_PROCESS_STDIO_MAXBUFFER'` and the message `stdout maxBuffer length exceeded` (or `stderr ...`), together with the truncated `stdout` and `stderr`. The third step is the one people miss. The limit does not only stop *collecting*; it **terminates the child**. A converter that streams a document to stdout is stopped part-way through, and if anything downstream trusted the partial output, the damage spreads past one log line. ## Bytes, not characters `maxBuffer` counts encoded bytes. Node's documentation gives the example that `console.log('中文测试')` sends 13 UTF-8 bytes for 4 characters (12 bytes of text plus the newline). Output in a non-Latin script or binary output therefore hits the limit far sooner than a character estimate suggests. The `encoding` option (default `'utf8'`; `'buffer'` returns `Buffer`s) changes what you receive, not how the limit is counted. ## Fixes, from best to worst | Fix | When it fits | Cost | |---|---|---| | Have the program write to a file path you pass as an argument | The converter accepts an output path | Nothing crosses the pipe; read or stream the file afterwards | | `spawn()` and pipe `subprocess.stdout` with `stream.pipeline` | Output must flow through the parent, for example into an HTTP response | You handle `'close'`, the exit code and stream errors yourself | | Raise `maxBuffer` to a measured bound | Output is bounded and small, such as a JSON summary | Each concurrent call can hold that much memory, twice if stderr is also large | | `maxBuffer: Infinity` | Almost never in a server | Node stops counting; one huge output can exhaust the heap | Streaming is the general answer because it keeps memory **flat**: `stream.pipeline` moves chunks as the destination accepts them, applies backpressure, and destroys both sides if either fails. The mechanics of backpressure belong to the streams material; the point here is that `spawn` gives you a stream to apply them to, and `execFile` does not. ## The opposite trap: a pipe nobody reads Moving to `spawn()` introduces the reverse failure. With the default `stdio: 'pipe'`, the child writes into an operating-system pipe of **limited capacity**. If the parent never reads `subprocess.stdout`, the pipe fills, the child blocks on its next write, and the job hangs with no error at all. So: - Read every stream you leave as `'pipe'`: pipe it, iterate it, or attach a `'data'` listener. - Use `stdio: ['ignore', 'pipe', 'inherit']` or a file descriptor for streams you do not need to capture. - Keep stderr bounded too: a chatty converter can fill the stderr pipe while you only read stdout. ## Checklist for a conversion endpoint - Prefer an output **file path** over output on stdout for anything document-sized. - If output must pass through the process, use `spawn` plus `stream.pipeline`, and wait for `'close'` to learn the exit code. - Keep `execFile` for small results and set `maxBuffer` to a deliberate value rather than relying on the default by accident. - Treat `ERR_CHILD_PROCESS_STDIO_MAXBUFFER` as a **failed job**, not a warning, since the child was terminated. - Remember that each in-flight request multiplies whatever buffer size you allow.

  • Does a timeout on execFile behave like the maxBuffer limit?
    It ends the same way. With `timeout` greater than `0`, Node sends `killSignal` (`'SIGTERM'` by default) once the child has run that many milliseconds. The callback's error then has `killed: true` and `signal` set to the signal name, while `code` is `null`. Both limits stop the child; neither is a soft warning.
  • Why can raising maxBuffer to 100 MB hurt even if every single conversion succeeds?
    Because the limit applies per call and per stream. Twenty concurrent conversions may each hold up to 100 MB of stdout as strings in the parent's heap, plus any stderr, which is a heap pressure and garbage-collection problem the single-request test never shows. Streaming keeps memory per request near the stream buffer size instead of the document size.

  • maxBuffer counts characters, so UTF-8 text gets the full limit.
  • When maxBuffer is exceeded the child keeps running and only the extra output is dropped.
  • stdout and stderr share a single combined maxBuffer.
  • spawn has the same maxBuffer limit as exec.
  • maxBuffer: Infinity is a sensible default for a server.