skip to content

In Ruby, how do Open3.capture3, capture2e and popen3 differ when you need a command's stderr and exit status?

level: middleimportance: should knowfreq 50%

answer

  1. require "open3"
  2. three strings, two, or streams
  3. stdin_data: keyword
  4. wait_thr.value is the status
  5. full pipe buffers deadlock

basics

~10 s

Open3.capture3 returns [stdout, stderr, status]; capture2e returns [merged stdout and stderr, status]; popen3 hands you stdin, stdout and stderr streams plus a wait thread, for streaming large or interactive output.

solid answer

~40 s

After `require "open3"`, the `capture*` methods run a command to completion and return strings plus a `Process::Status`: `capture3` gives `[stdout, stderr, status]`, `capture2e` gives `[stdout_and_stderr, status]` in one merged string, and `capture2` gives `[stdout, status]`. They accept `stdin_data:` to feed input and `binmode:` for binary output, plus the same env hash, argument list and options as `spawn`. `popen3` instead yields `stdin`, `stdout`, `stderr` and a wait thread whose `value` is the status, so you can stream output as it arrives. With `popen3` you should read stdout and stderr **concurrently**: each pipe has a fixed-size buffer, and a child blocked writing to a full stderr pipe never finishes while you wait on stdout. `capture3` does that for you with two reader threads, at the cost of holding all output in memory.

code

ruby · 15 lines
ruby
require "open3"

out, err, status = Open3.capture3(
  "ffprobe", "-v", "error", "-show_format", "-of", "json", path
)
raise "ffprobe failed: #{err}" unless status.success?

log, status = Open3.capture2e("ffmpeg", "-y", "-i", path, out_path)
warn log unless status.success?

Open3.popen2e("ffmpeg", "-y", "-i", path, out_path) do |stdin, out_err, wait_thr|
  stdin.close
  out_err.each_line { |line| progress(line) }
  wait_thr.value.success?
end

go deeper

for a junior

Recall that Open3.capture3 returns stdout, stderr and a status, and that it needs require "open3".

for a middle

Explain the capture versus popen families, stdin_data:, binmode:, the merged capture2e output, and the wait thread's value.

for a senior

Show the pipe-buffer deadlock and how capture3 or popen2e avoid it, and when streaming output beats buffering it all in memory.

for a principal

Decide how the team runs external tools: one wrapper that captures stderr, bounds output size, and reports status uniformly across every service.

## Why Open3 `system` shows you nothing but success, and backticks capture standard output only. Real tools, ffmpeg among them, write their diagnostics to **standard error**, and you need the **exit status** as well. The **`open3`** library, a default gem loaded with `require "open3"`, wraps `Process.spawn` with pipes so you get all three. ## The capture family The `capture*` methods start the command, wait for it, and return strings: | Method | Returns | |---|---| | `Open3.capture3(*cmd)` | `[stdout, stderr, status]` | | `Open3.capture2e(*cmd)` | `[stdout_and_stderr, status]`, merged | | `Open3.capture2(*cmd)` | `[stdout, status]`, stderr not captured | The `status` is a `Process::Status`, so `status.success?` and `status.exitstatus` work as usual. Two options are handled by the capture methods themselves: - `stdin_data:` sends a string, or anything with `readpartial`, to the command's standard input. - `binmode:` puts the pipes in binary mode, which you want for image, audio or video bytes. Everything else is passed through to `Process.spawn`: a leading env hash, the executable and its arguments as separate strings, and trailing options such as `chdir:`. Choose `capture3` when you want to log stderr separately, and `capture2e` when you want the combined transcript in the order the tool wrote it. If the executable cannot be found, the argument-list form raises **`Errno::ENOENT`** instead of returning a status. ## The popen family `Open3.popen3(*cmd)` does not wait. With a block it yields four objects and returns the block's value: 1. `stdin`: write to the child, then close it so the child sees end of input. 2. `stdout` and `stderr`: read the child's output as it arrives. 3. `wait_thr`: a thread whose `pid` is the child's pid and whose `value` blocks until the child exits and returns its `Process::Status`. In block form the streams are closed and the child is waited for when the block ends; without a block you must close the streams yourself. `popen2e` is the variant with a single merged output stream, and `popen2` ignores stderr. ## The deadlock trap Pipes have **fixed-size buffers**. If the child fills its stderr pipe while your code is blocked reading stdout to the end, the child stops, waiting for someone to drain stderr, and your read never finishes. Both sides wait forever. The fix is to read both streams **at the same time**: - use `capture3`, which starts one reader thread per stream internally; - or use `popen2e` so there is only one stream to read; - or read with threads or `IO.select` yourself. This bites exactly on tools that are chatty on stderr, which is why streaming a long ffmpeg run through `popen3` while reading only stdout hangs. ## Memory and streaming `capture*` keeps all output in memory as strings. That is fine for a probe that prints a few kilobytes of metadata. For a command whose output is large or unbounded, or when you want progress lines while the command runs, use `popen2e` or `popen3` and process lines as they arrive. ## Environment, options and pipelines Because Open3 hands everything to `Process.spawn`, the usual extras work unchanged: - a leading hash sets variables for the child only: `Open3.capture3({"TMPDIR" => dir}, "ffmpeg", ...)`; - trailing options such as `chdir: dir` apply to the child; - `stdin_data:` also accepts an object with `readpartial`, such as an open `File`, which is copied into the child's stdin with `IO.copy_stream` instead of being read into a string first. For chains of commands, `Open3.pipeline`, `pipeline_r` and `pipeline_w` connect several programs with pipes, each given as an argument list, again without a shell. ## Status, not $? The capture methods hand you the status as a return value, and `popen3` exposes it as `wait_thr.value`. Use those. `$?` is thread-local and Open3 reaps the child from its own waiter thread, so `$?` in your thread is not the place to look. ## Summary - Short command, need stderr separately: `capture3`. - Need one ordered transcript: `capture2e`. - Large or live output: `popen2e` or `popen3`, reading every stream you open. - Always pass the command as separate arguments.

  • In Ruby, why can Open3.popen3 hang when you read stdout to the end before reading stderr?
    Each pipe has a fixed-size buffer. If the child writes enough to stderr to fill it, the child blocks until someone reads stderr, while your code blocks waiting for stdout to reach end-of-file. Neither side proceeds. Read both streams concurrently, use `capture3`, which does, or merge them with `popen2e`.
  • In Ruby, how do you send input to a command run with Open3.capture3?
    Pass `stdin_data:`: `Open3.capture3("sort", stdin_data: names.join("\n"))`. The option is removed before the call reaches `spawn`, its value is written to the child's standard input, and the pipe is then closed so the command sees end of input.
  • In Ruby, what does Open3.capture3 do when the executable does not exist?
    In the argument-list form, `Open3.capture3("no_such_tool", "x")`, no shell is involved, so the failed exec surfaces as `Errno::ENOENT` raised from the call instead of a status with a non-zero exit code. Rescue it if a missing tool must be reported gracefully.

saying these in an interview costs you the question

  • Open3.capture3 returns only stdout and an exit code Integer.
  • Reading popen3's stdout fully and then stderr is always safe.
  • capture2e returns stderr and stdout as two separate strings.
  • After Open3.capture3 you should read the result from $?.
  • capture3 streams output, so memory use stays constant for huge outputs.