skip to content

Spawning and Reaping Children

Where a child process goes wrong in production: an environment it never inherited, pipes that fill and deadlock, a kill that misses the grandchildren, and an exit status nobody collects.

part ofPythonoverview, primer and where to startread it →
on this pageshow

questions

15

How do you make subprocess.run raise on a failing command, and which exception carries the exit code?

level: juniorimportance: must knowfreq 60%

answer

  1. The default is not an exception
  2. One keyword flips it to raising
  3. A method on the result does the same
  4. The exception carries code, cmd, streams
  5. Missing executable is a different error

basics

~20 s

Pass check=True to subprocess.run, or call check_returncode() on the CompletedProcess it returns. Both raise subprocess.CalledProcessError, whose returncode attribute holds the child's exit status. Without check, a non-zero status comes back silently and is easy to ignore.

solid answer

~40 s

`subprocess.run` does not raise on a non-zero exit status by default: it blocks until the child finishes and returns a `subprocess.CompletedProcess` whose `returncode` attribute is whatever the program exited with, and it is on the caller to look. Passing `check=True` makes the call raise `subprocess.CalledProcessError` instead, and `CompletedProcess.check_returncode` performs the same test after the fact, which is what you want when you need to inspect the captured output before deciding whether the run failed. The exception object carries the `returncode`, the `cmd` that was run, and — when the streams were captured — `stdout` and `stderr`, so an error message can quote what the tool actually said. Keep that distinct from `FileNotFoundError`, which means the executable never started at all; that is a different failure from a program that ran and reported an error.

code

python · 8 lines
python
import subprocess, sys

done = subprocess.run([sys.executable, "-c", "raise SystemExit(3)"])
print(done.returncode)
try:
    done.check_returncode()
except subprocess.CalledProcessError as exc:
    print(type(exc).__name__, exc.returncode, exc.cmd[-1])

go deeper

for a junior

Recall that subprocess.run returns rather than raises by default, and that check=True is what turns a non-zero exit status into subprocess.CalledProcessError. Know that exit status 0 means success.

for a middle

Explain what the exception carries — returncode, cmd, and the captured streams — and why the output fields are None when the child inherited the parent's streams. Be ready to contrast check=True with calling check_returncode() on the result.

for a senior

Show the judgement about which tools should not be run under check=True because their exit codes carry meaning, and demonstrate wrapping the failure into an error message that quotes the child's own stderr instead of a bare status number.

for a principal

Own the convention: whether the team's shared subprocess helper raises by default, how command failures surface in logs and alerts, and how you stop the silent-success failure mode from being reintroduced in code review.

## The default is silence `subprocess.run(args)` starts a child process, blocks until it finishes, and hands back a `subprocess.CompletedProcess`. That object's `returncode` attribute is the child's exit status — the small integer every program returns to the operating system when it ends. Zero means success; any other value means the program is telling you it did not do the job. By default `run` does not interpret that number at all. If you call `subprocess.run(["gzip", "report.pdf"])` and the file is missing, gzip exits non-zero, `run` returns normally, and your code sails on as though the file had been compressed. This is the single most common subprocess bug in real Python code, and it is silent by design: the library reports the outcome, it does not judge it. ## Two ways to turn a status into an exception The first is the `check=True` keyword argument: `subprocess.run(args, check=True)` raises `subprocess.CalledProcessError` the moment the child exits non-zero. The second is `CompletedProcess.check_returncode`, a method on the returned object that runs the same test and raises the same exception when you call it. They exist for different shapes of code. `check=True` is right when any failure is fatal and you want the exception at the call site. `check_returncode()` is right when you want the result object first — to log the captured stderr, to look at the numeric code, to decide that this particular non-zero value is acceptable — and only then to fail. ```python done = subprocess.run(cmd, capture_output=True, text=True) if done.returncode == 2: log.warning("tool reported differences: %s", done.stdout) else: done.check_returncode() ``` That pattern matters because not every non-zero status is an error. Plenty of command-line tools use exit codes as a channel: a comparison tool that exits 1 for "the inputs differ", a linter that exits 1 for "findings", a search tool that exits 1 for "no matches". Blanket `check=True` around such a tool turns normal output into a crash. ## What the exception actually carries `subprocess.CalledProcessError` is not just a marker. It holds the `returncode`, the `cmd` sequence exactly as you passed it, and — only if the corresponding stream was captured — `output` (aliased as `stdout`) and `stderr`. If you ran the child with its streams inherited from the parent, the exception's output fields are `None`, because the bytes went straight to your terminal and the library never saw them. That is why a diagnostic wrapper normally captures output: an error that says "command failed with exit status 1" is much less useful than one that can quote the tool's own message. ```python try: subprocess.run(cmd, check=True, capture_output=True, text=True) except subprocess.CalledProcessError as exc: raise RuntimeError(f"{exc.cmd[0]} failed ({exc.returncode}): {exc.stderr.strip()}") from exc ``` ## "Ran and failed" versus "never ran" These are two different worlds and interviewers probe the difference. If the executable cannot be found or is not executable, the failure happens while the parent is trying to start the child, and you get a `FileNotFoundError` or `PermissionError` — ordinary `OSError` subclasses, raised whether or not you passed `check=True`. Only once the program has actually run and exited does an exit status exist at all, and only then can `subprocess.CalledProcessError` be raised. Code that catches `Exception` around the call and reports "the command failed" flattens a configuration problem and a genuine tool error into the same log line. ## The older wrappers `subprocess.check_call` and `subprocess.check_output` predate `run` and behave as their names suggest: both raise `subprocess.CalledProcessError` on a non-zero status, and `check_output` additionally captures and returns standard output (as bytes unless you ask for text). They are thin wrappers over the same machinery and remain supported, but `run` with `check=True` and `capture_output=True` covers both and gives you a result object with `stderr` on it as well. `subprocess.CalledProcessError` derives from `subprocess.SubprocessError`, which is a useful base class if you want to catch the module's own failures without swallowing `OSError`. ## The judgement to carry into production code Make the raising form the default in whatever helper your codebase wraps around `subprocess`, and require callers to opt out for the tools whose exit codes carry information. The failure mode of forgetting `check=True` is invisible — a pipeline that reports success while producing nothing — and it is discovered a long way from where it was introduced.

  • What is the difference between subprocess.CalledProcessError and FileNotFoundError coming out of the same subprocess.run call?
    FileNotFoundError is raised in the parent while the child is being started — the executable was not found or was not executable — so no exit status ever existed. subprocess.CalledProcessError means the program did start, ran, and exited with a non-zero status. One is a configuration or packaging problem, the other is the tool reporting an error, and collapsing them into one handler hides which of the two you are looking at.
  • How does subprocess.check_output relate to subprocess.run with check=True?
    check_output is the older wrapper: it captures standard output, raises subprocess.CalledProcessError on a non-zero status, and returns bytes unless text mode is requested. subprocess.run with check=True and capture_output=True does the same job and additionally gives you stderr and a result object you can inspect. check_output remains supported; new code usually reaches for run.
  • When would you deliberately avoid check=True?
    When the tool uses exit codes as information rather than as failure — a comparison tool that exits 1 for "inputs differ", a linter that exits 1 for "findings". There you want the CompletedProcess back, branch on the specific code you understand, and call check_returncode() only for the values you did not expect.

subprocess.run hands you a receipt rather than reading it out: check=True is asking the cashier to shout when the transaction was declined.

saying these in an interview costs you the question

  • Thinks subprocess.run raises automatically when the command fails
  • Judges success by whether stderr had any output
  • Treats a non-zero exit code and a missing executable as the same failure
  • Assumes CalledProcessError carries output even when streams were not captured
  • Believes a returncode of 0 needs checking for truthiness

context

open as a page

How do subprocess.Popen.terminate() and Popen.kill() differ on Unix?

level: juniorimportance: must knowfreq 62%

basics

~20 s

Popen.terminate() sends SIGTERM, which the child may catch and use to shut down cleanly. Popen.kill() sends SIGKILL, which no process can catch, block or ignore, so the child dies immediately with no cleanup. Both signal only the direct child.

open as a page

Why can Popen.wait() deadlock when the child's stdout is subprocess.PIPE?

level: middleimportance: must knowfreq 65%

basics

~20 s

A kernel pipe holds a fixed amount, roughly 64 KiB on Linux. Once the child fills it, the child blocks in write() while the parent blocks in Popen.wait, so neither moves. Popen.communicate drains both pipes and waits in one call.

open as a page

Does subprocess.run's env argument merge with the parent environment or replace it?

level: middleimportance: must knowfreq 55%

basics

~20 s

It replaces it. The mapping you pass becomes the child's entire environment, so anything you leave out - PATH, HOME, LANG - is simply not there. Merge on purpose with a dict such as {**os.environ, 'KEY': 'value'}.

open as a page

Why does Popen.terminate() leave a shell=True command's real program running?

level: middleimportance: must knowfreq 58%

basics

~20 s

With shell=True the direct child is /bin/sh, so Popen.pid is the shell's pid and terminate() signals the shell, not the program it started. Start the child with start_new_session=True and signal the whole group with os.killpg(os.getpgid(p.pid), signal.SIGTERM).

open as a page

Why is stdout=subprocess.DEVNULL safer than subprocess.PIPE for output you never read?

level: juniorimportance: should knowfreq 45%

basics

~20 s

subprocess.DEVNULL points the child at the null device, so its writes always succeed and vanish. subprocess.PIPE creates a kernel pipe with a fixed capacity; once it fills, the child blocks until the parent reads it.

open as a page

How does subprocess.run find the program when args[0] is a bare command name?

level: juniorimportance: should knowfreq 42%

basics

~20 s

With shell=False, a name containing no directory separator is looked up in the PATH directories in order and the first match runs. A name with a separator is a path, resolved against the child's working directory. shutil.which runs the same search.

open as a page

How does os.environ relate to the real process environment a child inherits?

level: middleimportance: should knowfreq 38%

basics

~20 s

os.environ is a dict-like snapshot taken when the os module is first imported. Writing to it also pushes the value into the C environment, so children spawned afterwards see it; os.putenv changes only the C side and leaves the mapping stale.

open as a page

Why is a subprocess.Popen child's returncode -9 rather than 137 when a signal ends it?

level: middleimportance: should knowfreq 52%

basics

~20 s

Python's subprocess module decodes the raw wait status itself: a non-negative value is the child's own exit status, and -N means signal N killed it, so SIGKILL reports -9. The 128+N form is a shell convention, not Python's.

open as a page

What do you use instead of Popen.communicate() when a child streams gigabytes to stdout?

level: seniorimportance: should knowfreq 38%

basics

~20 s

Stop capturing into memory. Point the child's stdout at an open file object, or keep subprocess.PIPE and drain it incrementally — a loop over Popen.stdout, or one reader thread per stream when the streams must stay separate.

open as a page

When Popen.communicate(timeout=...) raises subprocess.TimeoutExpired, what is still running and what must you do?

level: seniorimportance: should knowfreq 45%

basics

~20 s

The child is still running, unsignalled and still holding its pipes; the timeout ended only your wait. Kill it, then call Popen.communicate again to drain the pipes and reap it, or you leak a process and a zombie.

open as a page

Why is os.chdir the wrong way to point each subprocess.run child at its own directory in a threaded worker?

level: seniorimportance: should knowfreq 35%

basics

~20 s

The working directory is process-global, not per thread, so concurrent workers overwrite each other's value and a child can start in another job's directory. Pass cwd= to subprocess.run instead: that change of directory happens inside the child, once per spawn.

open as a page

A renderer service spawns a subprocess.Popen per job and never calls poll() or wait(); ps fills with <defunct> children. What is leaking, and how do you fix it?

level: seniorimportance: should knowfreq 38%

basics

~20 s

Nothing leaks memory: a finished child keeps a process-table entry and its PID until the parent collects its exit status. Calling poll(), wait() or communicate() on the subprocess.Popen object reaps it; a parent that never does eventually cannot spawn.

open as a page

Popen.terminate() sometimes fails to stop a translation-memory updater's export child; how would you build a shutdown that always ends it?

level: seniorimportance: should knowfreq 46%

basics

~20 s

Signal the child's whole process group with SIGTERM, wait a bounded grace period, then send SIGKILL to the same group and wait again. Start the child with start_new_session=True so that group exists and excludes your own process.

open as a page

What happens to subprocess.Popen children when the Python parent exits first?

level: seniorimportance: nice to knowfreq 24%

basics

~20 s

They keep running. A child whose parent dies is an orphan and is reparented by the kernel to PID 1, or to the nearest ancestor marked as a subreaper. Python never kills them for you, so cleanup has to be explicit.

open as a page