skip to content

In subprocess.run, how is the args parameter interpreted with shell=True versus the default, and what does it cost?

level: middleimportance: must knowfreq 68%

answer

  1. One keyword changes what args even means
  2. A list versus a string, and who parses it
  3. The extra process nobody asked for
  4. A list under shell=True on POSIX misbehaves quietly
  5. shlex has quote, split and join

basics

~20 s

With the default shell=False, args is a sequence and each element becomes exactly one argv entry, executed directly with no shell parsing. With shell=True on POSIX, args should be a single string handed to /bin/sh -c, so the shell re-parses it and metacharacters inside interpolated values take effect.

solid answer

~50 s

By default `subprocess` executes the program directly: `args` is a sequence whose first element is the program and whose remaining elements become argv entries verbatim, spaces, quotes, semicolons and all. Nothing expands globs, `~`, `$VAR`, pipes or redirection, because no shell is involved. `shell=True` changes the meaning of the parameter entirely: on POSIX the string is run as `/bin/sh -c <string>`, and if you pass a *list* under `shell=True` on POSIX only the first element is the command - the rest become the shell's own positional parameters, which is a classic silent bug. The cost of the shell is an extra process, shell-level exit codes and the fact that any value you interpolate is re-parsed as source code. `shlex.quote()` exists for the rare case where a shell string is genuinely required, and `shlex.split()`/`shlex.join()` convert between the two shapes; the structural answer, though, is to keep the argument vector.

code

python · 8 lines
python
import subprocess
import sys

value = "shard 7; echo surprise"
subprocess.run(
    [sys.executable, "-c", "import sys; print('argv[1]:', sys.argv[1])", value],
    check=True,
)

go deeper

for a junior

Learn the default shape and use it: a list of strings, one element per argument, no shell. Be able to say that spaces and semicolons inside an element are just characters when there is no shell parsing them.

for a middle

Explain what shell=True actually launches - a shell process running your string as a command - and what that re-enables (globbing, pipes, redirection, substitution) and re-introduces. Know that shlex.split parses a string into a list and shlex.quote does the reverse for one value.

for a senior

Argue the tradeoff at a real call site: the extra process, the shell's own exit codes, the harder teardown, and the fact that interpolated values become source text. Show how you replace the shell features you lose with glob, an open file for stdout, and chained Popen objects.

for a principal

Set the rule for the codebase: shell=False everywhere by default, a reviewed exception list, and a lint or wrapper that catches new shell=True call sites - plus a story for the legacy scripts whose command strings live in config files.

The `args` parameter of `subprocess.run()` and `subprocess.Popen` means two different things depending on one keyword, and knowing which is which is most of the module. ## shell=False, the default `args` is a sequence - conventionally a list of `str`. Element zero names the program and is looked up on the path unless it contains a directory separator; the remaining elements are passed to the new program as its argv entries, one element to one entry, byte for byte. Nothing in between reinterprets them. A value containing spaces stays one argument. A value containing `;`, `|`, `&&`, `$(...)`, `*` or a newline is still one argument, because there is no parser looking for those characters: the operating system's exec family takes an array. That is why the argument vector removes an entire class of bug structurally rather than filtering it - there is no grammar left for a value to escape into. The consequence people trip over is the other direction: you also lose everything the shell used to do for you. No `*.log` expansion, no `~`, no `$HOME`, no `cmd1 | cmd2`, no `> out.txt`, no `&&`. Each of those has a plain Python replacement - `glob.glob` or `pathlib.Path.glob` for wildcards, `os.environ` for variables, an open file object passed as `stdout=` for redirection, two `Popen` objects wired with `stdout=subprocess.PIPE` for a pipeline, and ordinary `if` statements for `&&`. ## shell=True Now `args` is expected to be a single string, and `subprocess` runs the platform shell with it: on POSIX that is `/bin/sh -c <string>` (a different shell can be named with `executable=`), and on Windows the interpreter named by `COMSPEC`. The string is parsed by that shell, so word splitting, quote removal, globbing, variable and command substitution, pipelines and redirection all work again. Two mechanics matter in an interview. First, mixing the shapes: on POSIX, `subprocess.run(["echo", "hello"], shell=True)` does *not* echo hello - the shell runs `echo` with an empty argument list and `"hello"` becomes the shell's `$0`. The command silently does the wrong thing rather than failing. Second, what happens to values: any string you format into that command line is no longer data, it is source text for the shell. A directory name containing a space becomes two words; a value containing `;` becomes a second command; a value containing backticks or `$(...)` becomes a substitution the shell executes. This is the whole reason the module's own documentation pushes so hard toward the list form. ## Where shlex fits `shlex` is the standard library's POSIX-shell lexer. `shlex.split(s)` turns a command *string* into a list the way `/bin/sh` would tokenise it, which is useful for parsing a fixed, trusted command out of a config file into an argument vector. `shlex.join(parts)` is the inverse and quotes each element. `shlex.quote(s)` returns a single value wrapped so a POSIX shell will treat it as one word - the tool for the genuinely unavoidable case where an interpolated value must live inside a shell string. Two caveats: `shlex.quote` models POSIX shell rules only and is not correct for `cmd.exe`, and quoting is a modelling exercise you can get subtly wrong, whereas the argument vector requires no model at all. ## The other costs of the shell Even with trusted input, `shell=True` buys you an extra process between you and the program you care about. The exit code you observe may be the shell's translation (for example 127 for a command it could not find) rather than the program's. Killing the child kills the shell, not necessarily what the shell started. Error reporting is worse: with `shell=False`, a missing program raises `FileNotFoundError` immediately and unambiguously; with `shell=True` you get a normal completion carrying a shell error message on stderr. And portability shrinks, because the string is now written in one platform's shell language. The practical rule: default to a list, reach for the shell only when you actually want shell behaviour, and when you do, keep every interpolated value out of the string - pass it as an argument to the shell command instead, or quote it deliberately with `shlex.quote`.

  • You need a pipeline of two programs. How do you do it without shell=True?
    Start the first with `stdout=subprocess.PIPE`, pass that object as the second one's `stdin=`, then close the first's `stdout` in the parent so it sees EOF when the writer goes away, and read from the second. It is more code than a shell one-liner, but every argument stays a separate list element and each stage's exit code is separately visible.
  • On POSIX, what actually happens if you pass a list while shell=True is set?
    Only the first element becomes the shell command string; the remaining elements are handed to the shell as its own positional parameters, so they are invisible to the command unless it references `$1`, `$2` and so on. The call usually runs and does the wrong thing rather than raising, which is what makes the mistake expensive.
  • When is shlex.quote the right tool rather than a workaround?
    When a shell string is genuinely required - you are generating a command for a remote shell, a crontab line, or a script file to be executed later - and there is no argument vector to hand across that boundary. Inside a single subprocess call there almost always is one, so quoting there is effort spent recreating a guarantee the list form already gives you for free. Note it models POSIX shells, not cmd.exe.

saying these in an interview costs you the question

  • Says shell=True is fine as long as you escape the input
  • Believes a list under shell=True on POSIX passes normal arguments
  • Thinks the list form still expands wildcards and $VAR
  • Uses shell=True merely to avoid splitting a command string
  • Assumes shlex.quote is correct for Windows cmd.exe

context

resolving links…