Subprocesses

The subprocess module lets Python launch and control other programs — capturing their output, feeding them input, checking their exit status, and doing it safely.

Real programs often need to run other programs: a compiler, a shell utility, a database client, another script. The subprocess module is the standard, safe way to do this — replacing older, less safe tools like os.system() — and understanding it means understanding processes, pipes, exit codes, and a critical security consideration around shells.

What subprocess Does and Why It Replaced os.system

subprocess launches a child process, connects to its standard input/output/error streams as needed, and reports its exit status back to your program. It supersedes older, cruder tools like os.system() (which just runs a shell command string and gives you back only an exit code, with weak control over I/O) and os.popen().

Two entry points cover almost everything: subprocess.run() for the common case of 'run this, wait for it to finish, get the result', and subprocess.Popen for finer-grained control (streaming output as it arrives, talking to a long-running process interactively).

subprocess.run: The High-Level Interface

subprocess.run(args) runs a command, waits for it to complete, and returns a CompletedProcess object with the exit code and any captured output.

args is a list of the program name followed by its arguments — each element is passed to the OS separately, with no shell involved to (mis)interpret quoting or special characters.

text=True (or the equivalent encoding=) decodes captured output as str; without it, stdout/stderr are raw bytes.

import subprocess

result = subprocess.run(
    ["ls", "-la", "/tmp"],
    capture_output=True,
    text=True,          # decode stdout/stderr as str instead of bytes
    timeout=5,
)

print("exit code:", result.returncode)
print("stdout:", result.stdout)
print("stderr:", result.stderr)

Checking Success: returncode and check=True

By convention, a subprocess exit code of 0 means success and any nonzero value means failure (the specific nonzero value is program-defined). subprocess.run does not raise on a nonzero exit code by default — you either inspect .returncode yourself or pass check=True to have it raise CalledProcessError automatically.

import subprocess

# Manual check
result = subprocess.run(["grep", "pattern", "file.txt"], capture_output=True, text=True)
if result.returncode != 0:
    print("grep found nothing or failed")

# Automatic check
try:
    subprocess.run(["false"], check=True)   # 'false' always exits with status 1
except subprocess.CalledProcessError as e:
    print(f"Command {e.cmd} failed with exit code {e.returncode}")

Capturing stdout and stderr

By default, a child process's stdout/stderr are inherited from the parent (so they print directly to your terminal). capture_output=True redirects both into the returned object instead; you can also route them independently or merge stderr into stdout.

import subprocess

# Capture stdout and stderr separately
result = subprocess.run(["python3", "-c", "import sys; print('out'); print('err', file=sys.stderr)"],
                         capture_output=True, text=True)
print(repr(result.stdout), repr(result.stderr))

# Merge stderr into stdout
result = subprocess.run(["python3", "-c", "print('a')"],
                         stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
print(result.stdout)

# Discard output entirely
subprocess.run(["python3", "-c", "print('ignored')"], stdout=subprocess.DEVNULL)

Passing Input

input= feeds bytes (or str, with text=True) to the child's stdin, then closes it — convenient for piping data into a command in one call.

You cannot use both input= and stdin= at the same time — input= implicitly sets stdin=PIPE and writes to it for you.

import subprocess

result = subprocess.run(
    ["python3", "-c", "import sys; print(sys.stdin.read().upper())"],
    input="hello from parent\n",
    capture_output=True,
    text=True,
)
print(result.stdout)   # "HELLO FROM PARENT\n"

Timeouts

timeout= bounds how long run() will wait; if the process hasn't finished in time, it's killed and subprocess.TimeoutExpired is raised.

On timeout, subprocess.run terminates the child process for you before raising — you do not need to kill it manually.

import subprocess

try:
    subprocess.run(["sleep", "10"], timeout=2)
except subprocess.TimeoutExpired as e:
    print(f"Command {e.cmd} timed out after {e.timeout} seconds")

Popen: Lower-Level, Streaming Control

Popen is what run() is built on. Use it directly when you need to interact with a process while it's still running — reading output line by line as it arrives, or writing to its stdin over time — rather than waiting for it to finish first.

-u forces the child's Python interpreter to run unbuffered, so lines appear as they're printed rather than only after the child's stdout buffer fills up.

import subprocess

with subprocess.Popen(
    ["python3", "-u", "-c", "import time\nfor i in range(3):\n print(i); time.sleep(1)"],
    stdout=subprocess.PIPE,
    text=True,
) as proc:
    for line in proc.stdout:            # stream output as it's produced
        print("got:", line.rstrip())
    proc.wait()                          # ensure the process has fully exited
    print("exit code:", proc.returncode)

Pipes Between Two Processes

You can connect one process's stdout directly to another's stdin, mirroring a shell pipeline like cmd1 | cmd2, without invoking an actual shell.

Closing p1.stdout in the parent after wiring it into p2 is important: otherwise the parent holds an extra reference to the pipe, and p1 never sees EOF if p2 exits early.

import subprocess

p1 = subprocess.Popen(["cat", "access.log"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["grep", "ERROR"], stdin=p1.stdout, stdout=subprocess.PIPE, text=True)
p1.stdout.close()   # allow p1 to receive SIGPIPE if p2 exits early
output, _ = p2.communicate()
print(output)

Shell Security Considerations

run()/Popen accept either a list of arguments (preferred: no shell involved) or, with shell=True, a single string executed through the system shell. shell=True is dangerous when any part of the command string comes from untrusted input, because shell metacharacters (;, |, &&, backticks, $()) let an attacker inject arbitrary commands.

In the list form, 'filename' is passed to cat as a single literal argument — there is no shell to interpret the semicolon as a separate command.

Rule of thumb: prefer the list form always. If you truly need shell features (globbing, pipelines, environment expansion), use shell=True only with a fully trusted, hardcoded command string — never with any value built from external or user input.

import subprocess

filename = "report.txt; rm -rf ~"   # attacker-controlled input

# DANGEROUS: shell=True interprets the ';' as a command separator
subprocess.run(f"cat {filename}", shell=True)   # NEVER do this with untrusted input

# SAFE: list form passes the whole string as one literal argument to cat
subprocess.run(["cat", filename])   # 'cat' just fails to find a file with that odd name

Environment and Working Directory

cwd= runs the child in a different working directory; env= replaces the child's entire environment (commonly {**os.environ, "MY_VAR": "value"} to extend rather than replace it).

import os
import subprocess

result = subprocess.run(
    ["python3", "-c", "import os; print(os.environ['MY_VAR'])"],
    cwd="/tmp",
    env={**os.environ, "MY_VAR": "hello"},
    capture_output=True,
    text=True,
)
print(result.stdout)   # "hello\n"

Practical Usage: A Robust Command Runner

import subprocess

def run_command(args: list[str], timeout: float = 30.0) -> str:
    """Run a command, return its stdout, raise a clear error on failure."""
    try:
        result = subprocess.run(
            args,
            capture_output=True,
            text=True,
            timeout=timeout,
            check=True,
        )
        return result.stdout
    except subprocess.CalledProcessError as e:
        raise RuntimeError(
            f"Command {args!r} failed (exit {e.returncode}): {e.stderr.strip()}"
        ) from e
    except subprocess.TimeoutExpired as e:
        raise RuntimeError(f"Command {args!r} timed out after {e.timeout}s") from e
    except FileNotFoundError as e:
        raise RuntimeError(f"Command not found: {args[0]!r}") from e

Edge Cases

FileNotFoundError is raised (not a nonzero exit code) if the program itself doesn't exist. Deadlocks can occur with Popen if you write a large amount to stdin and also try to read stdout using the pipe's file objects directly without communicate() — the child's output buffer can fill and block the child while your parent is blocked writing, a classic deadlock; communicate() avoids this by handling both streams concurrently. Zombie processes: always wait()/communicate() so the OS can reap the finished child.

import subprocess

try:
    subprocess.run(["definitely-not-a-real-command"])
except FileNotFoundError:
    print("The executable itself doesn't exist")

Key Takeaways

Prefer subprocess.run() for the common 'run and wait' case; use Popen for streaming or interactive control.

Pass arguments as a list, not a shell string, unless you deliberately need shell features and fully trust the command.

Never build a shell=True command string from untrusted input — it enables command injection.

check=True raises CalledProcessError on a nonzero exit code; without it, inspect .returncode yourself.

Always set a timeout on subprocess calls that could hang, and handle TimeoutExpired.

text=True decodes stdout/stderr as str; without it you get bytes.

user_input = get_untrusted_filename() subprocess.run(f"cat {user_input}", shell=True) user_input = get_untrusted_filename() subprocess.run(["cat", user_input]) shell=True with an interpolated string lets shell metacharacters in user_input execute arbitrary commands; the list form passes the value as one literal argument with no shell involved.

result = subprocess.run(["risky-tool"]) # assumes success, uses result.stdout without checking result = subprocess.run(["risky-tool"], check=True, capture_output=True, text=True) # CalledProcessError raised automatically on nonzero exit

Without check=True (or a manual returncode check), a failed command is silently treated as if it succeeded.

subprocess.run(["some-command"]) # no timeout; could hang forever subprocess.run(["some-command"], timeout=30)

A subprocess that hangs (waiting on input, stuck in a loop) will block your program indefinitely without a timeout.

os.system("rm " + filename) # legacy, no output capture, shell-based subprocess.run(["rm", filename], check=True) os.system always goes through the shell and gives only a raw exit status; subprocess.run is safer, more flexible, and the modern recommended tool.

Why does subprocess.run() with a list of arguments avoid shell injection risks that shell=True with a string does not?

With a list, each element is passed directly to the operating system as a distinct argument, with no shell parsing involved, so characters like ; | & $() in the data have no special meaning; shell=True hands the whole string to a shell, which interprets those characters as command syntax.

What is the difference between subprocess.run() and subprocess.Popen()?

run() is a high-level convenience wrapper that starts a process, waits for it to finish, and returns a CompletedProcess with the result; Popen is the lower-level class run() is built on, used when you need to interact with the process while it's still running (streaming output, writing to stdin over time) rather than waiting for it to finish first.

What happens by default if the child process exits with a nonzero status when using subprocess.run() without check=True?

Nothing is raised automatically — run() returns normally with the nonzero value in result.returncode, and it's the caller's responsibility to check it (or pass check=True to have CalledProcessError raised).

Why should you pass a timeout to subprocess.run(), and what happens if the timeout is exceeded?

Without a timeout, a hung or unresponsive child process blocks the parent indefinitely; if the timeout is exceeded, subprocess.run() kills the child process and raises subprocess.TimeoutExpired.

What exception is raised if the executable named in the command doesn't exist on the system?

FileNotFoundError, not a nonzero return code — the OS never manages to start the process at all.

When is it appropriate to use shell=True?

Only when you genuinely need shell features like pipelines, globbing, or environment variable expansion, and the entire command string is fully trusted/hardcoded — never when any part of the command is built from external or user-supplied input.

Write a function run_and_capture(args) that runs a command, returns its stdout as text on success, and returns None if the command fails for any reason (nonzero exit, missing executable, or timeout after 10 seconds), without letting an exception propagate.

import subprocess def run_and_capture(args: list[str]) -> str | None: try: result = subprocess.run( args, capture_output=True, text=True, timeout=10, check=True) return result.stdout except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError): return None check=True turns a nonzero exit into CalledProcessError; timeout=10 turns a hang into TimeoutExpired; FileNotFoundError covers a missing executable. Catching all three and returning None satisfies the 'never propagate' requirement.

What does this print?

3 — subprocess.run() without check=True returns normally regardless of exit status, storing whatever the child process exited with in result.returncode.

Given untrusted input directory (a string from a web form), write safe code to list its contents using subprocess, and explain why it's safe.

import subprocess def list_directory(directory: str) -> str: result = subprocess.run(["ls", "-la", directory], capture_output=True, text=True, check=True) return result.stdout

Using the list form (['ls', '-la', directory]) with no shell=True means 'directory' is passed to ls as one literal argument regardless of its contents — there is no shell to interpret embedded semicolons, pipes, or other metacharacters as separate commands.

import subprocess
result = subprocess.run(["python3", "-c", "exit(3)"])
print(result.returncode)