Running External Commands with subprocess

March 13, 2026 · 6 min read ·Updated May 31, 2026 ·intermediate

pythonstdlibsubprocessprocesses

The subprocess module is your go-to tool for running external commands from Python. Whether you need to invoke a shell script, run a system utility, or orchestrate other programs, subprocess gives you the control you need. This guide covers practical patterns for everyday tasks.

When to Use subprocess

You should use subprocess when you need to:

Run system commands like git, docker, or curl
Execute shell scripts or batch files
Chain together command-line tools
Capture or pipe data between processes

Avoid subprocess when you can use Python built-ins. For file operations, use os or pathlib. For environment variables, use os.environ. Subprocess is for spawning new processes, not for in-process work.

Your first command

The simplest way to run an external command is with subprocess.run():

import subprocess

result = subprocess.run(["ls", "-la"])
print(result.returncode)

This runs ls -la and waits for it to finish. The return code tells you whether the command succeeded — a zero means success and any non-zero value signals an error. Using a list of arguments rather than a single string is the recommended pattern because it avoids shell interpretation of special characters and makes your code more portable across platforms.

Capturing Output

Most of the time, you need to see what the command produced. Use capture_output=True to grab both stdout and stderr:

import subprocess

result = subprocess.run(
    ["python", "--version"],
    capture_output=True,
    text=True
)

print(result.stdout)   # Python 3.12.0\n
print(result.stderr)   # (usually empty)
print(result.returncode)  # 0

The text=True argument decodes the bytes to strings automatically, using the system’s default encoding. Without it you get raw bytes, which is sometimes what you want for binary output like image data or compiled code. For most scripting tasks involving human-readable text, text=True combined with capture_output=True is the standard incantation that gives you clean string output with minimal boilerplate.

Running shell commands

Sometimes you need shell features like variable expansion, pipes, or glob patterns. Use shell=True for these cases:

import subprocess

result = subprocess.run(
    "echo $HOME && ls *.py 2>/dev/null | wc -l",
    shell=True,
    capture_output=True,
    text=True
)
print(result.stdout)

Be careful with shell=True combined with user-supplied input. Shell injection is a real security risk — an attacker who controls part of the command string can execute arbitrary commands on your system. If you must use shell mode with dynamic input, sanitise the values thoroughly or use shlex.quote() to escape them. When in doubt, stick with the list-of-arguments form, which avoids the shell entirely.

Handling Errors

Commands can fail. Use check=True to raise an exception when a command returns a non-zero exit code:

import subprocess

try:
    subprocess.run(
        ["git", "commit", "-m", "fix bug"],
        check=True,
        capture_output=True,
        text=True
    )
except subprocess.CalledProcessError as e:
    print(f"Command failed: {e.stderr}")

Using check=True is cleaner than manually inspecting the return code after every run() call. The exception object gives you access to the full command, exit code, and any output that was captured, so you can log the failure details without writing separate error-handling branches. For scripts that run multiple commands in sequence, failing fast with an exception is usually the right default.

Working with long-running processes

For commands that take a while, you might want to show progress or interact with them. Use Popen for full control:

import subprocess

proc = subprocess.Popen(
    ["tail", "-f", "/var/log/syslog"],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    text=True
)

try:
    for line in proc.stdout:
        print(line, end="")
except KeyboardInterrupt:
    proc.terminate()
    proc.wait()

The communicate() method is safer for sending input:

proc = subprocess.Popen(
    ["python"],
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    text=True
)

stdout, stderr = proc.communicate(input="print(hello from python)")
print(stdout)

The communicate() method handles both sending data to stdin and reading from stdout and stderr, then waits for the process to finish. This avoids deadlocks that can occur when you manually write to stdin and read from stdout on the same process, because the internal buffers might fill up and block. For interactive subprocess work where you need back-and-forth communication, communicate() is the safest single-exchange mechanism in the standard library.

Timeouts

Prevent commands from hanging forever by setting a timeout:

import subprocess

try:
    result = subprocess.run(
        ["sleep", "30"],
        timeout=5,
        capture_output=True,
        text=True
    )
except subprocess.TimeoutExpired:
    print("Command took too long and was killed")

Setting a timeout is a defensive measure that prevents a single slow command from stalling your entire script. When a timeout fires, the child process is killed automatically, so you do not need to clean up after it yourself. For commands that might legitimately take a long time, consider using a generous timeout combined with progress reporting rather than leaving the timeout unset.

Piping Data Between Commands

You can connect processes together using pipes:

import subprocess

# First process: generate data
proc1 = subprocess.Popen(
    ["echo", "line1\nline2\nline3"],
    stdout=subprocess.PIPE,
    text=True
)

# Second process: filter the data
proc2 = subprocess.Popen(
    ["grep", "line1"],
    stdin=proc1.stdout,
    stdout=subprocess.PIPE,
    text=True
)

proc1.stdout.close()  # Allow proc1 to receive SIGPIPE if proc2 exits
output = proc2.communicate()[0]
print(output)  # line1

Connecting processes with pipes recreates the Unix pipeline pattern inside Python. The key detail to remember is closing the parent’s copy of the upstream pipe after passing it to the downstream process — without that proc1.stdout.close() call, the first process won’t receive SIGPIPE if the second process exits early, and your script could hang. For simpler pipeline cases, the shell mode with a single string may be more readable, but the explicit Popen approach gives you finer control over error handling and intermediate processing.

Environment variables and working directory

Sometimes you need to control the environment or location:

import subprocess

result = subprocess.run(
    ["python", "-c", "import os; print(os.getcwd())"],
    cwd="/tmp",
    env={"PYTHONPATH": "/opt/mylib"},
    capture_output=True,
    text=True
)
print(result.stdout)  # /tmp

The cwd parameter changes the working directory for the child process without affecting the parent, which is useful for running commands that expect to operate in a specific location. The env parameter lets you control environment variables — when you pass a custom env dictionary, the child process sees only those variables and not the parent’s environment, so use it carefully if the command depends on inherited settings like PATH or HOME.

Best Practices

Always use a list of arguments rather than a shell string when possible. This avoids shell injection and is more portable:

# Good
subprocess.run(["ls", "-la"])

# Avoid unless you need shell features
subprocess.run("ls -la", shell=True)

Set capture_output=True instead of manually redirecting stdout and stderr. It is shorter and clearer.

Use text=True for human-readable output. It handles encoding for you.

Check out the shlex module if you need to parse shell-like strings into argument lists:

import shlex
import subprocess

cmd = echo Hello World
args = shlex.split(cmd)
subprocess.run(args)

Getting Started

The subprocess module handles most external command needs. Start with run() for simple cases, add error handling with check=True, and reach for Popen when you need streaming or complex process management.