Running Shell Commands from Python

The subprocess module is how Python programs spawn new processes, run shell commands, and interact with the operating system. Whether you’re automating deployments, running git commands, or querying system information, subprocess is the tool for the job.

The Basics: Running a Simple Command

The simplest way to run a command is with subprocess.run(). It takes a list of arguments (similar to how you’d type them in a terminal) and waits for the command to finish:

import subprocess

result = subprocess.run(["ls", "-la"])
print(f"Exit code: {result.returncode}")

This runs ls -la and prints the directory listing. The returncode attribute tells you whether the command succeeded (0) or failed (non-zero).

The argument list is a sequence where the first element is the command and subsequent elements are its arguments. This is safer than passing a single string because Python handles escaping automatically.

Capturing Command Output

Most real-world use cases need to capture what the command outputs. Use capture_output=True to grab both stdout and stderr:

import subprocess

result = subprocess.run(
    ["echo", "Hello from subprocess"],
    capture_output=True,
    text=True
)

print(result.stdout)  # Hello from subprocess\n
print(result.stderr)  # (empty if no errors)
print(result.returncode)  # 0

The text=True parameter converts the output from bytes to strings, making it easier to work with. Without it, you’d get b'Hello from subprocess\n' (bytes) instead of a readable string.

Running Commands That Might Fail

Sometimes a command failing is expected behavior. Use check=True to raise an exception when the exit code is non-zero:

import subprocess

try:
    result = subprocess.run(
        ["ls", "/nonexistent-directory"],
        capture_output=True,
        text=True,
        check=True
    )
except subprocess.CalledProcessError as e:
    print(f"Command failed with exit code {e.returncode}")
    print(f"Error output: {e.stderr}")

This pattern is useful when you’re chaining commands together and need to stop the chain if something breaks.

Working with Shell Features

The shell=True option lets you use shell features like pipes, environment variables, and glob patterns:

import subprocess

# Using shell features like pipes
result = subprocess.run(
    "ls -la | grep '.py' | wc -l",
    shell=True,
    capture_output=True,
    text=True
)

print(result.stdout)  # Count of Python files

Be careful with shell=True and user input. If you’re passing any input from users through the command, you risk shell injection attacks. Use the list form (["ls", "-la"]) whenever possible.

Passing Input to Commands

Some commands need input. Use the input parameter to send data to stdin:

import subprocess

# Send input to a command (like typing at a prompt)
result = subprocess.run(
    ["python3", "-c", "print(input('Enter: ').upper())"],
    input="hello world",
    capture_output=True,
    text=True
)

print(result.stdout)  # HELLO WORLD

This is particularly useful for interactive CLI tools that you want to automate. For example, you could feed answers to a command-line questionnaire or provide a password to a program that prompts for one.

Setting Timeouts

Long-running commands can hang your program indefinitely. The timeout parameter prevents this:

import subprocess

try:
    result = subprocess.run(
        ["sleep", "10"],
        timeout=5,
        capture_output=True
    )
except subprocess.TimeoutExpired:
    print("Command took too long and was killed")

The timeout is measured in seconds. If the command doesn’t complete in time, Python kills the subprocess and raises TimeoutExpired. This is essential for any automation script that might encounter a stuck process.

Changing the Working Directory

The cwd parameter runs the command in a specific directory:

import subprocess

# Run git status in a specific project directory
result = subprocess.run(
    ["git", "status"],
    cwd="/home/user/my-project",
    capture_output=True,
    text=True
)

print(result.stdout)

This is useful when you’re working with multiple project directories and need to run commands in each one. It’s common in build scripts that operate on different parts of a monorepo.

Environment Variables

Pass custom environment variables with the env parameter:

import subprocess

# Run a command with custom environment
result = subprocess.run(
    ["python3", "-c", "import os; print(os.environ['MY_VAR'])"],
    env={"MY_VAR": "custom_value"},
    capture_output=True,
    text=True
)

print(result.stdout)  # custom_value

When you set env, the new process doesn’t inherit your current environment variables. If you need the current environment plus some extras, copy it first:

import os
import subprocess

env = os.environ.copy()
env["NEW_VAR"] = "value"
result = subprocess.run(["my_command"], env=env, ...)

This pattern is useful when you’re setting up isolated environments for different build stages or running tests with specific environment configurations.

Beyond run(): Using Popen for Streaming

subprocess.run() waits for the command to finish before returning. For interactive use or streaming output, use the Popen class directly:

import subprocess

process = subprocess.Popen(
    ["ping", "-c", "4", "localhost"],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    text=True
)

# Read output line by line as it comes
for line in process.stdout:
    print(f"Received: {line.rstrip()}")

process.wait()
print(f"Process exited with code {process.returncode}")

This lets you process output in real-time instead of waiting for the entire command to complete. It’s particularly useful for long-running commands like servers or watchers.

Common Pitfalls

A few things trip up developers new to subprocess:

The most common mistake is using shell=True with a list of arguments instead of a string. This doesn’t work as expected:

# WRONG - this passes the list as a single argument
subprocess.run(["echo", "Hello", "$HOME"], shell=True)

# RIGHT - use a string for shell=True
subprocess.run("echo Hello $HOME", shell=True)

Another issue is forgetting that stdout and stderr are captured separately. If you want both combined, use stderr=subprocess.STDOUT:

result = subprocess.run(
    ["ls", "-la", "/some/path"],
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,
    text=True
)

# Both stdout and stderr are in result.stdout now
print(result.stdout)

People also sometimes forget that run() returns immediately. If you need the process to finish before continuing, call process.wait() explicitly when using Popen.

When to Use Popen vs run()

Use subprocess.run() for most cases. It’s simpler and handles the common scenarios well.

Use Popen when you need:

• Real-time streaming of output (like building a progress bar)
• Independent control of stdin, stdout, and stderr
• Non-blocking reads or writes
• Complex inter-process communication

For example, Popen is often used in deployment scripts that need to show live output from a build process.

Security Considerations

The main security concern with subprocess is shell injection. If you ever pass user input to a command with shell=True, a malicious user could run arbitrary commands on your system.

Always validate and sanitize any input that goes into shell commands. When possible, use the list form of arguments instead of shell strings, as it avoids shell interpretation entirely.

Next Steps

You now know how to run commands, capture output, handle errors, and manage process behavior. These skills are the foundation for automating DevOps tasks like deployment scripts, build automation, and server management.

The next tutorial in this series covers working with files and directories in Python, which pairs well with subprocess for building complete automation scripts. You’ll learn how to read, write, and navigate the filesystem programmatically.