Parallelism with multiprocessing

The Global Interpreter Lock (GIL) prevents Python from running multiple threads simultaneously in the same process. For CPU-bound tasks, this limits your code to a single CPU core. The multiprocessing module solves this by spawning separate processes, each with its own Python interpreter and memory space.

When to Use multiprocessing

Multiprocessing works best when your code is CPU-bound. These are tasks that spend most of their time doing calculations, not waiting for I/O:

• Image processing and video encoding
• Machine learning model training
• Scientific simulations
• Data processing pipelines

For I/O-bound tasks (network requests, file operations), threading or asyncio typically works better.

The multiprocessing Module

Python’s multiprocessing module provides a clean way to spawn and manage processes. The most common approach uses the Pool class, which manages a pool of worker processes for you.

Creating a Simple Pool

import multiprocessing
from multiprocessing import Pool

def square(x):
    return x ** 2

if __name__ == "__main__":
    with Pool(4) as pool:
        results = pool.map(square, range(10))
    print(results)
    # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

The Pool(4) creates a pool of 4 worker processes. The pool.map() function distributes the work across those processes and collects the results.

This pattern works identically to the built-in map() function, but executes in parallel.

Using Multiple Arguments

When your function needs multiple arguments, use starmap:

def power(base, exponent):
    return base ** exponent

if __name__ == "__main__":
    with Pool(4) as pool:
        # Pass tuples that get unpacked as arguments
        results = pool.starmap(power, [(2, 3), (3, 3), (4, 3)])
    print(results)
    # [8, 27, 64]

A Real Example: Processing Data

Suppose you have a list of numbers and need to apply an expensive computation to each:

import multiprocessing
import math

def fibonacci(n):
    """Calculate nth Fibonacci number recursively."""
    if n < 2:
        return n
    return fibonacci(n-1) + fibonacci(n-2)

def process_number(n):
    """Apply multiple operations to a number."""
    return {
        'input': n,
        'fib': fibonacci(min(n, 20)),  # Cap to avoid overflow
        'sqrt': math.sqrt(n),
        'log': math.log(n) if n > 0 else None
    }

if __name__ == "__main__":
    numbers = [10, 15, 20, 25, 30, 35, 40]
    
    with multiprocessing.Pool(4) as pool:
        results = pool.map(process_number, numbers)
    
    for r in results:
        print(f"Number: {r['input']}, Fib: {r['fib']}, Sqrt: {r['sqrt']:.2f}")

Running this on a 4-core machine will process all 7 numbers nearly 4 times faster than doing them sequentially.

Sharing Data Between Processes

Processes don’t share memory by default. Each process has its own copy of variables. If you need to share data, use multiprocessing.Value or multiprocessing.Array:

import multiprocessing

def worker(counter, lock):
    """Increment a shared counter."""
    for _ in range(1000):
        with lock:
            counter.value += 1

if __name__ == "__main__":
    # Create shared integer with a lock
    counter = multiprocessing.Value('i', 0)
    lock = multiprocessing.Lock()
    
    # Spawn 4 processes that all increment the same counter
    processes = [
        multiprocessing.Process(target=worker, args=(counter, lock))
        for _ in range(4)
    ]
    
    for p in processes:
        p.start()
    for p in processes:
        p.join()
    
    print(f"Final counter: {counter.value}")
    # Final counter: 4000

The lock ensures only one process modifies the counter at a time.

Process vs Pool

You have two main approaches:

Approach	Use When
Process directly	You need fine-grained control over each process
Pool	You have many similar tasks to run in parallel

Pool is simpler and handles the mechanics of process creation and cleanup for you. Most of the time, Pool is the right choice.

# Direct process — more control, more boilerplate
p = multiprocessing.Process(target=my_function, args=(arg,))
p.start()
p.join()

# Pool — simpler for parallelizable tasks
with Pool(4) as pool:
    results = pool.map(my_function, args)

Common Gotchas

The if name == “main” Guard

On Windows, you must wrap pool creation in this guard:

if __name__ == "__main__":
    with Pool(4) as pool:
        # Pool code here

Windows spawns new Python processes rather than forking, so it re-imports your module. Without the guard, you’ll create infinite processes.

Passing Objects

You can only pass objects that can be pickled. Most built-in types work fine. Custom classes need to be importable at the module level:

# This works
def my_func(x):
    return x * 2

# This may fail in some cases
class MyClass:
    def method(self, x):
        return x * 2

Too Many Processes

Don’t create more processes than CPU cores. With hyperthreading, aim for 2x cores at most:

# Good: match to available cores
cpu_count = multiprocessing.cpu_count()
pool = Pool(cpu_count)

Next Steps

Now that you understand multiprocessing, explore related topics:

• concurrent.futures.ProcessPoolExecutor — a higher-level API for process pools
• multiprocessing.Queue — for passing messages between processes
• asyncio — for I/O-bound concurrency in a single process

For CPU-heavy work where you need even more control, consider Cython or NumPy with vectorized operations.