Concurrency with threading
Python’s threading module lets you run multiple tasks at the same time. This is useful when your program needs to do several things simultaneously, like handling multiple user connections, downloading files in parallel, or keeping a UI responsive while doing background work.
In this tutorial, you’ll learn how to create threads, manage their execution, synchronize access to shared data, and understand when threading is the right tool for your problem.
Understanding Threads
A thread is the smallest unit of execution that a CPU can schedule. Think of it as a separate path of execution within your program—code that runs alongside the main program flow. Multiple threads share the same memory space, which makes communication between them easy but also introduces risks if you’re not careful.
Python’s Global Interpreter Lock (GIL) means only one thread executes Python bytecode at a time. This limits threading’s usefulness for CPU-bound tasks, but threads still help significantly for I/O-bound operations where your program waits for external resources.
Creating Threads
The threading module provides several ways to create threads. The most straightforward is to instantiate the Thread class with a target function:
import threading
import time
def download_file(filename, delay):
print(f"Starting download: {filename}")
time.sleep(delay)
print(f"Finished download: {filename}")
# Create threads
thread1 = threading.Thread(target=download_file, args=("data.csv", 2))
thread2 = threading.Thread(target=download_file, args=("image.jpg", 3))
# Start them
thread1.start()
thread2.start()
# Wait for both to complete
thread1.join()
thread2.join()
print("All downloads complete")This code creates two threads that download files concurrently. The start() method begins thread execution, and join() blocks until that thread finishes. Running this shows both downloads happening in parallel—the total time is roughly 3 seconds (the longer one) rather than 5 seconds if they ran sequentially.
Creating Threads with a Subclass
For more control, you can subclass Thread and override its run() method:
import threading
import time
class DownloadWorker(threading.Thread):
def __init__(self, filename, delay):
super().__init__()
self.filename = filename
self.delay = delay
def run(self):
print(f"Starting: {self.filename}")
time.sleep(self.delay)
print(f"Finished: {self.filename}")
# Use the subclass
workers = [
DownloadWorker("file1.txt", 1),
DownloadWorker("file2.txt", 2),
DownloadWorker("file3.txt", 3),
]
for worker in workers:
worker.start()
for worker in workers:
worker.join()
print("All work complete")This approach lets you add initialization logic in __init__ and encapsulate thread-specific data in the class.
Thread Synchronization with Locks
When multiple threads access shared data, you need to coordinate their access to prevent race conditions. A lock ensures only one thread can access a resource at a time:
import threading
class BankAccount:
def __init__(self, initial_balance=0):
self.balance = initial_balance
self.lock = threading.Lock()
def deposit(self, amount):
with self.lock:
new_balance = self.balance + amount
self.balance = new_balance
return new_balance
def withdraw(self, amount):
with self.lock:
if self.balance >= amount:
self.balance -= amount
return True
return False
def get_balance(self):
with self.lock:
return self.balance
account = BankAccount(100)
def worker(amount, is_deposit):
if is_deposit:
account.deposit(amount)
else:
account.withdraw(amount)
print(f"Balance after operation: {account.get_balance()}")
# Run concurrent operations
threads = []
for i in range(10):
t = threading.Thread(target=worker, args=(10, True))
threads.append(t)
t = threading.Thread(target=worker, args=(5, False))
threads.append(t)
for t in threads:
t.start()
for t in threads:
t.join()
print(f"Final balance: {account.get_balance()}")The with statement acquires and releases the lock automatically. Without this lock, concurrent deposits and withdrawals could interleave in ways that corrupt the balance.
Multiple Locks and Deadlocks
More complex programs often need multiple locks. This introduces the risk of deadlock—two threads each holding a lock the other needs:
# This demonstrates the problem (don't do this)
lock_a = threading.Lock()
lock_b = threading.Lock()
def thread_one():
with lock_a:
time.sleep(0.1) # Simulate work
with lock_b:
print("Thread one acquired both locks")
def thread_two():
with lock_b:
time.sleep(0.1)
with lock_a:
print("Thread two acquired both locks")
# This will likely deadlock
t1 = threading.Thread(target=thread_one)
t2 = threading.Thread(target=thread_two)
t1.start()
t2.start()To avoid deadlocks, always acquire multiple locks in the same order across all threads, or use threading.RLock() (reentrant lock) when a thread needs to re-acquire a lock it already holds.
The Threading Module’s Built-in Utilities
The threading module provides several other utilities for common patterns.
Event Objects
Events let threads signal each other:
import threading
import time
def waiter(event, name):
print(f"{name} waiting for event")
event.wait() # Block until event is set
print(f"{name} received the event!")
def setter(event):
print("Doing some work...")
time.sleep(2)
event.set() # Signal the waiting thread
event = threading.Event()
thread1 = threading.Thread(target=waiter, args=(event, "Worker"))
thread2 = threading.Thread(target=setter, args=(event,))
thread1.start()
thread2.start()
thread1.join()
thread2.join()Thread-Local Data
For data that should be unique to each thread, use threading.local():
import threading
import time
local_data = threading.local()
def process_request(request_id):
local_data.request_id = request_id
local_data.start_time = time.time()
# Do some work
time.sleep(0.1)
# Access thread-local data
duration = time.time() - local_data.start_time
print(f"Request {local_data.request_id} took {duration:.2f}s")
threads = [threading.Thread(target=process_request, args=(i,)) for i in range(5)]
for t in threads:
t.start()
for t in threads:
t.join()Each thread sees its own request_id and start_time—no synchronization needed.
When to Use Threading
Threading excels for I/O-bound tasks where your program spends time waiting: network requests, file operations, database queries, or user interface responsiveness. The GIL isn’t released during most I/O operations, but the operating system handles the waiting—your threads can proceed in parallel.
For CPU-bound tasks like mathematical computations or data processing, multiprocessing is usually better. Each process has its own Python interpreter and GIL, allowing true parallel execution across multiple CPU cores.
A good rule: if your code spends most of its time waiting for external resources, threading helps. If your code spends most of its time calculating, use multiprocessing.
When Not to Use Threading
Avoid threading when you need true parallel processing for CPU-intensive work. Avoid it for simple problems where the overhead of managing threads exceeds the benefit. And avoid sharing mutable data between threads without proper synchronization—the bugs are hard to reproduce and diagnose.
For many modern Python applications, asyncio provides a more efficient model for handling I/O concurrency, especially for network servers. Consider asyncio before threading for new projects involving many concurrent I/O operations.
Summary
Python’s threading module lets you run tasks concurrently. Create threads with threading.Thread(), use start() to begin execution and join() to wait for completion. When threads share data, use locks to prevent race conditions. Remember the GIL limits threading’s usefulness for CPU-bound work—for that, consider multiprocessing instead.
Threading remains valuable for I/O-bound tasks, background workers, and keeping applications responsive. The key is understanding when concurrency helps and when it adds unnecessary complexity.