Concurrent File Downloads in Python

How long should downloading 50 small files take? At 200ms per request on a serial script, ten seconds, with the network sitting idle for most of it. Switching to concurrent file downloads collapses the same job to a couple of seconds, and the standard library plus a thin third-party HTTP layer gets you there in well under 100 lines of code.

This guide walks through four working examples, simplest first, and finishes with the production touches you will want before pointing any of them at the open internet. The whole thing runs on Python 3.10+ and uses only urllib, concurrent.futures, asyncio, aiohttp, and httpx. Pick the one that matches your stack.

TL;DR

• For 5 to 50 files, use concurrent.futures.ThreadPoolExecutor with urllib.request or requests. It is the shortest path to a working concurrent file downloader.
• For hundreds of concurrent downloads, HTTP/2, or an already-async program, use asyncio with aiohttp or httpx.AsyncClient, plus an asyncio.Semaphore to cap concurrency.
• On every layer, pass an explicit timeout, call raise_for_status() before writing the body, and create one ClientSession per program.
• Skip multiprocessing for concurrent file downloads. Downloads are I/O-bound, so threads or asyncio are the right tools, not processes.

Which concurrency model should you pick?

Python ships three options for doing work in parallel. Most concurrent file download scripts should reach for the first.

Threads via concurrent.futures.ThreadPoolExecutor are the right tool for I/O-bound work. Socket reads release the GIL, so 4 to 32 worker threads can keep dozens of downloads in flight at once. Reach for this when the HTTP layer is blocking, like requests or urllib.request, and when the rest of the program is also synchronous. The implementation is short, the failure modes are familiar, and there is no event loop to reason about.

asyncio combined with an async HTTP client such as aiohttp or httpx is the right tool when you need hundreds of concurrent downloads, when the rest of the program is already async, or when you want HTTP/2 multiplexing. One thread, one event loop, no thread overhead per request. The trade-off is a more constrained style: no blocking calls inside the loop, careful cancellation, and a different way to bound concurrency than max_workers.

multiprocessing and ProcessPoolExecutor are the wrong tool for downloads. Downloads wait on sockets, not on the CPU, so process startup cost dominates and you gain nothing from extra cores. If you find yourself reaching for them here, stop and use threads instead.

If you are not sure which to pick, start with ThreadPoolExecutor. It is the shortest path to a working downloader, and you can switch to asyncio later if the workload justifies the rewrite.

For a quick mental model, here is the shortest possible concurrent file downloader for two URLs, about ten lines with no third-party imports:

import urllib.request
from pathlib import Path

URLS = ["https://www.example.com/", "https://www.iana.org/"]
DEST = Path("downloads")
DEST.mkdir(exist_ok=True)

for url in URLS:
    name = Path(urllib.parse.urlparse(url).path).name or "index.html"
    urllib.request.urlretrieve(url, DEST / name)

This serial version is the baseline. Every example below replaces the for loop with a pool that runs several requests at once, and the rest of the guide walks through why each design choice matters when you are building concurrent file downloads in Python.

What does the simplest parallel downloader look like?

The pure-stdlib version is the right place to start. It runs on any Python install with no pip install step.

# downloader_threaded.py
import concurrent.futures
import urllib.request
from pathlib import Path
from urllib.parse import urlparse

URLS = [
    "https://www.example.com/",
    "https://www.iana.org/",
]

DEST = Path("downloads")
DEST.mkdir(exist_ok=True)


def fetch(url: str, timeout: float = 10.0) -> tuple[str, int]:
    req = urllib.request.Request(url, headers={"User-Agent": "pyguides/1.0"})
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        name = Path(urlparse(url).path).name or "index.html"
        with (DEST / name).open("wb") as f:
            f.write(resp.read())
    return name, (DEST / name).stat().st_size


def main() -> None:
    with concurrent.futures.ThreadPoolExecutor(max_workers=4) as ex:
        futures = {ex.submit(fetch, u): u for u in URLS}
        for fut in concurrent.futures.as_completed(futures):
            url = futures[fut]
            try:
                name, size = fut.result()
            except Exception as exc:
                print(f"FAILED {url}: {exc!r}")
            else:
                print(f"OK     {url} -> {name} ({size} bytes)")


if __name__ == "__main__":
    main()

Running this script prints one summary line per file. Sizes shift a little between runs because example.com and iana.org serve slightly different markup depending on the request, but the shape stays the same: one OK line per URL, with whichever response happens to finish first reported first:

# output (approximate; sizes depend on the live page):
OK     https://www.iana.org/ -> index.html (5083 bytes)
OK     https://www.example.com/ -> index.html (1256 bytes)

The pattern is submit a callable per URL, then walk the futures in completion order with as_completed. Two details matter:

• Use urllib.parse.urlparse to get the filename, not url.split("/")[-1]. URLs can end in /, contain query strings, or be empty. urlparse(url).path gives you the path component, and Path(...).name extracts the last segment safely.
• Pass a User-Agent header. Some hosts reject bare urllib requests or return different content to unknown agents, which makes a “working” script mysteriously inconsistent across hosts.

For a one-shot script this is enough. The next examples show what to change when you need to scale up, switch to async, or add structured error handling.

How do you log progress and isolate failures with submit and as_completed?

executor.map(URLS, fetch) is shorter, but it has two rough edges. It returns results in input order, not completion order, and a single exception aborts iteration once you reach the failing item. For real concurrent file download lists, prefer submit plus as_completed so you can log progress and isolate failures.

The previous example already uses that pattern. The trick is the try / except around fut.result(): one bad URL becomes a logged failure instead of a hard crash. as_completed keeps yielding the rest of the futures in completion order, so the loop finishes even after several failures.

If you want a progress line per file, count completions:

with concurrent.futures.ThreadPoolExecutor(max_workers=8) as ex:
    futures = {ex.submit(fetch, u): u for u in URLS}
    completed = 0
    for fut in concurrent.futures.as_completed(futures):
        completed += 1
        url = futures[fut]
        try:
            name, size = fut.result()
        except Exception as exc:
            print(f"[{completed}/{len(URLS)}] FAILED {url}: {exc!r}")
        else:
            print(f"[{completed}/{len(URLS)}] OK     {url} -> {name} ({size} bytes)")

That is usually enough for a CLI tool. For a service with structured logs, swap print for logging.getLogger(__name__).info(...). The logging best practices guide walks through the standard dictConfig setup.

How do you switch from threads to asyncio for hundreds of overlapping downloads?

Threads are great up to a point. Past a few hundred concurrent downloads, the thread pool overhead, the file descriptor limit, and the connection pool start to bite. asyncio handles the same load with one thread and a single connection pool.

The pattern is three pieces: a ClientSession shared across all downloads, an asyncio.Semaphore to cap concurrency, and a coroutine that fetches one URL. The session owns a connection pool, so creating a new one per request is the most common aiohttp performance bug. Make one at the top of main, then pass it down.

# downloader_aiohttp.py
# pip install aiohttp
import asyncio
import aiohttp
from pathlib import Path
from urllib.parse import urlparse

URLS = [
    "https://www.example.com/",
    "https://www.iana.org/",
]

DEST = Path("downloads")
DEST.mkdir(exist_ok=True)
MAX_CONCURRENT = 4
TIMEOUT = aiohttp.ClientTimeout(total=15)


async def download(
    session: aiohttp.ClientSession,
    sem: asyncio.Semaphore,
    url: str,
) -> tuple[str, int]:
    async with sem:
        async with session.get(url, timeout=TIMEOUT) as resp:
            resp.raise_for_status()
            name = Path(urlparse(url).path).name or "index.html"
            with (DEST / name).open("wb") as f:
                async for chunk in resp.content.iter_chunked(8192):
                    f.write(chunk)
            return name, (DEST / name).stat().st_size


async def main() -> None:
    sem = asyncio.Semaphore(MAX_CONCURRENT)
    async with aiohttp.ClientSession(
        headers={"User-Agent": "pyguides/1.0"}
    ) as session:
        results = await asyncio.gather(
            *(download(session, sem, u) for u in URLS),
            return_exceptions=True,
        )
    for r in results:
        if isinstance(r, Exception):
            print(f"FAILED: {r!r}")
        else:
            name, n = r
            print(f"OK     {name} ({n} bytes)")


if __name__ == "__main__":
    asyncio.run(main())

Running the async version prints the same kind of output. The order between example.com and iana.org depends on which server returns first, and the byte counts shift slightly with each request, but the script always reports one line per file and never blocks the event loop on a single slow host:

# output:
OK     index.html (1256 bytes)
OK     index.html (5083 bytes)

Three things to notice:

• iter_chunked(8192) streams the body in 8KB pieces. Without it, resp.content is a StreamReader and a naive await resp.read() can buffer the whole file in memory. For large files this is the difference between working and OOM.
• resp.raise_for_status() runs before the file is written. ClientResponse does not raise on 4xx/5xx by default, so without this check you get empty files with no error, which is a worse failure mode than a loud exception.
• asyncio.gather(..., return_exceptions=True) collects failures as values instead of cancelling the rest. Drop the flag and the first failure tears down every sibling coroutine, which is rarely what you want for a downloader.

For a deeper walkthrough of aiohttp.ClientSession and timeouts, see the aiohttp client tutorial. The fundamentals of asyncio.gather and asyncio.Semaphore are covered in the async patterns tutorial. The full aiohttp client API lives in the upstream docs.

How does httpx compare to aiohttp?

httpx is the newer alternative to aiohttp for async HTTP. It has a cleaner API, optional HTTP/2 support, and a sync interface that matches requests if you ever need to drop back to blocking code. The streaming version uses client.stream(...) as an async context manager.

# downloader_httpx.py
# pip install httpx
import asyncio
import httpx
from pathlib import Path
from urllib.parse import urlparse

URLS = [
    "https://www.example.com/",
    "https://www.iana.org/",
]

DEST = Path("downloads")
DEST.mkdir(exist_ok=True)
MAX_CONCURRENT = 4


async def download(
    client: httpx.AsyncClient,
    sem: asyncio.Semaphore,
    url: str,
) -> tuple[str, int]:
    async with sem:
        async with client.stream("GET", url, follow_redirects=True) as resp:
            resp.raise_for_status()
            name = Path(urlparse(url).path).name or "index.html"
            with (DEST / name).open("wb") as f:
                async for chunk in resp.aiter_bytes():
                    f.write(chunk)
            return name, (DEST / name).stat().st_size


async def main() -> None:
    sem = asyncio.Semaphore(MAX_CONCURRENT)
    async with httpx.AsyncClient(
        timeout=15.0,
        follow_redirects=True,
        headers={"User-Agent": "pyguides/1.0"},
    ) as client:
        results = await asyncio.gather(
            *(download(client, sem, u) for u in URLS),
            return_exceptions=True,
        )
    for r in results:
        if isinstance(r, Exception):
            print(f"FAILED: {r!r}")
        else:
            name, n = r
            print(f"OK     {name} ({n} bytes)")


if __name__ == "__main__":
    asyncio.run(main())

resp.aiter_bytes() with no argument streams whatever chunk size the server uses, which is usually a few kilobytes. For HTTP/2 multiplexing, install the extra dependencies and flip the flag. The [http2] extra pulls in the h2 package that the multiplexing implementation is built on:

pip install 'httpx[http2]' h2

Once the extras are installed, httpx.AsyncClient accepts an http2=True argument that enables the multiplexed transport. The rest of the script is unchanged, because the streaming surface in client.stream(...) and the aiter_bytes() iteration work the same way over HTTP/1.1 and HTTP/2:

async with httpx.AsyncClient(http2=True, timeout=15.0) as client:
    ...

For many small concurrent file downloads from the same host, HTTP/2 multiplexing lets several streams share a single TCP connection. For mixed hosts or large files, the win is smaller. The httpx async docs cover the streaming surface in depth, and the httpx guide covers the full client surface.

What production touches turn a demo into a real script?

The examples above run against a couple of well-behaved hosts and will succeed. Real download lists hit redirects, 429 rate limits, flaky networks, and content the server has not finished uploading. The list below covers the gaps between a demo and a script you would actually run on a cron.

• Timeouts on every layer. urllib.request.urlopen, requests.get, aiohttp.ClientSession, and httpx.AsyncClient all accept a timeout argument. Without one, a single slow host with a half-open connection stalls the whole pool until the kernel TCP timeout kicks in. A 10 to 30 second total timeout is a reasonable starting point.
• Retries on transient failures. A 3-attempt loop with exponential backoff handles most 5xx, 429, and connection-reset cases. For something more polished, tenacity is the de facto choice. Skip the retry on 4xx other than 408, 425, 429; those are usually permanent.
• Skip files you already have. A HEAD request, or a check against Content-Length on disk, avoids re-downloading gigabytes because the script ran twice. Combine with a hash check via hashlib.sha256 for the paranoid case.
• Filename safety. Path(urlparse(url).path).name covers most cases, but the result can still be empty, contain a .., or be a reserved name on Windows. Fall back to a counter, hash, or sanitized slug if the parsed name is empty.
• Logging, not print. A logging.getLogger(__name__) with a dictConfig setup gives you levels, structured fields, and easy redirection to a file. See the logging best practices guide.
• Bounded concurrency, even for threads. The OS file-descriptor limit and the host’s connection cap are real. A semaphore-style cap inside the worker, or just a conservative max_workers=8 to 16, is usually enough.
• Cancellation cleanup. ThreadPoolExecutor.__exit__ waits for running workers on KeyboardInterrupt. For a fast exit, call executor.shutdown(wait=False, cancel_futures=True) (Python 3.9+) instead of relying on the context manager.

What mistakes come up most in concurrent file downloads?

A few patterns come up over and over in bug reports. They are easy to avoid once you know the trap.

• Creating one ClientSession per request. Defeats keep-alive, costs a TCP handshake per request, and is the top performance bug in aiohttp and httpx code. Make the session at the top of main and pass it down.
• Reaching for ProcessPoolExecutor. Downloads are I/O-bound. The process startup cost dominates for any reasonable file list, and you gain nothing from the extra cores.
• Skipping raise_for_status(). aiohttp and httpx do not raise on 4xx/5xx. Without the explicit check, an HTTP 404 gives you a zero-byte file and a successful print, which is a worse failure mode than a loud error.
• No timeout. A single slow host stalls the whole pool. Pass timeout= to every blocking call.
• url.split("/")[-1] for filenames. Empty paths, query strings, and trailing slashes all break it. Use urlparse(url).path and check the result.
• asyncio.gather with return_exceptions=False. Cancels every other download the moment one URL fails. For a downloader you almost always want return_exceptions=True and post-hoc error handling.

Frequently asked questions

How many concurrent downloads should I run at once?

Start with max_workers=8 for ThreadPoolExecutor or MAX_CONCURRENT=8 for an asyncio.Semaphore. Beyond ~32 workers you start hitting the OS file-descriptor limit and the per-host connection cap. If you have many URLs from the same host, prefer MAX_CONCURRENT=10–20 plus HTTP/2 over a larger thread pool.

Do I need to handle SSL certificate verification?

Yes, leave it on. urllib, requests, aiohttp, and httpx all verify certificates by default, and disabling verification (verify=False) is a security hole. If you sit behind a corporate proxy, pass the CA bundle path explicitly (verify='/path/to/ca-bundle.pem').

What happens if a download is interrupted partway through?

The partially written file stays on disk. Add a .part suffix while the download is in flight and rename it to the final filename only after the body has flushed and resp.raise_for_status() has passed. For a quick-and-dirty fix, just delete the file in the except branch and let the next run re-download it.

Should I use aiohttp or httpx for a new project?

For a new project with no existing dependency, httpx is the cleaner default. It has a sync mode that matches requests, optional HTTP/2 support, and a smaller API surface. Reach for aiohttp if you need its mature middleware ecosystem or if you are already invested in its session and connector model.

Conclusion

A working concurrent file downloader is short. The first example, ThreadPoolExecutor plus urllib.request, is enough for most scripts. Reach for asyncio with aiohttp or httpx when you have hundreds of concurrent downloads in flight or the rest of your program is already async. Either way, add timeouts, retries, bounded concurrency, and a ClientSession you create once per program before you point the script at the open internet.

The interesting design questions are not which library to use, but how to cap concurrency, how to name output files safely, and how to surface progress. Those are the pieces that turn a working concurrent downloader into one you would actually run as a service, and they are the same questions you will face the next time you need to build concurrent file downloads for a new workload.

See also

• concurrent.futures module reference
• asyncio module reference
• threading module reference