Task Groups and Exception Handling
Prerequisites
This is the fifth entry in a Python asyncio series. Earlier parts covered the event loop, coroutines and tasks, and the lower-level fan-out primitives asyncio.gather and asyncio.wait. asyncio.TaskGroup is the wrapper that sits on top of those: it ties the lifecycle of every spawned task to a single async with block, so cleanup is guaranteed by structure rather than by manual try / finally choreography.
Before continuing, you should be comfortable with the basics: what a coroutine is, what a Task is, how the event loop schedules them, and how await suspends a coroutine. If you are new to async Python, the gather and wait guide is a worthwhile prerequisite — the examples below assume you already know those fundamentals. You will also need Python 3.11 or later, since TaskGroup, ExceptionGroup, and except* are all new in that release.
Why TaskGroup? The orphan-task problem
Run three coroutines concurrently and one of them throws. What happens to the other two? With the old asyncio.gather(*coros) pattern, the first exception propagates and the remaining tasks are cancelled — but gather does not actually wait for those cancellations to finish. If your tasks own resources (HTTP sessions, locks, file handles), they can be left in a half-closed state. Worse, code after the gather call resumes while sibling tasks may still be winding down.
asyncio.TaskGroup (added in Python 3.11) is the structured-concurrency answer. Inside an async with TaskGroup(): block, all spawned tasks live and die together. If any task raises, the group cancels the rest and waits for them to actually finish before re-raising a combined ExceptionGroup. Nothing escapes the block while it is still doing cleanup.
The trade-off is that you cannot get “all results, no matter what” the way gather(return_exceptions=True) gives you. TaskGroup always treats failure as a hard error. That is usually what you want, but it is worth knowing.
Your first TaskGroup
A TaskGroup is an async context manager. You enter it, schedule coroutines with tg.create_task(coro), and the context manager’s __aexit__ waits for every task to complete before letting your code move on.
import asyncio
async def fetch(name: str) -> str:
await asyncio.sleep(0.1)
return f"data from {name}"
async def main() -> None:
async with asyncio.TaskGroup() as tg:
t1 = tg.create_task(fetch("a"))
t2 = tg.create_task(fetch("b"))
t3 = tg.create_task(fetch("c"))
# All three are done here.
print(t1.result(), t2.result(), t3.result())
asyncio.run(main())
# output: data from a data from b data from cA few details worth knowing:
- The constructor takes no arguments. The whole API is
enter/create_task/exit. create_task(coro, *, name=None, context=None)mirrorsasyncio.create_task. Passname=and the task shows up nicely in tracebacks andasyncio.all_tasks(). Always do this when you have more than a couple of children.- The group holds strong references to its tasks, unlike a bare
asyncio.create_task()call. You do not need to keep a separate list to avoid garbage collection. - You can add tasks dynamically from inside a coroutine that has the
tgreference. The pattern is useful for fan-out work:
async def spawn(tg: asyncio.TaskGroup, n: int) -> None:
for i in range(n):
tg.create_task(fetch(f"item-{i}"))
async def main() -> None:
async with asyncio.TaskGroup() as tg:
tg.create_task(spawn(tg, 3))
# All three children of spawn are also done.Reading results from tasks
Hold onto the task object before the async with block exits. After exit, the task is finished and task.result() returns its value (or raises, if it failed). If the group raised, sibling tasks are in the CANCELLED state, so calling task.result() on a cancelled task raises CancelledError — check task.cancelled() first if you are walking a mixed batch.
For partial-success cases, a done callback is the cleanest way to collect whatever succeeded:
async def might_fail(kind: str) -> str:
await asyncio.sleep(0.05)
if kind == "boom":
raise ValueError(f"bad value: {kind}")
if kind == "crash":
raise KeyError(kind)
return kind
async def main() -> tuple[list[str], list[Exception]]:
results: list[str] = []
errors: list[Exception] = []
try:
async with asyncio.TaskGroup() as tg:
for kind in ("a", "boom", "b", "crash", "c"):
t = tg.create_task(might_fail(kind))
t.add_done_callback(lambda fut: (
results.append(fut.result())
if fut.exception() is None
else errors.append(fut.exception())
))
except* (ValueError, KeyError):
pass
return results, errors
print(asyncio.run(main()))
# output: (['a', 'b', 'c'], [ValueError("bad value: boom"), KeyError('crash')])The order of done callbacks is not strictly the launch order when a failure cancels siblings, so do not rely on it. The contents of results and errors are correct; the order is not guaranteed.
When a task fails — the ExceptionGroup
The first time you write a TaskGroup that fails in production, the traceback will look weird. Instead of ValueError: bad value, you will see something like ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) followed by a ---------- separator and the original ValueError. That is the point.
ExceptionGroup (and its parent BaseExceptionGroup) are new exception types added in 3.11. The constructor is ExceptionGroup(message, exceptions) where exceptions is a sequence of BaseException instances. The grouped exceptions live in eg.exceptions as a tuple, and groups can nest: one of the leaves can itself be another group.
The crucial design fact: BaseExceptionGroup extends BaseException, not Exception. That means a plain except Exception: clause will catch an ExceptionGroup but not a BaseExceptionGroup. If you upgrade a library and suddenly see uncaught errors at the top of your program, this is almost always why.
Splitting a Group With eg.split()
eg.split(cond) returns a (matching, rest) tuple. cond is a type or tuple of types. One side can be None if no exceptions matched.
try:
async with asyncio.TaskGroup() as tg:
tg.create_task(might_fail("boom"))
tg.create_task(might_fail("crash"))
except BaseExceptionGroup as eg:
value_errors, rest = eg.split(ValueError)
if value_errors is not None:
print("handle value errors:", value_errors.exceptions)
if rest is not None:
raise rest
# output: handle value errors: (ValueError("bad value: boom"),)Re-raising rest keeps the original group structure for any upstream handler. eg.subset(cond) and eg.derive(excs) exist for finer work — subset returns a new group containing only matching leaves, derive returns a copy with a replaced exceptions tuple.
Handling groups with except* (Python 3.11+)
PEP 654 introduced except* syntax that does the splitting for you. A single try may mix regular except and except* clauses; at least one is required. Each except* clause runs independently and only matches the leaves it can handle.
try:
async with asyncio.TaskGroup() as tg:
tg.create_task(might_fail("boom"))
tg.create_task(might_fail("crash"))
tg.create_task(might_fail("ok"))
except* ValueError as eg:
print("got value errors:", [type(e).__name__ for e in eg.exceptions])
except* KeyError as eg:
print("got key errors:", [type(e).__name__ for e in eg.exceptions])
# output (line order may vary):
# got value errors: ['ValueError']
# got key errors: ['KeyError']Notes that matter in practice:
except*accepts a tuple of types:except* (TypeError, KeyError):.- An
elseclause works, and afinallyclause also works. except*does not replace aCancelledError— that case is special-cased (see below).
Gotchas that bite in practice
CancelledError is not wrapped. If a child task is cancelled and that is the only failure, the parent receives a CancelledError directly, not wrapped in an ExceptionGroup. Task groups only wrap “exception other than CancelledError”. Similarly, KeyboardInterrupt and SystemExit are re-raised bare; they are not grouped.
Do not swallow CancelledError in a child task. If a task catches asyncio.CancelledError and never re-raises, the group’s wait-for-cancellation can hang. The docs are explicit about this. The general rule: do cleanup, then re-raise.
except Exception misses BaseExceptionGroup. This is the single most common upgrade surprise. Use except BaseExceptionGroup or except* Exception for the broader catch.
gather(return_exceptions=True) is not a substitute. It returns exceptions as values in a result list and keeps running normally. TaskGroup cancels siblings and raises. Pick the one that matches your intent — “fan out and get all results, no matter what” still belongs to gather.
Cannot use TaskGroup as a one-off outside async with. There is no public start() / join() API. If you really need manual lifecycle, you are back to create_task() plus gather().
Naming tasks helps debugging. tg.create_task(coro, name=f"fetch-{url}") shows up in tracebacks. This is the cheapest improvement you can make to any group with more than two or three children.
Backport caveat. The exceptiongroup PyPI package restores the runtime classes on Python 3.10, but it does not enable the except* syntax — that is parsed by the compiler and requires 3.11+. Most projects just require 3.11+ for except*.
Conclusion
asyncio.TaskGroup plus ExceptionGroup plus except* is the modern way to run a batch of coroutines that should succeed or fail as a unit. It gives you the strongest guarantee asyncio offers: no orphaned tasks, no silently stranded resources, no surprise half-cleanups. The cost is a new exception model to learn, and the one-trick pony of always treating the first failure as fatal.
A reasonable default in new code: use TaskGroup for any place you would have reached for asyncio.gather(*aws) and you actually want the all-or-nothing semantics. Reach for gather(return_exceptions=True) only when you specifically need the “collect whatever happened” pattern.
Next steps
After TaskGroup, the natural next step is putting it to work on real I/O. The async file I/O guide shows how to fan out chunked reads across a TaskGroup, and the aiohttp client guide builds a small parallel HTTP fetcher that uses asyncio.TaskGroup as its main concurrency primitive. For the lower-level surface area, the asyncio module reference and the asyncio.Task reference list every method that TaskGroup ultimately delegates to. If your library still has to support 3.10, the exceptiongroup backport is worth a look — just remember it restores the runtime classes but not the except* syntax, which needs the 3.11+ compiler.