Python AST Metaprogramming: How to Parse and Transform Code
What is AST metaprogramming?
Every “magic” tool in the Python ecosystem runs your code through a parser before doing anything interesting with it. pytest collects test functions. attrs and pydantic build __init__ methods. mypy, ruff, and black all reason about the structure of your code. jedi powers autocomplete. They all share a starting point: the ast module.
ast turns Python source into a tree of objects. Once your code is a tree, you can count things in it, replace things in it, or build an entirely new tree from scratch and feed it back to the interpreter.
The module sits next to metaclasses and decorators in the metaprogramming toolchain, but it does a different job. Metaclasses reshape class creation. Decorators wrap callables. ast rewrites source, and that source can be a file on disk, a string you built, or the body of a live function pulled out with inspect.getsource().
This guide covers the core API: parse, walk, transform, compile. The examples are short. The patterns are what you reach for in real tools.
TL;DR
The ast module turns Python source into a tree of objects you can read, modify, and rebuild. Use ast.parse to get a tree, ast.walk or a NodeVisitor to read it, a NodeTransformer to rewrite it, and compile to run the result. Most ast bugs trace back to three things: missing fix_missing_locations calls, forgotten generic_visit recursion, or treating ast.unparse as a pretty-printer.
The ast module at a glance
Three functions cover most of the work.
import ast
source = """
def add(a: int, b: int) -> int:
return a + b
"""
tree = ast.parse(source)
print(ast.dump(tree, indent=2))
# output:
# Module(body=[FunctionDef(name='add',
# args=arguments(posonlyargs=[], args=[arg(arg='a', annotation=Name(id='int')),
# arg(arg='b', annotation=Name(id='int'))],
# kwonlyargs=[], kw_defaults=[], defaults=[]),
# body=[Return(value=BinOp(left=Name(id='a', ctx=Load()),
# op=Add(), right=Name(id='b', ctx=Load())))],
# returns=Name(id='int'), type_params=[])])Worth flagging up front:
ast.parsereturns aModulewrapping the top-level statements. Passmode='eval'and you get anExpressionwrapping a single expression. Mixing those up is a common bug, becausecompile()andliteral_eval()care which root node you hand them.- The
BinOp(left, op, right)shape is the universal “operator expression”. Everya + bever written, everya * b, everya ** b, looks exactly like that.a == bis aComparewith anEqoperator, but the same idea applies. ast.dumpis a debugging tool. Do not parse its output. Its formatting changes between versions and there is no API contract on it.
For walking the tree you have two options. ast.walk(node) is a breadth-first generator that yields every descendant, with no order guarantee. ast.iter_child_nodes(node) yields only direct children. Most code uses ast.walk for quick scans and a custom NodeVisitor for serious work.
ast.parse also accepts a few keyword arguments worth knowing. feature_version=(3, 12) parses code with the 3.12 grammar even on older interpreters, which matters when you build cross-version tools. type_comments=True keeps # type: ... annotations in the tree instead of discarding them. optimize=-1 (the default) forwards to compile(); pass 1 or 2 to drop asserts and string docstrings.
How do you read code with NodeVisitor?
NodeVisitor is the read-only workhorse. You subclass it, override visit_<NodeName> for the types you care about, and the base class dispatches for you.
import ast
class CallCounter(ast.NodeVisitor):
def __init__(self):
self.count = 0
def visit_Call(self, node: ast.Call):
self.count += 1
self.generic_visit(node) # walk into args/keywords
print(CallCounter().visit(ast.parse("f(g(1, 2), h(x.y), z)")).count)
# output: 3self.generic_visit(node) is the easy step to forget. visit only calls your visit_<Name> method. Without explicit recursion you count the outermost f(...) and stop, missing the g(1, 2) and h(x.y) inside the arguments. Any visitor that cares about nested occurrences has to call generic_visit (or descend into the relevant fields by hand).
A second example: scan a file for every function whose name starts with test_. That is essentially the slice of work pytest does at collection time.
import ast
class TestFinder(ast.NodeVisitor):
def __init__(self):
self.names = []
def visit_FunctionDef(self, node: ast.FunctionDef):
if node.name.startswith("test_"):
self.names.append(node.name)
self.generic_visit(node)
source = """
def test_add(): ...
def helper(): ...
def test_sub(): ...
"""
print(TestFinder().visit(ast.parse(source)).names)
# output: ['test_add', 'test_sub']ast.walk() works too for trivial cases. A hand-rolled visitor is faster on large files because you skip the queue machinery. In my experience that gap is roughly 2x on multi-thousand-line modules.
How do you modify code with NodeTransformer?
NodeTransformer is NodeVisitor with a twist: the return value of each visit_* method replaces the original node. Return None to delete it. Return a list to splice multiple nodes into statement-list fields. The rest of the parent tree gets walked automatically.
The canonical example, from the official ast docs, rewrites every free name into a dictionary lookup.
import ast
from ast import NodeTransformer, Name, Subscript, Constant, Load
class RewriteName(NodeTransformer):
def visit_Name(self, node):
if isinstance(node.ctx, ast.Load):
return Subscript(
value=Name(id='data', ctx=Load()),
slice=Constant(value=node.id),
ctx=node.ctx,
)
return node # leave Store/Del names alone
print(ast.unparse(RewriteName().visit(ast.parse("print(foo + bar)"))))
# output: print(data['foo'] + data['bar'])Watch the isinstance(node.ctx, ast.Load) guard. The unmodified example in the docs rewrites assignments too, which would turn foo = 1 into data['foo'] = 1 and crash at runtime. Always check ctx on Name and Attribute nodes. Load, Store, and Del mean three different things and your transformer usually only wants one of them.
For a more realistic transformer, the decorator-injection pattern is worth knowing. It is how tools like attrs and dataclasses add @dataclass-style behavior to a class without changing the class’s source on disk.
import ast
from ast import NodeTransformer, FunctionDef, Name, Load, Call, Constant
SRC = """
def add(a, b):
return a + b
def sub(a, b):
return a - b
"""
class TraceAll(NodeTransformer):
def visit_FunctionDef(self, node: FunctionDef):
self.generic_visit(node)
node.decorator_list.insert(
0,
Call(
func=Name(id="trace", ctx=Load()),
args=[Constant(value=node.name)],
keywords=[],
),
)
return node
tree = ast.parse(SRC)
new = TraceAll().visit(tree)
ast.fix_missing_locations(new)
print(ast.unparse(new))Running that prints back the same two functions, each now prefixed with @trace('add') and @trace('sub') respectively. self.generic_visit(node) inside visit_FunctionDef makes sure decorators already present on the function get walked too, so a transformer that targets decorators composes safely with one that adds them.
Building code from scratch
Sometimes you do not have source to parse. You want to generate a function from a config dict, or a dataclass from a schema, or a tiny helper from a template. In that case you build the AST by hand and feed it to compile().
import ast
module = ast.Module(
body=[
ast.FunctionDef(
name="greet",
args=ast.arguments(
posonlyargs=[],
args=[ast.arg(arg="name")],
kwonlyargs=[],
kw_defaults=[],
defaults=[],
),
body=[
ast.Return(value=ast.Call(
func=ast.Name(id="print", ctx=ast.Load()),
args=[ast.JoinedStr(values=[
ast.Constant(value="Hello, "),
ast.FormattedValue(
value=ast.Name(id="name", ctx=ast.Load()),
conversion=-1,
format_spec=None,
),
ast.Constant(value="!"),
])],
keywords=[],
)),
],
decorator_list=[],
),
],
type_ignores=[],
)
ast.fix_missing_locations(module)
ns = {"print": print}
exec(compile(module, "<gen>", "exec"), ns)
ns["greet"]("world")
# output: Hello, world!The fully-explicit node construction is verbose but portable to Python 3.9+. On 3.13 and later, the node classes ship with sensible defaults for the omitted fields, so the constructor gets much terser. Same pattern with a quarter of the code still works, but writing the long form once is the fastest way to internalize the shape of these objects.
One more line worth highlighting: ast.fix_missing_locations(module). Hand-built nodes have no lineno and no col_offset. compile() requires both. The helper walks the tree and inherits source positions from the nearest parent that has them, defaulting to (1, 0, 1, 0) for the root.
What is the full parse-transform-compile loop?
Most metaprogramming tools fit in five lines.
import ast
source = "..." # or read from a file
tree = ast.parse(source) # 1. parse
tree = MyTransformer().visit(tree) # 2. transform
ast.fix_missing_locations(tree) # 2b. if you added nodes
code = compile(tree, "<ast>", "exec") # 3a. compile
exec(code, namespace) # 3b. runFor read-only inspection, drop step 2. For codegen, drop step 1 and build the tree by hand, but keep fix_missing_locations and the compile() call.
A complete, runnable example that reads a file, decorates every function with a @trace call, and writes the result back:
import ast
import sys
from ast import NodeTransformer, FunctionDef, Name, Load, Call, Constant
class TraceAll(NodeTransformer):
def visit_FunctionDef(self, node: FunctionDef):
self.generic_visit(node)
node.decorator_list.insert(
0,
Call(
func=Name(id="trace", ctx=Load()),
args=[Constant(value=node.name)],
keywords=[],
),
)
return node
src = open(sys.argv[1]).read()
tree = ast.parse(src)
tree = TraceAll().visit(tree)
ast.fix_missing_locations(tree)
print(ast.unparse(tree))Think of this as the skeleton of every codegen tool that ships in the Python ecosystem. The hard parts are not the AST manipulation itself. They are the source-position bookkeeping, the round-trip fidelity, and the constant-folding edge cases.
How does ast.literal_eval safely evaluate code?
ast.literal_eval is the one AST function most developers should reach for first. It evaluates a string containing only Python literals: numbers, strings, tuples, lists, dicts, sets, booleans, None. Nothing else.
import ast
config = ast.literal_eval(open("config.txt").read())Use it as the safe replacement for eval() on data that is supposed to be data. Anything that is not a literal, a name, a call, or an operator, raises ValueError with a useful message. Roughly equivalent to json.loads for primitive Python literals, with the bonus that tuples and Python constants like True and None work out of the box.
One real trade-off to know about: deeply nested literals can blow the C stack and crash the interpreter. The function documents this in its own source. If you accept user input, set a sensible depth limit on the parsed tree before evaluating it.
What are the most common AST pitfalls?
A short list of the bugs that show up over and over.
Hand-built nodes have no lineno and no col_offset. compile() raises a confusing “unknown location” error pointing at line 0 with no useful context. Always call ast.fix_missing_locations(tree) before you compile() anything you built yourself (or anything your transformer injected synthetically).
visit_Call (or any other visit_* method) only fires once per node. To count nested calls, assignments inside if blocks, or anything below the surface, call self.generic_visit(node) (or descend into the relevant fields by hand). Forget this and you miss every nested occurrence.
NodeTransformer deletes a node whenever you return None. A method that falls off the end of its body without an explicit return node silently drops the node. Modern linters (and Python 3.13+) can warn about implicit None returns; on older interpreters, watch your method bodies.
ast.unparse is not a formatter. It discards comments, normalizes whitespace, and converts constant tuples to lists and frozensets to sets. Run the output through ruff format or black if you need formatted source.
ast.Num, ast.Str, ast.Bytes, ast.NameConstant, and ast.Ellipsis are gone in 3.14. Everything is ast.Constant now. The same goes for ast.Index, which is now a plain ast.Tuple for multi-axis slicing. Code online that imports those names from ast is pre-3.8 and needs a rewrite.
Pass feature_version=(3, 12) (or whatever the minimum target is) when you parse code that may contain newer syntax than your interpreter supports. Without it the parser uses the current grammar and chokes on valid source from the future.
What’s next
AST sits one stop into the compilation pipeline. The next step down is the dis module, which prints out the bytecode compile() produces; the bytecode guide has the tour. Working upstream from AST, the tokenize module is what you reach for when whitespace and comments matter to you.
For the rest of the metaprogramming toolkit, the metaclasses guide covers class-creation magic and the decorators deep dive covers callable wrapping. The ast module reference has the full function signatures, and the canonical reference is the Python ast documentation. The compile() and eval() builtins are the other end of the pipeline.
To feed live code into ast.parse(), the inspect module is the standard way to pull source from a function or class with inspect.getsource(). And if you find yourself writing the same transformer twice, it is probably time to reach for a real parser-generator like libcst. The ast module is great for tooling you control, less great for refactors that have to preserve formatting.
Frequently asked questions
What is the ast module used for? It parses Python source into a tree of ast.AST nodes. Tools like pytest, attrs, mypy, black, and ruff rely on it to read and reason about code. You can also feed a hand-built tree to compile() and run it.
Is ast the same as libcst? No. ast gives you a structural tree without comments or formatting, and ast.unparse does not round-trip. libcst preserves formatting and comments, which makes it better for refactoring tools that have to leave the file looking the way the user wrote it.
Can ast.literal_eval replace eval? Yes, for data. It only accepts Python literals (numbers, strings, tuples, lists, dicts, sets, booleans, None) and raises ValueError on anything else. Use eval only when you actually need to run arbitrary code.
Why does my hand-built AST raise SyntaxError: unknown location? You forgot ast.fix_missing_locations(tree). The compiler needs lineno and col_offset on every node, and hand-built nodes start with neither. The fix is one line.
Does ast.parse run my code? No. It only parses. To execute the tree, pass it to compile(tree, filename, mode) and then exec (or eval for an expression) the result inside a controlled namespace.
Conclusion
The ast module is small. The mental model is even smaller: parse, walk, transform, compile. The complications come from the edges: source positions, cross-version grammar, the gap between unparse and a real formatter.
A quick way to choose the right tool:
| Task | Right tool |
|---|---|
| Match a known pattern in code | ast.parse plus NodeVisitor |
| Rewrite source on disk | libcst (preserves formatting) |
| Generate a function or class at runtime | ast codegen plus compile |
| Evaluate data from a config file | ast.literal_eval |
| Hot-swap a value in a live object | not ast, use a normal import or attribute write |
Most of the time, if you can describe what you want as “find all the X in this code” or “turn every Y into a Z”, Python ast metaprogramming can do it. The work is the boilerplate around parse, visit, and fix_missing_locations. Once that loops cleanly in your head, every other metaprogramming tool built on ast becomes much easier to read.