Writing Python C extensions: a complete developer's guide

What a Python C extension actually is

A Python C extension is a shared library (.so on Linux, .pyd on Windows, .dylib on macOS) that exposes a C function named PyInit_<modulename>. When Python’s import machinery loads your file, it looks for that symbol, calls it, and gets back a module object. The object carries a method table that maps Python names like spam.system to C functions you wrote.

You write one because you have C code you want to call from Python, or because you’ve found a hot loop that needs to be faster than pure Python can manage. Neither is a small project; cffi, pybind11, Cython, and maturin exist precisely because the raw C API is unforgiving. The C API gives you the most control, at the cost of doing everything yourself: argument parsing, return-value construction, reference counting, and exception signaling.

This guide is for the “I want to understand what the others are doing under the hood” case, and the “I really do need a hand-rolled extension” case. It targets CPython 3.12+; the multi-phase initialization pattern (Py_mod_exec) and PyModule_AddObjectRef are both standard from that point on. If you’re not sure whether you need the C API at all, the decision table at the end of this article will help you choose.

Setting up the build

Modern Python extension projects use a pyproject.toml with setuptools as the build backend. No setup.py required.

[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "spam"
version = "0.1.0"

[tool.setuptools]
ext-modules = [
  {name = "spam", sources = ["spammodule.c"]},
]

Once your pyproject.toml is in place, build the extension in place using the standard editable install. This compiles spammodule.c into a .so shared library and drops it on sys.path so you can import spam from anywhere in the project:

python -m pip install -e .
# output: Successfully built spam
# output: Successfully installed spam-0.1.0

When the extension grows beyond a single file, list each .c source in the sources array. Setuptools compiles them all into one shared library, so the final import name is whatever you set in name. For deeply nested projects, dotted module names like spam._core mirror the package layout:

[tool.setuptools]
ext-modules = [
  {name = "spam._core", sources = ["src/core.c", "src/parser.c", "src/util.c"]},
]

A common foot-gun here is the wrong Python finding the extension. If you run pip install with python3.12 but your setup.py happens to use /usr/bin/python3.11 for the compile step, the resulting .so will refuse to load on 3.12. Use virtual environments and PEP 517 build isolation to keep the build Python pinned to the one you’ll run with. The pyproject.toml guide walks through the full project layout in more detail.

A minimal module: spam.system

The canonical “first extension” in CPython’s own tutorial wraps the C system() call. Here it is, modernized for 3.12+ with multi-phase initialization. Save it as spammodule.c.

#define PY_SSIZE_T_CLEAN
#include <Python.h>

static PyObject *SpamError = NULL;

/* Module function */
static PyObject *
spam_system(PyObject *self, PyObject *args)
{
    const char *command;
    int sts;

    if (!PyArg_ParseTuple(args, "s", &command)) {
        return NULL;  /* PyArg_ParseTuple already set the exception */
    }
    sts = system(command);
    if (sts < 0) {
        PyErr_SetString(SpamError, "System command failed");
        return NULL;
    }
    return PyLong_FromLong(sts);
}

static PyMethodDef spam_methods[] = {
    {"system", spam_system, METH_VARARGS,
     "Execute a shell command."},
    {NULL, NULL, 0, NULL}  /* Sentinel */
};

static int
spam_module_exec(PyObject *m)
{
    SpamError = PyErr_NewException("spam.error", NULL, NULL);
    if (SpamError == NULL) return -1;
    if (PyModule_AddObjectRef(m, "SpamError", SpamError) < 0) {
        Py_DECREF(SpamError);
        SpamError = NULL;
        return -1;
    }
    return 0;
}

static PyModuleDef_Slot spam_module_slots[] = {
    {Py_mod_exec, spam_module_exec},
    {0, NULL}
};

static struct PyModuleDef spam_module = {
    .m_base = PyModuleDef_HEAD_INIT,
    .m_name = "spam",
    .m_doc = "Example C extension that wraps the C system() call.",
    .m_size = 0,
    .m_methods = spam_methods,
    .m_slots = spam_module_slots,
};

PyMODINIT_FUNC
PyInit_spam(void)
{
    return PyModuleDef_Init(&spam_module);
}

There are three layers to read here. The method table (spam_methods) is the user-facing surface: each entry pairs a Python name with a C function and a flag describing the call signature. The module definition (spam_module) wires the table, a doc string, and a slot array together. The slot array holds initialization callbacks that the import system calls at controlled points: Py_mod_exec runs before the module is handed to sys.modules, which is the right time to attach SpamError.

Build and try it:

python -m pip install -e .
python -c "import spam; print(spam.system('echo hi'))"
# output: hi
# 0

The hi is the shell command’s stdout; the 0 is spam.system’s return value (the exit code of the shell). If you pass a non-existent command, you get a spam.error exception instead.

Parsing arguments safely

PyArg_ParseTuple is the front door to your function. It reads from a tuple of positional arguments, decodes them according to a format string, and writes the results into your local variables. If parsing fails it returns 0 and already sets an exception, so you return NULL immediately, without touching PyErr_SetString.

static PyObject *
spam_clip(PyObject *self, PyObject *args, PyObject *kwargs)
{
    static const char *kwlist[] = {"text", "lo", "hi", NULL};
    const char *text;
    double lo, hi;

    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "sdd", kwlist,
                                     &text, &lo, &hi)) {
        return NULL;
    }
    /* ... */
    Py_RETURN_NONE;
}

Note METH_VARARGS | METH_KEYWORDS in the method table when you accept keyword arguments, and the $ format marker to make everything after it keyword-only. Here are the format codes you’ll reach for most often:

Code	C type	What it does
s	const char *	UTF-8 string (no NUL terminator required)
s#	const char *, Py_ssize_t	String with explicit length
y	const char *	bytes object
i	int	Native C int
l	long	Native C long
d	double	C double
O	PyObject *	Any object (borrowed reference)
O!	PyTypeObject *, PyObject *	Type-checked object (borrowed)
O&	converter function	Custom C-level converter
|	(modifier)	Remaining args are optional
$	(modifier)	Remaining args are keyword-only

For functions on a hot path, METH_FASTCALL skips the tuple-then-parse step and gives you a PyObject *const *args array plus a Py_ssize_t nargsf count (the high bit flags keyword arguments). The signature is faster because it avoids one allocation per call.

Returning values

Functions that return PyObject * must return a new reference: a Py_INCREF’d object the caller now owns. The constructors below all return new references and NULL on failure (with an exception already set, so again, just return NULL).

PyObject *PyLong_FromLong(long v);
PyObject *PyFloat_FromDouble(double v);
PyObject *PyUnicode_FromString(const char *u);  /* UTF-8 in, str out */
PyObject *PyBytes_FromString(const char *s);
PyObject *PyBool_FromLong(long v);             /* 0 → Py_False, else Py_True */

For compound values like tuples and dicts, Py_BuildValue reads a format string the way printf reads one, except it produces Python objects instead of printing them. The format codes mirror the ones PyArg_ParseTuple accepts, so you can pass mixed-type data back to Python in a single call:

return Py_BuildValue("(sdO)", "answer", 42.0, some_list);
/* output: ('answer', 42.0, [...])  */

For the void case, use the Py_RETURN_NONE macro. It does the Py_INCREF(Py_None); return Py_None; dance for you.

Reference counting

Every PyObject * has a reference count. The count goes up with Py_INCREF and down with Py_DECREF. When it hits zero, the object’s deallocator runs. Because dealloc can call arbitrary Python code (finalizers, weakref callbacks, __del__), you can never hold a non-owning pointer across a Py_DECREF call.

Two flavors handle the NULL case:

• Py_INCREF / Py_DECREF: assume non-NULL; Py_DECREF(NULL) is a fatal error.
• Py_XINCREF / Py_XDECREF: NULL is a no-op. Use these for variables that may still be unset on the error path.

The rules that matter most in practice:

1. A constructor (PyLong_FromLong, PyUnicode_FromString, PyArg_ParseTuple with O! are exceptions, since those are borrowed) hands you a new reference. You own it; you must Py_DECREF (or hand it off) before returning.
2. An O or O! format spec gives you a borrowed reference. If you store the pointer past the function call, Py_INCREF it first.
3. On the error path, every owned object built so far must be Py_XDECREF’d.

Here’s a buggy version followed by the fix. The bug: a borrowed reference is stored in a static and then the function is re-entered; the previous object can be freed underneath you.

/* BUGGY: PyObject *cached (borrowed from PyArg_ParseTuple("O", ...)) */
static PyObject *cached = NULL;
static PyObject *
spam_cache(PyObject *self, PyObject *args)
{
    if (!PyArg_ParseTuple(args, "O", &cached)) return NULL;
    /* If Python frees that object later and we use cached → crash. */
    Py_RETURN_NONE;
}

Fix it by Py_INCREF’ing the borrowed reference, and Py_DECREF’ing the previous one. The Py_XDECREF on the old value matters because it might still be NULL on the very first call, when the static has not been initialized yet:

static PyObject *
spam_cache(PyObject *self, PyObject *args)
{
    PyObject *incoming;
    if (!PyArg_ParseTuple(args, "O", &incoming)) return NULL;
    Py_XDECREF(cached);     /* drop the old reference we owned */
    cached = incoming;
    Py_INCREF(cached);      /* upgrade borrowed → owned */
    Py_RETURN_NONE;
}

Refcount bugs are the #1 reason hand-written C extensions crash in production. If you find yourself staring at a SystemError: <built-in method __...> line in a stack trace, suspect a missing Py_INCREF. The memory model guide covers the broader picture.

Errors and exceptions

The C-API convention is strict: any function that returns PyObject * either returns a valid new reference or returns NULL with an exception already set. Callers check for NULL first and never touch the return value otherwise. There is no try/except; the runtime walks the C stack itself.

PyErr_SetString(SpamError, "System command failed");
PyErr_SetFromErrno(PyExc_OSError);                  /* uses errno */
PyErr_SetObject(PyExc_ValueError, detail_object);   /* exception + value */

A common mistake is calling PyErr_Occurred() to test for an exception, then continuing with normal logic. PyErr_Occurred only reports; it does not clear. If you want to swallow an exception, call PyErr_Clear(), but think twice, because an un-cleared exception will surface at the next Python frame boundary and that’s usually what you want.

For module-level exceptions, create a new exception class with PyErr_NewException and attach it with PyModule_AddObjectRef:

SpamError = PyErr_NewException("spam.error", NULL, NULL);
if (SpamError == NULL) return -1;
if (PyModule_AddObjectRef(m, "SpamError", SpamError) < 0) {
    Py_DECREF(SpamError);
    SpamError = NULL;
    return -1;
}

PyModule_AddObjectRef (added in 3.10) is the right call. The older PyModule_AddObject is famous for leaking references on the error path because it steals the reference even when the add fails. Use the modern variant.

The Limited API and stable ABI

The full CPython C API exposes struct internals, including PyObject fields, concrete PyTypeObject layout, and so on. Those change between minor versions, which means an extension compiled against Python 3.12 won’t load on 3.13. The Limited API is a curated subset that CPython promises to keep ABI stable across versions.

To opt in, define Py_LIMITED_API before including Python.h:

#define Py_LIMITED_API 0x030C0000   /* declare "I work on 3.12 and up" */
#include <Python.h>

The hex value is 0x03XX0000 where XX is the minimum minor version. Values below 0x03090000 (3.9) are not allowed. Restrictions include: no direct struct field access, no subclassing of heap types without PyType_FromMetaclass (added in 3.12), and no touching PyObject internals.

In pyproject.toml, one line gets you an abi3 wheel that works across every supported 3.x:

[tool.setuptools]
ext-modules = [
  {name = "spam", sources = ["spammodule.c"], py_limited_api = true},
]

A py_limited_api = true build produces spam-0.1.0-cp312-abi3-manylinux_2_17_x86_64.whl, a single binary that the loader accepts on any CPython ≥ 3.12 on that platform. There’s a current caveat: on Python 3.13’s experimental free-threaded build (--disable-gil), abi3 wheels are not accepted. That’s tracked in CPython issue #111506 and may be resolved by 3.14. Don’t treat that as a permanent limitation; check release notes before you decide to skip the Limited API on those grounds. The GIL guide has more on the free-threaded story.

Alternatives worth knowing

Reaching for the raw C API is rarely the only path. Here’s a quick decision aid.

Need	Recommended tool	Why
Call a few C functions, prototyping	ctypes (stdlib)	No C compile step; lives in .py files
Call many C functions or write complex bindings	cffi	You write Python-side bindings; it generates C
Need C++ templates, NumPy buffers, many types	pybind11	Header-only C++, type-safe, mature
Want a near-Python feel, accept some build complexity	Cython (.pyx → C)	Type hints, large ecosystem
Need a stable ABI, writing C by hand	Raw C API + Py_LIMITED_API	Most control, most work
Want to support PyPy / GraalPython / future interpreters	HPy, cffi, or pure Python	Avoids the CPython-specific API
New project, want future-proofing	maturin + Rust, or pybind11	Modern tooling, type system catches refcount bugs at compile time

The profiling guide is worth reading before any of these; rewrite the hot loop only after you’ve measured it.

Common mistakes

The 80% case of bugs in hand-written C extensions maps to a small set of patterns. Each is one to two lines; each has bitten a lot of people.

1. Forgetting PY_SSIZE_T_CLEAN. Always define it before any system header to keep # formats from macro-expanding to Py_ssize_t cleanly. It is no longer required on 3.13+, but it is harmless there and helps with mixed-version toolchains.
2. Py_DECREF(NULL). Fatal. Use Py_XDECREF for any variable that may be unset on the error path.
3. Storing a borrowed reference past the call. PyArg_ParseTuple(..., "O", &obj) gives you a borrowed ref. Py_INCREF before you store it.
4. Forgetting to Py_XDECREF on the error path. If you built two objects and the second one failed, the first still has to be released. Long functions are where this slips.
5. Trying to clear an exception with PyErr_Occurred(). It only reports. Use PyErr_Clear() to actually swallow, and only when you mean to.
6. Mismatched METH_* flags. Pick the flag that matches the function: no args → METH_NOARGS + Py_UNUSED; one positional → METH_O; variable → METH_VARARGS; keyword support → add METH_KEYWORDS.
7. Returning a borrowed reference. A C function returning PyObject * must return a new reference. Returning a borrowed one causes a leak or use-after-free.
8. Calling the C API inside Py_BEGIN_ALLOW_THREADS. That block is for blocking C calls only, usually stdio, read, pthread_mutex_lock. Wrap the call, not the surrounding code.
9. Re-initializing globals under sub-interpreters or free-threaded builds. Two PyInit_spam calls will re-enter spam_module_exec. Use a guard or per-module state via m_size and PyModule_GetState.
10. Using PyModule_AddObject on the error path. It steals the reference even when it fails. Use PyModule_AddObjectRef.
11. Building with the wrong Python. Mixed versions of python on PATH produce a .so that only loads on one minor. Use a virtual environment and PEP 517 build isolation.
12. Letting Py_DECREF run while you still hold a non-owning pointer. Dealloc can call Python code. Save what you need, drop the ref, then use the saved pointer.

Conclusion

A hand-written C extension is the lowest-level way to talk to CPython and the most work to get right. The payoff is real: full access to the C API, the ability to ship a single abi3 wheel across Python versions, and zero overhead between Python and your code. The cost is everything else: parsing arguments, building return values, reference counting, and exception handling, all in C, with no help from the type system.

If you don’t need that level of control, start with cffi or pybind11 and come back to the raw API when a profiler points you here. If you do need it (when you’re wrapping an existing C library, when you’ve found a loop that nothing else speeds up, when you want a stable ABI wheel), the patterns above are the ones you’ll reuse: PyArg_ParseTuple for inputs, PyLong_FromLong for outputs, careful Py_INCREF / Py_DECREF everywhere in between, and a slot-based module definition that plays well with the multi-phase init scheme introduced in 3.5 and refined in every release since.

For a deeper look at how CPython itself is structured, the Python bytecode guide and the memory model guide round out the picture.

See also

• len(): the reference counting rules above apply to any extension that returns a length.
• struct module: the stdlib alternative for binary data layout (useful when you don’t quite need a C extension).
• ord(): character-code lookups, a good one-liner to start with when writing your first C extension.