Python/C Interoperability

CPython is implemented in C and exposes a C API, which is why Python can call into native libraries and be called from them — this topic maps out the landscape of tools that make that possible.

Python programs frequently need to talk to code written in C: calling a fast native library, reusing a legacy DLL, or squeezing out performance that a pure-Python loop cannot reach. Because the reference implementation, CPython, is itself written in C, the language has several well-established doors between the two worlds. This topic is a map of that landscape — the C API, extension modules, ctypes, the cffi ecosystem, and the ABI concerns that make all of it work (or break).

What Is Python/C Interoperability

Python/C interoperability means running Python code and native (C, C++, Rust, Fortran, ...) code in the same process and letting them call each other and share data. It exists because some tasks are better solved in a compiled language — raw numeric throughput, existing system libraries, hardware access — while Python remains the glue and scripting layer on top.

This is a CPython implementation topic in the sense that most of the concrete mechanisms (the C API, ctypes talking to shared libraries) are specific to CPython or to how CPython is built. Other Python implementations (PyPy, etc.) support interop differently or with more overhead.

Why It Matters

Interop is how the Python ecosystem gets its performance and its reach: NumPy's array math, cryptography's OpenSSL bindings, database drivers, image codecs, and GUI toolkits are all Python packages with a native core underneath a Python API. Understanding the layer beneath a familiar import lets you debug crashes that pure-Python reasoning can't explain, and lets you write your own bindings when a library only exists in C.

The CPython C API

CPython exposes a C API (declared in Python.h) that lets C code create Python objects, call Python functions, raise exceptions, and manage reference counts. Writing directly against this API produces the fastest, most capable bindings, but it is verbose, requires careful reference-counting discipline, and ties your code to CPython specifically (not other implementations).

PyArg_ParseTuple decodes the Python-level argument tuple into C values; PyLong_FromLong builds a new Python object to return. Every 'new reference' created this way must eventually be released (Py_DECREF) or it leaks — this bookkeeping is the main source of C API bugs.

// Sketch of a C API extension function (conceptual, not runnable here)
static PyObject *
spam_add(PyObject *self, PyObject *args) {
    long a, b;
    if (!PyArg_ParseTuple(args, "ll", &a, &b))
        return NULL;                 // exception already set
    return PyLong_FromLong(a + b);   // build a new Python int object
}

Extension Modules

An extension module is a compiled shared library (.so on Linux, .pyd on Windows, .dylib on macOS) that Python's import system loads exactly like a .py file, but whose code is native. Extension modules are typically built using the C API (directly or through a build helper) and distributed as compiled wheels so end users never need a compiler.

Many familiar standard-library modules (math, json's C accelerator, _socket) are extension modules; from the importing code's point of view, an extension module and a pure-Python module look identical.

ctypes

ctypes is a standard-library module that loads compiled shared libraries at runtime and calls their functions directly from pure Python — no C compiler and no separate build step required. It trades some safety and ergonomics for that convenience: you describe argument and return types yourself, and mistakes there can crash the process rather than raising a Python exception.

ctypes is covered in depth as its own topic; here it is one point on the interoperability map.

The cffi Ecosystem (Third-Party)

cffi is a third-party library (not part of the standard library) that offers an alternative to both the raw C API and ctypes. You give it a C declaration (either parsed from a header or written by hand) and it generates efficient bindings, either at runtime ('ABI mode', similar in spirit to ctypes) or by compiling a small C shim at build time ('API mode', similar in spirit to a C extension). PyPy's developers designed cffi partly because raw C API extensions are especially costly for non-CPython implementations to support.

cffi must be installed separately (pip install cffi); it is not built into Python.

Native Libraries and How They Get Loaded

A native library is compiled code — a shared object the operating system's dynamic linker loads into the process's address space. Python-side tools differ mainly in when and how they hook into that loaded code:

Mechanism        Compile step?   Runtime cost           Safety
C API extension   yes (build)     lowest                 manual refcounting; C-level bugs
ctypes             no             per-call overhead       no type checking; easy to crash
cffi (API mode)    yes (build)     low                    generated glue; safer than raw C API
cffi (ABI mode)    no             similar to ctypes       same class of risk as ctypes

ABI Concerns

The Application Binary Interface (ABI) is the set of low-level rules — how arguments are passed in registers/stack, how structs are laid out in memory, how symbols are named — that let compiled code from different sources call each other correctly. Interop tools must agree with the target library's ABI exactly: wrong argument types, wrong struct field order, or a mismatched calling convention produce silent memory corruption instead of a clean error.

ABI concerns are explored in full in the dedicated Foreign Function Interface Concepts topic; the short version here is: getting the Python-side type description exactly right is not optional, it's the whole game.

Choosing an Approach

A rough decision guide: reach for ctypes for a quick call into an existing shared library with a handful of functions; reach for cffi (third-party) for a more ergonomic and often faster version of the same job, especially in API mode; write a C API extension (often via a helper like pybind11, third-party) when you need maximum performance or deep control and are willing to compile; and reach for Cython (third-party, covered separately) when you want to write mostly-Python code that compiles to fast native code.

Practical Usage

This is the smallest useful interop example: a native function (sqrt, from the C math library) called from pure Python, with no build step.

import ctypes

# Load the C standard math library and call a native function directly.
libm = ctypes.CDLL("libm.so.6")  # platform-specific name; see the ctypes topic
libm.sqrt.restype = ctypes.c_double
libm.sqrt.argtypes = [ctypes.c_double]

result = libm.sqrt(2.0)
print(result)  # 1.4142135623730951

Edge Cases and Pitfalls

A segfault instead of a Python traceback is the classic symptom of an interop bug — it means native code touched invalid memory, and Python's own exception machinery never got a chance to intervene. Version skew between a compiled extension and the Python it's loaded into, missing shared libraries at runtime, and 32-bit/64-bit or endianness mismatches across platforms are all common real-world failure modes.

Best Practices

Prefer a well-maintained third-party binding over writing your own whenever one exists. When you do write bindings, be exact and conservative about types, validate at the Python/native boundary before trusting data on either side, and test on every target platform — interop code is exactly where 'works on my machine' is most dangerous.

Mental Model

Think of the Python process as a house with several different kinds of doors to the C world: the C API is a custom-built door installed at construction time (compiled in); ctypes is a universal door you can open on any wall at runtime, but you must describe exactly what's on the other side yourself; cffi is a similar door built with better tooling and more safety rails.

Quick Reference

C API            Native, compiled, full control      stdlib (Python.h) — no separate install
Extension module Compiled .so/.pyd loaded as import   built from C/C++/Rust source
ctypes           Runtime FFI, no compiler needed       stdlib
cffi             Runtime or build-time FFI             third-party (pip install cffi)
Cython           Python-like syntax -> compiled C       third-party

Key Takeaways

Python/C interoperability is how the ecosystem combines Python's productivity with native performance and existing C libraries.

The C API gives full control and best performance but is CPython-specific and demands manual reference counting.

ctypes is a standard-library, no-compiler way to call into shared libraries; cffi is a third-party alternative with better ergonomics.

Getting the ABI description right (types, struct layout, calling convention) is the crux of safe interop — mismatches crash the process rather than raising exceptions.

import ctypes lib = ctypes.CDLL("libm.so.6") lib.pow(2.0, 10.0) # argtypes/restype not set import ctypes lib = ctypes.CDLL("libm.so.6") lib.pow.argtypes = [ctypes.c_double, ctypes.c_double] lib.pow.restype = ctypes.c_double result = lib.pow(2.0, 10.0)

Without argtypes/restype, ctypes assumes everything is a C int, which silently corrupts doubles and can crash or return garbage. Always declare the signature explicitly.

Native code runs with the full privileges of the Python process and no sandboxing; loading an untrusted or mismatched shared library is a security and stability risk, not just a style issue.

Why can a bug in an interop layer produce a segmentation fault instead of a Python exception?

Because native code operates outside Python's object and exception system; if it dereferences invalid memory, the operating system terminates the process before Python's interpreter loop ever gets a chance to raise anything.

Name two ways Python code can call into a compiled C library, and one difference between them.

ctypes (standard library, load at runtime, no compiler needed, weaker type safety) and writing a C API extension module (compiled ahead of time, faster and safer once built, requires a build step and a compiler).

Is cffi part of the Python standard library?

No — cffi is a third-party package that must be installed separately with pip; ctypes is the standard-library equivalent.

What does 'ABI' stand for and why does it matter for interop?

Application Binary Interface — the low-level agreement on argument passing, struct layout, and calling conventions; interop code must match a native library's ABI exactly or the call corrupts memory instead of failing cleanly.

Why might a library's developers choose cffi over writing a raw C API extension?

cffi's generated bindings are less implementation-specific and cheaper to maintain across CPython and PyPy, and API mode still gets compiled-glue performance without hand-written reference-counting code.

Give one example of a widely used Python package whose core is a native extension.

NumPy — its array operations are implemented in C for performance, with a Python API layered on top.

Write the ctypes calls needed to load libm and call its native pow function correctly with two doubles.

import ctypes libm = ctypes.CDLL("libm.so.6") libm.pow.argtypes = [ctypes.c_double, ctypes.c_double] libm.pow.restype = ctypes.c_double print(libm.pow(2.0, 10.0)) # 1024.0

Load the library with CDLL, then explicitly set argtypes and restype before calling — ctypes defaults every argument and return value to a C int otherwise, which corrupts double-precision calls.

This snippet omits argtypes/restype. What is the realistic risk of running it as written?

ctypes defaults the return type to c_int, so the actual double result from sqrt is misinterpreted as an integer — the printed value will be wrong (garbage bits reinterpreted), not 2. In more complex cases mismatched types like this can crash the process instead.

Explain, in one or two sentences, why an extension module and a pure-Python module can be imported with the exact same import statement.

Python's import system doesn't care whether a module's code is Python bytecode or compiled native code loaded via a shared library loader — both present themselves as importable modules with the same module protocol, so the caller's import syntax is identical either way.

import ctypes
lib = ctypes.CDLL("libm.so.6")
print(lib.sqrt(4))