Why does pip compile a package with a C extension from source, and what does that build need?

juniormust knowfreq 55%

answer

Nothing on the index matched this machine
pip fell back to the source archive
A compile needs a toolchain present
The header CPython ships: Python.h
sysconfig reports the include directory

basics

~20 s

Because no prebuilt wheel matched this machine's interpreter version, ABI and platform, pip fell back to the source archive. Compiling it needs a C compiler and linker, CPython's development headers (Python.h), and the project's declared build backend.

solid answer

~40 s

A wheel is a zip that is installed by copying files into place, so it carries an already-compiled extension. Its filename records which interpreter, ABI and platform it is valid for; pip computes the tags the running interpreter supports and picks a match. When nothing matches — a brand-new interpreter minor version, an unusual architecture, a musl userspace, a free-threaded build — pip falls back to the source distribution and builds it here. That build needs a C toolchain, CPython's development headers (Python.h, under sysconfig.get_paths()["include"], split into a separate development package on most Linux distributions), plus headers for any external C library the extension includes. In production the safest posture is to refuse the fallback: pip install --only-binary=:all: fails loudly instead of quietly dragging a compiler into your runtime image.

A wheel is a zip that is installed by copying files into place, so it carries an already-compiled extension. Its filename records which interpreter, ABI and platform it is valid for; pip computes the tags the running interpreter supports and picks a match. When nothing matches — a brand-new interpreter minor version, an unusual architecture, a musl userspace, a free-threaded build — pip falls back to the source distribution and builds it here. That build needs a C toolchain, CPython's development headers (`Python.h`, under `sysconfig.get_paths()["include"]`, split into a separate development package on most Linux distributions), plus headers for any external C library the extension includes. In production the safest posture is to refuse the fallback: `pip install --only-binary=:all:` fails loudly instead of quietly dragging a compiler into your runtime image.

code

python · 6 lines

python

import os
import sysconfig

include_dir = sysconfig.get_paths()["include"]
print(include_dir)
print(os.path.exists(os.path.join(include_dir, "Python.h")))

go deeper

for a junior

Be ready to say what happened: no prebuilt wheel matched this interpreter and platform, so pip downloaded the source archive and tried to compile it. Know that compiling needs tools the machine may not have.

for a middle

Explain the mechanics: pip matches wheel tags first, and a source build needs a C toolchain plus CPython's development headers, whose directory sysconfig.get_paths() reports. Name the distinct failure messages and their separate fixes.

for a senior

Show you keep this out of production: assert binary-only installs so a missing wheel fails fast rather than pulling a compiler into a runtime image, and explain why wheels lag a fresh interpreter release.

for a principal

Own the policy: which interpreter versions, architectures and userspaces your organisation supports, whether any host may compile at install time, and what the plan is when a needed dependency publishes no wheel for a target you require.

Two shapes of distribution

A project on an index is published in two forms. A wheel (.whl) is a zip archive whose contents are already laid out the way they will sit in site-packages; installing one is a copy plus some metadata bookkeeping, with no build step at all. A source distribution (sdist, .tar.gz) is the project's source tree, and installing one means building it first. For a pure-Python project that difference is mostly speed. For a project containing a C extension it is the difference between copying an already-compiled .so/.pyd and running a compiler on the machine doing the install.

How pip decides

A wheel's filename encodes which interpreter, ABI and platform the archive is valid for. pip computes the set of tags the running interpreter supports and picks the best matching wheel. If nothing matches, it falls back to the sdist and builds it. That is why the same pip install is instantaneous on your laptop and prints pages of compiler output on a colleague's box. The usual reasons no wheel matches:

a Python minor version released more recently than the maintainer's last build run;
an architecture the project does not build for;
a Linux userspace the published wheels do not target;
a free-threaded interpreter, which is a different ABI from the ordinary build and therefore needs its own wheels — that build became officially supported in 3.14, so many projects are still filling that leg in.

What the source build actually needs

A build backend. Under PEP 517 the installer reads [build-system] from pyproject.toml, creates an isolated environment, installs the listed requirements into it and calls the backend's hooks. Nothing is implicitly available: Python 3.12 removed distutils from the standard library, so a project whose build relied on the stdlib copy fails on 3.12 and later unless it declares a backend that supplies its own.
A C compiler and linker. The distribution's C toolchain on Linux, the command line developer tools on macOS, and on Windows the Microsoft C++ build tools whose absence produces the familiar "Microsoft Visual C++ 14.0 or greater is required" message.
CPython's development headers. The extension includes Python.h, which lives in the interpreter's include directory — sysconfig.get_paths()["include"] will print it. Interpreters from an installer or a version manager ship the headers; Linux distributions split them into a separate development package, which is why fatal error: Python.h: No such file or directory is the single most common source-build failure on a server. The trap is that the headers must belong to the same interpreter you are installing into, not to whichever Python the distribution treats as default.
Headers and link libraries for any external C library the extension binds to, plus whatever else the build itself wants — a make, a cmake, sometimes a Rust toolchain for a project with a Rust extension module.

Reading the failure

Three failure shapes, three different fixes, and none of them is a bug in the package. The build frontend cannot find a compiler at all: install the toolchain. Python.h is missing: install the interpreter's development package. A third-party header is missing, or the link step reports an unresolved symbol: install that library's development package, or find a wheel that has it vendored already.

The fixes, in the order to reach for them

Prefer not compiling. Upgrade pip so it recognises newer tags; install into an interpreter version the project actually publishes wheels for; after a new Python release, give maintainers time to extend their build matrix rather than compiling their code yourself. If you must build, install the toolchain and the development headers. If you already have the build requirements installed and do not want an isolated environment resolving its own, pip install --no-build-isolation uses the ambient environment instead.

Make the fallback explicit

The dangerous case is the silent one: a deployment that used to install a wheel starts compiling because a wheel went missing, so now the host needs a compiler, the install takes minutes instead of seconds, and the artefact is not the one you tested. pip install --only-binary=:all: refuses source builds and fails immediately; --no-binary=:all: forces the opposite, which is occasionally what a distribution packager wants. Asserting binary-only installs, and keeping a compiler out of the runtime image, turns a slow surprise into a fast and obvious error.

## Two shapes of distribution A project on an index is published in two forms. A **wheel** (`.whl`) is a zip archive whose contents are already laid out the way they will sit in `site-packages`; installing one is a copy plus some metadata bookkeeping, with no build step at all. A **source distribution** (sdist, `.tar.gz`) is the project's source tree, and installing one means *building* it first. For a pure-Python project that difference is mostly speed. For a project containing a C extension it is the difference between copying an already-compiled `.so`/`.pyd` and running a compiler on the machine doing the install. ## How pip decides A wheel's filename encodes which interpreter, ABI and platform the archive is valid for. pip computes the set of tags the running interpreter supports and picks the best matching wheel. If nothing matches, it falls back to the sdist and builds it. That is why the same `pip install` is instantaneous on your laptop and prints pages of compiler output on a colleague's box. The usual reasons no wheel matches: - a Python minor version released more recently than the maintainer's last build run; - an architecture the project does not build for; - a Linux userspace the published wheels do not target; - a **free-threaded** interpreter, which is a different ABI from the ordinary build and therefore needs its own wheels — that build became officially supported in 3.14, so many projects are still filling that leg in. ## What the source build actually needs 1. **A build backend.** Under PEP 517 the installer reads `[build-system]` from `pyproject.toml`, creates an isolated environment, installs the listed requirements into it and calls the backend's hooks. Nothing is implicitly available: Python 3.12 removed `distutils` from the standard library, so a project whose build relied on the stdlib copy fails on 3.12 and later unless it declares a backend that supplies its own. 2. **A C compiler and linker.** The distribution's C toolchain on Linux, the command line developer tools on macOS, and on Windows the Microsoft C++ build tools whose absence produces the familiar "Microsoft Visual C++ 14.0 or greater is required" message. 3. **CPython's development headers.** The extension includes `Python.h`, which lives in the interpreter's include directory — `sysconfig.get_paths()["include"]` will print it. Interpreters from an installer or a version manager ship the headers; Linux distributions split them into a separate development package, which is why `fatal error: Python.h: No such file or directory` is the single most common source-build failure on a server. The trap is that the headers must belong to the *same* interpreter you are installing into, not to whichever Python the distribution treats as default. 4. **Headers and link libraries for any external C library** the extension binds to, plus whatever else the build itself wants — a `make`, a `cmake`, sometimes a Rust toolchain for a project with a Rust extension module. ## Reading the failure Three failure shapes, three different fixes, and none of them is a bug in the package. The build frontend cannot find a compiler at all: install the toolchain. `Python.h` is missing: install the interpreter's development package. A third-party header is missing, or the link step reports an unresolved symbol: install that library's development package, or find a wheel that has it vendored already. ## The fixes, in the order to reach for them Prefer *not compiling*. Upgrade pip so it recognises newer tags; install into an interpreter version the project actually publishes wheels for; after a new Python release, give maintainers time to extend their build matrix rather than compiling their code yourself. If you must build, install the toolchain and the development headers. If you already have the build requirements installed and do not want an isolated environment resolving its own, `pip install --no-build-isolation` uses the ambient environment instead. ## Make the fallback explicit The dangerous case is the silent one: a deployment that used to install a wheel starts compiling because a wheel went missing, so now the host needs a compiler, the install takes minutes instead of seconds, and the artefact is not the one you tested. `pip install --only-binary=:all:` refuses source builds and fails immediately; `--no-binary=:all:` forces the opposite, which is occasionally what a distribution packager wants. Asserting binary-only installs, and keeping a compiler out of the runtime image, turns a slow surprise into a fast and obvious error.

How do you make an install fail immediately instead of quietly compiling from source?
Pass `--only-binary=:all:` to pip, which refuses to build any sdist and errors out when no wheel matches. Do that in production image builds and in CI so a missing wheel is a loud, fast failure rather than a slow build that silently changes what you ship. `--no-binary=:all:` is the mirror image, forcing source builds everywhere.
Why do wheels for a brand-new Python release often go missing for weeks?
Each CPython minor version is a new ABI for an ordinary extension, so maintainers must add that interpreter to their build matrix and rebuild every platform leg. Until that release run happens there is no matching wheel and installers fall back to the sdist. Projects that ship a stable-ABI wheel avoid the per-version rebuild and are usable on the new release immediately.
Where do the build requirements go while pip compiles the package?
Into a throwaway isolated environment that pip creates for the build, populated from `[build-system] requires` in `pyproject.toml`. It is discarded afterwards and its contents never reach the target environment, which is why a build requirement is not automatically an install requirement.

A wheel is flat-pack furniture that arrives assembled for your exact room; an sdist is the raw timber, and if nobody cut a piece for your room you need the saw, the bench and the plans yourself.

saying these in an interview costs you the question

Believes pip always downloads a prebuilt binary
Thinks a compiler alone is enough, forgetting the headers
Assumes any wheel on an index works on any machine
Confuses the runtime interpreter package with its development package
Treats every compile error as a bug in the package

Why does pip compile a package with a C extension from source, and what does that build need?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Two shapes of distribution

How pip decides

What the source build actually needs

Reading the failure

The fixes, in the order to reach for them

Make the fallback explicit

▾common follow-ups3 questions

▾analogy~20 s

▾red flags5 flags

§ context

▾related questions5 questions

▾related topics4 topics

Two shapes of distribution

How pip decides

What the source build actually needs

Reading the failure

The fixes, in the order to reach for them

Make the fallback explicit

answer

go deeper

context