22Environments & dependencies — venv, pip, reproducibility

In Chapter 21 we organized our own code into packages — modules nested into a clean import tree we controlled end to end. Now that tidy code has to leave home and be useful. Useful means running thousands of files we did not write: other people's libraries, each pinned to versions that quietly fight one another. Here's the plan. First we'll watch that fight break out on a single shared site-packages. Then we'll build the one thing that ends it: the virtual environment. We'll open up pyvenv.cfg, the three-line contract that turns one shared interpreter into an isolated one. We'll watch pip unzip a wheel into a folder and write the receipt that lets it uninstall the exact same files again. We'll watch its resolver search a space of version numbers the way a SAT solver hunts for a satisfying assignment. And we'll climb the reproducibility ladder from a loose name up to a hash-verified lockfile. The whole way through, one question is the one that matters — what actually crosses the boundary into your project, and how do we make it the same bytes every single time? By the end, a venv stops being a spell you chant and becomes what it plainly is: a directory of inert files and a PATH string we reordered ourselves. No service, no magic — just the machine doing exactly what we told it to.

01The need: global site-packages starts a version war

Let's start where the trouble is born — a tool that just works. You type pip install requests, a second passes, and import requests answers. So you keep going. Weeks later you start a second project that wants a newer requests. You type pip install requests again, and it too says done. Nothing warned you. Nothing asked. But now go back to the first project and run it — and watch it die deep inside a library you never touched: ImportError: cannot import name ... from urllib3. You edited none of A's files. You changed nothing in A. And yet A is broken.

Here's the collision, concretely. Project A needs requests 2.20, and requests 2.20 pins its own dependency: urllib3<1.25. Project B needs requests 2.31, which demands urllib3>=1.26. So two projects have two flatly incompatible requirements for a third library that neither of them names out loud. The second install didn't sit politely beside the first — it overwrote it. That's because on the whole machine there is exactly one directory named requests, and exactly one named urllib3, and an install writes into that one directory.

To see why, watch where imports actually look. When you write import requests, Python doesn't scan your disk. It walks one specific ordered list of directories — sys.path — from left to right, and binds the first requests/__init__.py it finds. One entry on that list is site-packages: the single folder where installed third-party code lives.

>>> import sys, requests
>>> requests.__file__
'/usr/lib/python3.11/site-packages/requests/__init__.py'
>>> for p in sys.path: print(p)

/home/you/project
/usr/lib/python3.11
/usr/lib/python3.11/site-packages   # <- requests was found HERE, the first hit

Look hard at what that resolution is. Import is name-only. The string "requests" is the entire key; there is no version anywhere in the lookup. The filesystem has no slot for "requests 2.20" sitting next to "requests 2.31" — a directory is either named requests or it is not. So site-packages is, precisely, a key/value store with one physical slot per top-level import name. And pip install is a mutating write into that store. To put 2.31 in, it first deletes the files 2.20 recorded, then unpacks the new files in their place. There is never an instant where both versions coexist. The store simply has no room for two.

This is why no amount of care saves you. It is not that pip is careless, or that you forgot a flag. The data model itself has nowhere to put a second version — one hole, and you are trying to seat two pigeons. One global mutable directory, N projects, mutually incompatible pins: the shape guarantees a loser. On a system Python the wound goes deeper. There, site-packages is root-owned, and the operating system's own tooling — apt, dnf, the package manager that installs your entire OS — imports out of it. A stray sudo pip install that bumps a shared library can leave the machine unable to manage its own packages. You reach for a Python tool and break the system's own.

If one global directory is the disease, the cure writes itself: give every project its own site-packages so the single-slot store is no longer shared. That private directory has a name — a virtual environment — and the next section opens one up to find it is nothing but files. →

02What a venv actually is

People chant "activate your venv" like an incantation. Then python still points at the wrong interpreter after they've said the magic words, and they have no model to debug it with. So let's drain the magic out completely. A virtual environment is a directory of files on disk — inert data, not a running service. Nothing about it is live until you act. Once you can picture the files, "activate" stops being a spell and becomes something you can inspect, break, and fix.

Create one with python -m venv .venv. That command writes a concrete, boring tree:

$ python -m venv .venv
$ ls -F .venv
bin/            # Scripts\ on Windows
include/
lib/
pyvenv.cfg      # a 3-line text file — the whole contract (next section)

$ ls -F .venv/bin
activate        # an ordinary shell script
pip*
python -> /usr/bin/python3.11   # a SYMLINK to the base interpreter (POSIX)

$ ls .venv/lib/python3.11/site-packages
# ...empty. Just pip and setuptools bookkeeping. Nothing else yet.

Read that tree closely. The python inside bin/ is not a fresh copy of Python. On POSIX it is a symlink to the base interpreter you already had; on Windows it is a tiny launcher stub. There is a pyvenv.cfg text file at the root. And there is an empty site-packages/ — that emptiness is the whole point of section 1's fix. Most important of all: nothing is running. There is no daemon, no background process, no environment variable set. It is just a folder. It will sit there unchanged whether you "activate" it or not.

So what does activation do? source .venv/bin/activate runs an ordinary shell script that performs exactly four boring steps. It saves your current $PATH and prompt into backup variables. It prepends .venv/bin to the front of $PATH. It exports VIRTUAL_ENV=/path/to/.venv. And it tells the shell to rehash its cache of command locations. That is the entire trick.

# before
$ echo $PATH
/usr/bin:/bin

$ source .venv/bin/activate

# after — one directory spliced onto the front
(.venv) $ echo $PATH
/proj/.venv/bin:/usr/bin:/bin

(.venv) $ which python
/proj/.venv/bin/python        # the shell scans PATH left-to-right, stops at the first hit

After activation, typing python makes the shell walk $PATH from left to right and stop at the first directory that contains a python — now .venv/bin, because you just moved it to the front. The (.venv) in your prompt is cosmetic, painted on by the same script. And deactivate is the mirror image: it restores the saved $PATH and prompt and unsets VIRTUAL_ENV. There is no state to tear down, because there was no state — only a string that got reordered and then put back.

But this raises a sharp question. PATH only decides which python binary runs — and that binary is a symlink to the same base interpreter you already had. So how does it know to use the venv's empty site-packages instead of the base one, while still finding the standard library back in the base install? The isolation can't come from PATH. It has to happen inside the interpreter, at startup. →

03How the interpreter finds its packages

PATH chose which python runs — but that binary is a symlink to the base interpreter, byte-for-byte the same executable you had before you ever made a venv. Run it, and somehow it must use the venv's empty site-packages while still importing the standard library, which physically lives back in the base installation. Two different roots, one shared binary. The isolation cannot come from PATH, and it cannot come from the executable, because they're identical. It has to be computed by the interpreter itself, in the first milliseconds of startup. This is PEP 405, and it is exact.

At startup CPython must compute sys.prefix — its answer to "where do I find the standard library and site-packages?" To do it, the running python looks in its own directory and the parent directory for a file named pyvenv.cfg. That file is the entire contract, and it is three lines:

home = /usr/bin
include-system-site-packages = false
version = 3.11.4

If that file is present, the interpreter flips into venv mode. It sets sys.prefix to the venv root, so its site-packages resolves to .venv/lib/python3.11/site-packages — the empty directory. Then it reads the home key, which points back at the base install's binary directory. From home it locates the real installation by finding a landmark module (os.py), and sets sys.base_prefix to that base root. The standard library is then sourced from base_prefix; the third-party site-packages is sourced from prefix. The include-system-site-packages = false key decides whether the base's site-packages is also tacked on — normally not, for clean isolation. Finally site.py runs and assembles the two into sys.path.

>>> import sys
>>> sys.prefix        # where site-packages comes from
'/proj/.venv'
>>> sys.base_prefix   # where the stdlib comes from
'/usr'
>>> # outside a venv these two are EQUAL. Inside one, they differ.

So the final sys.path is a two-color stack. The standard-library entries come from base_prefix, the shared base install. And a single third-party entry — the venv's own site-packages — comes from prefix. The base's site-packages is skipped. That is how one shared binary serves an isolated environment: the stdlib is borrowed from home, and the packages are private.

The environment is now isolated, and its site-packages is empty. Time to fill it. When you type pip install rich it's importable in a second; pip install numpy sometimes grinds for a minute and fires up a C compiler. Same command, wildly different work — what physically lands in that directory? →

04pip installs wheels into site-packages

pip install rich finishes before you've let go of Enter, and import rich works. pip install numpy sometimes chews for a minute, prints "Building wheel...", and spins up a C compiler. Same command, two utterly different machines behind it. You deserve to know exactly what happens to that empty site-packages directory during an install — and why one case is a plain file-copy and the other is a build.

A wheel (.whl) is nothing exotic: it is a ZIP archive with a standardized filename and internal layout. The filename alone encodes compatibility, in five dash-separated fields:

rich-13.7.0-py3-none-any.whl
     |      |   |    |__ platform tag: any OS      -> pure python
     |      |   |______ abi tag:      none
     |      |__________ python tag:   py3 (any Python 3)
     |_________________ version

numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.whl
             |     |     |__ platform: Linux x86_64  -> COMPILED binary
             |     |________ abi: CPython 3.11 ABI
             |______________ python: CPython 3.11 only

Inside the ZIP are the package directories plus one <name>-<version>.dist-info/ folder of bookkeeping: METADATA (name, version, and the all-important Requires-Dist lines), RECORD (every installed file with its sha256 and byte size), WHEEL, and entry_points.txt. Installing a pure-python wheel is mechanically trivial. Unzip the files into site-packages, write the RECORD, generate console-script shims from entry_points, and optionally byte-compile .pyc files. No build step. No arbitrary code runs. That is exactly why it is fast — and why it is safe.

>>> import zipfile
>>> z = zipfile.ZipFile("rich-13.7.0-py3-none-any.whl")
>>> z.namelist()[:4]
['rich/__init__.py', 'rich/console.py', 'rich/text.py',
 'rich-13.7.0.dist-info/RECORD']
>>> print(z.read("rich-13.7.0.dist-info/RECORD").decode()[:90])
rich/__init__.py,sha256=Q...,6072
rich/console.py,sha256=v...,99123     # path, hash, size — the receipt

The contrast is the sdist (.tar.gz): raw source plus a build backend (setup.py or pyproject.toml). Installing an sdist means pip must first build a wheel — run the backend, possibly invoke a C compiler, link against system libraries — and only then unpack the wheel it just made. That is numpy's lost minute: it isn't downloading slowly, it's compiling. And the platform tag told you which path you were on before anything ran. none-any means pure python, just files; cp311-...-manylinux means a precompiled binary for one interpreter ABI and OS.

And uninstall is the mirror image, which is why it is reliable: pip reads RECORD and deletes exactly the files it lists — no guessing, no leftovers. So site-packages is fully demystified. It is nothing but unpacked wheel files plus dist-info bookkeeping. The one-second install and the one-minute compile are the same act — get a wheel, unzip it — differing only in whether the wheel already existed or had to be built first.

Unzipping a wheel is easy. The hard part came one step earlier: which version of each package should pip download? Ask for two libraries and pip must pick one version of each so that everyone's Requires-Dist pins agree at once. That is not a lookup. It is a search — and sometimes it hangs. →

05Dependency resolution is a constraint problem

You run pip install flask some-analytics-lib and pip stalls, printing "This is taking longer than usual... backtracking", sometimes ending in a wall of text titled ResolutionImpossible. It feels like it should just grab the latest of each and be done. So why is choosing version numbers genuinely hard — and why does "no solution exists" ever happen, instead of a clean error? Because resolution is search, not lookup, and this section makes you see the search.

Every release declares its own requirements as version constraints, in the Requires-Dist lines you saw in METADATA:

Requires-Dist: urllib3>=1.21.1,<3
Requires-Dist: certifi>=2017.4.17
Requires-Dist: charset-normalizer>=2,<4
Requires-Dist: idna>=2.5,<4

Your top-level requirements, plus every transitive dependency's requirements, form a system of constraints over version variables — one variable per package, each with an allowed set of versions. And because site-packages holds exactly one version per name (section 1's root cause, back to collect its debt), the resolver must find a single assignment. It has to pick one version for every package such that every constraint on it holds simultaneously. That is a constraint-satisfaction problem — a SAT problem in disguise.

pip's resolver (a library called resolvelib) does backtracking search. Choose a candidate version for a package. Propagate its constraints to narrow the allowed sets of the others. If any package's allowed set collapses to empty, backtrack — undo the most recent choice, drop to the next-older version, and try again. Repeat until a consistent assignment is found, or the search space is exhausted.

def resolve(work, chosen, allowed):
    if not work:
        return chosen                 # every package assigned -> success
    pkg, rest = work[0], work[1:]
    for ver in newest_first(allowed[pkg]):
        narrowed = apply_deps(index[pkg][ver], allowed)
        if narrowed is None:      # some set went empty -> dead end
            continue               # BACKTRACK: try the next-older version
        got = resolve(rest, {**chosen, pkg: ver}, narrowed)
        if got: return got
    return None                     # no candidate works -> ResolutionImpossible

Worst case, that search is exponential — which is exactly why pip can hang. It is not stuck; it is exploring a combinatorial tree of version choices. Historically the pain was worse: to read a version's Requires-Dist, pip sometimes had to download that version's wheel or sdist first, so every probe carried heavy network I/O. And ResolutionImpossible is not a crash — it is a proof. It means the constraint set is unsatisfiable. Package A needs C<2 while package B needs C>=2, so no single version of C can satisfy both, and no amount of backtracking will conjure one.

The classic pinch point is the diamond: your project depends on A and B, and both of them depend on a shared C. If A wants C>=1,<2 and B wants C>=1,<3, the overlap [1,2) holds a solution. Tighten B to C>=2 and the overlap vanishes — the diamond becomes unsatisfiable, and pip hands you the proof.

Say the resolver wins and prints a clean set. There's a catch that section 6 exists to close: it re-solves every time, against an index that keeps growing new releases. Same requirements file, different day, different answer. To freeze the exact set forever — the same bytes on every machine — you need a lockfile. →

06Reproducibility: from requirements to a lockfile

"Works on my machine." Your requirements.txt says one word: flask. Six months later a teammate runs the same file on a fresh laptop. The resolver picks flask 3.1 instead of your 2.3, a transitive dependency crosses a major version, and CI turns red. Identical file, different bytes. The install has to become byte-for-byte identical everywhere and every time — and you deserve to know precisely which artifact delivers that guarantee. It is a ladder, and we climb it rung by rung.

Rung 1 — a loose name. flask means "whatever the resolver picks at install time." It is non-deterministic across time, because PyPI keeps gaining new releases and section 5's resolver re-solves against that moving index on every install. Nothing is pinned; the ground moves under you.

Rung 2 — pin the direct dependency. flask==2.3.3 freezes your top level, but flask's own dependencies — jinja2, werkzeug, click, markupsafe — still float. A werkzeug point release with a behavior change can still slip in. You've nailed the trunk and left the branches loose.

Rung 3 — a fully-resolved lockfile. pip freeze, or pip-tools' compiled requirements.txt, or uv.lock / poetry.lock, pins every package in the resolved graph to an exact version. This does something profound: it bypasses the resolver entirely at install time. The install stops being a search and becomes a straight download-and-unpack of a known set. Same graph, every machine, every day — because there is no longer any choice left to make.

Rung 4 — content hashes. Add --hash=sha256:... to each line. Before installing, pip computes the sha256 of the downloaded artifact and refuses to proceed unless it matches the recorded hash. The install becomes tamper-evident: a swapped PyPI file, a poisoned mirror, a man-in-the-middle — all rejected at the gate.

flask==2.3.3 \
    --hash=sha256:09c347a92aa7ff4a8e7f3206795f30d826654baf38b873d0744cd571ca609efc
werkzeug==2.3.7 \
    --hash=sha256:2b8c0e447b4b9dbcc85dd97b6eeb4dcbaf6c8b6c00f7bf1932d2b1b0b0b0b0b0
jinja2==3.1.2 \
    --hash=sha256:6088930bfe239f0e6710546ab9c19c9ef35e29792895fed6e6e31a023a182a61
# every package pinned; every artifact verified before it lands

Notice the layering. Pinned versions give you the same graph. The lockfile gives you the same files. The hashes give you the same, verified, bytes. Three independent guarantees, stacked — and this is where the chapter's law finally shows its full face. What crosses the boundary into your environment is always bytes, and reproducibility is nothing but the discipline of proving those bytes are the ones you meant.

The professional discipline is two files, not one. An abstract requirements.in holds what you want — flask, loosely. You compile it once, with the resolver, into a locked requirements.txt that spells out exactly what you get — the full pinned, hashed closure. You regenerate that lockfile only when you deliberately choose to upgrade, never as a silent side effect of an install. uv and Poetry bake this split into their *.lock files; pip-tools does it with pip-compile. That is the exact moment when "works on my machine" dies.

You've closed the last leak. The venv gave each project a private shelf; pyvenv.cfg made it isolated; the wheel is bytes on disk; the resolver chose a consistent set; the lockfile froze those exact bytes onto every machine. The same law governed all of it — objects live inside one process, but only bytes cross a boundary. The next boundary this volume crosses is the one where your program stops being alone and starts running beside others — other cores, other processes — and the same flatten-and-rebuild machine will be waiting. →