20Inside import — sys.path, finders, loaders, and .pyc

In Chapter 19 we met import from the outside — four honest verbs, find, compile, exec, cache, named but still half-trusted. In this chapter we pry the lid off and watch each one actually run. Here is the plan. First we print sys.path, the ordered list of folders import walks, and watch a stray random.py in your own directory beat the entire standard library. Then we follow the winning file through the finder/loader handshake, into the .pyc bytecode cache with its sixteen-byte header, and out the other side as a live module object. The whole way through, we keep asking the one question that matters — when you type import X, what exactly decides which bytes on disk become the module you get back? By the end, import is no longer a keyword you trust but a machine you can single-step: an ordered search you can print, a finder/loader protocol you could implement yourself, and a compile pipeline that turns the text you type into the same bytecode the Volume 1 VM already knows how to run. It is the same story as every boundary in this volume. What crosses from disk into your process is always bytes, and import is the machine that rebuilds live objects on the near side.

01sys.path: an ordered search, first match wins

Let's start with the bug everyone hits. You're debugging something fiddly with randomness, so you make a throwaway file to poke at it and name it, reasonably, random.py. You run your real program and it breaks: AttributeError: module 'random' has no attribute 'random'. You did not touch the standard library. You did not mistype. And yet import random has handed you your file instead of Python's. The same code that works on your laptop will break the moment it runs in a directory that does not contain your little file. Every Python programmer meets this bug, stares at it for an hour, and walks away convinced import is haunted. It is not. Import is a deterministic walk down a list you can print, and the moment you can see the list, the haunting stops for good.

Here is the whole secret in one sentence: import X does not search your disk and it does not know where the standard library lives. It walks sys.path — a plain list[str] of directory names — from top to bottom. At each location it asks the same blunt question, "do you have X?", and it stops at the first one that says yes. First match wins. That rule is the entire mystery. So print the list and look at the thing that has been deciding your imports all along:

>>> import sys
>>> for i, p in enumerate(sys.path):
...     print(i, repr(p))
0 ''                                  # cwd / the script's own folder — checked FIRST
1 'C:\\Python312\\python312.zip'
2 'C:\\Python312\\Lib'                # the standard library lives here
3 'C:\\Python312'
4 'C:\\Python312\\Lib\\site-packages'  # where pip installs land

Read the entries in the order the interpreter built them at startup, because the order is the rule. Index 0 is the special one. Run python foo.py and it becomes the folder that foo.py lives in; run -c, or -m, or the bare REPL and it becomes '', meaning the current working directory. That single entry is checked before everything else, which is exactly why a sibling random.py beats the standard library: your project folder is first in line. After it come the $PYTHONPATH entries you set in the environment. Then the standard-library directories and the frozen modules baked into the executable. And finally the site-packages directory that the site module appends, where every pip install lands. Top to bottom, first match wins, no exceptions.

The most useful fact is that sys.path is just data. It is a list. You can sys.path.insert(0, ...), .append(...), or reorder it at runtime, and the very next import obeys the new order — import re-reads the list on every call and never snapshots it. But before the walk even begins, import checks a faster cache: sys.modules, the dictionary of everything already imported in this process. A hit there means no search at all; the module object is returned instantly. That is why the second import json is nearly free, and why a module's top-level code runs exactly once per process no matter how many files import it.

>>> import random
>>> random.__file__
'C:\\Users\\you\\project\\random.py'   # not the stdlib — YOUR file won the search
>>> del sys.modules['random']        # evict the wrong module from the cache
>>> # now delete your local random.py, then import again → it falls through to the stdlib

The walk finds a directory that can produce the name — but who decides whether a folder "has" the module, and who turns the winning file into a live object? Those are two different jobs, done by two different kinds of object, and pulling them apart is what turns import from a keyword into a machine. Finders and loaders →

02Finders and loaders: the import protocol

You have watched import pull a module out of a .zip file. You have seen a Jupyter notebook import a sibling notebook that has no .py at all. If import were simply "open a .py and run it," none of that could exist. So there must be a seam: a place where the question "how do we find this?" is cleanly separated from "how do we build it?" Only a separated, swappable design could serve modules from a zip, a notebook, a database row, or thin air. Finding that seam is what turns import from a magic word into a machine with named parts you could implement yourself.

The real shape of import X is a two-phase protocol driven by importlib. Phase FIND: the machinery iterates sys.meta_path, a list of meta path finders, and asks each one find_spec(name, path, target). A finder returns either None — "not mine, ask the next one" — or a ModuleSpec. The three defaults, in order, are BuiltinImporter (the C builtins from the last section), FrozenImporter (modules frozen into the binary), and PathFinder, the one that finally consults sys.path. Print the chain and it is exactly what the mechanism predicts:

>>> import sys, json
>>> sys.meta_path
[<class '_frozen_importlib.BuiltinImporter'>,
 <class '_frozen_importlib.FrozenImporter'>,
 <class '_frozen_importlib_external.PathFinder'>]
>>> json.__spec__
ModuleSpec(name='json', loader=<...SourceFileLoader...>,
           origin='C:\\Python312\\Lib\\json\\__init__.py',
           submodule_search_locations=['C:\\Python312\\Lib\\json'])

That ModuleSpec is the pivot of the whole system, the blueprint the rest of import consumes. Learn its fields, because they answer every "how did import know…" question you will ever have. .name is the dotted name. .loader is the object that will actually build the module. .origin is usually the source filename (or the literal string '(built-in)'). .submodule_search_locations is the __path__ a package hands to its children. .cached points at the .pyc. A finder's entire job is to fill in that form and hand it back.

Phase LOAD is where the spec becomes a living thing. Import calls importlib.util.module_from_spec(spec) to make an empty module object, then inserts it into sys.modules before running a single line of it. This is the subtle move that lets circular imports partially work: a module that imports back into a half-built one finds the half-built object already registered, rather than triggering an infinite re-import. Only then does import call spec.loader.exec_module(module), which runs the module's code object into the module's __dict__. The division of labour is clean: a finder answers "where and how," a loader does "create and exec." Nothing is hidden. You can perform the entire sequence by hand:

import importlib.util, sys

def manual_import(name):
    spec = importlib.util.find_spec(name)          # PHASE FIND → a ModuleSpec
    module = importlib.util.module_from_spec(spec)  # create the empty module
    sys.modules[name] = module                      # register BEFORE executing
    spec.loader.exec_module(module)                 # PHASE LOAD → run the body
    return module

textwrap = manual_import('textwrap')
print(textwrap.shorten("the whole import protocol, by hand", 20))

One layer remains, and it explains the zip trick. PathFinder does not read files itself. For each entry on sys.path it consults sys.path_hooks to obtain a path entry finder for that specific location: the default hook handles ordinary directories, another handles zip archives. It then memoises the result in sys.path_importer_cache, so it only computes it once per directory. So there are genuinely two finder layers: meta path finders are global strategies, and path entry finders are per-sys.path-entry workers. That is why a .zip on sys.path "just works": a path hook claims the archive and returns a finder that reads modules out of it.

The loader ran the module's code object — but where did that code object come from, and why is the second import so much faster than the first? The answer is a folder you have seen appear next to your code and always been slightly afraid to delete. __pycache__ →

03Source becomes bytecode; __pycache__ caches it

The first time you import a large library there is a noticeable pause; the second time it is instant. And sitting next to your code, unasked for, is a folder called __pycache__ full of .pyc files. You have wondered three things about it and never quite asked out loud: is it safe to delete, why does it exist, and did my source just get "compiled" like C? Answer those and you have explained startup time, a whole family of "stale bytecode" bugs, and exactly what Python is trading memory for speed on.

Python is compiled — just not to machine code. Every time a .py is imported, its source is compiled into a code object full of bytecode. (The script you run directly as __main__ is the exception: that one is never cached.) To avoid paying that compile cost on every future import, the loader marshals the code object to bytes and writes them to __pycache__/NAME.cpython-3XX.pyc. The whole trick lives in that file's header, so let us take it apart byte by byte. A modern .pyc begins with sixteen bytes: a 4-byte magic number that encodes the exact CPython version and bytecode format (this is why the filename is tagged cpython-312, and why a 3.11 .pyc is silently ignored by 3.12); a 4-byte bit-field of flags; and then, in the default timestamp scheme, the source's 4-byte modification time plus its 4-byte size.

Now the payoff. On the next import the loader reads only that header. It compares the magic number: a mismatch means a different interpreter, so it recompiles. If the magic matches, in timestamp mode it compares the stored mtime and size against a fresh os.stat of the source. If they match, it concludes "the source is unchanged, trust the cache," skips tokenizing, parsing and compiling entirely, and simply marshal.loads the code object straight back into memory to execute. That is why the second import is fast: the expensive front of the pipeline is skipped, and only unmarshal-plus-exec remain. Decode the header yourself and the machine stops being mysterious:

import struct, importlib.util, os

with open("__pycache__/mymod.cpython-312.pyc", "rb") as f:
    magic = f.read(4)
    flags = struct.unpack("<I", f.read(4))[0]
    mtime = struct.unpack("<I", f.read(4))[0]
    size  = struct.unpack("<I", f.read(4))[0]

print(magic == importlib.util.MAGIC_NUMBER)      # True — this interpreter's format
print(mtime, int(os.stat("mymod.py").st_mtime))  # equal → cache is trusted

Be exact about the failure mode, because it bites real builds. Timestamp invalidation trusts the filesystem clock. Restore an older source whose mtime is now newer than the cache, or build on a machine with a skewed clock, and Python can serve you stale bytecode: it runs code that no longer matches the source you are reading. That is exactly why hash-based .pyc files exist (PEP 552). When the flags bit is set, those mtime and size fields are replaced by a truncated SipHash of the source, and the loader validates by content rather than by clock. You opt in with py_compile using CHECKED_HASH (rehash and compare every time) or UNCHECKED_HASH (trust the hash blindly, for reproducible builds where the toolchain guarantees freshness).

The cache stores a code object — the compiled unit the loader executes. But where does that object come from? Between the characters you type and the bytecode the VM runs sit three transformations, and the next section freeze-frames every one of them. Text → AST → code object →

04The compile pipeline: source → AST → code object

Volume 1 handed you a virtual machine that eats bytecode and pushes values on a stack. But you do not write bytecode — you write text. Somewhere between the characters your fingers produce and the instructions the VM consumes, three transformations happen. While they stay invisible, the VM feels disconnected from the language you actually speak. This section closes the loop. It shows the exact assembly line that turns total = a + b into the LOAD_FAST/BINARY_OP/STORE_FAST instructions your Volume 1 stack machine already knows how to run, and it hands you two tools, ast and dis, to stop the line at any station and look.

Station 1 — TOKENIZE. The tokenizer sweeps the raw source string into a stream of typed tokens: NAME, OP, NUMBER, NEWLINE, and — the detail that explains a whole category of beginner pain — INDENT and DEDENT. Python's significant indentation is not a special case buried in the parser. It is literal tokens the lexer synthesises when the leading whitespace changes. That is why mixing tabs and spaces is a tokenizer-level error, thrown before the grammar is ever consulted. Watch the stream directly:

import tokenize, io
src = "total = a + b\n"
for tok in tokenize.generate_tokens(io.StringIO(src).readline):
    print(tokenize.tok_name[tok.type], repr(tok.string))
# NAME 'total' · OP '=' · NAME 'a' · OP '+' · NAME 'b' · NEWLINE '\n' · ENDMARKER ''

Station 2 — PARSE to an AST. Since 3.9, CPython uses a PEG parser that assembles those tokens into an Abstract Syntax Tree: a tree of typed nodes where precedence is baked into the shape and all formatting is gone. total = a + b becomes an Assign whose target is Name('total') and whose value is a BinOp(Name('a'), Add(), Name('b')). This is the same tree that decorators, linters, and this book's own tracer inspect — structure without a single stray space or comment:

import ast
print(ast.dump(ast.parse("total = a + b"), indent=2))
# Module(body=[
#   Assign(targets=[Name(id='total', ctx=Store())],
#          value=BinOp(left=Name(id='a'), op=Add(), right=Name(id='b')))])

Station 3 — COMPILE to a code object. The compiler walks the AST, builds a control-flow graph, and emits bytecode into a code object (types.CodeType). This is the real unit of executable Python, the thing a function has (func.__code__) and a module is. Its fields are the accumulated payload of everything upstream: co_code (the raw bytecode bytes), co_consts (literals and nested code objects), co_names (global and attribute names), co_varnames (locals), co_argcount, co_flags, and a line-number table mapping bytecode offsets back to source lines for tracebacks. Then dis.dis disassembles co_code into readable opcodes — and here the whole book snaps together, because these are the exact stack instructions from Volume 1:

import dis
def f(a, b):
    total = a + b
    return total
dis.dis(f)
# LOAD_FAST    a       push local a onto the value stack        stack: [a]
# LOAD_FAST    b       push local b                             stack: [a, b]
# BINARY_OP    +       pop two, push their sum                  stack: [445]
# STORE_FAST   total   pop into the local `total`               stack: []
# LOAD_FAST    total   push it again to return                  stack: [445]
# RETURN_VALUE         pop and hand it back to the caller

And now the tie that binds the chapter: compile(src, name, 'exec') is literally stations 1 through 3 in a single call, returning the code object. Import's compile step is this pipeline. The .pyc from the last section caches its station-3 output, and the loader from section two runs that output into a module's __dict__. One unbroken arc: text → tokens → AST → code object → (marshalled into a .pyc) → executed by the VM. compile even hands you a code object you can eval directly:

code = compile("a + b", "<demo>", "eval")   # tokenize + parse + compile
print(code.co_names)                       # ('a', 'b')
print(eval(code, {"a": 200, "b": 245}))    # 445 — Titanium meets Blinding Lights again

You now know how one file becomes a module. But a real codebase splits one logical package across many folders — and the rule you were taught, "a package needs an __init__.py," turns out to be only half true. Namespace packages →

05Namespace packages: folders without __init__.py

Someone taught you a firm rule: a package needs an __init__.py. Then you clone a large monorepo where import company.billing and import company.auth resolve to two completely different directories, with no __init__.py in sight, yet every import succeeds. Either the rule you learned is wrong, or something deeper is running underneath it. Something deeper is running underneath it, and it is exactly how big organisations split one logical package across many folders, repositories, and separately installed distributions.

There are two kinds of package, and the difference is one file. A regular package is a directory containing __init__.py. Importing it runs that file and sets the package's __path__ to a single-element list: the one directory it came from. A namespace package (PEP 420, since 3.3) is what you get when the import system finds directories matching the name across sys.path but none of them contains __init__.py. Here is the exact algorithm PathFinder runs while scanning for company. If an entry has company/__init__.py, that is a regular package, so stop. If an entry has a plain directory company/ with no __init__.py, record that directory as a portion and keep scanning the rest of sys.path, collecting every matching directory. If the scan ends without ever finding a regular package or a company.py module, import synthesises a namespace package whose __path__ is not a plain list but a live _NamespacePath object holding all the collected portions.

dirA/company/billing.py      # no __init__.py anywhere under company/
dirB/company/auth.py         # a second, separate portion of the same name

>>> import sys
>>> sys.path[:0] = ['dirA', 'dirB']
>>> import company.billing, company.auth   # both succeed
>>> company.__file__                        # None — no single file backs it
>>> type(company.__path__)
<class '_frozen_importlib_external._NamespacePath'>
>>> list(company.__path__)
['dirA/company', 'dirB/company']                 # both portions, merged

That merged __path__ is the entire mechanism. import company from dirA and import company.auth where auth lives in dirB both work because company.__path__ spans both directories, and submodule search walks every portion. Contrast the two sharply. A regular package is one directory that runs __init__.py — a real place to put initialisation code, an __all__, side effects, a version string. A namespace package is zero-or-many directories, runs no code, has __file__ set to None, and carries a live multi-location __path__ that is recomputed lazily as sys.path changes.

The design tradeoff is worth naming, because it is why the feature exists at all. Namespace packages let separately installed distributions contribute subpackages under one shared top-level name. The classic case is pip install azure-storage and pip install azure-identity landing in different site-packages folders yet both importable as azure.storage and azure.identity. You gain that composability, you lose a place to put package-init code, and you inherit a subtler hazard. An accidental empty folder named like a real package — created by a botched checkout, or left behind as a stray build artefact — can silently convert a working regular import into an empty namespace package and shadow the real one. The tell is always the same pair of fields:

>>> import json                 # a regular package
>>> json.__file__                # '.../Lib/json/__init__.py' — a real file
>>> json.__path__                # ['.../Lib/json'] — a plain list, one entry
>>> # namespace package: __file__ is None, type(__path__) is _NamespacePath

You have now seen finders on a list, a spec protocol, and a default filesystem implementation with two flavours of package. The obvious, powerful question remains: if finders are just objects on a list — can you add one? Import hooks →

06Import hooks: why the protocol is pluggable

You know the shape now: import is find_spec → module_from_spec → exec_module, and finders live on a plain Python list. Which raises the question this whole chapter has been walking toward: if it is just objects on a list, can I add my own? Yes. And that yes is the entire difference between a language feature you use and a mechanism you own. Import hooks are how tools import notebooks as modules, how packages get lazily loaded, and how encrypted or remote code gets fetched and run. More to the point, they are how you would build any of those yourself, in pure Python, at runtime.

There are two extension points, and choosing between them is choosing your blast radius. Meta path hooks are the sledgehammer. Put a finder object on sys.meta_path and, because that list is consulted first and for every single import, your finder can intercept any name from any source — a database row, a URL, an in-memory dictionary, a decrypted blob. It needs one method, find_spec(name, path, target), returning a ModuleSpec or None to politely decline. Here is the minimal shape: a finder that serves modules whose source is just a string in memory, with no file behind them at all:

import sys, importlib.util, importlib.abc

SOURCES = {"greet": "def hello():\n    return 'hi from memory'\n"}

class DictLoader(importlib.abc.Loader):
    def __init__(self, src): self.src = src
    def create_module(self, spec): return None          # use the default empty module
    def exec_module(self, module):
        code = compile(self.src, module.__name__, "exec")  # the section-4 pipeline!
        exec(code, module.__dict__)                     # run into the module namespace

class DictFinder(importlib.abc.MetaPathFinder):
    def find_spec(self, name, path, target=None):
        if name in SOURCES:
            return importlib.util.spec_from_loader(name, DictLoader(SOURCES[name]))
        return None                                    # decline → the next finder is asked

sys.meta_path.insert(0, DictFinder())
import greet                                        # no file on disk exists
print(greet.hello())                                 # hi from memory

Notice what you just did. Import fetched source from a Python dict, ran it through compile — the exact pipeline from section four — and executed the resulting code object into a module's __dict__, the exact load phase from section two. Every piece of this chapter reappears, now under your control.

The second extension point, path hooks, is the scalpel. Put a callable on sys.path_hooks. Each hook is handed a sys.path entry string and either returns a path entry finder for it or raises ImportError to decline. This only affects things reachable through sys.path entries, which is exactly how zipimport works. The zipimporter is a path hook that claims .zip entries, and the sys.path_importer_cache memoises the mapping so the archive is opened once. That is the full explanation of a trick that has always looked like magic:

>>> import sys
>>> sys.path.insert(0, "libs.zip")   # a path to an ARCHIVE, not a folder
>>> import packed_module            # read straight out of the zip
>>> sys.path_importer_cache["libs.zip"]
<zipimporter object "libs.zip">      # the path hook that claimed it

Hold the precedence in your head as one ordered pipeline. The sys.modules cache comes first. Then the sys.meta_path finders in order — yours can go at the front to override, or at the back to be a fallback. PathFinder is itself just the last meta path finder, and it fans out to sys.path_hooks per entry. For correctness, a well-behaved loader sets module.__spec__, __name__, __loader__, and __file__ (or leaves __file__ unset for non-file sources) and executes into module.__dict__ via exec_module. The machinery, not your loader, handles inserting into sys.modules and unwinding on error.

You can now see, extend, and even hijack the machine that turns disk bytes into live modules. Next: once a module is live, what happens when your program needs to send its objects to another process — where nothing but bytes can cross? Serialization and the pickle pipe →