19Modules & import — find, compile, exec, cache

All through Volume 1 we lived inside a single process. One file, read top to bottom by the interpreter, every object at a real address, every name pointing straight at it. In this chapter your code outgrows that one file. The moment code here wants to reuse code over there, you hit the first real boundary of Volume 2. import is the machine that crosses it. I want to slow down on it, because nearly everyone reads it as "load a file" — and it is nothing of the sort. So here's the plan. We take a single import prices and watch it run four steps — find, compile, exec, cache. Those steps turn a source file on disk into a live module object whose __dict__ is just an ordinary namespace, and they run the file's top-level code exactly once per process, no matter how many times you ask for it. The whole way through, we keep asking the one thing that matters: when you write import, what does Python do to that file, and how does it decide never to do it again? By the end you'll read a module as a plain object you can poke at, watch sys.modules enforce run-once, see why if __name__ == "__main__": works at all, and know exactly what from X import Y copies — and why it can quietly go stale.

01The need: one file that only grows

Let's start where every program starts: with one file, back when one file was the honest size of the problem. shop.py read a CSV of inventory off disk, ran the price math — discounts, tax, the rounding rule your accountant insists on — and drove the menu loop a cashier uses to ring things up. It fit on a screen. It worked. Then it grew, the way working code always does, and now it's nine hundred lines. Watch what that costs you. To change the rounding rule, you scroll past two hundred lines of storage code you didn't come to read. To fix the menu, you edit dangerously close to the math. And because every name in the file shares one flat global namespace, a total in the menu and a total in the tax code are literally the same name. Rebind one and you've silently changed the other.

Then the real pain shows up. Your teammate is building a different program and wants just your price math — forty good lines she doesn't want to rewrite. But there's no way to hand someone forty lines of a file. A file is all-or-nothing. So she does the only thing the language seems to allow: she copies the forty lines into her file. It works, for now. Two weeks later you find a bug — a discount could go negative — and you fix it in your copy. Hers still has the bug. Nobody edited both, because nobody could. They're two independent copies now, and they've quietly drifted. That's the felt need in one sentence: a single file cannot be partially reused, and copy-paste reuse rots the instant either copy changes. What you want is one authoritative copy of the price code that many programs can pull in by name.

Name the constraint precisely before naming the cure. A running Python program is one process with one interpreter, but your source lives in many files on disk. Without import there is exactly one path from file to running program: the interpreter reads the single file you named on the command line — python shop.py — compiles it, and runs it. One file per run. So every function, class, and constant your program uses must live in that one file, or be pasted into it. import is the mechanism that breaks that one-file ceiling. It lets code in file A cause code in file B to be found, compiled, run, and made reachable from A — all inside the same process, sharing the same heap and the very same object identities.

To see exactly what import buys, watch two tempting non-solutions fail. The first is copy-paste, which we've already met. It duplicates the source text, so fixes never propagate, and it gives each file its own separate discount function object — two functions that merely look alike. The second is closer, and more instructive, because it actually runs the other file:

>>> exec(open('prices.py').read(), globals())
prices module loading...        # a top-level print in prices.py fired — right now
>>> discount(100)                # the name leaked STRAIGHT into your globals, no prefix
90.0
>>> exec(open('prices.py').read(), globals())
prices module loading...        # ran the WHOLE file AGAIN — re-read from disk, re-printed

exec does read and run the file, but it comes with three problems. It dumps every name straight into your globals with no namespace boundary, so a helper in prices.py now silently overwrites any helper of yours. It has no memory, so a second call re-reads the file off disk and re-runs it top to bottom. And it re-fires every side effect. import is the disciplined version of exactly this. It runs the file once, wraps the resulting names in a single named container so they cannot collide with yours, and remembers the result so the next import is nearly free. That discipline is not one feature but four mechanisms stacked in sequence — find, compile, exec, cache — and the rest of this chapter takes them apart one at a time.

You've been told import "just works." Before you trust it, you need the shape of what it actually does — four steps, so that every weird symptom later maps to exactly one of them. →

02The four steps, once

You type import json and it just works. But "just works" is precisely the kind of magic that bites you later — when a module prints something the moment you import it, or imports the wrong file that happens to share a name, or seems to ignore an edit you just saved. To debug any of those, you need the silhouette of what import does, because each symptom lives at exactly one stage. So before popping any hood, fix the shape. When you write import prices, the import machinery — ultimately the function importlib._bootstrap.__import__ — does four things in order.

STEP 1 — FIND. It first checks the cache, sys.modules (that's the next section). On a miss, it walks sys.meta_path, a list of finders. The default finders walk sys.path — a list of directories — looking for prices.py, or a package directory, or a compiled C extension. A finder that recognises the name returns a ModuleSpec: a small record describing where the code is and how to load it (which loader, what origin path). Find answers one question — "given this name, what file, and who loads it?"

STEP 2 — COMPILE. The loader reads the source bytes off disk. Compilation is expensive, so CPython caches its result: it looks in a __pycache__ directory for a matching .pyc. If a fresh one exists — matched against the source file's modification time and size, or optionally a hash of the source — it skips straight to loading that pre-compiled bytecode. Otherwise it calls compile() to turn the source string into a code object — the same bytecode you met when you disassembled a function — and writes a .pyc so the next process starts faster.

STEP 3 — EXEC. It creates a fresh, empty module object. Then — and this detail is load-bearing — it inserts that object into sys.modules before executing anything. It seeds the module's __dict__ with __name__, __file__, __spec__, and friends, then execs the code object with that __dict__ serving as both globals and locals. Running the top-level code is what actually fills the module: every top-level def, class, and assignment binds a name in that dict.

STEP 4 — CACHE. The module is already sitting in sys.modules from step 3. On success it simply stays there, and the name prices is bound in your namespace to that same module object. The whole point of the cache is the load-bearing claim of this chapter: steps 1–3 happen at most once per process, because step 4's stored module short-circuits step 1 on every later import of the same name.

These stages are not a metaphor. Every artifact between them is a real object you can hold and inspect. Import a small module and interrogate what find and compile produced:

>>> import prices
>>> type(prices)
<class 'module'>                 # step 3 built a real object of type 'module'
>>> prices.__name__
'prices'
>>> prices.__file__                 # the file FIND located
'/home/you/shop/prices.py'
>>> prices.__spec__                 # the ModuleSpec ticket, still attached
ModuleSpec(name='prices', loader=..., origin='/home/you/shop/prices.py')
>>> import os
>>> os.listdir('__pycache__')   # step 2's compiled bytecode, written to disk
['prices.cpython-312.pyc']    # the name encodes the interpreter version

That .pyc filename carries cpython-312 for a reason: bytecode is not portable across interpreter versions, so each version keeps its own cache and never runs another's. And the run-once claim is directly observable. Put a top-level print in the module, import it twice in one session, and the print fires only on the first import. Steps 1–3 ran once; the second import was a pure cache lookup. Each of the next four sections pops the hood on exactly one of these stages. This one only fixes the silhouette, so you always know which stage a symptom belongs to.

Step 3 built "a real object of type module." That phrase is the whole next section — a module is not a special language construct, it is an ordinary object with an ordinary __dict__. →

03A module is an object with a __dict__

You write json.dumps(data) and treat json as a special, keyword-ish thing — a namespace handed down from the language. But if a module is magic, you cannot reason about it. You cannot ask what is inside it, store it in a variable, or explain why json.dumps and a plain attribute lookup on any object feel identical. Demystifying the module is what turns import from an incantation into a thing you can inspect and predict. So here is the chapter's core reframe, stated flatly: a module is an ordinary Python object. It is an instance of types.ModuleType, and like almost every object in the language, it carries a __dict__ — a plain dict mapping name strings to value objects. That __dict__ is the module's global namespace.

Recall step 3 from the last section. When exec ran the module's top-level code, it used the module's __dict__ as the globals. So every top-level PI = 3.14159, every def area(r):, every class Circle: bound a key in that one dict. After import json, then, the name json in your namespace is bound to one ModuleType instance. And json.dumps is not special syntax at all. It is attribute access on that object, which for a module resolves by looking up the string 'dumps' in json.__dict__. Make the identity chain explicit and it stops being mysterious. import json binds the name json to a module object; that object's __dict__['dumps'] is a function object; and json.dumps(x) fetches that function, then calls it. Two steps you already know — a name lookup and a dict lookup — with nothing extra in between.

The convincing way to prove a module is ordinary is to do ordinary-object things to it. Alias it into another variable. Drop it into a list. Open its namespace with vars(). Even bolt a brand-new name onto it with setattr. Every one of these works, because there is nothing special to violate:

import mymath                 # mymath.py defines PI = 3.14159 and def area(r): ...

print(type(mymath))            # <class 'module'>  — an instance of types.ModuleType
print(vars(mymath).keys())    # dict_keys(['__name__', '__file__', ..., 'PI', 'area'])

j = mymath                       # a module is a value: alias it like anything else
toolbox = [mymath, str, len]    # put it in a list
print(j.area(2))              # 12.566...  — the alias reaches the same function

print(mymath.PI is mymath.__dict__['PI'])   # True — attribute access IS the dict lookup
setattr(mymath, 'E', 2.71828)             # write a new key into its namespace
print(mymath.E)               # 2.71828 — you just edited the module's __dict__

The line mymath.PI is mymath.__dict__['PI'] returning True is the whole reframe in one is. The dotted access and the raw dict lookup reach the identical object, because they are the same operation. This connects straight back to the binding chapters of Volume 1. Back then you learned that a module's global namespace "is just a dict" — the same fact that let module-level globals work. Now you can see that dict, print its keys, and write into it. There is a small, honest subtlety worth stating exactly. For a module, attribute access goes through __dict__ directly: a module does not run the full instance-and-class descriptor machinery for its own globals the way a class instance does. That is why module.__dict__, vars(module), and the namespace the code executed in are not three related things but literally one and the same object.

You bolted a new name onto mymath in one file. If another file imported it, would that file see your E? To answer that, you have to know whether the two imports share one object or hold two copies. →

04exec runs top-level code — and its side effects — once

You import a module and something prints — or a config file gets read, or a network call fires — before you have called a single function. That surprise is a message: you do not yet believe that importing a module runs it. Until that fact is bone-deep, import-order bugs and import-time side effects feel random and malicious. In truth they are perfectly deterministic. So zoom all the way into step 3. Executing a module is not conceptually different from executing your main script. The interpreter takes the module's code object and runs every top-level statement, in source order, in a frame whose globals are the fresh module's __dict__. There is no separate "loading" phase that skips the interesting lines. The lines are the loading.

Concretely: a top-level data = compute() calls compute() now. A top-level print('loading') prints now. A top-level config = open('config.json').read() does real file I/O now — all during the import statement, before control returns to the importer. This is exactly why a def at module scope defines a function without calling it. The def statement, when executed, builds a function object and binds its name. That is all a def does. A bare function call at module scope, by contrast, actually runs the function, because a call expression is executed like any other. The distinction is not about def being special. It is that one statement binds a name and the other invokes a call.

import time

print('boot loading...')        # TOP-LEVEL CALL — runs during import
START = time.time()             # TOP-LEVEL CALL — the clock is read at import time

def greet():                    # def executes: it BINDS the name 'greet'...
    print('hi')               # ...but the body does NOT run yet

greet()                        # a BARE call — THIS line runs the body, at import time

Import boot and, before you touch anything, you see boot loading... then hi. The def line bound greet; the line after it called greet. Ordering is strict and total. Statements run top to bottom, so a name must be bound above the line that uses it, or you get a NameError. This is not a style rule. It is the direct consequence of executing lines in order. Move the definition below its first use and it fails; move it back up and it works.

The ordering rule reaches across files too, and this is where import-order surprises are born. If module A's top-level code does import B and then uses something from B, then all of B's top-level code runs to completion before A's next line executes. Import is a synchronous, depth-first descent. A pauses at its import B, B runs entirely, control returns to A, and A continues. Trace the interleaving once and it is never mysterious again:

# --- B.py ---
def setup():
    print('B.setup ran')
    return 42

# --- A.py ---
import B                        # B is found/compiled/exec'd fully before the next line
VALUE = B.setup()             # prints 'B.setup ran', binds VALUE = 42 — all at A's import time
print('A finished, VALUE =', VALUE)

Now tie it to the run-once law you'll formalize next. Because the finished module object is cached, this top-level execution — and every side effect it causes — happens exactly once per process, even if ten different files import the module. That is a gift and a trap. The gift: expensive setup runs once and is shared. The trap: heavy or side-effecting work at module top level — a network call, reading a large file, mutating shared state — fires at import time, in import order, often long before you meant for the program to "start." That fragility is precisely the pressure that produces the __main__ guard in the next section. The guard is a way to keep the runnable side effects out of the top-level path that import executes.

"Exactly once per process" keeps coming up. What actually enforces it? One dict you can print — and it is the same dict step 1 checks first and step 4 writes last. →

05sys.modules is the run-once cache

You have heard "importing runs the code once" — but why once? If two different files both write import prices, what actually stops prices.py from running twice? And here is the flip side that bites every REPL user: you edit a module, re-type import prices in the same session, and your edits seem ignored. Why? Both answers live in a single dict you can print. It is called sys.modules, and seeing it turns two memorized rules into one mechanism you can inspect. sys.modules is an ordinary dict mapping import name → module object, and it is the very first thing import consults. The front of step 1 and the storage of step 4 are the two ends of this one cache.

Here is the real algorithm, in full. On import prices, check whether 'prices' is a key in sys.modules. If yes — skip find, compile, and exec entirely, and just bind the local name to the already-stored object. If no — run find → compile → create a blank module → insert it into sys.modules → exec its code → leave it cached. Two consequences fall straight out. First, run-once: the top-level code execs only on the cache-miss path, so no matter how many files import prices, or how many times, its code runs exactly once and every importer receives the same module object. Second, and equally important, the namespace is a process-wide singleton. Because everyone holds the same object, if one file does prices.RATE = 0.2, every other importer sees 0.2. There is one __dict__, shared.

>>> import sys, prices
prices module loading...            # cache MISS — top-level code ran
>>> 'prices' in sys.modules
True
>>> sys.modules['prices'] is prices   # the cache entry and your name are ONE object
True
>>> import prices                   # cache HIT — no 'loading' print this time
>>> prices.RATE = 0.2              # mutate the shared singleton...
>>> sys.modules['prices'].RATE       # ...and every importer sees it
0.2

Now the edits-ignored mystery dissolves. In a long-lived REPL or server process, re-typing import prices after saving an edit is a cache hit, so it does nothing at all. You are handed the old object, and the new source on disk is never read. To force a re-exec you must reach past the cache. Either delete the key (del sys.modules['prices']) so the next import misses, or call importlib.reload(prices), which re-execs the source into the existing module object. Nothing was broken; the cache was doing its job perfectly.

The pre-exec insertion from step 3 is not a detail. It is what makes circular imports survivable. The blank module is placed in sys.modules before its top-level code runs. So if that code, directly or through another module, imports itself, the second import finds the partially-built module already in the cache and returns it, instead of recursing forever into find/compile/exec. The cost is honest and worth knowing. The second importer may receive a half-populated module — only the names bound above the import line exist yet. That is exactly why circular imports fail with "cannot import name X" when X is defined lower down. The cache trades infinite recursion for the possibility of a temporarily incomplete module.

Step back and the whole design snaps into focus. sys.modules is the registry of object identity for modules, and import is fundamentally a cached lookup keyed by name. This is the deepest connection to the thread of this volume. Volume 1 taught that a name points at one real object at one address. Here that idea scales up a level: an import name points at one real module object, shared across every file in the process. There are not "copies of json" floating around. There is one json object, and import json anywhere is a lookup that hands back that same one.

You've seen that top-level code always runs on import. So how does a file act as a quiet library when imported but a runnable program when launched? One string, set by the interpreter, decides. →

06__name__ and the __main__ guard

Every real Python file you have ever read ends with the same incantation — if __name__ == '__main__': — and you have cargo-culted it faithfully without knowing what __name__ actually is or why the guard works. That ignorance has a concrete cost: you cannot decide what belongs inside the block and what belongs outside it. The felt need is real and specific. You want one file that is both a reusable library — safe for anyone to import for its functions — and a runnable program that does something when you launch it. And you want that without its "run the program" code firing every time someone merely imports it for a function. Section 4 showed the danger: top-level code runs on import. So an unguarded "start the menu loop" line at module scope would ambush every importer. The guard is the fix, and it is built from a single fact.

Every module has a __name__ entry in its __dict__, and the interpreter chooses its value at load time based on how the module was entered. When a module is imported, its __name__ is set to its import name — the string 'prices'. When a file is run directly as the program — python prices.py, or python -m prices — the interpreter runs it as the top-level module and sets its __name__ to the special string '__main__'. Same file. Same code object. Same bytecode. Different __name__, decided purely by which door the code was entered through.

Once you know that, the guard is not clever — it is obvious. if __name__ == '__main__': is a plain runtime comparison of that one string. So the block beneath it runs only when the file was launched directly, and never when it was imported. That single test lets you give the file two faces. The library face — your defs, classes, and constants — goes at top level, where every importer receives it. The program face — argument parsing, the menu loop, a demo — goes inside the guard, where it fires only on direct launch:

def discount(price, pct):        # LIBRARY FACE — importers get this, silently
    return round(price * (1 - pct), 2)

if __name__ == '__main__':       # PROGRAM FACE — runs ONLY on `python prices.py`
    print('demo:', discount(100, 0.1))   # importers never see this fire

Import this module from another file and you get discount quietly, with no demo output. Run python prices.py and it prints demo: 90.0. One file, two behaviors, chosen by the entry door. And the guard's value is exactly what sections 4 and 5 set up. Because import execs the whole top level, an unguarded "run the program" call at module scope would fire on every import. And because that exec is cached and run-once, the unwanted program-run would be smuggled into whichever file imported you first, then never repeat — a maddening, once-only ghost. The guard is the discipline that keeps import-time side effects out of the top-level path entirely.

Finally, dispel the last bit of magic. __main__ is not a keyword or a special construct. It is an ordinary module too. When you launch a file directly, the interpreter registers it in sys.modules under the key '__main__', exactly like any other module, and gives it the name '__main__'. You can prove it from inside the guard with print('__main__' in sys.modules) → True. There is no separate mechanism for "the main program." There is just the entry-point module, which happens to be assigned the reserved name '__main__' instead of a filename-derived one. The whole idiom, then, is two ordinary facts stacked: modules have a __name__, and the entry-point module's __name__ is '__main__'.

You now import cleanly. But the other import form — from config import SETTINGS — behaves differently in a way that silently breaks configs and monkeypatches. What exactly does it bind? →

07from X import Y copies the reference

You wrote from config import SETTINGS at the top of five files, because it read better than config.SETTINGS everywhere. Weeks later, some initialization code reassigns config.SETTINGS to a fresh dict at runtime, and your five files keep seeing the old one. Or you write from utils import helper, a test monkeypatches utils.helper to a stub, and your imported name blithely runs the real function, unpatched. This aliasing surprise silently defeats configs, test patches, and hot-swaps. The only cure is to know exactly what from X import Y binds. So state it precisely: from X import Y is two operations, and the second is a reference copy.

Operation one is a full, ordinary import of X — find, compile, exec, cache, identical to import X. X's top-level code runs, and X ends up in sys.modules. The only difference from plain import X is that X's own name may not be left bound in your namespace. Operation two looks up the attribute Y on the module object — a lookup of 'Y' in X.__dict__, the dict lookup from section 3 — and binds that value to a name Y in your namespace. The crucial word is value. It copies the reference — the object Y points at right now — not a live link to X's slot. After from config import SETTINGS, your SETTINGS and config.SETTINGS are two names, in two different namespaces, both pointing at the same object at this instant. And from this instant on, the two names have independent futures.

Two consequences follow, and the difference between them is the entire lesson — the same names-versus-objects model from the binding chapters, now stretched across a module boundary. Consequence one: rebinding does not reach you. If someone later does config.SETTINGS = {new dict}, that only changes config.__dict__['SETTINGS'] to point at a new object. Your name still points at the old one. Your Y is now stale. This is precisely why monkeypatching module.func fails to affect callers who did from module import func. Consequence two: mutation does reach you. If instead someone mutates the shared object in place — config.SETTINGS['debug'] = True — both names still point at that same object, so you do see the change. Rebind versus mutate is the whole story:

from config import SETTINGS       # copy the reference: your SETTINGS -> Object#1
import config

config.SETTINGS = {'v': 2}          # REBIND: config.SETTINGS -> a NEW Object#2
print(SETTINGS)                 # {'v': 1}  — your name still on Object#1 (STALE)
print(config.SETTINGS)          # {'v': 2}  — the module moved on without you

config.SETTINGS['debug'] = True   # MUTATE the object both names still share
print(SETTINGS)                 # {'v': 2, 'debug': True} — seen, because same object

Contrast this with plain import config. Because you kept the module object and reach the value fresh each time through config.SETTINGS — a live __dict__ lookup performed at the moment of use — you always observe the current binding. That is exactly why import module then module.thing is more patch-friendly than from module import thing. The first re-reads the slot every time; the second froze a snapshot of it at import. Neither is "correct." They answer different questions. from module import thing asks "what object is thing right now?" and remembers the answer. import module; module.thing asks the question again on every use.

The reference-copy semantics are the same no matter how you dress the syntax. import numpy as np and from utils import helper as h only change which local name you bind. They do not change the fact that you bound a reference to whatever object was there at import time. And the mechanism is nothing exotic — it is the plainest possible reference copy. Import a name bound to a shared list, append to the list through either name, and both the module's name and your imported name see the new element, because there is one list and two names for it. That single object with two names, reachable from two files, is the through-line of this whole chapter. import does not copy objects between files. It gives many files names for the same objects — which is exactly the law the rest of Volume 2 will test against every harder boundary, where objects genuinely cannot cross and only bytes can.

One object, many names, all inside one process — the easy case. Next volume-step: what happens when the boundary is real, an object cannot cross it, and only a byte stream can. That is where import's cousins — encoding, files, and pickle — take over. →