21Packages & program structure

In Chapter 20 we cracked the import machine open — sys.path, the chain of finders, the loaders, the .pyc cache — and watched a single module get found, compiled, and cached. Now we point that same machine at a whole directory. Your playlist has outgrown a single file. Shuffle logic, storage, and the data classes sprawl across a dozen .py files, and import shuffle sits next to import config with no hint of which belongs to whom. So we fold the pile under one name — import playlist — the way import json hands you a whole folder of code through a single handle. Here's the plan. We'll see exactly what runs when a directory imports, meet __init__.py as runnable module code rather than ceremony, sort absolute imports from relative ones, learn what python -m quietly repairs, and watch two innocent imports poison each other. The whole way through we keep asking the one question that decides all of it — when a folder becomes a name, what code runs, and in what order? By the end we've restructured the whole playlist into a real package: one clean front door, the private plumbing hidden behind it, and python -m playlist starting the program from a single command.

01A package is a directory that imports as one name

Let's start with the mess. You've been writing one file per idea — engine.py, storage.py, model.py — and reaching across them with import storage. It worked, but the folder is now a junk drawer: nothing says these files are one thing. Meanwhile import json hands you a whole directory of standard-library code through a single word. That's exactly what you want here — import playlist, one name for the whole project. So here's the question, and it's sharper than it looks. How does a folder become a single importable thing, and what code — if any — runs the moment it does?

Watch what import actually does, because it isn't what most people picture. import playlist does not go hunting for a file called playlist. Instead it walks a list of finders — the objects sitting in sys.meta_path, the same crew you met last chapter — and asks each one the same question: do you know how to load the name 'playlist'? The finder that handles the filesystem, FileFinder, scans the directories on sys.path. The instant it spots a directory named playlist holding an __init__.py, it says yes. It hands back a module spec, a little record whose loader is a SourceFileLoader aimed straight at playlist/__init__.py. That's the handle the rest of the import is built on.

Now the import machinery runs a fixed four-step ritual. Every step matters later:

# 1. create an empty module object — a namespace, nothing in it yet
mod = new_module("playlist")

# 2. mark it as a PACKAGE by giving it __path__
mod.__path__ = [".../playlist"]   # where future sub-imports will search

# 3. register it BEFORE running any of its code
sys.modules["playlist"] = mod    # remember this line — §5 turns on it

# 4. exec __init__.py's bytecode, using mod.__dict__ as its globals
exec(compile(init_source), mod.__dict__)

Read step 2 twice. The single thing that separates a package from an ordinary module is the attribute __path__ — a list holding the package's own directory. An ordinary module like math has no __path__. A package has one, and that list is the map future sub-imports consult when they need to find playlist's children. Step 4 is the reveal: __init__.py is not configuration, not a manifest, not boilerplate the interpreter reads for settings. It is the package's module body — ordinary Python that runs top to bottom, exactly like a script, with the fresh module's __dict__ serving as its global namespace. Whatever names it binds become the package's attributes. Print something at the top of __init__.py and it prints the instant someone imports the package.

print("init running")
print("__name__    =", __name__)      # playlist
print("__package__ =", __package__)   # playlist
print("__path__    =", __path__)      # ['/abs/path/to/playlist']

Here is the surprise that trips everyone: importing the package does not import its submodules. import playlist runs __init__.py and stops. There is no playlist.engine yet — touch it and you get AttributeError. A submodule springs into existence only when something explicitly imports it. At that moment the machinery does something quietly elegant: after loading engine.py, it binds the submodule onto its parent, so playlist.engine becomes a real attribute of the playlist module object. The package is a namespace that fills in on demand.

That sys.modules line in step 3 also explains why a package initialises exactly once. Import playlist twice in the same program and __init__.py runs a single time. On the second import, the machinery finds playlist already sitting in sys.modules and hands back the cached object without re-executing a line. Every module is a process-wide singleton keyed by its dotted name. This is the same cache that, from Ch 22 onward, you'll watch your program lean on as it starts reaching past its own memory.

You have one name for the whole folder. But right now users can reach every file and every helper inside it. Next: how __init__.py lets you publish a clean front counter and hide the plumbing behind it. →

02__init__.py defines the public surface

Your users type from playlist import Track and it works. Good. But nothing stops them typing from playlist.engine._internal_shuffle import _fisher_yates — and now someone's code depends on a private helper you meant to rewrite next week. The moment they do, your freedom to change it is gone. You need a way to say, out loud and enforceable-in-spirit: this is the API I promise to keep; everything else is plumbing I reserve the right to move. Where does that line live, and how do you draw it?

The answer falls straight out of Section 1. A package's public surface is nothing more mystical than the set of names bound in the __init__ module's namespace after it finishes executing. There is no separate export table, no public: keyword. If a name is an attribute of the package module, it's reachable as playlist.that_name. If it isn't, it isn't. So you curate the surface by choosing what __init__.py binds. The tool is the re-export:

# pull the good names UP from the submodules where they live
from .model import Track, Playlist
from .engine import Player
from .storage import save, load

__all__ = ["Track", "Playlist", "Player", "save", "load"]

That from .engine import Player runs inside __init__.py, so it binds Player as an attribute of the playlist module. Now from playlist import Player resolves directly. The user never learns, and never needs to learn, that Player physically lives in engine.py. This is facade construction. Your folder layout is your implementation, free to be reorganised; the import path is your API, held stable. Move Player from engine.py into engine/core.py next year, fix the one line in __init__.py, and every caller keeps working. The facade decouples the two.

Now the two things people expect to be privacy controls — and neither one is. First, __all__. It is a list of strings, and it governs exactly one thing: what from playlist import * pulls into the caller's namespace. That's it. It does not hide anything. With the __all__ above, from playlist import * brings in those five names. But from playlist import debug_dump still works fine for any name that actually exists, listed or not. __all__ is a curated star-export manifest and a documentation signal ("these are the headline names"), never a lock.

Second, the leading underscore. Naming a helper _fisher_yates or _seed also changes exactly one behaviour: names starting with _ are skipped by import * when there's no __all__. Attribute access still reaches them perfectly — playlist.engine._fisher_yates reads out the function without complaint. The underscore is a message to humans and linters: touch this and you're on your own; I may delete it. Be exact about the mechanism, because half-truths here breed superstition. Python has no access control. There is no private keyword and no runtime that forbids reaching a name. "Private" in Python means precisely two non-enforcing things — omitted from __all__, and marked with an underscore — plus the social contract that goes with them. The interpreter will hand out every name you ask for. The convention is a fence, not a wall.

You've published names. But those from .engine import Player lines with a leading dot are doing something specific — and they behave differently depending on how the file is run. Time to understand the dot. →

03Absolute vs relative imports

You reorganise. engine.py moves into a subfolder, and suddenly import storage throws ModuleNotFoundError — even though storage.py is right there next to it. You patch it to from .storage import save, and imported through the package, it works. Relieved, you run python engine.py to test that file alone — and it detonates: ImportError: attempted relative import with no known parent package. Same line, same file, opposite outcomes. You cannot move on until you understand why.

There are two import styles, and they resolve through completely different machinery. An absolute import names the full dotted path from a top-level entry on sys.path: from playlist.storage import save. The finder walks it segment by segment — find playlist, descend into it, find storage, pull out save. It depends on nothing but sys.path and the name.

A relative import — from .storage import save, from ..util import log — never touches a filesystem name directly. The compiler encodes the leading dots as a level count: one dot is level=1, two dots level=2. At runtime the interpreter resolves the target in three moves. It takes the current module's __package__ string, walks level-1 dots up from it, then appends the name after the dots. The whole thing is relative to one attribute — __package__ — and hangs entirely on that string being set correctly.

# inside module  playlist.audio.engine   →  __package__ = "playlist.audio"
from ..util import log        # level = 2 (two dots)

# resolver: start at __package__      -> "playlist.audio"
#           strip (level-1)=1 name     -> "playlist"
#           append the target "util"   -> "playlist.util"
# result: from playlist.util import log

Now the crash makes sense. When you import playlist.engine, the machinery stamps engine.__package__ = "playlist", so .storage resolves to playlist.storage — clean. But when you run python engine.py, that file is loaded as the startup module. Its __name__ becomes "__main__" and its __package__ becomes "" — the empty string. Now the dot resolver is asked to walk up from nothing. There is no parent package string to strip a name from, so the interpreter can only raise: attempted relative import with no known parent package. The line isn't wrong. The context is missing. This is the exact hole that python -m, the next section, was built to fill.

So which style should you use? The trade is real, and it's about who you're protecting. Relative imports say "my neighbour." from .storage import save means the sibling module in whatever package I live in. It never spells the top package's name, so if you rename playlist to setlist tomorrow, every relative import inside keeps working untouched. They make a package renamable and relocatable. Absolute imports say the full address. from playlist.storage import save is explicit and unambiguous, and it always resolves under -m. That's why it's the safe default for anything you might run as a program. The discipline that keeps you out of trouble is a single rule: never rely on a relative import in a file you also run directly as a script. For library code deep in the package, relative is lovely. For an entry point, use absolute, or run it through -m.

If relative imports need a real package context, and running a file directly destroys that context, there must be a way to run a package as a package. There is — one command, and it's the front door your project has been missing. →

04python -m and __main__.py: running a package

You want your project to run with one clean command — python -m playlist — not python playlist/some_file_i_happened_to_pick.py. And you're done with the relative-import crash from the last section. It turns out both problems have a single answer, and it's a flag you've probably typed without knowing what it repairs. To use it well, you need to see how differently CPython starts up in the two cases.

When you run python playlist/engine.py, the interpreter takes the script path. It sets sys.path[0] to the script's own directory (playlist/), then executes the file as the startup module — __name__ = "__main__", __package__ = "". That's the broken world of Section 3. There's no parent package, so every relative import inside dies. And because playlist/ itself is on the path but its parent isn't, import playlist may not even resolve.

When you run python -m playlist, a different subsystem — runpy — takes over, and it does three things in order:

# 1. put the CURRENT WORKING DIRECTORY on sys.path[0]
#    so the top-level package 'playlist' is importable
sys.path.insert(0, os.getcwd())

# 2. IMPORT the package normally -> runs playlist/__init__.py,
#    so 'playlist' is a real, correctly-initialised package
import playlist

# 3. find playlist/__main__.py and exec it AS the startup module,
#    but with package context preserved:
#      __name__    = "__main__"     (so the guard fires)
#      __package__ = "playlist"     (so relative imports RESOLVE)

Step 3 is the whole trick, and it's genuinely clever. The entry module runs under the name __main__, so if __name__ == "__main__": fires and it behaves like the program's start. And it carries __package__ = "playlist", so relative imports inside it resolve exactly as they would in any library module. Both truths hold at once: I am the program and I live inside a package. A plain script can never have both. That is why python -m is the invocation that makes from .engine import Player work in an entry point.

The file that runs is __main__.py, and now its role is clear. It is the package's designated entry point, the folder-level equivalent of the if __name__ == "__main__": block you'd put at the bottom of a single script. You don't pick a random file to be the front door. You write playlist/__main__.py, and python -m playlist finds it by convention.

from .engine import Player       # resolves under -m, would crash as a script
from .storage import load

print("__name__    =", __name__)     # __main__
print("__package__ =", __package__)  # playlist  — both true at once

tracks = load("library.json")
Player(tracks).run()

This also demystifies __name__ itself, the most cargo-culted line in Python. __name__ is simply the string the loader stamped on the module when it loaded it: "playlist.engine" when imported as a library, "__main__" when it's the startup module. So if __name__ == "__main__": is not magic. It's a plain string comparison asking one question: am I the program being run, or am I being imported by someone else? Run this only when I'm the program; stay quiet when I'm a library. The guard has meant that, and only that, all along.

You have a package that imports as one name, publishes a clean surface, and runs from a real front door. One structural landmine remains — the one where two perfectly correct imports poison each other. →

05Circular imports and the half-built module

You split a class across two files — model.py needs a type from engine.py, engine.py needs a type from model.py — and import playlist dies with a line that sounds like an accusation, the circular import: ImportError: cannot import name 'Player' from partially initialized module 'playlist.engine' (most likely due to a circular import). Nothing is misspelled. Both files are correct in isolation. You could shuffle import lines until it stops crashing, but that's superstition. You want to know precisely why a valid pair of imports poisons each other, and the exact fix.

This builds directly on Section 1, on that one line you were told to remember: import inserts an empty module object into sys.modules before running the module's code. Watch the timeline of import playlist.model when the two files import each other at the top:

t0  import model      -> sys.modules["...model"]  = <empty>   # seed, before code
t1  model line 1:  from .engine import Player
t2    import engine   -> sys.modules["...engine"] = <empty>   # seed, before code
t3    engine line 1: from .model import Track
t4      model already in sys.modules  -> DO NOT re-run it

Read t4 slowly, because it's the crux. When engine does from .model import Track, the machinery finds model already in sys.modules — the empty seed from t0 — and, correctly, does not re-run it, since re-running would loop forever. It hands engine the half-built model object exactly as it stands. But at t4, model has only executed its first line; it never reached class Track. So the name Track isn't in model's namespace yet. from .model import Track demands that name right now, can't find it, and raises cannot import name 'Track' from partially initialized module. The module isn't broken. It's just unfinished, caught mid-execution by a neighbour who reached across too early.

Now the distinction that unlocks everything, and it's the difference between two import forms you probably thought were interchangeable:

from .model import Track   # needs the NAME 'Track' to exist RIGHT NOW, at import time
import playlist.model     # binds the MODULE object; you touch .Track LATER, at call time

from .model import Track copies a name out of model's namespace at the instant the import runs — during module execution, when model may still be half-built. import playlist.model only binds the module object, which exists from t0, empty or not. You don't read model.Track until later, at call time, by which point both module bodies have finished and Track is present. So the cycle itself is not fatal. What's fatal is depending on a not-yet-executed name during module execution. Change when you need the name, and the cycle becomes harmless.

That gives four fixes, worst to best:

# 1. move the import to CALL time — runs after both bodies finished
def shuffle(self):
    from .model import Track   # local import, deferred until called
    ...

# 2. bind the module, read the attribute lazily
import playlist.model
# ...later...  playlist.model.Track

# 3. type-checking only — exists for the checker, never at runtime
from typing import TYPE_CHECKING
if TYPE_CHECKING:
    from .model import Track   # used only in string annotations

# 4. BEST — break the cycle: extract Track into a leaf module both import

Every mechanism is now on the table: the package name, the facade, the dot, the front door, the cycle. Time to cash them all in — turn the sprawling playlist into a real, layered, cycle-proof program. →

06Restructuring the playlist into a package

Everything so far has been mechanism. Now you spend it. The playlist has swollen into one bloated file mixing data classes, playback logic, and file-saving. You can't test the storage in isolation, can't hand a teammate "just the engine," can't change the save format without risking the shuffle. You want to convert the pile into a program: a package with clear modules, one public surface, one entry point, and — by construction — no import cycles. This is the exact shape the capstone in Ch 23 will ship.

The single design decision that makes all of it fall into place is dependency direction. Decide which module depends on which, and make every arrow point the same way — toward a foundation. Here's the layout:

playlist/
  __init__.py     # facade: re-exports the public names + __all__
  model.py        # LEAF — Track, Playlist. imports nothing internal
  storage.py      # imports model. save/load to JSON
  engine.py       # imports model. Player, shuffle, next/prev
  __main__.py     # entry point: wires storage + engine. python -m playlist

Look at the arrows. model.py is a leaf: pure data — the Track and Playlist dataclasses — importing nothing else in the package. storage.py imports model, one direction. engine.py imports model, one direction. Neither storage nor engine imports the other; they meet only at the top, in __main__.py, which wires them together. Because every arrow points down toward the model leaf, there is no cycle possible — not "we avoided one," but "the shape forbids one." This is the deep lesson of Section 5 turned into architecture. You don't dodge circular imports with clever import placement; you make them structurally impossible by layering.

# model.py   — the leaf, imports nothing internal
from dataclasses import dataclass

# storage.py — one arrow down to model
from .model import Track, Playlist

# engine.py  — one arrow down to model
from .model import Track, Playlist

# __init__.py — the facade on top
from .model import Track, Playlist
from .engine import Player
from .storage import save, load
__all__ = ["Track", "Playlist", "Player", "save", "load"]

Every prior mechanism is doing a job here, in situ. The relative imports (from .model import ...) mean the package is renamable and works under -m — Section 3. Those imports resolve because __package__ is set correctly when the package is imported, and __main__.py keeps that context under python -m playlist — Section 4. __init__.py re-exports the public trio behind a stable facade and declares __all__ — Section 2. And the acyclic layering is the structural cure for the poisoning of Section 5. None of the modules is meant to be run directly. The only front door is __main__.py.

The payoff is the seams, and this is why the layering is worth the ceremony. Because storage.py isolates all the I/O, you can swap JSON for SQLite later without touching a line of engine.py. The engine never knew where tracks came from, only that load() returns them. Because engine.py holds the algorithm, you can swap the shuffle strategy without touching storage. And model.py is the stable contract both sides agree on — the Track shape that storage produces and engine consumes. Layering doesn't just prevent cycles; it hands you a program you can change one piece at a time. That's exactly the surface Ch 23's capstone builds on.

Your program has a shape. Next volume-thread: the moment it needs to speak to anything outside itself — a file, a socket, another process — the objects stop crossing and the bytes begin. That's Ch 22. →