Python, from the metal up — Volume 2

15Files & persistence

Every structure you built in Volume 1 — the playlist list, the song dicts — lived in your process's RAM, and chapter 0 told you the deal: RAM is rented. This chapter moves the playlist somewhere that survives the end of the program, the closed laptop lid, the power cut. By the end you will save the playlist to disk, read it back byte-for-byte, and know the two exact moments where a careless program silently loses data.

RAM forgets. Disk remembers.

Run any Volume 1 program twice and the second run starts blank. That is not a Python quirk — it is physics plus housekeeping. The moment your process ends, the operating system reclaims every byte of its RAM for the next program (ch0's addressed boxes get new tenants). Cut the power and the DRAM cells lose their charge anyway. A file is different: it is a named sequence of bytes on a storage device — flash or magnetic — that keeps its state without power.

Python's bridge between the two worlds is open(). It asks the operating system for access to a file and hands you back a file object — a Python object (run type(f): it prints <class '_io.TextIOWrapper'>) that wraps the OS's file handle. Call .write() on it with a str; call .read() to get one back.

# save the playlist — "w" means: open for writing
with open("playlist.txt", "w") as f:
    f.write("Levels|Avicii|200\n")          # write() takes a str
    f.write("Wake Me Up|Avicii|247\n")      # \n ends each line
    f.write("Titanium|David Guetta|245\n")
# the with-block ended, so the file is closed and on disk

with open("playlist.txt", "r") as f:        # "r" means: open for reading
    print(f.read())                         # read the whole thing back
# Levels|Avicii|200
# Wake Me Up|Avicii|247
# Titanium|David Guetta|245

Two modes so far — "r" and "w". But "w" has teeth: it destroys before it writes →

Three modes, one trap: "w" erases first

The mode string tells open() your intent. "r" reads an existing file — if it isn't there, you get a real error; we ran it: FileNotFoundError: [Errno 2] No such file or directory: 'no_such_playlist.txt'. "a" appends: the file is created if missing, and every write lands at the end. "w" is the dangerous one: it creates the file if missing — and if the file exists, it truncates it, meaning it cuts the contents down to zero bytes. Watch a playlist die:

with open("playlist.txt", "a") as f:        # append: safe
    f.write("Hey Brother|Avicii|255\n")     # lands after Titanium
# file now holds all four songs — we printed it and checked

with open("playlist.txt", "w") as f:        # the mistake: "w" to add a song
    f.write("Strobe|deadmau5|634\n")        # the wipe already happened at open()

with open("playlist.txt", "r") as f:
    print(f.read())                         # Strobe|deadmau5|634
# four songs in, one song out — 93 bytes shrank to 21

Playlist restored, saved safely. Now read it twice in a row — the second read comes back empty →

The cursor: why reading twice returns nothing

Every open file object carries a cursor — a position marker recording how far into the file you have read or written. read() starts at the cursor, returns everything from there to the end, and leaves the cursor at the end. So a second read() starts at the end, finds nothing ahead of it, and returns ''. The file is fine; your position is spent.

f = open("playlist.txt", "r")   # the three-song file again
print(repr(f.read()))   # 'Levels|Avicii|200\nWake Me Up|Avicii|247\nTitanium|David Guetta|245\n'
print(repr(f.read()))   # ''  ← empty: the cursor is parked at the end
f.seek(0)               # teleport the cursor back to position 0
print(repr(f.read()))   # the full 66-character string again
f.close()

seek(0) moves the cursor back to the start; read(20) reads exactly 20 characters and stops — we ran it and got 'Levels|Avicii|200\nWa', cursor parked mid-word at character 20. Drag that cursor yourself:

Text is a costume — a file holds bytes

A file never holds a str, a list, or any Python object. It holds bytes, full stop. When you open in text mode (the default), Python straps a translator onto the byte pipe: every str you write is encoded to bytes on the way out, every read is decoded back (ch2: str vs bytes). Volume 1 chapter 0 told you "one letter per byte" — that was a labeled simplification, and here is the payoff. UTF-8 spends one byte on English letters and 2–4 bytes on the rest:

print(len("Tiësto"))               # 6 — six characters (a str)
print("Tiësto".encode("utf-8"))    # b'Ti\xc3\xabsto' — ë became TWO bytes
print(len("Tiësto".encode()))      # 7 — seven bytes

with open("tiesto.txt", "w", encoding="utf-8") as f:
    f.write("Tiësto")              # 6 characters in…

import os
print(os.path.getsize("tiesto.txt"))   # 7 — …7 bytes on disk

with open("tiesto.txt", "rb") as f:    # "rb" = binary mode: raw bytes, no translator
    print(f.read())                    # b'Ti\xc3\xabsto'

Skip encoding="utf-8" and Python guesses from the machine's locale. On this Windows machine the guess is cp1252, so reading the same file back without it printed Tiësto — same 7 bytes, wrong decoder, garbage name. That failure mode has a name: mojibake.

Bytes explain what lands on disk. Now the scarier question — when. Your write may not be on the disk at all yet →

The write buffer: your data's last stop in RAM

Talking to a disk is orders of magnitude slower than talking to RAM, so Python does not send each write() to the disk immediately. It parks your characters in a write buffer — a chunk of the process's own RAM (8192 bytes by default here; io.DEFAULT_BUFFER_SIZE prints 8192) — and ships the whole batch when the buffer fills, when you call flush(), or when the file closes. Efficient, and dangerous: if the process dies before the shipment, the buffered tail dies with it. We killed a process for real:

# crash_test.py — os._exit() ends the process instantly: no close, no cleanup
import os
f = open("crash_test.txt", "w")
f.write("Levels|Avicii|200\n")          # → buffer (RAM)
f.write("Wake Me Up|Avicii|247\n")      # → buffer (RAM)
f.flush()                               # ship the buffer to the OS NOW
f.write("Titanium|David Guetta|245\n")  # → buffer again
os._exit(1)                             # killed here — Titanium never left RAM

# afterwards, from a fresh program:
# os.path.getsize("crash_test.txt")  →  42
# open("crash_test.txt").read()      →  'Levels|Avicii|200\nWake Me Up|Avicii|247\n'

Two lines survived — the two that were flushed. Delete the flush() line and run again: we did, and the file measured 0 bytes. All three writes were sitting in the buffer when the process died. Nothing reached the disk.

Crash-safety handled. Time to save something richer than pipe-separated lines — the whole playlist dict, shape and all →

JSON: the dict → text → dict round trip

First, a better way to point at files. pathlib.Path turns path strings into path objects with behaviour:

from pathlib import Path
p = Path("gym_mix.json")                   # a path object, not a plain str
print(p.suffix, p.stem)                    # .json gym_mix
print(Path("exports") / "2026" / p.name)   # exports\2026\gym_mix.json  (/ on macOS+Linux)
print(p.exists())                          # False — nothing saved yet

Now the main event. Our playlist is a dict of lists of dicts — pipe-separated lines can't hold that shape. JSON (JavaScript Object Notation) is a text format whose vocabulary maps almost one-to-one onto Python's: objects↔dicts, arrays↔lists, plus strings, numbers, true/false/null. json.dump renders a dict into characters; text mode encodes them to bytes; disk stores them:

import json
playlist = {
    "name": "Gym Mix",
    "songs": [
        {"title": "Levels",     "artist": "Avicii",       "seconds": 200},
        {"title": "Wake Me Up", "artist": "Avicii",       "seconds": 247},
        {"title": "Titanium",   "artist": "David Guetta", "seconds": 245},
    ],
}
with open("gym_mix.json", "w", encoding="utf-8") as f:
    json.dump(playlist, f)                 # dict → characters → bytes → disk
print(p.stat().st_size)                    # 213 — the file exists, 213 bytes

print(p.read_text(encoding="utf-8"))       # the exact characters on disk
# (all 213 on one line; middle shortened here to fit the page):
# {"name": "Gym Mix", "songs": [{"title": "Levels", "artist": "Avicii",
#  "seconds": 200}, {"title": "Wake Me Up", … "seconds": 245}]}

with open("gym_mix.json", "r", encoding="utf-8") as f:
    loaded = json.load(f)                  # characters → a brand-new dict
print(loaded == playlist)                  # True  — same shape, same values
print(loaded is playlist)                  # False — a different object
print(id(playlist), id(loaded))            # 1455405821952 1455402595008 (yours differ)

JSON's vocabulary is smaller than Python's, and it fails in two different ways — one silent, one loud. We ran both:

print(json.loads(json.dumps((200, 247, 245))))
# [200, 247, 245]  ← tuple went in, LIST came out. Silent shape change.

json.dumps({"Levels", "Titanium"})
# TypeError: Object of type set is not JSON serializable

json.dumps(3 + 4j)
# TypeError: Object of type complex is not JSON serializable

import csv
rows = [["title", "artist", "seconds"],
        ["Levels", "Avicii", 200],
        ["Wake Me Up", "Avicii", 247],
        ["Titanium", "Guetta, David", 245]]      # a comma INSIDE a field
with open("gym_mix.csv", "w", newline="", encoding="utf-8") as f:
    csv.writer(f).writerows(rows)

print(Path("gym_mix.csv").read_text(encoding="utf-8"))
# title,artist,seconds
# Levels,Avicii,200
# Wake Me Up,Avicii,247
# Titanium,"Guetta, David",245      ← quoted automatically

with open("gym_mix.csv", newline="", encoding="utf-8") as f:
    print(list(csv.reader(f))[3])
# ['Titanium', 'Guetta, David', '245']   ← note: '245' is a str now!

The playlist now survives the process — but the program grew: songs, saving, loading, formats, all crammed into one file. Chapter 16 slices it into modules, and shows exactly what Python does, line by line, the first time you write import →

16Modules & import

In Chapter 15 the playlist learned to survive a power cut. But the whole program still lives in one ever-growing file. This chapter splits it across files: you will watch import do its four-step job — find, compile, run, cache — prove that a module's code executes exactly once per process, and give every project a private shelf of packages so versions never fight.

One file gets crowded

By the end of Chapter 15, playlist.py was doing three jobs at once: the JSON disk code, the total-time maths, the menu loop. Every feature makes the scroll longer, and reusing the disk code in another project means copy-paste — two copies that immediately start drifting apart.

Python's unit of splitting is the module. A module starts life as an ordinary .py file; the statement import playlist_store makes Python load that file and hand you one object holding everything the file defined. Keep the two levels separate: the file on disk holds bytes (ch 15), the module object lives on the heap (ch 0). So the program becomes three files in one folder — here are the two that matter today:

"""Disk code from Chapter 15, now living in its own file."""
import json                     # a module importing modules
from pathlib import Path

print("playlist_store.py is running, top to bottom")   # a tracer

def save_playlist(songs, path):
    Path(path).write_text(json.dumps(songs, indent=2), encoding="utf-8")

def load_playlist(path):
    return json.loads(Path(path).read_text(encoding="utf-8"))

import playlist_store                # the whole machinery fires here

songs = [
    {"title": "Levels",     "artist": "Avicii",       "seconds": 200},
    {"title": "Wake Me Up", "artist": "Avicii",       "seconds": 247},
    {"title": "Titanium",   "artist": "David Guetta", "seconds": 245},
]

playlist_store.save_playlist(songs, "playlist.json")   # reach through the module
back = playlist_store.load_playlist("playlist.json")
print(len(back), "songs back from disk")

$ python playlist_app.py
playlist_store.py is running, top to bottom
3 songs back from disk

The tracer print() at the top of playlist_store.py is planted on purpose: it reports the exact moment that file's code runs. It is about to earn its keep.

Same behaviour, three tidy files. But look again at the first line of playlist_app.py: it names no folder, no drive, not even the .py. How did Python find that file? →

Step 1: FIND — the sys.path search

import playlist_store gives Python only a name, so step 1 is a search. The list of places to look sits in sys.path — an ordinary list (ch 6) of folder paths, checked front to back:

import sys
for place in sys.path:      # a plain list of folder paths (ch 6)
    print(place)

$ python show_path.py
C:\…\ch16demo
C:\Users\ajair\anaconda3\python312.zip
C:\Users\ajair\anaconda3\DLLs
C:\Users\ajair\anaconda3\Lib
C:\Users\ajair\anaconda3
C:\Users\ajair\AppData\Roaming\Python\Python312\site-packages
C:\Users\ajair\anaconda3\Lib\site-packages
…

Real output from my machine, trimmed at two marked spots: the first path is shortened (it is simply the folder the script lives in) and the … folds three more site-packages entries. Yours will differ — but the shape won't: slot 0 is the running script's own folder (that is how playlist_app.py found playlist_store.py with no path at all), then the standard library that shipped with Python, then site-packages, the shelf pip fills (§ end of chapter). The first folder holding a match wins and the search stops. Two staging notes for honesty: a few modules like sys are baked into the interpreter itself and answer before any folder is searched, and yes — that .zip is importable as-is. If every slot misses:

$ python bad_import.py
Traceback (most recent call last):
  File "C:\…\ch16demo\bad_import.py", line 1, in <module>
    import playlist_shop
ModuleNotFoundError: No module named 'playlist_shop'

Python now holds a path to a file full of text. A CPU cannot execute text (ch 0). Step 2 fixes that →

Steps 2, 3, 4: compile, run once, cache

Step 2: COMPILE. My playlist_store.py is 408 bytes of source text, and Chapter 0's pipeline applies to every module: source → bytecode → PVM. Python compiles the file and saves the result beside it, in a folder it creates itself: __pycache__\playlist_store.cpython-312.pyc — module name, interpreter, version baked into the filename; 1,053 bytes on my disk. Next import, even days later in a fresh process, Python sees the source unchanged and skips straight to the saved bytecode. Only imported files get this: playlist_app.py, run directly, never appears in __pycache__.

Step 3: RUN. The bytecode executes — the whole file, top to bottom. The import json line runs, the def lines build function objects (ch 9), the tracer fires. And it happens once per process:

import playlist_store   # first import: runs the file
import playlist_store   # second import: cache hit, runs nothing
print("imported twice, but the print above fired once")

$ python import_twice.py
playlist_store.py is running, top to bottom
imported twice, but the print above fired once

Step 4: CACHE. The finished module object goes into sys.modules, a dict (ch 7) mapping name strings to module objects. Every later import of that name is one dict lookup:

import sys
print("json" in sys.modules)          # before we import it
import json
print("json" in sys.modules)          # after
print(sys.modules["json"] is json)    # the cache entry IS our module

$ python cache_check.py
False
True
True

The price gap is brutal: on my machine the first import playlist_store took about 14.5 ms (find, load bytecode, run the file — which itself imports json and pathlib); the second took about 0.001 ms. Machine-dependent numbers, but the ratio is the lesson: one dict lookup.

So import handed playlist_app.py a "module object". What is that thing made of? Time to crack it open →

What you got: an object with a dict inside

Everything in Python is an object (ch 2) — modules included. Crack one open:

import playlist_store                 # (its tracer line prints first)

print(type(playlist_store))
print(vars(playlist_store) is playlist_store.__dict__)
print([k for k in vars(playlist_store) if not k.startswith("__")])
print(vars(playlist_store)["save_playlist"])

$ python inspect2.py
playlist_store.py is running, top to bottom
<class 'module'>
True
['json', 'Path', 'save_playlist', 'load_playlist']
<function save_playlist at 0x000001D09E175BC0>

A module object's namespace is a dict — vars(playlist_store) is playlist_store.__dict__, the same object (that heap address in the last line changes every run). Its keys read like the file itself: json and Path were created by the import lines inside playlist_store.py, then come the two functions, plus dunder entries like __name__ and __file__. So playlist_store.save_playlist(...) is a lookup in that dict followed by a call. Which makes from ... import ... easy to say precisely — it does not import less:

from playlist_store import load_playlist   # watch the print fire anyway

import sys
print("playlist_store" in dir())          # did I get a label for the module?
print("playlist_store" in sys.modules)    # is the module object still there?
print(load_playlist is sys.modules["playlist_store"].load_playlist)

$ python from_import.py
playlist_store.py is running, top to bottom
False
True
True

Same four steps, same full run — the tracer proves it. The only difference is which label lands in your namespace: import x binds x to the module object; from x import y binds y straight to one object out of the module's dict (label model, ch 3).

One dunder in that dict — '__name__' — gets a different value depending on how the file was started. That little fact powers a line you'll meet in every real Python project →

__name__: one file, two lives

Before running a file's code, the interpreter stamps __name__ by one rule: started directly (python playlist_stats.py), the module is named "__main__"; imported, it gets its own name. Watch both lives:

"""Total-time maths for the playlist."""

def total_seconds(songs):
    return sum(s["seconds"] for s in songs)

print("my __name__ is", __name__)

if __name__ == "__main__":               # true only when run directly
    demo = [{"title": "Levels", "seconds": 200},
            {"title": "Wake Me Up", "seconds": 247},
            {"title": "Titanium", "seconds": 245}]
    print("demo total:", total_seconds(demo), "seconds")

$ python playlist_stats.py
my __name__ is __main__
demo total: 692 seconds

$ python use_stats.py        # a 2-line file: import playlist_stats + a print
my __name__ is playlist_stats
imported it; no demo ran

That is the entire secret of the guard: code under if __name__ == "__main__": runs only when the file is the program, never when it is someone else's building block. playlist_stats.py can carry its own demo drive and still be imported silently by playlist_app.py — no stray prints, no double work. (692 seconds, by the way: Levels 200 + Wake Me Up 247 + Titanium 245.)

Your shelf now holds three modules. Python ships with a few hundred more, and the internet holds hundreds of thousands. Where do they all live — and how do you keep them from fighting? →

Shelves: packages, the stdlib, pip and venv

One more word, then the map. A package is a folder of modules. You have been using one all chapter — json is not a single file:

import importlib.util
print(importlib.util.find_spec("json").origin)

$ python where_json.py
C:\Users\ajair\anaconda3\Lib\json\__init__.py

Importing a package finds the folder and runs its __init__.py — the same four steps. (Publishing your own package to PyPI is a real road; it's just not this book's road.) The standard library is the shelf that ships with Python — five picks, playlist-flavoured:

import random, math, datetime, pathlib, json

songs = ["Levels", "Wake Me Up", "Titanium"]
print(random.choice(songs))                       # different every run
print(math.fsum([200, 247, 245]), "seconds")      # exact float sum
print(datetime.timedelta(seconds=692))            # seconds -> h:mm:ss
print(pathlib.Path("playlist.json").stat().st_size, "bytes on disk")
print(json.dumps({"title": "Levels"}))            # object -> text (ch 15)

$ python shelf_demo.py
Titanium
692.0 seconds
0:11:32
255 bytes on disk
{"title": "Levels"}

Everything beyond the stdlib comes from pip, which downloads third-party packages into site-packages — the tail slots of our sys.path printout. But a shelf has one slot per package name: it cannot hold requests 2.31 for project A and requests 2.32 for project B. The fix is venv: a private shelf per project. Built for real on my machine (the ls and pip --version outputs are captured; the last two lines are for your terminal — activation changes your live shell, and installs need internet):

$ python -m venv demo_env            # build the private shelf
$ ls demo_env/Lib/site-packages      # a brand-new shelf holds only…
pip
pip-24.2.dist-info
$ demo_env/Scripts/pip --version     # …and that pip serves THIS shelf
pip 24.2 from C:\…\demo_env\Lib\site-packages\pip (python 3.12)

demo_env\Scripts\activate            # run in YOUR terminal (macOS/Linux: source demo_env/bin/activate)
pip install rich                     # lands on this project's shelf, no other

The playlist program is now three honest modules — but look at save_playlist(songs, path). It will accept anything: a list, a string, the number 7. Hand it a song whose seconds are the string "200" and nothing complains — until the maths quietly breaks three files away. Chapter 17 puts the contract in writing, right on the label: type hints. →

17Type hints — intent the tools can check

The playlist program now lives in modules (ch 16), but every function still keeps a secret: what kind of object each parameter expects. In this chapter you write that intent down — seconds: int — watch Python ignore it completely, then meet the tool that reads it and catches wrong-type bugs before the program ever runs. By the end, one decorator will turn those written-down intents into three methods you no longer have to write.

Write the intent down

Chapter 3 settled it: a name is a typeless label. seconds = 200 ties the label to an int object; nothing stops the very next line from re-tying it to a string. The object has a type. The label never did.

Here is the Volume 2 twist: Python lets you write down what a label is meant to reference. Put a colon and a type after the name. Functions get the same treatment — each parameter takes its own : type, and an arrow before the final colon declares the return:

seconds: int = 200                      # "seconds is meant to reference an int"

def total(lengths: list[int]) -> int:   # takes a list of ints, hands back an int
    return sum(lengths)

print(total([200, 247, 245]))           # Levels + Wake Me Up + Titanium
# 692

The official word is annotation; everyone says type hint. Be precise about where it hangs: the hint describes the name, not the object. The int object 200 is the same object it was in Chapter 3 — no sticker on it, no extra bytes in it. The note is pinned to the label.

You just made a promise in writing. Let's break it and see who complains. →

Break the promise — nothing happens

Break it in the dumbest way possible: promise an int, deliver a string.

seconds: int = "oops"    # the hint says int; the object is a str
print(seconds)           # oops                      <- runs fine
print(type(seconds))     # <class 'str'>             <- really a str
print(__annotations__)   # {'seconds': <class 'int'>} <- where the hint went

No error. Not even a warning. CPython evaluated the hint (the name int), filed it in __annotations__ — an ordinary dict that modules and functions carry — and never looked at it again. Functions store theirs the same way: total.__annotations__ from last section prints {'lengths': list[int], 'return': <class 'int'>}, evaluated once when the def statement ran (ch 9), then left alone.

So a type hint is metadata: data about your program, stored inside your program, enforcing nothing.

A promise nobody checks is decoration. For hints to be worth anything, something has to read the file — without running it. →

The checker — bugs caught before the run

That something is a type checker: a separate program that reads your source top to bottom, follows every hint, and reports each place where the promises contradict. It never executes a line. The two you'll meet everywhere are mypy (pip install mypy) and pyright, the engine behind VS Code's red underlines. Point mypy at last section's file:

$ python -m mypy hints.py
hints.py:1: error: Incompatible types in assignment (expression has type "str", variable has type "int")  [assignment]
Found 1 error in 1 file (checked 1 source file)

Now the bug this chapter exists for. Chapter 15 read the playlist back from disk, and files hold text — a duration is a string until you convert it. Suppose one slips through:

def total(lengths: list[int]) -> int:
    return sum(lengths)

print(total([200, 247, "245"]))   # "245" — fresh from a file, never converted

Run it and CPython dies mid-sum — but only when that line executes, with exactly that data: TypeError: unsupported operand type(s) for +: 'int' and 'str'. The same file through the checker, before any run:

$ python -m mypy playlist.py
playlist.py:4: error: List item 2 has incompatible type "str"; expected "int"  [list-item]
Found 1 error in 1 file (checked 1 source file)

The checker earns its keep hardest on values that are only sometimes there. →

str | None — the honest hint

Chapter 7's .get returns None on a miss, and None is a real object (ch 2) with almost no methods. Ask the library for an artist it doesn't have:

def find_artist(title: str) -> str | None:   # honest: a str, OR None on a miss
    artists = {"Levels": "Avicii", "Titanium": "David Guetta"}
    return artists.get(title)                # .get -> None when the key is absent (ch 7)

artist = find_artist("Wake Me Up")           # not in this dict -> None
print(artist.title())
# AttributeError: 'NoneType' object has no attribute 'title'

The return hint str | None reads "a str, or None" — the bar means either. (Older code spells it Optional[str]; identical meaning. It does not mean "optional parameter" — it means "or None".) The moment the hint is honest, the checker forces the what if it's missing? question at every call site:

$ python -m mypy library.py
library.py:6: error: Item "None" of "str | None" has no attribute "title"  [union-attr]
Found 1 error in 1 file (checked 1 source file)

The fix is a guard, and mypy follows it — inside the else, artist can only be a str (checkers call this narrowing):

if artist is None:                # 'is' — identity check, ch 4
    print("no artist on file")    # the miss case, handled on purpose
else:
    print(artist.title())         # here the checker KNOWS artist is a str
# prints: no artist on file      -- and mypy: Success: no issues found in 1 source file

Single values, done. But the playlist itself is a container — how do you hint what's inside? →

Containers — hint what's inside

A container hint answers two questions: which container, and what it holds. Square brackets carry the inside part:

playlist: list[tuple[str, str, int]] = [      # each row: (title, artist, seconds)
    ("Levels", "Avicii", 200),
    ("Wake Me Up", "Avicii", 247),
    ("Titanium", "David Guetta", 245),
]
durations: dict[str, int] = {"Levels": 200, "Wake Me Up": 247, "Titanium": 245}

print(durations["Levels"].title())            # an int has no .title() ...

Read tuple[str, str, int] position by position: slot 0 a str, slot 1 a str, slot 2 an int — a fixed shape, exactly what Chapter 6 said tuples were for. And remember what a list actually stores (ch 6): 8-byte references. list[int] declares what those references are meant to point at; the running list never checks. The checker does, and it follows hints through every lookup — it knows durations["Levels"] is an int:

$ python -m mypy durations.py
durations.py:8: error: "int" has no attribute "title"  [attr-defined]
Found 1 error in 1 file (checked 1 source file)

Every hint so far cost keystrokes and paid in caught bugs. One decorator flips the trade: hints that write code for you. →

@dataclass — hints that write your dunders

In Chapter 12 you wrote the same three dunders for every data-shaped class: an __init__ that copies parameters onto self, a __repr__ so printing is honest, an __eq__ so value comparison works. @dataclass reads the field hints and writes all three while the class statement executes:

from dataclasses import dataclass

@dataclass                    # reads the hints below, generates the dunders
class Song:
    title: str                # field 1 -> first __init__ parameter
    artist: str               # field 2
    seconds: int              # field 3

levels = Song("Levels", "Avicii", 200)   # the __init__ you didn't write
copy   = Song("Levels", "Avicii", 200)

print(levels)                 # Song(title='Levels', artist='Avicii', seconds=200)
print(levels == copy)         # True   -- generated __eq__ compares field values
print(levels is copy)         # False  -- still two objects, two labels (ch 3)

The free __repr__ prints every field; the free __eq__ makes == mean "same field values" while is still means "same heap object". And because the generated __init__ carries the hints, construction is checked too — Song("Levels", "Avicii", "200") runs without a squeak (the rule from section 2 never bends), but the checker reports:

$ python -m mypy songs.py
songs.py:9: error: Argument 3 to "Song" has incompatible type "str"; expected "int"  [arg-type]
Found 1 error in 1 file (checked 1 source file)

So when do hints earn their keep? Wherever code has more than one reader — teammates, code review, you in six months. Wherever the crash site sits far from the mistake, which is precisely the big-program disease. And every time you type a dot: hint artist as str and your editor lists string methods instantly, because autocomplete is a type checker in a party hat. When are they ceremony? A five-line script you run once and delete — hinting it catches nothing the first run wouldn't.

The checker just certified every type in the playlist program. It cannot certify a single answer: a total() that returns the wrong sum with the right type sails straight through. Catching wrong answers takes a different machine — Chapter 18 puts the program on a test bench. →

18Testing & debugging

The playlist program now survives restarts (ch15), lives in tidy modules (ch16), and declares its types (ch17). This chapter is where you stop hoping it works and start checking it works — automatically, on every change. You will write your first test file, read pytest's red and green for real, and learn the six-step loop that turns a 2 a.m. crash into a permanent upgrade.

The bug we already shipped once

Rewind to trace T15 (Vol 1 ch10). The prime checker, as first written:

def is_prime(n):
    for d in range(2, n):        # try every candidate divisor
        if n % d == 0:           # a clean split? not prime
            return False
    return True                  # survived the loop: prime

print(is_prime(1))               # → True   (wrong — 1 is not prime)
print([n for n in range(1, 20) if is_prime(n)])
# → [1, 2, 3, 5, 7, 11, 13, 17, 19]   ← 1 crashed the party
print(is_prime(0), is_prime(-7)) # → True True   (worse)

Walk n=1 like a trace: range(2, 1) is empty, so the loop body runs zero times, so execution falls through to return True. The fallback answer wins by default. "Loop never runs, default escapes" is a bug class that ships in real products constantly — and we only caught ours because we hand-traced it in Volume 1, then fixed it with a guard. Nothing pins that fix. Delete the guard in a refactor — it looks redundant, guards often do — and Python runs the broken version without a squeak.

Python has a one-word statement for exactly that →

assert — a claim with teeth

assert is a statement that takes an expression and, optionally, a message: assert claim, "message". Python evaluates the claim. Truthy (ch4)? Nothing happens — no output, no trace, the next line runs. Falsy? Python raises an AssertionError — a real exception riding the ch13 machinery — carrying your message.

assert is_prime(17)                             # true claim — silence
assert not is_prime(1), "1 must not be prime"   # false claim — bang
print("never reached")

Run against the unguarded is_prime, CPython 3.12 answers — arrows pointing at the exact sub-expression that lied:

Traceback (most recent call last):
  File "assert_demo.py", line 8, in <module>
    assert not is_prime(1), "1 must not be prime"   # false claim — bang
           ^^^^^^^^^^^^^^^
AssertionError: 1 must not be prime

One assert in a script dies on the first failure and takes the whole run with it. You want fifty claims, each run in isolation, all reported at once →

pytest — red, then green

A test runner is a program whose only job is to find your claims, run each one separately, and report. The standard choice is pytest — not in the stdlib, so: pip install pytest (this machine got 7.4.4). Discovery is pure naming convention: it collects files named test_*.py, and inside them, functions named test_*. Each test is a plain function making three moves — Arrange the data, Act on it once, Assert the result. Here is test_playlist.py, sitting next to our ch16 module playlist.py:

from playlist import total_seconds, is_prime

def test_three_songs():
    songs = [                                  # Arrange: build the world
        {"title": "Levels",     "seconds": 200},
        {"title": "Wake Me Up", "seconds": 247},
        {"title": "Titanium",   "seconds": 245},
    ]
    assert total_seconds(songs) == 692         # Act + Assert: poke once, pin the answer

def test_empty_playlist():
    assert total_seconds([]) == 0              # the sum of nothing is 0

def test_one_is_not_prime():
    assert not is_prime(1)                     # the T15 claim, made permanent

def test_seventeen_is_prime():
    assert is_prime(17)                        # and a claim from the happy middle

Rewind playlist.py to the morning before T15's fix — no guard — and run pytest in that folder (output below is real; paths and a plugin line trimmed):

======================= test session starts ========================
platform win32 -- Python 3.12.7, pytest-7.4.4, pluggy-1.0.0
collected 4 items

test_playlist.py ..F.                                         [100%]

============================ FAILURES ===============================
_____________________ test_one_is_not_prime _________________________

    def test_one_is_not_prime():
>       assert not is_prime(1)                     # the T15 claim, made permanent
E       assert not True
E        +  where True = is_prime(1)

test_playlist.py:15: AssertionError
==================== short test summary info ========================
FAILED test_playlist.py::test_one_is_not_prime - assert not True
==================== 1 failed, 3 passed in 0.05s ====================

Read the dot line: one character per test — . pass, F fail — and the failure report even shows where the True came from. Add the two-line guard (if n < 2: return False), run again:

test_playlist.py ....                                         [100%]
======================== 4 passed in 0.01s ==========================

Four tests, all green. But out of infinitely many possible inputs, which ones deserve a test? There's a checklist →

The edge-case drill

Bugs cluster at the edges of an input's range, not in the comfortable middle — T15's bug lived at n=1, not n=6. So drill every function with six probes: empty, zero, one, negative, huge, wrong type. Here is the drill on total_seconds and the fixed is_prime, every line actually run:

levels = {"title": "Levels", "seconds": 200}
levels_x_million = [levels] * 1_000_000

total_seconds([])                # → 0            empty: sum of nothing — sensible, now pin it
total_seconds([levels])          # → 200          one: no fencepost stumble
total_seconds(levels_x_million)  # → 200000000    huge: correct (a million dict lookups — ch19 will care)
total_seconds(None)              # → TypeError: 'NoneType' object is not iterable

is_prime(0), is_prime(1), is_prime(-7)   # → (False, False, False)   the guard holds all three
is_prime(15_485_863)             # → True — in 0.55 s on this machine (the millionth prime)

Note what the wrong-type probe revealed: the TypeError comes from Python's for asking None for items — Python chose that behaviour, not us. And the negative probe? A song with seconds=-5 just quietly shrinks the total. No crash, no answer — a question.

Now watch the drill work as a suite — several claims holding one function from different angles at once:

The drill gives you candidates. But what deserves a test at all — and what does a wall of green actually certify? →

What to test — and what green really proves

Test the behaviour you own and the boundaries you chose. total_seconds returning 692 for our three songs: yours. Whether int + int adds correctly: not yours — CPython ships and runs its own test suite before every release, and re-proving list.append inside your project spends minutes proving nothing about your code. The subtle territory is the seam: your assumptions about their behaviour ("csv hands every field back as str") — those are yours to pin, and one of them bites us in the next section.

Here is the honest seam, made tangible. This impostor passes our entire four-test suite — yet it is not a prime checker:

So every test is a bug you imagined in advance. The one you didn't imagine still arrives — as a traceback, at the worst possible hour →

Debugging is a loop, not a panic

The playlist now loads from playlist.csv through the ch15 loader — and the program dies (real output, path trimmed):

Traceback (most recent call last):
  File "app.py", line 4, in <module>
    print(total_seconds(songs))
          ^^^^^^^^^^^^^^^^^^^^
  File "playlist.py", line 4, in total_seconds
    total += song["seconds"]
TypeError: unsupported operand type(s) for +=: 'int' and 'str'

Method, not panic — six steps. 1 Read it bottom-up (ch13): last line first — what (adding int to str); one frame up — where (playlist.py line 4); above that — who called. 2 Reproduce it small: in the REPL, 200 + "247" raises the same TypeError: unsupported operand type(s) for +: 'int' and 'str' — two characters of code now stand in for the whole crash. 3 Form ONE hypothesis — singular: "the seconds came back from the CSV as text." Not five guesses at once; one, falsifiable. 4 Check it: print(repr(song["seconds"])) → '247' — quotes, so str. (Or drop breakpoint() on the line above: Python's built-in door into the pdb debugger, where n steps and p prints.) 5 Fix at the boundary: a file holds bytes; the csv module decodes them to text and hands back str — restoring numbers is your job (ch15). So the loader converts once: row["seconds"] = int(row["seconds"]), and every downstream function sees ints. 6 The step everyone skips: add the test that would have caught it — a regression test, a claim pinning a bug you actually had so it can never quietly return:

from playlist import load_playlist

def test_loaded_seconds_survive_the_sum():
    songs = load_playlist("playlist.csv")   # Arrange: the real file, the real loader
    assert total_seconds(songs) == 692      # Act + Assert: str-seconds would crash here

test_playlist.py .....                                        [100%]
======================== 5 passed in 0.02s ==========================

Five tests, all green — and one of them whispered a warning: is_prime(15_485_863) answered correctly but took over half a second on this machine, and while it ground away, the program could do nothing else. A correct program can still freeze mid-thought. Chapter 19 opens exactly there: where the time goes, how to measure it, and how to keep the playlist responsive while the heavy work runs →

19Threads, async & the GIL

Chapter 18 proved the playlist app correct. Now give it a real job — fetch cover art for "Levels", "Wake Me Up" and "Titanium" — and it freezes for three straight seconds while the network answers. After this chapter you can overlap waits with threads, explain the GIL from chapter 3's refcounts instead of quoting folklore, buy true parallelism with processes, juggle thousands of waits with async/await — and know which tool to reach for before you write a line.

Two kinds of slow

Every slow program is slow in one of exactly two ways, and the fix for one is poison for the other. CPU-bound means the processor is the bottleneck: summing 20 million song durations keeps it busy every microsecond, and only more compute helps. I/O-bound means the processor is idle: you asked the network for the "Levels" cover, and the reply takes on the order of 100 ms to arrive. Chapter 0's numbers make that shocking — a CPU steps through instructions by the billions per second, so in the time one network reply crawls back it could have executed tens to hundreds of millions of instructions (machine-dependent, but always a huge number). Instead it sits there. Disk reads, database queries, input() — same shape: the program is "running", the CPU is doing nothing.

If the CPU is a spectator for most of a download, could it run the other two downloads in the gaps? Python ships a module for exactly that →

Three downloads, one second

A thread is an independent line of execution inside one process: it gets its own call stack of frames (ch 9) but shares the same heap of objects (ch 0) with every other thread. The threading module lets you start more of them. One honest simplification before the code: a real download needs a network; time.sleep(1.0) is our stand-in for that one-second wait — it idles the CPU the same way a socket does, but reproducibly, so the timings below are about waiting, not about my Wi-Fi.

import threading
import time

titles = ["Levels", "Wake Me Up", "Titanium"]

def fetch_cover(title):
    time.sleep(1.0)          # stand-in for one network wait
    print(f"cover art ready: {title}")

# --- one after another ---
start = time.perf_counter()
for title in titles:
    fetch_cover(title)                 # wait, wait, wait
print(f"sequential: {time.perf_counter() - start:.2f} s")

# --- three threads ---
start = time.perf_counter()
workers = [threading.Thread(target=fetch_cover, args=(t,)) for t in titles]
for w in workers:
    w.start()                          # begin all three waits NOW
for w in workers:
    w.join()                           # block until each finishes
print(f"threaded:   {time.perf_counter() - start:.2f} s")

Note the shape: Thread(target=fetch_cover, args=(t,)) takes the function object (ch 9 — no parentheses) plus its arguments; start() launches it; join() waits for it. On this machine (yours will differ by a few hundredths):

cover art ready: Levels
cover art ready: Wake Me Up
cover art ready: Titanium
sequential: 3.00 s
cover art ready: Levels
cover art ready: Titanium
cover art ready: Wake Me Up
threaded:   1.00 s

Three one-second waits, one second of wall-clock — the time a clock on the wall measures, the time your user feels. The waits overlapped.

Free speedup! So point two threads at pure arithmetic and it halves too, right? Chapter 3's refcounts are about to say no — loudly →

The GIL, derived from a refcount

Recall chapter 3: every object carries ob_refcnt, a plain integer in its header, counting the references to it. Every name you bind, every list slot you fill, ticks some object's count up; every del, every scope exit, ticks one down; at zero, CPython frees the object on the spot. A tick is three machine steps: read the integer, add one, write it back.

Now add a second thread — same heap, remember. Suppose ob_refcnt is 1 and both threads bind a new name to that object at the same instant. Both read 1. Both compute 2. Both write 2. The truth is 3; memory says 2. Later, two of those references go away: 2 → 1 → 0 — and CPython frees an object a third name still points at. That is not an exception you can catch; that is a corrupted interpreter.

CPython's fix is blunt: one Global Interpreter Lock. At any instant, at most one thread in the process may execute Python bytecode, so no two ticks can ever collide. And here is why the download demo still flew: a thread about to wait — time.sleep, a socket read, a file read — is C code that releases the GIL before asking the operating system to pause it. Waiting needs no bytecode.

That's the theory of the crime. Let's time it →

The crunch test: threads can't, processes can

crunch below is pure bytecode — 20 million loop iterations, no waits, about 0.6 s per call on this machine. If threads ran bytecode in parallel, two threads should finish two jobs in about 0.6 s of wall-clock.

import threading
import time

N = 20_000_000

def crunch(n):                 # pure bytecode: no waits anywhere
    total = 0
    for _ in range(n):
        total += 1
    return total

start = time.perf_counter()
crunch(N); crunch(N)           # one thread, both jobs back to back
one = time.perf_counter() - start

start = time.perf_counter()
a = threading.Thread(target=crunch, args=(N,))
b = threading.Thread(target=crunch, args=(N,))
a.start(); b.start()           # two threads, one job each
a.join(); b.join()
two = time.perf_counter() - start

print(f"one thread, both jobs: {one:.2f} s")
print(f"two threads:           {two:.2f} s")

one thread, both jobs: 1.21 s
two threads:           1.21 s

Identical, run after run (this machine; the pattern, not the digits, is the point). crunch never waits, so it never releases the GIL voluntarily; the two threads just take 5 ms turns at one stream of bytecode. Same total bytecode, same total time. The escape hatch is multiprocessing: a process is a whole separate running program — separate interpreter, separate heap, separate refcounts, and therefore its own GIL. The switch is nearly mechanical:

import multiprocessing
import time

# N and crunch(n) exactly as above

if __name__ == "__main__":         # required — see the warning below
    start = time.perf_counter()
    crunch(N); crunch(N)           # one process, both jobs
    one = time.perf_counter() - start

    start = time.perf_counter()    # timer includes process launch
    a = multiprocessing.Process(target=crunch, args=(N,))
    b = multiprocessing.Process(target=crunch, args=(N,))
    a.start(); b.start()           # two interpreters, one job each
    a.join(); b.join()
    two = time.perf_counter() - start

    print(f"one process, both jobs: {one:.2f} s")
    print(f"two processes:          {two:.2f} s")

one process, both jobs: 1.17 s
two processes:          0.68 s

Faster — but 0.68, not 0.585: launching a process is heavyweight, and data pays a toll at the border. Arguments and results cross between processes as pickled bytes through a pipe — chapter 15's law again: what leaves a process is bytes, never objects.

Processes buy real parallel compute — paid in startup time and copied bytes. But for the download problem there is something lighter than threads: no second anything →

async/await: one thread, parked frames

A thread is the operating system pausing and resuming your code wherever it likes. A coroutine is Python doing the same trick to itself, in one thread, at positions you mark. async def defines a coroutine function; calling it runs nothing — it builds a coroutine object, a not-yet-started call wrapped around its own frame. await marks the exact line where that frame may suspend: it is parked on the heap, not destroyed — chapter 9's frames doing chapter 10's generator pause-and-resume trick. While one frame is parked, the event loop — a scheduler running in your single thread — resumes whichever other coroutine's wait has finished. One thread, so the GIL never even changes hands.

import asyncio
import time

async def fetch_cover(title):
    await asyncio.sleep(1.0)      # park this frame; free the loop
    return f"cover art ready: {title}"

async def main():
    start = time.perf_counter()
    results = await asyncio.gather(     # run all three, collect results
        fetch_cover("Levels"),
        fetch_cover("Wake Me Up"),
        fetch_cover("Titanium"),
    )
    for line in results:
        print(line)
    print(f"wall clock: {time.perf_counter() - start:.2f} s")

asyncio.run(main())                     # start the event loop

cover art ready: Levels
cover art ready: Wake Me Up
cover art ready: Titanium
wall clock: 1.01 s

Three seconds of waiting, 1.01 s of wall-clock (this machine) — and unlike the threads, gather hands results back in the order you listed them, whoever finished first. One catch: the loop only switches at await. Swap asyncio.sleep for the blocking time.sleep and the measured wall-clock snaps back to 3.00 s — a C-level block the loop cannot see freezes everyone.

Threads, processes, coroutines — three tools, one question left: which one, when? →

Choosing: waits versus work

The whole chapter compresses into one sorting rule. Count your waits and weigh your work. Hundreds or thousands of waits (a cover for every song in a 10,000-track library): async — one thread juggles them all for the price of parked frames. A few waits, or a library that blocks and offers no await: threads — the OS does the switching and waiting threads release the GIL. Real computation: processes — the only tool here that puts two cores to work on bytecode, if the win beats the copy-toll. And sequential remains the right default until a measurement says otherwise (ch 18's discipline).

The playlist app now saves (ch 15), splits into modules (ch 16), declares its types (ch 17), proves itself (ch 18), and downloads without freezing. One thing left — the best part: putting every tool from both volumes into a single build. Chapter 20 assembles the complete app — four files, one job each: the model, the engine, the storage, the loop — and ships it →

20The capstone — ship the playlist app

This is the chapter where you stop doing exercises. You will assemble a complete, working app — model, engine, storage, command loop, tests — in five stages, four files, 171 lines, standard library only (plus ch18's pytest to run the proof). Every stage below was run for real; every output is pasted from an actual session, not imagined.

The blueprint — four files, one job each

Volume 1 gave you the machine. Chapters 15–19 gave you files, modules, hints, tests, and concurrency. Now spend all of it at once, on an app called musicbox: a playlist you talk to from the terminal, that still knows your songs tomorrow.

An app is just a program with a job description, and ours splits into four modules (a module is one .py file — ch16), each with exactly one reason to change. playlist.py knows what a song is and how a playlist behaves. storage.py knows how to turn that into bytes on disk and back (ch15). app.py talks to a human. test_playlist.py checks that the other three keep their promises. Look at the arrows in the figure: app.py imports the engine and the vault, storage.py imports the model, and nothing imports app.py. Dependencies flowing one way is nine-tenths of what the word architecture means.

Build order: start with the smallest thing that can be true — one song. →

Stage 1 — the model: one Song, zero boilerplate

A song is three facts travelling together: a title, an artist, a duration in seconds. Chapter 17's @dataclass turns that sentence directly into a class — the top of playlist.py:

"""The engine: the Song model and the Playlist that manages it."""
from dataclasses import dataclass


@dataclass
class Song:
    title: str        # annotations become fields (ch17)
    artist: str
    seconds: int      # duration - an int, not a str

Nine lines, and @dataclass writes __init__, __repr__, and __eq__ for you (the dunder machinery from ch12). Prove it — this ran for real:

levels = Song("Levels", "Avicii", 200)
print(levels)                                   # __repr__, free
print(levels.seconds)                           # field access
print(levels == Song("Levels", "Avicii", 200))  # __eq__, free
print(levels is Song("Levels", "Avicii", 200))  # two objects (ch3)

Song(title='Levels', artist='Avicii', seconds=200)
200
True
False

== is True because the generated __eq__ compares fields; is is False because these are two separate objects at two addresses (the label model — Vol 1 ch 3). File that away: when Stage 3 loads songs back from disk, you will get equal-but-not-identical Songs, and a test in Stage 5 pins exactly that down.

A Song can't add itself, count itself, or save itself. It needs an engine. →

Stage 2 — the engine: one object, two routes to it

The Playlist class keeps two structures pointed at the same Song objects: a list for play order (a dynamic array of 8-byte references — ch6) and a dict keyed by title for instant lookup (a hash table whose buckets record references — ch7). No Song is ever copied; each one simply has two routes leading to it. The rest of playlist.py:

class Playlist:
    def __init__(self, name: str) -> None:
        self.name = name
        self._songs: list[Song] = []           # keeps play order (ch6)
        self._by_title: dict[str, Song] = {}   # O(1) lookup (ch7)

    def add(self, song: Song) -> None:
        if song.title in self._by_title:
            raise ValueError(f"duplicate title: {song.title!r}")
        self._songs.append(song)
        self._by_title[song.title] = song      # update BOTH structures

    def remove(self, title: str) -> Song:
        if title not in self._by_title:
            raise KeyError(f"no song called {title!r}")
        song = self._by_title.pop(title)
        self._songs.remove(song)               # update BOTH structures
        return song

    def find(self, title: str) -> Song | None:
        return self._by_title.get(title)       # one hash, no scan

    def total_seconds(self) -> int:
        return sum(song.seconds for song in self._songs)

    def __len__(self) -> int:                  # so len(playlist) works
        return len(self._songs)

    def __iter__(self):                        # so for song in playlist works
        return iter(self._songs)

Walk the moves: add asks the index "seen this title?" before touching anything — a duplicate raises ValueError and the playlist stays clean. find is one hash and a bucket check, no scan. remove pops the dict entry and deletes from the list. The underscore prefix on _songs and _by_title is the "internal — hands off" convention from ch12. Run it:

road_trip = Playlist("Road Trip")
road_trip.add(Song("Levels", "Avicii", 200))
road_trip.add(Song("Wake Me Up", "Avicii", 247))
road_trip.add(Song("Titanium", "David Guetta", 245))

print(len(road_trip), "songs,", road_trip.total_seconds(), "seconds")
print(road_trip.find("Titanium"))
print(road_trip.find("Alone"))
road_trip.add(Song("Levels", "Impostor", 999))   # the trap

3 songs, 692 seconds
Song(title='Titanium', artist='David Guetta', seconds=245)
None
Traceback (most recent call last):
  ...
  File "...\playlist.py", line 20, in add
    raise ValueError(f"duplicate title: {song.title!r}")
ValueError: duplicate title: 'Levels'

list : 88 bytes (shallow)
dict : 184 bytes (shallow)
same object? True

Quit Python and all of this — list, dict, Songs — vanishes with the process's heap (ch3). Time to make it survive. →

Stage 3 — persistence, and the JSON wrinkle

Saving lives in its own module, storage.py — all 21 lines of it. asdict() (from ch17's dataclasses toolbox) flattens each Song into a plain dict; json.dump writes the lot as text (ch15):

"""Persistence: turn a Playlist into JSON text on disk, and back."""
import json
from dataclasses import asdict

from playlist import Playlist, Song


def save(playlist: Playlist, path: str) -> None:
    data = {"name": playlist.name,
            "songs": [asdict(song) for song in playlist]}
    with open(path, "w", encoding="utf-8") as f:   # with = ch11
        json.dump(data, f, indent=2)


def load(path: str) -> Playlist:
    with open(path, encoding="utf-8") as f:
        data = json.load(f)                    # gives back dicts, not Songs
    playlist = Playlist(data["name"])
    for d in data["songs"]:                    # rebuild real objects
        playlist.add(Song(d["title"], d["artist"], d["seconds"]))
    return playlist

Here is the honest wrinkle, and it matters: json.load has never heard of Song. It returns plain dicts and lists — JSON is a text format with six shapes, not a Python object freezer. So load() rebuilds each Song by hand, and it does so through playlist.add(), which means a hand-edited file with a duplicate title gets rejected by the same rule as everything else. One door in. The proof run:

328 bytes on disk
<class 'dict'>            <- what json.load returns
<class 'playlist.Song'>   <- what OUR load() returns
692

>>> json.loads(json.dumps((200, 247)))     # tuple in…
[200, 247]                                 # …list out
>>> json.dumps({1: 'Levels'})              # int key in…
'{"1": "Levels"}'                          # …str key out
>>> json.dumps({'Levels', 'Titanium'})    # a set?
Traceback (most recent call last):
  ...
TypeError: Object of type set is not JSON serializable

The data now outlives the process. But nobody can drive it yet. Give it a face. →

Stage 4 — the loop that won't die

app.py makes three moves. Move one: load the saved playlist, or start fresh if there's no file yet — try/except FileNotFoundError (ch13), because a missing file is a normal Tuesday, not a disaster. Move two: a while True loop (ch5) reads a line and partitions it into a command and the rest. Move three: four branches — and the add branch wears armour.

"""The face: a command loop. Run it:  python app.py"""
from playlist import Playlist, Song
import storage

PATH = "playlist.json"


def parse_song(rest: str) -> Song:
    """'Levels; Avicii; 200' -> Song. Raises ValueError on bad input."""
    title, artist, seconds = (part.strip() for part in rest.split(";"))
    return Song(title, artist, int(seconds))


def main() -> None:
    try:
        playlist = storage.load(PATH)
        print(f"Loaded {len(playlist)} songs from {PATH}")
    except FileNotFoundError:                  # no file = first run (ch13)
        playlist = Playlist("Road Trip")
        print("No saved playlist - starting fresh.")

    while True:
        line = input("> ").strip()
        if not line:                           # empty line? just re-prompt
            continue
        command, _, rest = line.partition(" ") # 'find Levels' -> 'find', 'Levels'
        if command == "quit":
            storage.save(playlist, PATH)       # save happens HERE, once
            print(f"Saved {len(playlist)} songs to {PATH}. Bye.")
            break
        elif command == "list":
            for song in playlist:              # __iter__ walks _songs
                print(f"  {song.title} - {song.artist} ({song.seconds}s)")
            print(f"  total: {playlist.total_seconds()}s")
        elif command == "add":
            try:                               # the shield (ch13)
                song = parse_song(rest)
                playlist.add(song)
                print(f"  added {song.title!r} ({song.seconds}s)")
            except ValueError as err:          # bad split, bad int, duplicate
                print(f"  can't add that: {err}")
        elif command == "find":
            song = playlist.find(rest)
            if song is None:
                print(f"  no song called {rest!r}")
            else:
                print(f"  {song.title} - {song.artist} ({song.seconds}s)")
        else:
            print(f"  unknown command {command!r} (add/list/find/quit)")


if __name__ == "__main__":                     # script vs import (ch16)
    main()

Look at the shield. parse_song can fail two ways — the split(";") unpack gets the wrong count, or int() chokes — and playlist.add can fail a third way, on a duplicate. All three raise ValueError, so one except catches all three and the loop answers instead of dying. A real session (the text after each > is what I typed; the rest is the program):

$ python app.py
No saved playlist - starting fresh.
> add Levels; Avicii; 200
  added 'Levels' (200s)
> add Wake Me Up; Avicii; 247
  added 'Wake Me Up' (247s)
> add Titanium; David Guetta; 245
  added 'Titanium' (245s)
> find Titanium
  Titanium - David Guetta (245s)
> add Levels; Avicii; 200
  can't add that: duplicate title: 'Levels'
> add Levels 200
  can't add that: not enough values to unpack (expected 3, got 1)
> add Faded; Alan Walker; two hundred
  can't add that: invalid literal for int() with base 10: 'two hundred'
> quit
Saved 3 songs to playlist.json. Bye.

Three different attacks — a duplicate, a missing field, a word where a number belongs — and the loop shrugged three times. Those error strings are CPython 3.12's own words, surfaced instead of splattered.

It survives bad input when you drive. Will it survive next month, when you edit remove() at midnight? Prove it. →

Stage 5 — the proof: test the traps

Chapter 18's rule: don't test that code works, test the places it would break. Our traps: a duplicate sneaking in, removing what isn't there, the two structures drifting apart, and the disk round trip. test_playlist.py:

"""The proof: tests for the traps. Run:  python -m pytest -q"""
import os

import pytest

import storage
from playlist import Playlist, Song

LEVELS = Song("Levels", "Avicii", 200)
WAKE = Song("Wake Me Up", "Avicii", 247)
TITANIUM = Song("Titanium", "David Guetta", 245)


def make_playlist() -> Playlist:          # plain helper - no fixtures needed
    playlist = Playlist("Road Trip")
    playlist.add(LEVELS)
    playlist.add(WAKE)
    return playlist


def test_add_and_total():
    playlist = make_playlist()
    assert len(playlist) == 2
    assert playlist.total_seconds() == 447


def test_duplicate_add_is_rejected():
    playlist = make_playlist()
    with pytest.raises(ValueError):        # the trap must spring (ch18)
        playlist.add(Song("Levels", "Impostor", 999))
    assert len(playlist) == 2              # and nothing slipped in


def test_remove_missing_raises():
    playlist = make_playlist()
    with pytest.raises(KeyError):          # Titanium was never added
        playlist.remove("Titanium")


def test_remove_updates_both_structures():
    playlist = make_playlist()
    playlist.remove("Levels")
    assert playlist.find("Levels") is None                 # gone from the index
    assert [s.title for s in playlist] == ["Wake Me Up"]   # and from the list


def test_save_load_round_trip():
    playlist = make_playlist()
    playlist.add(TITANIUM)
    storage.save(playlist, "roundtrip_test.json")
    loaded = storage.load("roundtrip_test.json")
    os.remove("roundtrip_test.json")             # clean up our test file
    assert [s.title for s in loaded] == ["Levels", "Wake Me Up", "Titanium"]
    assert loaded.total_seconds() == 692
    assert loaded.find("Levels") == LEVELS       # equal fields (ch12 __eq__)
    assert loaded.find("Levels") is not LEVELS   # but a NEW object (ch3)

The last two asserts are Stage 1's promise collected: the reloaded Levels is equal to the original and is not the original. Green run, for real:

$ python -m pytest -q
.....                                                                    [100%]
5 passed in 0.02s

Now the payoff. I deleted one line from remove() — the self._songs.remove(song) — exactly the ghost-song bug from Stage 2, and ran pytest again:

>       assert [s.title for s in playlist] == ["Wake Me Up"]   # and from the list
E       AssertionError: assert ['Levels', 'Wake Me Up'] == ['Wake Me Up']
E         At index 0 diff: 'Levels' != 'Wake Me Up'
E         Left contains one more item: 'Wake Me Up'
E         Use -v to get more diff

test_playlist.py:44: AssertionError
=========================== short test summary info ===========================
FAILED test_playlist.py::test_remove_updates_both_structures - AssertionError...
1 failed, 4 passed in 0.04s

The test caught the ghost before any user did, and even printed which song refused to leave. That is what the 56 lines buy: permission to change the engine without fear.

Five stages, all green. Step back and look at what you're holding. →

Ship it — the whole app, running

Here is everything, and you have already read every line of it: playlist.py in Stages 1–2, storage.py in Stage 3, app.py in Stage 4, test_playlist.py in Stage 5. That's the complete source — 171 lines, nothing hidden; the app itself needs only the standard library (the proof runs on ch18's pytest).

The final exam is the second launch. Kill the process, start it again, and ask what it remembers:

$ python app.py
Loaded 3 songs from playlist.json
> list
  Levels - Avicii (200s)
  Wake Me Up - Avicii (247s)
  Titanium - David Guetta (245s)
  total: 692s
> quit
Saved 3 songs to playlist.json. Bye.

Every object from the last session is gone — this is a new heap, new addresses, new Songs rebuilt from 328 bytes of text. The music survived the death of the process. That's the whole point of Volume 2 in one line of output.

Now watch your own architecture run. Fire a command and follow it through the layers you built:

One page left: everything Volume 2 handed you, on a single card — and the road past it. →

The Volume 2 card — and where the road goes

Everything this volume added to your machine model, one row each:

And ten terms for the Volume 1 glossary's new neighbours:

buffer

a holding area in RAM where written bytes wait before the OS flushes them to disk (ch15)

module

one .py file; runs once on first import, then served from cache (ch16)

cache

a stored result reused so work isn't redone — sys.modules, lru_cache (ch11/16)

annotation

the type label after : or ->; stored on the function/class, not enforced at runtime (ch17)

dataclass

a class whose __init__/__repr__/__eq__ are generated from its field annotations (ch17)

test

a function that asserts one promise your code makes; collected and run by pytest (ch18)

red/green

watch a test fail before making it pass — the only proof the test can actually catch the bug (ch18)

GIL

CPython's global interpreter lock: one thread executes bytecode at any instant (ch19)

coroutine

a function defined with async def; pauses at await instead of blocking the thread (ch19)

event loop

the scheduler that runs coroutines, switching between them at every await (ch19)

Where this goes next. Three roads leave this chapter. Databases: our save() rewrites all 328 bytes every time — fine for 3 songs, brutal for 3 million; when that bill comes due, sqlite3 is already sitting in the standard library. The web: a server is a while True loop that reads bytes from a network socket instead of input() — you already built the hard half. Sharing: a package is modules in a folder with a name tag; pip fetches other people's. This course won't walk those roads — deliberately. You now hold the machine model underneath all three, and that's the thing documentation always assumes and never teaches. You can take it from the docs now. That was the plan all along.

Chapter 0 began with a switch that was either on or off. Twenty chapters later you typed quit and 328 bytes settled onto a disk — a playlist that will still be there tomorrow, written by an app you built, tested, and understood at every level. From a bit to a shipped app. Go build the next one. ∎