24Testing — what green actually proves

In Chapter 23 we watched the type checker judge our code without ever running it. Those were static hints, settled before line 1 executes. Now we actually run the thing and ask a harder question: does it do the right thing? Picture the moment. You changed one function, and your stomach dropped. Did you just break the three callers you forgot existed? A test is the machine that answers that, so you don't have to re-check by hand. Here's the plan. We take pytest apart down to the metal: how it finds your claims by walking the filesystem, how it rewrites assert at the syntax-tree level so a failure explains itself, how fixtures inject dependencies, and how parametrize turns one test body into many. All the way through, we keep asking the one thing that really matters: when the bar goes green, what has that green actually proven? By the end you can pin a claim like total([200,245]) is 445 to your code and have a machine re-check it forever — knowing exactly, and honestly, what that green does and doesn't buy you.

01A test is an executable claim

Let's start with the fear, because the fear is the whole reason testing exists. You changed total() to handle a new edge case, and now you're afraid. Somewhere in the codebase three functions call it, and you can't remember which. So you do what everyone does the first year: you run the program by hand, type a few inputs, squint at the output, and ship because nothing looked wrong. That works for a weekend. It does not survive a month. The number of things you'd have to re-check by hand grows faster than any afternoon, so in practice you check almost nothing and call it confidence. What you actually need is simple: write down, once, a sentence like total([200,245]) is 445, and have a machine re-check that sentence forever. A test is not a chore bolted onto real work. It is the only way to keep a claim about your code alive after the moment you first believed it.

Now throw the word 'test' away and look at the primitive underneath it: assert. Look at what it actually is. It's not a function, but a statement the compiler itself knows about, and it compiles down to something you could have written by hand. assert x == 5 becomes, in bytecode, almost exactly if not (x == 5): raise AssertionError. Watch the machine do it:

import dis
dis.dis(compile("assert x == 5", "<s>", "exec"))

# LOAD_NAME            x
# LOAD_CONST           5
# COMPARE_OP           ==
# POP_JUMP_IF_TRUE     --> end     # true? jump clean over the raise
# LOAD_ASSERTION_ERROR             # false? build the exception
# RAISE_VARARGS        1           # and throw it
# end: LOAD_CONST None / RETURN_VALUE

Read those six lines and the whole mystery of testing evaporates. An assert evaluates an expression. If it is truthy, control jumps clean over the raise and the function keeps going. If it is falsy, it constructs an AssertionError and throws. Two exact facts ride along. First, assert takes a second operand: assert expr, msg. The message becomes the exception's argument, so a failure can carry a sentence. Second, and this is the fact that decides where you're allowed to use it: asserts are compiled entirely out when Python runs with -O. The name __debug__ is a compile-time constant, true normally and false under -O, so under optimization assert emits zero bytecode. That is why production code must never validate untrusted input with assert: flip on -O and your check simply isn't there. But a test is debug-time-only code by definition. Assert is not merely allowed there; it is exactly right.

So here is the definition the rest of the chapter builds on. A test is just a function whose body makes claims with assert. It takes no arguments you supply by hand, returns None on success, and raises on failure. 'The test passed' means, with no magic whatsoever, the function returned without raising. pytest does not inspect your soul. It calls your function inside a try/except. If the call returns, it paints the dot green. If any exception propagates out, it catches it and paints the dot red. That exception can be an AssertionError you wrote, or a TypeError you didn't expect.

def total(items):
    return sum(items)

def test_total():
    assert total([200, 245]) == 445

# pytest adds nothing but the calling loop — you can BE pytest:
try:
    test_total()
    print("green")          # returned → passed
except AssertionError as e:
    print("red", repr(e))  # raised → failed

That try/except around a function call is the entire runner in miniature. Everything downstream is a refinement of it. Collection answers 'which functions do I call?'. The assert rewrite answers 'how do I make the raised exception informative?'. Fixtures answer 'what do I pass in as arguments?'. Hold the exception-as-signal model in your head and none of pytest will ever feel like sorcery again.

You can write a claim. But you never told pytest your ten functions exist — no registry, no list, no main(). So how does it find them? Next: collection, the search pytest runs against your filesystem. →

02Collection: discovery by convention

You wrote ten test functions across four files. You never wired them together. There is no list of tests, no registry, no main() that calls them in order. Yet you type pytest and it finds all ten and runs them. A program you never told about your functions somehow knows they exist. And when a colleague insists the file be named test_orders.py and the function test_ships, it feels like superstition, some etiquette the tool enforces for no reason. What you need to see is that the naming rules are not etiquette. They are the literal query pytest runs against your disk and your module objects. Once you see collection as a search with rules, the worst failure in testing — a test that silently never runs — stops being a mystery you shrug at and becomes something you can predict exactly.

Collection is a two-stage reflection process, and both stages are pure convention matched against real runtime objects. Stage one is the filesystem. pytest walks the directory tree from the rootdir (respecting testpaths if you set it) and selects files whose names match test_*.py or *_test.py. A file called helpers.py is invisible to this walk, no matter how many asserts it contains. Stage two is import plus introspection. For each matched file, pytest imports it as a module. Here is the load-bearing detail of the entire section: importing a module runs it top to bottom. Every line at module level executes during collection, before a single test runs. That is why an import error, or a top-level line that raises, surfaces as a collection error. pytest never got far enough to call any test, because it crashed while reading the file.

import re, importlib

# stage one: filesystem — keep only files matching the convention
candidates = ["test_orders.py", "helpers.py", "cart_test.py"]
matched = [f for f in candidates
           if re.match(r"test_.*\.py$|.*_test\.py$", f)]
# ['test_orders.py', 'cart_test.py']  — helpers.py is simply not seen

# stage two: import each (RUNS it), then walk its namespace
module = importlib.import_module("test_orders")
items = [obj for name, obj in vars(module).items()
         if name.startswith("test") and callable(obj)]

After the import, pytest walks the module's namespace with the same machinery as dir() and vars(). It introspects the live module object. It plucks out functions whose name starts with test, and classes named Test* (with no __init__) whose methods start with test. Each match becomes a Test item: an object pytest will later call. This is why a mis-named function vanishes without a word. def check_total() is a perfectly good function that asserts perfectly good things, and pytest walks right past it, because check does not start with test. No error, no warning. It simply never runs, and your green bar is green over a claim that was never checked.

There is one subtlety worth meeting precisely, because it bites everyone once. In pytest's default 'prepend' import mode, it inserts each test file's directory onto sys.path and imports the file by its bare basename. So two files both named test_utils.py in different folders collide. Python's import system sees the same module name twice, and one shadows the other, producing a baffling error. The fixes are exactly what the mechanism predicts. Add __init__.py files so the packages give the modules distinct dotted names, or switch to importmode=importlib, which imports by full path and never touches sys.path.

Now the payoff that ties collection to section 1. Each collected item is run as its own function call inside its own try/except. When item three raises an AssertionError, that exception unwinds item three's stack and is caught by the runner, and the loop simply moves to item four. One test's failure cannot poison the next, because they were never on the same stack. Contrast a hand-rolled main() that calls your tests in sequence. There, the first uncaught exception kills every test after it, and you learn about one failure while five real ones hide behind it. Isolation is not a mystical property of tests. It is a structural gift pytest hands you by choosing to run N independent calls instead of one long function.

pytest can find your claim and run it in isolation. But when assert response.status == 200 fails, plain Python tells you only 'AssertionError' — not the value it saw. Next: the syntax-tree surgery that makes failures talk. →

03The assert rewrite makes failures talk

You wrote assert response.status == expected. It fails. Plain Python hands you exactly one bit of information: AssertionError. Not the value of status, not the value of expected, nothing you can act on. That single poverty is why an entire vocabulary exists: assertEqual(a, b), assertGreater, assertIn, assertAlmostEqual, a zoo of methods you memorize purely so the failure message will finally show you the two numbers. What you want is the informative failure without the ceremony. And pytest's answer is genuinely startling the first time you see it. A bare assert a == b fails with assert 404 == 200, both operands, printed. That is not normal Python. It cannot be, because you just saw normal Python print nothing but the exception's name. Understanding the mechanism behind it dissolves the entire assertEqual family at once, and explains why the magic works only inside pytest.

The trick is compile-time surgery on the syntax tree, and it happens at import, not at call. Here is the exact pipeline. Before Python source becomes bytecode, it is parsed into an Abstract Syntax Tree: a tree of nodes where assert a == b is an Assert node wrapping a Compare node. pytest registers an import hook, a finder on sys.meta_path. So when a module pytest wants to collect is imported, pytest intercepts the import, reads the source, parses it to an AST, and runs a transformer over it. That transformer rewrites every Assert node into something much larger.

# you wrote:
assert get_status() == 200

# pytest rewrites it to roughly this, at import time:
_left  = get_status()          # evaluate each operand ONCE, into a temp
_right = 200
_ok    = _left == _right
if not _ok:
    raise AssertionError(
        f"assert {_left!r} == {_right!r}")   # reprs of the saved values

Read what that buys. The operands are saved into hidden temporaries, so each subexpression is evaluated exactly once. Even get_status(), which might have a side effect, is not called twice. And because their runtime values are sitting in _left and _right, the explanation can repr() them into a message that shows you precisely what pytest saw. The transformed AST is compiled to a .pyc cached under __pycache__ with a pytest-specific tag, so this rewrite cost is paid once per file, not once per run.

Three exact consequences fall out of 'it's an import-time transform', and each one explains a real confusion. One: it only fires for files pytest imports through its hook. An assert buried in a plain helper module you import normally is not rewritten. It gives you the bare, useless AssertionError, unless you explicitly call pytest.register_assert_rewrite('mymodule') before it's imported. Two: a module already imported before pytest's hook installs won't be rewritten at all, so ordering matters. Three: the introspection is type-aware. For == on two lists, dicts, or multiline strings, pytest emits a structured diff, driven by the operands' runtime types: the first differing element, or a unified diff:

def test_playlist():
    assert [200, 245, 300] == [200, 245, 301]

# pytest's failure isn't just 'AssertionError' — it points at the culprit:
#   assert [200, 245, 300] == [200, 245, 301]
#     At index 2 diff: 300 != 301

Now land the payoff. assertEqual, assertGreater, and the rest of that zoo exist because vanilla unittest could not see inside the expression. By the time self.assertTrue(a == b) runs, the comparison has already collapsed to a lone True or False, and the operands are gone. So unittest made you name the relation, assertEqual, precisely so it could receive a and b separately and print them. pytest never has that problem. It sees the whole expression tree at import time, so one operator — ==, <, in, is — plus the rewrite reconstructs a better message than any hand-written assertEqual ever managed. You get to write the plainest possible Python and receive the richest possible failure. That is not a convenience layer bolted on top. It is a smarter place to stand.

Failures talk now. But every test still opens the same database, builds the same temp dir, logs in the same client — ten lines of setup drowning the one line of claim. Next: fixtures, and dependency injection by name. →

04Fixtures are dependency injection

Every test in the file needs the same thing before it can start: a fresh database connection, a temp directory, a logged-in client. So the top of every test function grows the same six lines of setup and the same four lines of teardown, and now the boilerplate outweighs the claim. Worse, when the setup changes you edit it in twenty places. And if you get clever and build the object once to share it, test three mutates it and test seven fails, but only when they run in that order, which is the most maddening bug there is. What you need is precise: declare 'this test needs a fresh X', and have the machine build a clean X, hand it over, and tear it down afterward, with no boilerplate in the test body and no test bleeding state into the next. That mechanism is dependency injection, and pytest's version of it is the feature people most misread as magic.

A fixture is a function decorated with @pytest.fixture. The wiring is pure name-matching through introspection. Before pytest calls a test, it inspects the test function's parameter names with inspect.signature. For each parameter it looks up a fixture of that exact name, calls it, and passes the return value as that argument. That is the whole of 'injection': the test declares a dependency by naming it as a parameter, and pytest resolves the name to a provider. Nothing is global, and the test never constructs the object itself.

import pytest

@pytest.fixture
def playlist():
    return [200, 245]        # Blinding Lights, Titanium

def test_total(playlist):        # names 'playlist' → pytest injects it
    assert total(playlist) == 445

Fixtures can themselves take fixture parameters, so pytest builds a dependency graph and resolves it depth-first. A db fixture that names a config parameter forces config to be built first, its result fed into db, and db's result fed into the test. Now the two mechanisms that matter most. Setup and teardown ride on yield. A fixture that yields is a generator. pytest calls next() to run everything up to the yield, which is the setup, and hands the yielded value to the test. When the test finishes, pytest calls next() again, driving the generator past the yield to StopIteration, which runs everything after the yield, which is the teardown. The yield is the literal seam between the two phases. And because pytest drives that second next() inside a finally, teardown runs even if the test failed.

@pytest.fixture
def db():
    conn = connect()        # setup — runs before the test
    yield conn              # <- the seam: hand conn to the test
    conn.close()            # teardown — runs after, even on failure

Scope controls how long the built object lives. The default, scope='function', means the fixture is re-called and re-torn-down for every single test, giving a fresh object each time. This is exactly why order-independence is the default: test seven can't be poisoned by test three mutating the object, because test seven received a different object, built by a fresh call. Widen to scope='module' or 'session' and pytest calls the fixture once and memoizes the return value, reusing the same object across many tests. That is faster, since you connect to the database once, not four hundred times. But now shared mutable state is possible again, and you must understand the trade you are buying: speed on one side, the risk of order-dependent tests on the other.

Two final threads. First, connect fixtures to the chapter's spine: collection decides which functions to call; fixtures decide what to pass to them. The parameter name is the whole interface between the two. Second, where do fixtures live so many files can share them? In conftest.py. pytest auto-discovers fixtures defined there for every test in that directory tree, with no import needed and no wiring. It is the exact same convention-over-configuration idea that powered collection. You never register a fixture any more than you register a test. You name it, put it in the right place, and the search finds it.

One fixture feeds one object to one test. But you want to run the same claim over empty lists, single elements, negatives, and huge inputs — without copy-pasting the body four times. Next: parametrize, one body into many cases. →

05Parametrize: one body, many cases

You want to test total() on the empty list, one element, some negatives, and a big list. So you copy the test function four times, changing two literals in each. Now the assertion logic lives in four places, and if you fix a bug in how you assert, you fix it four times. And when case three fails, the traceback says test_total_negatives, and you have to go read the body to remember what input that even was. The obvious escape is a loop inside one test, for case in cases: assert ..., and it is worse. The first failing case raises and unwinds the whole function, so you learn about failure one and nothing about two through four, and the whole thing reports as a single red. What you need is to run one claim over a table of inputs, where each row is its own independently-reported case that runs even if its neighbors fail, without duplicating the body. That is parametrization, and it is the difference between a test and a test suite.

import pytest

@pytest.mark.parametrize("items,expected", [
    ([],            0),
    ([200],         200),
    ([200, 245],    445),
    ([-5, 5],       0),
])
def test_total(items, expected):
    assert total(items) == expected

Here is the mechanism, and it is not what it looks like. @pytest.mark.parametrize is not a runtime loop. It acts at collection time. The mark attaches metadata to the test function. During collection — the same stage from section 2 that turned files into items — pytest reads that metadata and multiplies the single function into N distinct test items, one per row of the table, binding the named parameters to that row's values. It uses the very same parameter-injection machinery as fixtures. Parametrize is essentially a fixture whose values you supply explicitly. So the four-row table above does not produce one test that loops four times. It produces four separate items.

Three consequences fall out, and every one of them is a reason parametrize beats the loop. One: each row becomes its own item with its own generated id. pytest builds an id like test_total[items3-expected3] from the values, or from your ids= labels if you give them. So the report shows four results, not one, and a failure names the exact input that broke. Two: because they are separate items, each runs in its own call inside its own try/except, so case two failing does not stop case three. You get four independent verdicts and the full picture. Three: stacking two parametrize decorators produces the Cartesian product. A 3-row and a 2-row decorator generate 3 × 2 = 6 items, one per pair, which pytest builds by nesting the multiplication.

Now feel the contrast sharply against the loop-inside-a-test. That loop is one item. The first assert that fails raises and unwinds the entire function, so cases after it never execute, and a single red hides how many cases actually broke. One? All four? You cannot tell. Parametrize inverts every part of that: N items, N independent calls, N verdicts, and each failure self-labeling its input. Same four claims, two completely different execution structures, and only one of them tells you the truth.

That last line is the tell that ties this section back to collection. When pytest prints collected 6 items from a file with two test functions, it is not miscounting. Parametrize is collection-time item generation. The same discovery machine that turned files into items also expands one function into a whole family of them. Discovery and multiplication are one mechanism, seen twice.

Now your suite is green — 47 passed — and you feel safe. That feeling is the most dangerous thing in this chapter, because it is largely unearned. Next: exactly what a green bar licenses you to believe. →

06What green proves — and what it doesn't

The suite is green. 47 passed. You feel safe, and that feeling is the most dangerous thing in this whole chapter, because it is largely unearned. Green feels like 'my code is correct.' It is not. It means, exactly, 'the specific claims I remembered to write still hold.' The gap between those two sentences is not a technicality. It is where every shipped bug lives. This section is the intellectual core of the chapter, the epistemics of testing. Its job is to install a permanent, accurate model of what a green bar does and does not license. That way you neither over-trust it (ship on green with three shallow tests) nor over-fear it (chase 100% coverage believing the number means correctness).

Formalize it. A test suite is a finite set of claims {c₁ … cₙ}. Green means every cᵢ evaluated true on this run. It says exactly nothing about any behavior you did not express as some cᵢ. This is Dijkstra's razor made concrete: testing shows the presence of bugs, never their absence. The input space of any nontrivial function is astronomically large, often infinitely so, and your tests sample a handful of points in it. Green proves the sampled points behave. The unsampled continuum, which is essentially all of it, is uncharted. A test that never fails on the inputs you chose tells you nothing about the input you didn't.

Now dismantle the metric everyone reaches for to feel safe: coverage. Line coverage instruments the run. A tracer (via sys.settrace, or sys.monitoring in 3.12+) records which line numbers executed while your tests ran, and coverage percent is executed lines over total lines. Here is its exact, fatal failure: executed is not verified. A line runs during your test whether or not any assert checks what that line produced. Watch it fail live:

def total(items):
    return len(items)     # BUG: should be sum(items)

def test_total():
    result = total([200, 245])
    assert isinstance(result, int)   # passes! len returns an int too

# coverage: 100% — the buggy line ran.
# correctness: 0%  — result is 2, not 445, and no assert pinned the VALUE.

Read that and let it sting. The buggy return len(items) executed, so coverage proudly reports 100% of the function covered. The test passed green, because len returns an int just like sum does, and the only thing the assert pinned was the type. The wrong value, 2 instead of 445, sailed straight through, on a fully-covered line, because no assertion ever looked at it. Coverage measured reach. It structurally cannot measure correctness, because correctness lives in the assertions, not in the execution. Push to branch coverage, which asks whether both sides of every if ran, and it is a genuine refinement. But it does not touch this core gap: you can traverse every branch of a function and still assert nothing meaningful about what any of them produced.

So what actually determines a suite's proof-strength? Two things, neither of them coverage. First, assertion quality: what each test pins down, whether that is a value, a type, a side effect, or a raised error. An isinstance check pins almost nothing; assert result == 445 pins the answer. Second, input sampling: which regions you probe, such as boundaries, empties, negatives, and the off-by-one edges where bugs actually cluster. A function correct on the three medium inputs you tried and wrong on the empty list is green until the day you add the empty-list case. The bug was always there; your sample just never touched it.

If you want an honest measure of a suite, one that can't be fooled by lines that merely ran, the tool is mutation testing. You deliberately corrupt the code (flip a + to a -, an < to <=) and rerun. If a test dies, that behavior was really being checked. If the suite stays green on the mutant, no test ever pinned that behavior, however high your coverage number. Mutation testing measures whether your assertions can feel a bug; coverage only measures whether your feet walked the floor. Keep the distinction for life.

If green only proves what you asserted, then the discipline is in when you write the assert. Next: red-green-refactor, and how a killed bug becomes a permanent antibody. →

07Red-green-refactor and the regression test

Two fears converge here, and one answer resolves both. The first: you are about to change working code to make it cleaner, and you are paralyzed. How do you restructure without breaking something you can't see? The second, and deeper: you just spent three hours hunting a bug, found it, fixed one character, and moved on. In six months someone will reintroduce that exact bug and you will hunt it again from scratch, because the three hours of knowledge you paid for evaporated the moment you deleted your debug prints. Both fears have the same cure: a test written at the right moment. Red-green-refactor gives you a rhythm that makes refactoring safe by construction, and the regression test turns each bug you kill into a permanent antibody. Here is the truth to land: a bug's most valuable byproduct is not the fix. It is the test that guarantees the bug never returns.

Red-green-refactor is a discipline that cashes in the isolation and re-runnability the earlier sections built. The loop, mechanically:

# 1. RED — write the test FIRST, for behavior that doesn't exist yet, and RUN it.
def test_empty():
    assert total([]) == 0       # fails today — total([]) raises

# 2. GREEN — write the MINIMUM code to pass. The test is now your spec.
def total(items):
    return sum(items)             # sum([]) is 0 — green

# 3. REFACTOR — change structure freely; the green test is a tripwire.

RED is the step people skip, and skipping it is a quiet disaster. You write the test before the code and run it to watch it fail, because a test you never saw fail might be asserting nothing at all (recall the empty def test_nothing(): pass from section 1, and the coverage-lies trap from section 6). Watching it go red proves the test can fail, which proves it actually exercises the thing you think it does. GREEN is writing the least code that passes; the test is now your executable definition of done. REFACTOR is the payoff. You change the implementation freely — sum() to a hand-rolled loop, one function into three — and the green test is a tripwire that fires the instant you break observable behavior. This is the whole reason 'refactor' is a real word and not a synonym for 'rewrite and hope'. Refactoring means changing structure without changing behavior, and 'without changing behavior' is only a checkable claim if behavior is pinned by tests.

Now the highest-value instance of RED: the regression test. A bug gets reported: total() crashes on the empty list. Before you fix it, you write a test that reproduces it: assert total([]) == 0, which is red right now, because of the bug. Then you fix the bug, and the test turns green. And here is the part that pays forever. That test now lives in the suite permanently, and its single job is to fail loudly if this exact bug ever comes back, through a careless edit, a bad merge, or a refactor gone wrong. You paid three hours to understand the bug once; the regression test banks that understanding so no one ever has to pay it again.

Step back and see what a mature suite actually is. Section 6 killed the idea that it proves correctness. It doesn't, and it can't. So what is it? It is institutional memory. Every regression test is a dated entry in a diary: on this day, this input broke this function; here is the claim that ensures it never breaks the same way again. A suite is not a proof of correctness. It is the accumulated scar tissue of every bug the code has survived. And that reframes an old intuition: old code with a real suite is safer to change than new code without one, precisely because the old code's suite remembers hundreds of specific ways it once broke, and stands guard over every one of them.

And that closes the whole chapter's circle. The regression test is an executable claim (§1), discovered by collection (§2), whose failure talks through the rewrite (§3), often fed by a fixture (§4) and generalized by parametrize (§5). And it is written by someone who knows, without illusion, that it proves only what it asserts (§6). Green never meant 'correct.' It meant 'the claims I was disciplined enough to write are still true.' Make those claims sharp, sample the edges where bugs hide, and turn every bug you kill into a claim that outlives you. That is the whole craft.

You can now keep a claim about your code alive forever — and you know exactly what that claim does and doesn't prove. Next volume-thread stop: when the thing crossing the boundary isn't a value you can assert on, but bytes on a wire. →