25Debugging — the traceback is the unwound stack

In Chapter 24 a green test bar certified that our code did exactly what we predicted. This chapter is about the other moment — the one where it doesn't, and the terminal fills with red. We take it slowly, because most tutorials skip one question: when Python prints a traceback, is it printing a message about the crash, or the crash itself? Here's the plan. First we read a traceback as what it physically is — the call stack you built in Volume 1, unwound and printed one frame at a time. Then we watch an exception stop being an instant and become a process. It climbs that stack, popping one frame after another, asking each one whether it has a rescuer. Then we meet pdb. It is not an external program reaching in from outside. It is ordinary Python that the interpreter itself calls on every single line, through a hook that was in the language the whole time. And we turn debugging from panicked flailing into a binary search. It converges, in about log₂(N) steps, on the one line where your prediction and reality first split. By the end, the scariest wall of red in Python reads like a map.

01A traceback is the call stack, printed

Let's start with the thing everyone flinches at. You pressed Enter, and the terminal answered with a wall of red — fifteen lines of File this and line that, arrows and indentation, and a word you half-recognise at the very bottom. The reflex is to flinch, scan for a keyword, and paste the whole thing into a search box. Kill that reflex now. That wall is not the machine yelling at you. It is the machine handing you a diagnostic image of itself — a traceback, free of charge. Every line in it is a hard fact about where your program was standing at the instant it fell over. Learn to read it and you diagnose in two seconds what print-debugging takes twenty minutes to chase.

To read it you need just one thing you already own. In Volume 1 you built the call stack. Every time one function calls another, Python pushes a frame — a small workspace holding that call's local variables and its place in the code — and pops it when the call returns. Here is the reframe: a traceback is that stack. Not a description of it, not a summary. It is the stack itself, unwound and printed one frame per block. That single sentence is the whole chapter; everything else follows from it.

Here is the machinery, exactly. CPython keeps a per-thread record, the PyThreadState, which holds a pointer to the frame currently running. Each frame carries a field f_back that points at the frame that called it. So the frames form a singly-linked chain from the innermost call all the way back to the module that started everything. (Since 3.11 the hot per-call data lives in a lean _PyInterpreterFrame on the interpreter's own data stack. The heavier PyFrameObject you can poke at from Python is materialised lazily, only when something needs to hold onto it — and a crash is exactly such a something.) When an exception reaches a frame that does not catch it, the interpreter calls a C function named PyTraceBack_Here. It allocates one PyTracebackObject that records three fields: tb_frame, a pointer to that very frame; tb_lineno, the line number; and tb_next, a link to the traceback node already built for the frame inside this one. The result is a second linked list running parallel to the f_back chain — a faithful shadow of the stack that was open at the moment raise fired.

Where does that line number come from? The frame never stored "I am on line 12." It stored f_lasti — the byte offset of the last bytecode instruction it ran. Python recovers the source line by looking that offset up in a compact table the compiler baked into the code object, co_linetable (and, in 3.11+, co_positions() for exact column spans, which is how modern Python underlines the precise sub-expression that failed). The filename and function name in each block come straight off the frame's code object, from co_filename and co_name. Nothing in a traceback is invented. Every field is copied or decoded from a frame that genuinely existed.

Now watch a three-deep chain fall over. Four calls are open — module, then load_config, then read_field, then to_int — so four frames are on the stack when int("x") refuses:

def to_int(raw):
    return int(raw)             # line 2 — int() raises here on 'x'

def read_field(row, key):
    return to_int(row[key])     # line 5

def load_config(rows):
    return read_field(rows[0], "port")   # line 8

load_config([{"port": "x"}])           # line 10

Traceback (most recent call last):
  File "config.py", line 10, in <module>
    load_config([{"port": "x"}])
  File "config.py", line 8, in load_config
    return read_field(rows[0], "port")
  File "config.py", line 5, in read_field
    return to_int(row[key])
  File "config.py", line 2, in to_int
    return int(raw)
ValueError: invalid literal for int() with base 10: 'x'

Count them. Four calls were open, so four frames were on the stack, so the traceback has exactly four blocks. They appear outermost-first and innermost-last, in the same order as the f_back chain read from the module end. Block one is the <module> frame (co_name is literally "<module>" for top-level code). Block four is to_int, the crash site, its line 2 decoded from that frame's f_lasti. There is no fifth block for int itself, because int is C code with no Python frame to capture. The mapping is total and mechanical: frame N of the stack becomes block N of the text.

Step back, because this is the whole volume in miniature. The call stack is a graph of live objects — frames pointing at frames, each holding real Python values at real RAM addresses. To get it out of the process and onto your screen, Python had to flatten that graph into a flat stream of characters: one File …, line …, in … block per frame, in order. That is a serialization, the exact move this whole volume studies, whether the far side is a disk file, a socket, or a pickle. A traceback is the call stack serialized to text so it can cross the boundary out of the process. What crosses a boundary is always bytes, never objects. A traceback is the stack, flattened.

You can now map a traceback back to the stack that made it. But that stack didn't just sit there — it unwound, frame by frame, and the exception was moving the whole time. Next: watch it travel. →

02Exceptions unwind the stack

"An exception happened" sounds like a single instant — a switch flips, the program stops. That picture will fail you the moment you try to catch something and it slips past your except anyway. An exception is not an instant. It is a motion. It starts at the frame where raise fired and it travels up the stack, one frame at a time, stopping at each to ask a single question: does anyone here rescue me? If not, that frame is torn down, the exception rises to the caller, and it asks again. Understanding this search lets you predict, before you run anything, exactly where a try has to sit to catch a given error — and why a finally or a with block still runs on the way out.

The moment raise executes, it does not jump anywhere. It sets the pending exception on the thread state — the exc_info slots — and hands control back to the eval loop, _PyEval_EvalFrameDefault. Now comes the Volume-2 upgrade over the story you may have heard. Since Python 3.11, entering a try block costs nothing at runtime. That is what "zero-cost exceptions" means. There is no longer a SETUP_FINALLY instruction pushing a block onto a runtime block-stack; the happy path executes zero extra instructions. Instead, the compiler bakes a side table into each code object: co_exceptiontable, a compact map from ranges of bytecode offsets to the handler offset that covers them.

So here is what actually happens when the exception is pending. The interpreter takes the current instruction offset in this frame and binary-scans the exception table. There are two outcomes, and only two. If some row covers this offset — meaning the failing instruction lies inside the byte-range of a try in this frame — the interpreter jumps to that row's handler offset, pushes the exception object, and runs your except or finally or __exit__ logic. The search is over. If no row covers this offset, the frame is finalized. Its locals are decref'd, any pending finally clauses and with-statement __exit__ methods fire as part of the unwind, and PyTraceBack_Here records this frame into the growing traceback (section 1's list, built one node per pop). Then control returns to the caller frame, where the identical table lookup repeats against that frame's exception table. Pop, search, pop, search, up the f_back chain, until either a frame catches it or the module frame is reached with nothing caught. At that point PyErr_PrintEx hands the finished traceback to sys.excepthook to print. "Zero-cost" means the whole price is paid once, at throw time, as table lookups during unwinding — never on the code that doesn't throw.

The consequence that trips everyone: placement is a range test. A handler catches an exception only if the failing instruction's offset falls inside the byte-range the compiler recorded for that try. Put the try around the wrong line and the offset misses the range, the row doesn't match, and the exception goes straight past a handler that looks like it should apply. Watch it happen, and watch a with block clean up on the way out:

def parse(raw):
    with open("cache.tmp", "w") as f:   # f.__exit__ will still run on the way out
        f.write(raw)
        n = int(raw)                    # raises ValueError on 'x' — NO try in this frame
        return n * 2                     # never reached

def run():
    try:
        return parse("x")          # the covered range lives HERE, one frame up
    except ValueError:
        return -1                       # the rescuer

print(run())                          # -> -1, and cache.tmp was closed during the unwind

Trace the motion. int("x") raises inside parse. The interpreter scans parse's exception table for a row covering that offset, and there is none, because parse has no try. So parse is finalized. On the way out, the with block's f.__exit__ fires and the file is flushed and closed. Cleanup rides the unwind, guaranteed, whether you exit normally or by exception. The frame pops, is recorded into the traceback, and the exception rises into run. Now the scan is against run's table, and the failing call site parse("x") is inside the covered range, so the row matches: jump to the except, bind the exception, return -1. The search stops. Move that try to wrap the wrong line and the offset misses, so the handler never fires. Catching is geometry, not magic.

The exception has landed and the traceback is printed. Now you have to read it — and the order it prints in is the deliberate reverse of the order it searched. Next: the one habit that turns forty lines of red into a two-second diagnosis. →

03Read it bottom-up

The instinct is to read a traceback like English — top to bottom, first line first. Follow that instinct and you march straight into someone else's framework, three libraries deep, past files you have never opened, and you give up before you reach the part that concerns you. Reverse it. Go to the bottom first. The last line tells you what broke and why. The block directly above it tells you where. Everything above that is just the road that led there, so skim it only if you need to. This one habit is the single highest-leverage debugging skill there is, and it takes thirty seconds to learn.

The reason it works is structural, not stylistic. In section 1 you saw the traceback printed outermost-frame-first, innermost-frame-last. That print order is the deliberate inverse of the search order. The exception searched from the inside out; the printer lists from the outside in, so the last block is always the frame nearest the crash. Beneath that final block sits one more line, of the form ExceptionType: message. That line is not a frame at all. It is type(exc).__name__ followed by str(exc): the actual failure, the type of thing that went wrong and the human-readable detail. So bottom-up reading is a fixed three-step protocol. (1) The last line is the diagnosis — what and why. (2) The frame block just above it is the crash site — the exact line of code where raise fired, recovered from that frame's f_lasti. (3) Scan upward only far enough to find the first frame you own rather than library code. Nine times out of ten you never reach step three.

There is one structure you must learn to recognise, because it stacks multiple tracebacks in a single dump: chained exceptions. When code raises a new exception while it is already handling one, Python links them so you don't lose the original. There are two flavours, and the wording tells them apart. An implicit chain prints During handling of the above exception, another exception occurred:. Python sets this up automatically by storing the first exception in the second one's __context__ attribute the moment a new raise runs inside an except block. An explicit chain prints The above exception was the direct cause of the following exception:. You asked for it with raise NewError(...) from original, which sets __cause__ and signals deliberate intent: "I am translating that low-level error into this meaningful one." Each chained segment is a complete traceback of its own. One rule survives all the structure: the lowest exception line in the entire dump is still the one that ultimately crashed the program, and you read its cause chain by walking upward through the segments.

def get_port(cfg):
    return int(cfg["port"])            # KeyError if 'port' is missing

def start(cfg):
    try:
        return get_port(cfg)
    except KeyError:
        raise RuntimeError("config is missing a port")   # implicit chain via __context__

start({})

Traceback (most recent call last):
  File "chain.py", line 6, in start
    return get_port(cfg)
  File "chain.py", line 2, in get_port
    return int(cfg["port"])
KeyError: 'port'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "chain.py", line 10, in <module>
    start({})
  File "chain.py", line 8, in start
    raise RuntimeError("config is missing a port")
RuntimeError: config is missing a port

Two segments, one story. The bottom line, RuntimeError: config is missing a port, is the exception that actually reached the top and crashed the program. That is your diagnosis. The bridge line tells you it did not appear from nowhere: it was raised while handling the KeyError in the top segment, which is the real root cause — a missing "port" key. The implicit "During handling…" wording is Python's honest confession that the second error happened because the first one did. If get_port had instead written raise RuntimeError("config is missing a port") from err, the bridge would read The above exception was the direct cause… — the same shape, but now stating that the translation was intentional. And raise ... from None suppresses the chain entirely and hides the KeyError — occasionally kind to a user, often cruel to the next debugger.

Reading a traceback tells you where a crash landed — after the fact. But sometimes you need to stop the machine mid-stride and look around while it's still alive. That means a debugger. And a debugger is not what you think it is. →

04pdb rides the trace hook

The first time you type breakpoint(), the program freezes and a (Pdb) prompt appears, it feels like sorcery. Something reached inside your running code, stopped time, and is now letting you walk around in the frozen machine. It feels like an external program with special powers over the interpreter. It is nothing of the kind. Demystifying it completely is worth more than a hundred pdb commands, because the same underlying trick powers coverage tools, profilers, and line tracers — and you can build a working one yourself in fifteen lines.

The foundation is a hook that has been in the language the whole time. CPython's eval loop can call back into Python between operations. You arm it with sys.settrace(fn), which stores fn as the thread's global trace function. Once tracing is active, _PyEval_EvalFrameDefault invokes that function on four well-defined events: 'call' when a frame is entered, 'line' when execution crosses into a new source line, 'return' when a frame exits, and 'exception' when one is raised. The 'line' event is the interesting one. The interpreter detects it by comparing the current instruction offset against the line boundaries decoded from co_linetable — the very same table section 1 used to build the traceback — gated by the frame's f_trace_lines flag. So a debugger does not "watch" your code from outside. It is a function the interpreter calls, from the inside, at every line boundary.

One more piece makes it precise. The trace function returns a value, and that return value is treated as the per-frame local trace function. The interpreter stashes it in the new frame's f_trace and calls it for that frame's later line events. Return None and that frame stops being traced; return the function itself and you keep getting its lines. This is how a debugger follows one specific frame instead of drowning in every line of every library. Prove the whole thing is real and public with a tracer you can write yourself:

import sys

def tracer(frame, event, arg):
    print(f"{event:<9} line {frame.f_lineno:>3}  {frame.f_code.co_name}")
    return tracer          # return per-frame tracer so we keep getting 'line' events

def total(xs):
    s = 0
    for x in xs:
        s += x
    return s

sys.settrace(tracer)
total([1, 2, 3])
sys.settrace(None)

call      line   8  total
line      line   9  total
line      line  10  total
line      line  11  total
line      line  10  total     # loop back — the 'for' line fires again each iteration
line      line  11  total
...
return    line  12  total

There is the machine, exposed: a 'call' at entry, a 'line' for every source line the interpreter crosses (the loop body reappears once per iteration, because it genuinely re-executes that line), and a 'return' at the end. Nothing is hidden. Now the reveal: pdb is just a well-dressed version of exactly this. It is built on the standard library's bdb.Bdb base class, which implements those four callbacks. On each 'line' event, bdb asks one question: is there a breakpoint here, or am I single-stepping? If yes, it calls user_line, which starts pdb's Cmd-based interactive prompt via cmdloop(). That is a REPL: it reads commands (n, s, c, p, l…) from stdin, mutates the debugger's stepping state, and then returns so the eval loop continues. The (Pdb) prompt is a Python REPL running inside the interpreter's own per-line callback. Typing next simply tells bdb to stop again at the next line event in this frame.

And breakpoint()? It is the convenient front door and nothing more. It calls sys.breakpointhook(), which by default imports pdb and calls pdb.set_trace(), which calls sys.settrace to arm the hook starting at the current frame. Three hops from a builtin to the same primitive you just used by hand.

Now you can stop the machine anywhere and inspect it. But where should you stop? Poking at random is just slow flailing with a nicer prompt. Next: the method that guarantees progress — treat the bug as an experiment. →

05The scientific loop

Under pressure, almost everyone debugs the same bad way: change a line that looks suspicious, sprinkle print everywhere, rerun, squint, change something else, rerun again. It feels like work — you're typing, the program's running — but it converges slowly and teaches you nothing, because none of it is testing a specific idea. There is a method that guarantees progress on every step, and it comes straight out of a first principle you already trust from Volume 1: the machine is deterministic. Identical bytecode, identical inputs, and identical starting state produce identical output, every single time.

Sit with what that implies, because it dissolves the whole mystery. A bug is not a spooky, intermittent event. It is a fixed, reproducible divergence between the state you predicted and the state that actually exists, and it first appears at one specific line: the first line where your prediction and reality disagree. Everything before that line is correct by definition; if it weren't, the divergence would have shown up earlier. Everything after it is downstream corruption — wrong values spreading from a fault that already happened. So debugging is not a hunt through fog. It is a search for a single boundary on the execution timeline: the last known-good point and the first known-bad point, with the bug pinned exactly between them.

And a search for a boundary on an ordered line is the oldest efficient algorithm there is: binary search. If the program runs N steps and you can observe state at any step, checking the midpoint tells you which half holds the first divergence. Is the state still correct at step N/2? Then the bug is in the later half, and everything up to the midpoint is cleared. Is it already wrong there? Then the bug is in the earlier half. Each observation halves the suspect region, so you localise the fault in about log₂(N) checks, not N. A thousand-step program yields to ten well-placed probes. That is the difference between a bug that takes two minutes and one that takes two hours — not talent, just refusing to look anywhere but the midpoint of the current suspect region.

Two preconditions make the experiment valid, and skipping either is why debugging sometimes feels impossible. First, reproduce reliably. A bug that appears "randomly" is lying to you. It means there is a hidden input you have not pinned. The usual culprits are wall-clock time, dict or set ordering under a fresh PYTHONHASHSEED, thread interleaving, an un-seeded RNG, or the classic one, a reused mutable default argument. Pin every input until the failure is deterministic: seed the RNG, fix the hash seed, freeze the clock, force one thread. Determinism is what makes an experiment repeatable, and an experiment you can't repeat can't be halved. Second, every hypothesis must be falsifiable — it must predict a concrete, observable value. "After line 14, total should be 42" can be confirmed or refuted by running line 14. "Something's wrong with the loop" predicts nothing and can't be tested against anything. Vague hunches cannot be binary-searched; specific predictions can.

Here is the most infamous "random" bug in Python, and how pinning it reveals the hidden input:

def collect(item, bucket=[]):     # bucket is created ONCE, at def-time — the hidden state
    bucket.append(item)
    return bucket

print(collect("a"))    # ['a']
print(collect("b"))    # ['a', 'b']  — 'randomly' remembers the previous call!

It doesn't remember randomly; it remembers deterministically. The default [] is one list object, created a single time when the def statement ran, and bound to the function forever. Every call that omits bucket shares that same list. So the "randomness" was a hidden input in plain sight: the accumulated state of one object across calls. Predict the value of bucket at the top of the second call ("it should be empty") and reality says ['a']. Prediction and reality diverge right there, at the parameter binding, and the bug is pinned. The fix follows from the diagnosis, not from guessing: def collect(item, bucket=None): then bucket = [] if bucket is None else bucket, so a fresh list is made on each call. The lesson generalises past this one trap: reproduce first, and "random" resolves into a variable you forgot to pin.

The scientific loop finds bugs while the program runs. But the best evidence is often at the crash scene itself — and that scene doesn't have to vanish when the program dies. Next: re-enter the dead frames and interrogate the real objects that were alive at the instant of failure. →

06Post-mortem: inspecting the dead frames

Print debugging has a cruel structure. You must decide in advance which variables to print, then rerun, discover you printed the wrong ones, add more prints, and rerun again. Each cycle is a guess about which value mattered, and each print shows you only a stringified snapshot frozen at print-time — str(x), not x itself. What you actually want is to arrive at the crash scene with everything still intact: after the program has already died, re-enter the exact frames that unwound and inspect the real, live objects that existed at the moment raise fired. No rerun. No guessing which value was the one. This is post-mortem debugging, and it works for a reason that ties this whole chapter together.

Normally, frames are freed as the stack unwinds. Section 2 showed each frame being finalized and its locals decref'd on the way out. So how can they still be there after the crash? Because of section 1: the traceback holds strong references to every frame it captured. Each PyTracebackObject.tb_frame keeps its PyFrameObject alive even after the lightweight _PyInterpreterFrame on the C data stack is gone. This is exactly what "frame materialization" was for. The frames are dead in the sense that execution has left them, but they are still sitting on the heap, held by the traceback's tb_next chain. And CPython puts that chain where you can reach it: after an unhandled top-level exception, sys.last_traceback (with sys.last_type and sys.last_value) points straight at it.

That is the entire mechanism behind post-mortem. pdb.pm() reads sys.last_traceback. pdb.post_mortem(tb) walks tb_next to the innermost node — the crash site — and opens the interactive prompt positioned in that frame, with frame.f_locals being the actual dict of live objects that existed when the exception fired. From there u and d walk the frame chain (tb_frame's f_back), so you can inspect any caller's real state too. Watch it turn a rerun-and-guess loop into a single shot:

>>> def crunch(rows):
...     total = 0
...     for r in rows:
...         total += int(r["n"])      # ValueError on a bad row
...     return total
...
>>> crunch([{"n": "10"}, {"n": "oops"}])
Traceback (most recent call last):
  ...
ValueError: invalid literal for int() with base 10: 'oops'
>>> import pdb; pdb.pm()          # re-enter the dead innermost frame
> <stdin>(4)crunch()
(Pdb) p r                          # the EXACT row that killed it — a live dict, not a string
{'n': 'oops'}
(Pdb) p total                      # the accumulator, frozen mid-loop at the crash
10
(Pdb) u                            # walk f_back up to the caller frame
(Pdb) p rows                       # the caller's real locals, still alive on the heap
[{'n': '10'}, {'n': 'oops'}]

No rerun, no advance guessing. The malformed row {'n': 'oops'} is right there in frame.f_locals as the actual dict object, and total shows the loop had accumulated 10 before it died — the state of the machine at the instant of failure, preserved. breakpoint() from the last section is the live counterpart: it arms the trace hook to drop you in before the crash, at a line you choose. Same frames, inspected either just before death or just after. Post-mortem is simply the after.

There is a sharp corollary hiding in all this, and it is a real memory bug people ship. Because tracebacks pin frames, and frames pin their locals, holding onto a traceback keeps an entire chain of frames — and every object they reference — alive. The most common way to do this by accident is to stash the exception itself, since except Exception as e gives you an e whose e.__traceback__ is the whole chain. Save that e in a list or a module global "for logging later" and you have quietly rooted a graph of dead frames and all their big local objects — a leak that grows with every handled error. This is exactly why Python automatically deletes the as e name at the end of an except block. When you must keep something, the fix is to keep the message or a formatted string — or to call traceback.clear_frames(e.__traceback__) — rather than the live exception.

You began this chapter flinching at a wall of red. You end it able to read that wall as the unwound stack, watch an exception travel and clean up as it goes, ride the interpreter's own hook into a frozen machine, binary-search to the first divergence, and re-enter the dead frames to read the truth. The traceback was never noise. It was Volume 1's stack, handing you the answer. →