16Durability — flush, fsync, and the page cache

In Chapter 15 we opened a file, handed it bytes, and closed it. Python let us walk away believing the data was safe on disk. It wasn't. In this chapter we follow those bytes all the way down to the metal and find the exact moment "saved" becomes true. There's a lie hiding in write() that most tutorials never make you look at. Here's the plan. First we catch write() returning a number before a single byte has touched the platter. Then we take apart the three volatile buffers stacked between your string and the disk — Python's own, the kernel's, and the drive's — and meet os.fsync, the one syscall that forces bytes past all of them. The whole way down we keep asking the one question that matters: if the power cut right now, which of my writes would survive? By the end we've built the atomic write-temp-then-rename save that never leaves a half-written file, and seen how a database pays fsync's steep price just once for a thousand records. "I saved it" stops being a hope and becomes a claim you can prove.

01write() lies: the page cache and write-back

Let's start by watching the lie happen. You wrote the code. f.write(record) returned 128, the length of your line, right there in the return value. The next statement, print("saved!"), ran, and you watched it print. Then the building lost power for one second, the machine rebooted, and you reopened the file. The record is gone. Not garbled. Not half-there. Gone, as though write() had never been called. Every beginner carries the same unspoken assumption: the function returned, so the data is on the disk. It is flat wrong. The gap between "write() returned" and "the platter holds my bytes" is exactly where real, customer-losing data loss lives. So before we can defend against it, we have to see it plainly. A successful return from write() is a promise the kernel has not yet kept.

So watch what physically happens on that call. write() is a syscall: you cross the border out of your process and into the kernel. (That's Volume 2's recurring boundary — what crosses is bytes, never live objects.) But the kernel does not turn around and speak to the storage device. It copies your bytes out of your process's memory into pages of the page cache, which is kernel-owned RAM that mirrors file contents. It flips those pages to dirty (the in-kernel flag is literally PG_dirty), bumps the file's in-memory size, and returns the byte count. That is the entire fast-path syscall: a memcpy plus a little bookkeeping. No seek, no platter, no flash — zero device I/O. Your "saved" record is sitting in volatile RAM that just happens to belong to the kernel instead of to you.

The dirty pages reach the device later, asynchronously, driven by kernel writeback threads (you may know the old names pdflush/bdflush; modern kernels use per-backing-device kworker writeback). "Later" is not vague. It is governed by three independent triggers, and knowing them tells you exactly how long your data hangs in the wind. The first is a timer. A page that has been dirty longer than vm.dirty_expire_centisecs (about 30 seconds) becomes eligible, and a sweep runs every vm.dirty_writeback_centisecs (about 5 seconds) to flush the eligible ones. The second is memory pressure. Once dirty pages exceed vm.dirty_background_ratio the kernel wakes writeback in the background, and if they climb to vm.dirty_ratio your own next write() is forced to block until pages drain — the one moment write-back stops feeling free. The third is an explicit sync/fsync, which is the whole subject of this chapter.

Write-back is a deliberate trade, not an accident, and it helps to see its two siblings. Write-through caching would push each write to the device before returning. That's durable immediately, but every write() pays a device round-trip, so it is painfully slow. O_DIRECT goes the other way and bypasses the page cache entirely, handing I/O straight to the device (databases use it to manage their own caching). Ordinary buffered I/O — what you get by default — chose write-back on purpose: speed now, durability on demand. The demand is the syscall you'll meet two sections from now.

You can watch the lie with a stopwatch. A hundred writes should touch the disk a hundred times and cost real milliseconds — unless they never touch the disk at all:

import time

with open("set.log", "wb") as f:
    start = time.perf_counter()
    for i in range(100):
        n = f.write(b"track played\n")   # returns 13 — bytes copied into cache
    elapsed = time.perf_counter() - start

print(elapsed)     # ~0.0001 s — 100 "writes", not one disk seek

A hundred microseconds for a hundred writes. That number is impossible if the platter were involved. A single seek on a spinning disk is milliseconds, tens of thousands of times longer. The speed is the tell: write() returned the byte count the moment it finished the memcpy into cache, and nothing has left RAM. The return value 13 means "13 bytes copied," not "13 bytes persisted." Read that sentence again, because the rest of the chapter is nothing but the machinery for closing the gap between those two claims.

So write() only reaches the kernel's cache. The responsible move is to call f.flush() — but that closes a different gap than you think, because there is a second buffer you didn't know you had, living inside Python itself. →

02Python's buffer sits above the kernel's

You've learned that write() only reaches the page cache, so you reach for f.flush(), feeling diligent. Here is the trap that catches even experienced programmers. They believe flush() means "now it's on disk." It does not. It means "now it's in the kernel." There is a second buffer between your string and the platter, and it lives inside Python, in your own process. flush() empties only that one. Until you can see both buffers with your own eyes, you cannot reason about how far any given call moves your data. So let's make them visible.

When you open() a file in the normal, buffered mode, Python does not hand you a bare file descriptor. It hands you a tower of wrappers. At the bottom is a FileIO object holding the raw OS file descriptor. Over it sits an io.BufferedWriter, an in-process bytearray sized to io.DEFAULT_BUFFER_SIZE (8192 bytes, or the filesystem's block size). When you open in text mode there's a third floor on top: a TextIOWrapper that encodes your str into bytes (UTF-8, per Chapter 15) with its own buffering. You can pull the tower apart by hand:

>>> f = open("set.log", "w")
>>> type(f)
<class '_io.TextIOWrapper'>      # floor 3: str -> bytes
>>> type(f.buffer)
<class '_io.BufferedWriter'>     # floor 2: the bytearray in YOUR heap
>>> type(f.buffer.raw)
<class '_io.FileIO'>             # floor 1: the raw OS file descriptor
>>> f.buffer.raw.fileno()
3                            # the integer the kernel knows you by

Now the crucial mechanics. When you call f.write(s) in buffered mode, Python appends the bytes to that BufferedWriter bytearray and no syscall happens. None. The data has not left your process, and the kernel has never heard of it. Python issues the actual write() syscall(s) only when one of three things happens: the bytearray fills to 8192 bytes, or you call f.flush(), or you close(). That's what pushes the bytes down into the kernel's page cache. So there are two distinct, stacked buffers between your string and the medium: (1) Python's BufferedWriter, in your process's heap; and (2) the kernel page cache, in kernel RAM. Two owners, two chunks of volatile memory, one border between them.

And f.flush() operates on exactly one of those borders. It drains Python's bytearray by issuing the write() syscall, pushing the bytes down into the kernel page cache, and it stops there. It issues no fsync. It triggers no device I/O. Nothing reaches the platter. After flush(), your data has simply moved from one volatile buffer to a different volatile buffer. You closed the near gap and left the more dangerous one wide open — the one that survives power loss. You can prove the whole story with os.stat, which reports the file size the kernel knows, not the bytes trapped in Python:

import os

p = "buf.log"
with open(p, "wb") as f:
    f.write(b"x" * 10)
    print(os.stat(p).st_size)   # 0  -> bytes still in Python's bytearray
    f.flush()
    print(os.stat(p).st_size)   # 10 -> flush() crossed into the kernel

Zero, then ten. Between those two prints your bytes physically travelled from your heap to the kernel's, and not one inch further. That jump from 0 to 10 is the Python→kernel boundary being crossed, shown as an integer you can watch change.

Two buffers down, and both are volatile RAM. So what actually forces bytes onto the physical medium — past the kernel cache and the drive's own cache? There is exactly one syscall for it. →

03fsync is the only real durability guarantee

Two buffers down, one to go, and this last one is the one that survives the power cut. You now know flush() lands your bytes in the kernel page cache, still volatile. So what actually drives them onto the physical medium? There is exactly one syscall for the job, and if you never call it, no amount of flushing will save your data from a crash. This is the section that hands you the only real durability primitive there is: os.fsync.

os.fsync(fd) tells the kernel: take every dirty page belonging to this file, schedule them to the device now, and do not return control to my process until the storage controller confirms they are physically persisted. That last clause is the whole difference. Where write() and flush() return after a memcpy, fsync performs real device I/O and blocks until it completes. That's why it costs orders of magnitude more: single-digit milliseconds on flash, tens of milliseconds on a spinning disk, versus microseconds for a buffer copy. When fsync returns, and only then, your bytes are on the medium.

There's a subtlety worth stating exactly, because it sharpens the mental model you've been building. On a correctly implemented stack, fsync doesn't only flush the kernel page cache. It also issues a cache-flush / FUA barrier to the drive's own volatile write cache, forcing bytes past that final buffer onto the platter or NAND. So the honest picture has three volatile stages, not two: Python's bytearray, the kernel page cache, and the drive cache. flush() defeats stage 1. fsync() defeats stages 2 and 3 together. (This is exactly why serious databases care about drives with "power-loss protection." Some consumer SSDs acknowledge the barrier without honoring it, lying about durability to win benchmarks.)

import os

def durable_write(path, data):
    with open(path, "wb") as f:
        f.write(data)          # -> Python's BufferedWriter (your heap)
        f.flush()              # -> kernel page cache (still volatile)
        os.fsync(f.fileno())   # -> the physical medium (durable at last)

Three lines, three floors of the tower, one call per floor. Notice what close() (the implicit exit of the with block) does and does not do. It flushes Python's buffer to the kernel, so closing never strands bytes in your heap. But it does not fsync. A closed file can still be a lost file. Durability is never automatic; you ask for it by name.

And you can feel its price with a stopwatch. Run ten thousand tiny records two ways, flush only versus flush-plus-fsync, and the gap is not subtle:

import os, time

def bench(use_fsync):
    with open("t.log", "wb") as f:
        start = time.perf_counter()
        for _ in range(10_000):
            f.write(b"row\n")
            f.flush()
            if use_fsync:
                os.fsync(f.fileno())
        return time.perf_counter() - start

print(bench(False))   # ~0.02 s   — all buffer copies
print(bench(True))    # ~20-30 s  — 1000x, each ms a device round-trip

A thousand-fold. That entire difference is device physics. The flush-only run never left RAM, while the fsync run waited on the storage controller ten thousand separate times. This is the idea that reframes the chapter. Durability has a fixed, unavoidable, per-call price measured in device round-trips, and every storage system ever built is, at heart, a scheme for paying that price as few times as possible. (You'll see the scheme in the final section.) If your file's mtime and size don't matter for readback, os.fdatasync is the cheaper cousin. It persists the file data but skips the extra metadata barrier — same data durability, one fewer barrier per call.

Now you hold all three primitives — write, flush, fsync — and each leaves your data in a different buffer. The payoff is being able to point at any line and name exactly what a crash there would destroy. →

04The two crash windows, side by side

You have the three primitives. Here is the payoff: you can look at a specific line of your save routine and say, with certainty, what a crash on that line would destroy. Vague dread — "I might lose data" — is useless. A real durability argument names the exact window and the exact bytes at risk. This section turns your buffer knowledge into a crash-survival map you can read off any code that touches a file. It hinges on a distinction most programmers never make: a process crash and a power cut are not the same event, and they claim different victims.

The method is simple. Lay a crash on a timeline of your three calls and ask one question: which buffers currently hold the only copy of the data? Each buffer has a different owner and a different lifetime, so the answer changes as your bytes descend the tower. There are two named windows, each defined by which call has not yet run.

Window A — after write(), before flush(). The bytes live only in Python's user-space BufferedWriter, inside your process's heap. The kernel has never seen them. A crash here loses them, even a clean kill -9 with the power cable untouched, because a killed process's heap simply evaporates and nothing flushes a dead process's buffer. Note the asymmetry that makes this window uniquely dangerous: it is lost on a mere process crash, not just on power loss. An unhandled exception, a segfault in a C extension, an OOM kill — any of these in Window A, and the bytes are gone.

Window B — after flush(), before fsync(). Now the bytes are out of Python and in the kernel page cache. Here the asymmetry flips in your favor: a process crash is now survivable. The kernel owns those pages, and it will write them back on its own schedule whether or not your program still exists. So kill -9, an exception, an interpreter crash — none of them lose the data in Window B, because the data no longer depends on your process being alive. But a power cut or a kernel panic in Window B still loses everything, because the page cache is volatile RAM that writeback hadn't yet drained. Only when fsync returns are you clear of both windows: the bytes are on the medium and survive power loss.

You can make the abstract windows concrete. Build a save routine that crashes on command, plus a reader that reports what a fresh process would recover:

import os

def save(f, data, crash_at=None):
    f.write(data)
    if crash_at == "window_A":      # bytes only in Python's heap
        raise SystemExit            # process dies: data LOST
    f.flush()
    if crash_at == "window_B":      # bytes in kernel page cache
        raise SystemExit            # process dies: data SURVIVES
    os.fsync(f.fileno())        # past both windows: durable

Crash in window_A, reopen the file in a brand-new process, and the record is missing — the kernel was never told. Crash in window_B and the same reopen finds the record intact, because the kernel finished the job your dead process couldn't. Same crash type, opposite outcome, decided entirely by whether flush() had run. That single behavioral difference is the two-window model made real. To model the power cut you need a simulator, because a real power cut also wipes the page cache. Use a three-band model where power_cut() clears both python_buf and page_cache but keeps medium; run it at each stage and it prints exactly which records survive.

You can now protect a single write. But real programs face a nastier failure — a crash halfway through overwriting a file that was already good — and there fsync alone isn't enough. →

05Write-temp-then-rename: the atomic save

fsync makes a write durable, but it does nothing about a nastier and more common disaster: a crash halfway through overwriting an existing good file. Picture updating config.json. You open it for writing, and before you've written a single new byte you have neither the old complete version nor a complete new one. You have a truncated husk, and the good data you were replacing is gone forever. Every config editor, every save-game, every document app must guarantee one invariant: a reader always sees SOME valid file, never a blend of old and new. This section teaches the filesystem trick that makes a half-written file impossible to observe: the atomic rename.

First, see the trap precisely. open(target, "wb") truncates the existing file to zero length as part of opening it, before your first write(). So the destruction happens up front, and the window of vulnerability is the entire duration of your write. A crash anywhere in it leaves target corrupt, and the old contents are already destroyed. You can reproduce it in two lines: open for writing, raise an exception before writing anything, and os.stat(target).st_size is now 0. The good file is gone and you never wrote the new one.

The fix relies on a POSIX guarantee that is one of the quiet load-bearing facts of computing: rename(2) — Python's os.replace — is atomic within a single filesystem. A rename doesn't move bytes. It flips a directory entry from pointing at the old inode to pointing at the new inode, in one indivisible metadata operation. No observer, and no crash, can catch it half-done. Any reader that looks sees either the complete old file or the complete new file, never a partial one. So instead of editing the target in place, you build the new version off to the side and then swing the name over to it in a single atomic instant.

import os, tempfile

def atomic_write(path, data):
    d = os.path.dirname(path) or "."
    fd, tmp = tempfile.mkstemp(dir=d)        # same dir -> same filesystem
    try:
        with os.fdopen(fd, "wb") as f:
            f.write(data)
            f.flush()
            os.fsync(f.fileno())            # step 2: temp's DATA is durable
        os.replace(tmp, path)                # step 3: atomic name flip
        dirfd = os.open(d, os.O_RDONLY)      # step 4: persist the rename itself
        try:
            os.fsync(dirfd)
        finally:
            os.close(dirfd)
    except BaseException:
        os.unlink(tmp)                        # never leave a stray temp behind
        raise

Walk the failure modes, because each line is defending one. If you skip step 1 and write in place, you already saw the truncated-husk disaster. If you put the temp in /tmp instead of the target's directory, you may cross a filesystem boundary — /tmp is often a separate mount — and os.replace silently degrades from an atomic rename into a non-atomic copy-then-delete, reopening exactly the window you were trying to close. If you skip step 2, you can atomically rename an empty or partial temp file over your good one. A crash then leaves target pointing at durable garbage: durability without the data, the worst of both worlds. And if you skip step 4, the rename is durable in name only. The directory entry change sat in the page cache, and a power cut right after replace can un-happen the swap even though the temp file's data was safely persisted. Do all four and the invariant holds through any crash: target always names a complete, fsync'd file.

Now you can save one file durably and atomically. But do it in a loop — one fsync per record — and your program crawls. Databases commit tens of thousands of durable writes a second on the same disk. Here's the trick. →

06Durability costs throughput — how databases cheat

You now know the honest, safe save: fsync every record. Put it in a loop and watch your program crawl to a few hundred durable writes per second, because each fsync blocks on slow physical I/O and you pay that price once per record. Yet real databases commit tens of thousands of durable transactions per second on the very same hardware. They are not cheating on durability. Every one of those transactions genuinely survives a power cut. They are amortizing durability's cost with one specific trick, and it is the design pattern this whole chapter has been building toward: the write-ahead log.

Start from the physics you measured two sections ago. fsync's guarantee comes from waiting on the device, and that wait is a fixed cost per call — roughly one device round-trip plus a cache-flush barrier — almost independent of how many bytes you're syncing. Persisting one record and persisting a thousand records in a single fsync cost nearly the same, because the barrier dominates and the extra bytes ride along for free. Read that again, because it's the entire insight: fsync's cost is per-call, not per-byte. If you call it once per record, throughput is capped at about 1/latency — a few hundred to a few thousand ops per second, forever, no matter how fast your CPU. The only way up is to make one fsync do the work of many.

A write-ahead log does exactly that by combining two moves. First, instead of updating scattered data structures in place — random I/O, an fsync for each — you append every change as a record to the end of a single log file. Appends are sequential writes: no seeks, cache-friendly, the fastest thing a storage device does. Second, you fsync the log once for a whole batch of appended records, making them all durable together with a single barrier. Many small durable writes collapse into one. Here are the two strategies side by side, and the second is the whole game:

import os

def commit_each(records):        # one barrier PER record — slow
    with open("wal.log", "ab") as f:
        for r in records:
            f.write(r + b"\n")
            f.flush()
            os.fsync(f.fileno())   # N records -> N device round-trips

def commit_group(records):       # ONE barrier for the whole batch — fast
    with open("wal.log", "ab") as f:
        for r in records:
            f.write(r + b"\n")   # sequential appends, all in cache
        f.flush()
        os.fsync(f.fileno())       # N records -> 1 device round-trip

Feed both a thousand records: commit_each pays a thousand barriers, commit_group pays one, and the throughput gap is roughly a thousand-fold. It's the same ratio you measured in section 3, now working for you instead of against you. This is group commit, and a real database doesn't even need the records to arrive in one call. It gathers the commit records of many concurrent transactions arriving within a small time window (a millisecond or two), fsyncs once, then acknowledges them all together. Each transaction pays a tiny added latency — it waits a moment for the group to form — in exchange for a big throughput gain and full durability. Latency up a hair, throughput up 100×, durability uncompromised.

def replay(path):                  # a database's crash recovery, in miniature
    state = {}
    with open(path, "rb") as f:
        for line in f:
            key, val = line.rstrip(b"\n").split(b"=", 1)
            state[key] = val          # redo every durable change, in order
    return state

That is the core of a storage engine in about twenty lines: append changes to a durable log, replay it to rebuild state after a crash. Everything else — B-trees, MVCC, checkpoints that truncate the log once the data files catch up — is optimization on top of this spine. And notice how it reframes the whole chapter. fsync is the expensive, irreducible primitive, and the write-ahead log is the design pattern that makes durability affordable. It is literally the temp-file and fsync ideas of the last two sections — write it down safely first, make the durable copy the source of truth — scaled from one file to a production database. You didn't learn six disconnected tricks. You learned one law (durability costs a device round-trip) and the single pattern the entire industry uses to pay it as rarely as possible.

You've flattened objects to bytes and forced those bytes durably onto disk. Next: what happens when the bytes have to cross to another process — where even the object's shape has to travel. Serialization and the pickle pipe. →