15Files are syscalls — the file descriptor

In Volume 1 we lived entirely inside one running process. Every object sat at a real address in RAM, every name pointed straight at it, and the whole world was one private room. In this chapter we cut a door in that wall. Here's the plan. A program that does anything useful has to reach past itself — to the disk, to other encodings, to other people's code, to future-you. Every one of those boundaries obeys a single law: what crosses a process boundary is always bytes, never objects. We start at the oldest boundary of all, the one between memory that forgets and memory that remembers. All the way through we keep asking the one question that matters: if a file isn't a Python object, then what is it, and what really happens the instant you type open('song.txt')? By the end you'll see the answer for yourself. A "file" is a kernel thing you can only ever touch through a tiny integer, and open, read, write, close are four traps into the operating system wearing a friendly object as a disguise.

01Why disk at all: the volatile / non-volatile split

Let's start with a small cruelty you've lived with since your very first program and probably never said out loud. Every script you have ever written forgot everything the instant it ended. That list you spent an hour building, a running total, a game's high score — all gone the moment the process exits, as if it had never existed. We tend to shrug it off: "that's just how programs work." But that shrug hides a real physical fact. Until you feel why memory forgets, open() looks like an arbitrary API instead of what it really is: the only door out of a burning building.

So look at where your data actually lives. In Volume 1 every object sat in DRAM — dynamic random-access memory, the main memory of your machine. Here is what DRAM really is under that name. It stores each single bit as a puff of electric charge on a microscopic capacitor, gated by one transistor. Engineers call it the 1T1C cell: one transistor and one capacitor per bit. And here is the fatal part. That capacitor is tiny, a few femtofarads, and it leaks. Charge drains through the transistor's imperfect off-state in a matter of milliseconds. Leave a stored 1 alone and it decays into an unreadable smear in well under a tenth of a second.

So DRAM cheats to stay alive. The memory controller marches through every row of every chip roughly every 64 milliseconds and performs a refresh. It reads each row and immediately writes it back, re-pouring charge into buckets that are always leaking. Your "static" variable is really a puddle that is constantly being topped back up. This is not a metaphor. It is happening billions of times a second in the machine reading this sentence. Now cut the power. Refresh stops within a single cycle, and every capacitor drains to the noise floor. The bits are not "lost by the operating system" or "cleared on shutdown." They are physically gone, because the physics that held them needed a power supply and the power supply is gone. That is what the word volatile means, all the way down.

That different hardware exists, and it works by a completely different trick. It stores state in a form that survives with zero power. A NAND flash cell traps a clump of electrons on a floating gate — a scrap of conductor completely sealed behind an oxide insulator. Those trapped electrons shift the transistor's threshold voltage, and the oxide wall holds them in place for years with nothing plugged in. A spinning hard disk uses an even older trick. It stores each bit as the magnetization direction of a grain of ferromagnetic material on a platter, and a magnetized grain simply stays magnetized when the lights go out. Flash and disk are non-volatile. Pull the plug and they lose nothing.

This is the axiom the entire chapter — the entire volume — rests on. There is a hard line running through your computer's hardware. On one side is fast volatile silicon that forgets the instant it is unpowered. On the other is slower non-volatile media that remembers without power. A "file" is nothing more grand than a byte that has crossed to the remembering side. That is the whole concept. The reason open() exists, the reason files exist as an idea, is that some hardware can hold a bit without being fed — and RAM cannot.

You can feel the split with three lines of code. Write a program that counts, then run it twice:

# counter.py — run me twice. I will always say 1.
count = 0
count = count + 1
print(count)            # 1 ... and 1 again on the next run, and forever

It restarts at zero every time because count lives in DRAM, and DRAM forgot it the microsecond the process died. Now change one thing. Read the previous value from a file at the top, and write the new one back at the bottom. Suddenly the count climbs across separate runs — 1, 2, 3 — surviving the death of the program each time. Nothing about the arithmetic changed. What changed is that one number physically left volatile silicon and landed on the remembering side. That crossing is what the rest of this chapter takes apart, mechanism by mechanism.

So a byte has to reach non-volatile hardware. But your Python process is a plain user program with no wire to the disk — if it tried to command the drive directly, the CPU would fault. So how does one innocent line, open('song.txt'), reach a device you are forbidden to touch? →

02open() is a trap into the kernel

You now know a byte has to reach non-volatile hardware to survive. But sit with an uncomfortable fact: your Python process is not allowed to touch that hardware. Your code cannot issue a command to the SATA or NVMe controller. It cannot poke the disk's registers or nudge the spinning platter. Suppose it tried. If the CPU, while running your bytecode, attempted the privileged instruction that talks to a device controller, the processor would raise a fault and the operating system would kill your program on the spot. So we have a paradox. One line, open('song.txt'), clearly does reach the disk. How, if your code is forbidden from doing so directly?

The answer lives in the CPU itself, in a mechanism called privilege rings. The processor is always running at some privilege level. Your program — and every ordinary application — runs in ring 3, the unprivileged, fenced-in level. The operating system kernel runs in ring 0, with full authority over the machine. Certain instructions are privileged: talking to device controllers, remapping the page tables that define memory, touching I/O ports. Attempt any of them from ring 3 and the hardware itself slams the door — a fault, not a warning. This is enforced by the silicon, not by politeness. By design, a user process has no instruction that reaches the disk.

So how do you ever get anything done? Through the one sanctioned door in the wall: a syscall. A syscall is a special CPU instruction — literally syscall on x86-64, svc on ARM64 — that does something no ordinary instruction can. It atomically switches the CPU into ring 0 and jumps to a single fixed entry point inside the kernel, an address the hardware was told about at boot. You don't get to choose where in the kernel you land. You get exactly one guarded gate. Which service you want is selected by a number you place in a register beforehand. On Linux x86-64, register rax holds the syscall number (openat is number 257). This controlled, one-way crossing is called a trap. The program traps into the kernel, the kernel does the privileged work on your behalf, and control traps back out to ring 3.

Follow one open through the gate. Your ring-3 code executes the syscall instruction. The CPU flips to ring 0 and lands at the kernel entry point. Now running with full privilege, the kernel walks the filesystem to find song.txt, talks through the block-device driver to the actual disk controller, sets up its bookkeeping for this newly opened file, and then hands back a single small non-negative integer. The CPU flips back to ring 3, and your program resumes with that integer in hand. That integer is the file descriptor — the kernel's receipt, the handle you will use for every later read and write. It is not the file. It is a claim ticket the kernel gave you.

CPython's friendly open() is many layers of Python and C sitting on top of exactly this syscall. To see the raw thing with nothing dressed on it, use os.open. It is the thin wrapper that does little more than the trap, then hands you the number back:

import os
fd = os.open("song.txt", os.O_RDONLY)
print(fd)          # 3 — a bare integer. Not an object. The kernel's receipt.

There it is: 3. Not a file object, not a path, not a stream — an int. That is the correction this section exists to make. open() is not "Python opening a file." It is Python asking the kernel, across a hardware-enforced mode switch, to open a file on Python's behalf and hand back a handle. The friendly f = open(...) object you're used to has that same integer buried inside it. Call f.fileno() and it will show you the very number the kernel returned. Everything else is wrapping paper around a receipt.

The kernel handed you the integer 3 — never 0, 1, or 2. That's not luck and it's not random. Why does counting start at 3, and where does the number even live? →

03The fd table, and why 0, 1, 2 are already taken

Open your first file and you get descriptor 3. Open one in a brand-new program tomorrow and you still get 3. Never 0, never 1, never 2. That is remarkably consistent for something that looks arbitrary, and consistency is always a clue that a structure is doing the choosing. There is one. If you don't understand the array these integers index into, then leaks, reuse, and "why did I just read the wrong file?" will confuse you for the rest of your career. So let's make the descriptor concrete.

Every process has its own private file-descriptor table — an array living inside the kernel, hanging off the kernel's bookkeeping struct for your process. It is indexed by the descriptor integer. Entry i doesn't hold the file itself. It holds a pointer to a kernel open-file description — a small record of the current read/write offset, the flags you opened with, and a link to the underlying file on disk (its inode). The descriptor you hold in Python is just the index. All the real state lives on the kernel's side of the wall. This is why the descriptor is a mere integer. It is an array subscript, and the array isn't yours to see.

Now the mystery of 0, 1, 2. When the shell (or whatever launched you) starts your process, it does not hand you an empty table. It pre-wires the first three slots. Index 0 is standard input, index 1 is standard output, index 2 is standard error — each already pointing at your terminal, or wherever they were redirected. This is why print reaches your screen even though you never opened anything. print ultimately writes to descriptor 1, which was bound to your terminal before your first line ran. You inherited a working output before you did anything at all.

And here is the rule that ties it together: open always installs the new file at the lowest free index in the table. With 0, 1, and 2 already occupied, your first open can only land at 3. The next at 4. The next at 5. There is no randomness. It is just "smallest empty slot," every time.

import os, sys
print(sys.stdin.fileno(), sys.stdout.fileno(), sys.stderr.fileno())  # 0 1 2
a = os.open("song.txt", os.O_RDONLY)
b = os.open("song.txt", os.O_RDONLY)
print(a, b)      # 3 4 — the two lowest free slots, in order

Two consequences fall straight out of "it's a private, finite array." First, the table is per-process. Your descriptor 3 and another program's descriptor 3 index two entirely different arrays pointing at two entirely different files. Descriptors are not global names. A 3 only means anything inside the process that owns the table. Second, the array is finite. Its capacity is capped by a limit named RLIMIT_NOFILE, the thing your shell's ulimit -n reports, often 1024 by default. You may hold only that many open descriptors at once.

That ceiling is not theoretical. Open files in a loop and never close them, and you watch the descriptors climb — 3, 4, 5, …, 1021, 1022, 1023 — and then the program dies with OSError: [Errno 24] Too many open files. That is the finite table, felt as a crash. A leaked descriptor is a slot you claimed and never gave back.

The flip side of "lowest free index" is reuse. Close descriptor 3 and its slot goes empty again, so the very next open hands you 3 back, not 5. This is the source of one of the nastiest bugs in systems programming. You close a file, keep a copy of the old number 3 lying around, and later something else opens a different file that also gets 3. Now your stale 3 silently reads the wrong file, and the kernel is perfectly happy to comply, because to it your 3 is a valid index into a valid slot. The number was reclaimed. Your memory of what it meant was not.

You hold descriptor 3. Now you actually want the file's bytes inside a Python variable — but your memory and the disk sit on opposite sides of the ring wall. How do bytes physically cross, and does the kernel have any idea what they mean? →

04read and write copy bytes across the ring boundary

You hold descriptor 3. You want two things: the file's contents pulled into a Python variable, and your own bytes pushed out onto the disk. But recall where these two live. Your process's memory sits in ring-3 user space. The disk and the kernel's buffers sit behind the ring-0 wall. They are different memory that the CPU deliberately keeps apart. So the real question of this section is physical: when data moves between your variable and the file, what actually happens to it? And a quieter question sits underneath. When you write the string "My Favorite Song", does the kernel understand it's a song title?

Both movements are syscalls, each a trap into ring 0 exactly like open was. read(fd, n) asks the kernel to copy up to n bytes from its buffers for that file into a buffer in your process's address space. write(fd, buf) copies the other direction: bytes from your memory into the kernel's buffers, which the kernel later flushes to the physical device. Notice the load-bearing word in both sentences: copy. Your user pages and the kernel's buffers are separate memory the CPU won't let mix, so the kernel must physically move the bytes across the boundary — a real memcpy, plus the cost of the mode switch. That double cost, a trap and a copy on every call, is the whole reason I/O is expensive and the whole reason we buffer instead of doing a syscall per byte.

Now the crucial part, the one that reframes what a file is. The payload is always raw bytes — an array of integers from 0 to 255, nothing more. The kernel has no notion that your bytes spell a song title, or encode a JSON object, or form a JPEG. When you write b"My Favorite Song", the kernel sees the sequence 77, 121, 32, 70, 97, 118, … and faithfully moves those numbers. It does not see "My Fav…". It cannot. Meaning is not something the kernel stores or checks. Meaning is imposed entirely by your program, above the wall. The file on disk is just a byte sequence. The fact that it "is" a song title or a picture is an agreement between the program that wrote it and the program that reads it. The kernel is a neutral courier of opaque numbers.

There is one more truth the friendly API hides, and it bites everyone eventually. These syscalls return a count. write returns how many bytes the kernel actually accepted. read returns how many it actually delivered, and that can be fewer than you asked for. Ask read for 4096 bytes near the end of a file and you'll get only the handful that remain. Read from a pipe or a socket and you may get a trickle. This is called a short read (and its sibling, the short write). Assuming a single call moved everything is a classic, painful bug.

import os
fd = os.open("song.txt", os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o644)
n = os.write(fd, b"My Favorite Song")
print(n)              # 16 — the count of bytes accepted, not True, not None
os.close(fd)

fd = os.open("song.txt", os.O_RDONLY)
print(os.read(fd, 4))    # b'My F' — only 4 bytes; a deliberate short read

Look at what write handed back: 16, not True. It is telling you it moved sixteen bytes. You're expected to check that number, because in the general case it might have moved fewer and you'd need to loop and send the rest. And read(fd, 4) gave you b'My F' — four raw bytes, prefixed b for bytes, because that is genuinely all the kernel deals in. Your program is the thing that decides those four numbers are letters.

You saw a leak crash a program by never closing. So closing matters — but you can't just add a close() at the end and relax, because what if the line before it raises? →

05close() and the context manager that guarantees it

The last section left you with a warning felt as a crash: leak descriptors and you eventually slam into the finite table and die. So closing matters. The naive fix is obvious — put a close() at the end and move on. But that fix has a hole you can drive a truck through, and every teacher who ever nagged you about with open(...) was quietly pointing at it. What if a line between the open and the close raises an exception? Then execution leaps over your close(), the descriptor leaks, and, worse, the bytes you thought you'd saved may never reach the disk at all. This section is about making cleanup a guarantee instead of a hope.

First, what close(fd) actually does. It is itself a syscall. It returns your descriptor's slot to the process's fd table, freeing that index for reuse — exactly the reclamation we saw in section 3. It also tells the kernel you're finished with this open-file description, so the kernel can drop its reference and, when the last reference falls, release the underlying bookkeeping. Two separate things depend on close running: the finite descriptor table (unclosed descriptors leak until ulimit stops you) and durability. That second one is subtle. Your writes usually don't go straight to the platter. They sit in the kernel's buffers, and above that, in Python's own userspace buffer. Closing flushes Python's buffer down through the write syscall and hands clean state to the kernel. Skip the close and your final writes may still be sitting in a buffer that dies with the process.

Now the hole. Consider the honest, well-meaning version:

f = open("song.txt", "w")
f.write("My Favorite Song")
risky()               # if this raises, control jumps AWAY...
f.close()             # ...and this line never runs. Leak + unflushed data.

If risky() raises, the exception propagates up and out of this block, straight past f.close(). The descriptor leaks. The buffered write may be lost. And this isn't an exotic edge case. Anything can raise: a bad value, a full disk, a keyboard interrupt. A cleanup line you merely hope executes is not cleanup.

The context manager closes the hole structurally. Write it the way you've been told to:

with open("song.txt", "w") as f:
    f.write("My Favorite Song")
    risky()           # even if THIS raises...
# ...__exit__ still fires here — buffer flushed, fd slot freed — then the
# exception keeps propagating. Cleanup happened on the way out regardless.

Here is the actual mechanism, no magic. with open(...) as f binds the file object to f and, more importantly, the with statement guarantees that f.__exit__() is called when the block ends — normally OR via an exception. The file object's __exit__ method calls f.close(). That's the whole trick. __exit__ runs on every path out of the block, so a crash mid-write still flushes the buffer and still frees the descriptor before the exception continues on its way. with is treated as "law" not because teachers love tidiness, but because it moves cleanup from a line you hope executes to a guarantee the language enforces on every exit path.

You can prove __exit__ fires under an exception with your own eyes. Write a tiny class with __enter__ and __exit__ that prints when it exits, wrap a deliberate crash in it, and watch EXIT called appear even as the traceback prints. Cleanup ran on the way out, and only then did the exception surface. Do the same with a real file and reopen it afterward: the bytes are there, flushed by the close that __exit__ guaranteed.

All chapter you've been told the file is always bytes — yet open('song.txt') hands you a str, while open('song.txt','rb') hands you bytes. Same descriptor, same disk, same byte-only kernel. So where do the strings come from? →

06Text mode is a codec layer bolted onto a byte stream

Something in this chapter should have been quietly bothering you. Five sections of "the file is always bytes; the kernel only moves numbers" — and yet the very first thing you ever did, open('song.txt'), handed you a str. Open the same file with 'rb' and you get bytes instead. But it is the same descriptor, the same disk, the same kernel that traffics only in numbers from 0 to 255. The bytes on the platter did not change. So where did the string come from? Something between the kernel and your variable is manufacturing str out of bytes. Until you can point at exactly which layer does it, the b'' prefix and every "works on my machine" encoding bug will feel like black magic.

Underneath every file mode there is one descriptor delivering raw bytes from the kernel. That is the ground truth, and it never changes. What changes is how many layers Python stacks on top of it. Open in binary mode — 'rb' or 'wb' — and Python gives you that byte stream almost bare: a BufferedReader or BufferedWriter sitting directly over the descriptor. read returns bytes, write demands bytes, and no interpretation happens, because none was asked for. This is the honest view of the file: exactly what the kernel moved.

Text mode — plain 'r' or 'w' — wraps that same buffered byte stream in one more object: an io.TextIOWrapper. That wrapper is a codec layer, a two-way translator. On the way in, it decodes incoming bytes into a str using an encoding. On the way out, it encodes your str back into bytes before they ever reach the buffer and the write syscall. It also quietly translates newlines (\r\n on Windows ⇆ \n) so your text looks the same everywhere. The consequence is the sentence that dissolves the whole mystery: a str never touches the disk and never crosses the syscall boundary. Bytes do. The str exists only above the TextIOWrapper, conjured by the codec.

See both branches at once. The bytes on disk are identical; only the number of layers differs:

open("song.txt", "rb").read()   # b'My Favorite Song' — bytes, raw off the fd
open("song.txt", "r").read()    # 'My Favorite Song'  — str, decoded by the codec layer

The gap between them widens the moment a character needs more than one byte. Save a file containing a musical note ♪ or an emoji, then count. In 'rb' mode the length is larger than the number of visible characters, because that one glyph is several bytes on disk (three, for ♪, in UTF-8). Read the same file in 'r' mode and the length equals the character count exactly, because the TextIOWrapper consumed those several bytes and produced one str element. That collapse of many bytes into one character is the decode step, happening at a layer you can now name and point to.

Which raises the sharpest, most practical point of the section: which encoding does text mode use? By default, whatever your platform's locale says — often UTF-8 on modern systems, but not always. That "not always" is the source of a thousand "but it worked on my laptop" bugs. A file written as UTF-8 on your machine and read with a different default encoding on a colleague's can decode into garbage, or raise outright. The codec layer is powerful, but its default is a guess made by the platform, and guesses are where correctness goes to die.

And because the str only exists if the codec succeeds, you can watch the whole illusion refuse to form. Write bytes that are not valid UTF-8 in 'wb' mode, then try to read that file in 'r' with encoding="utf-8". You get a UnicodeDecodeError. Not corrupted text — no text at all, because the decoder could not build a legal str from those bytes. That error is the codec layer telling you plainly that the string was never real disk data. It was always something the top layer manufactured, and manufacturing can fail.

You've crossed the first boundary: bytes leaving your process, over the ring wall, onto remembering hardware, and back again — with a codec layer deciding whether they come home as bytes or as text. That codec is a whole world of its own. Chapter 17 climbs inside it: what an encoding actually is, why UTF-8 won, and how a handful of bytes agree to mean a single character. →