17Bytes vs text — Unicode and UTF-8 at the byte level

In Chapter 16 we made a write survive — flush, fsync, the page cache, all the machinery it takes to be sure the bytes really reach the disk. But we never asked what those bytes were. This chapter does, and I want to go slowly. There's a wall hidden in here that Python 3 built on purpose — the wall between text and bytes — and most tutorials climb over it without ever naming it. Here's the plan. We'll set a str and a bytes object side by side and watch four visible letters turn into five integers. Then we'll derive UTF-8 by hand, one bit at a time, until no black box is left. The whole way through we keep asking the one thing that matters: what really crosses the line the moment a string leaves your process? The answer never changes. What crosses a process boundary is always bytes, never objects — a flat run of numbers from 0 to 255, with nothing left of the rich, structured str that RAM was holding. Get that crossing wrong and you get mojibake. That's not corruption and not a Python bug — just a decoder reading perfectly good bytes against the wrong table. By the end you'll have derived the whole of UTF-8 with your own hands, and never be surprised by an encoding again.

01Two alphabets: code points vs 0–255

Let's start with the smallest thing that cracks the illusion open. Open a fresh prompt and type this — four letters you can see — then watch what comes back:

>>> "café".encode("utf-8")
b'caf\xc3\xa9'                # four letters → five bytes, one shown as \xNN
>>> len("café"), len("café".encode("utf-8"))
(4, 5)

Look at what it did. You've spent your whole programming life treating text and a file's contents as the same substance. You read a file, you get text; you print text, it lands in a file. Python 3 flatly refuses to let those two blur. The é you see as one letter came back as two bytes, \xc3\xa9, and the object holding them is not a string at all. It's a different kind of thing that merely prints in a way that fools you. That's the wall this whole chapter is about. Until it's concrete in your hands, every codec idea here has nowhere to stand. So let's make it concrete.

A Python str is a C struct called PyUnicodeObject. Inside, it holds an array of code points — abstract character-identity numbers — plus a one-word kind tag. That tag is PEP 393's flexible string representation. It lets the array store each code point in the narrowest slot that fits the widest character present: 1 byte per character if every code point fits in Latin-1 (kind 1), 2 bytes if the largest fits in the Basic Multilingual Plane (kind 2, UCS-2), and 4 bytes otherwise (kind 4, UCS-4). Here's the fact that matters. When you write s[i] you get back a length-1 str, and ord(s[i]) is a code point somewhere in the range 0 to 0x10FFFF — about 1.1 million possible values. The str alphabet is over a million symbols wide.

A bytes object is a completely unrelated C struct, PyBytesObject: a flat C char array of raw uint8 values. When you write b[i] you get back a plain Python int, and that int is somewhere in 0 to 255 — exactly 256 possible values, no more. The bytes alphabet is 256 symbols wide, full stop. So a str and a bytes are two distinct types over two distinct alphabets: about 1.1 million code points versus exactly 256 byte values. They are as different as a musical score is from the grooves cut into a vinyl record. One is meaning; the other is a physical encoding of it.

Python 2 blurred this wall and paid for it. There, 'x' + u'y' silently decoded the byte string as ASCII to promote it to Unicode. That worked on your English test data, then detonated with a UnicodeDecodeError in production the first time a real name arrived. Python 3 tore that bridge down on purpose. Now 'x' + b'y' raises a TypeError immediately — no default codec, no ASCII guess, no runtime surprise. There is no implicit conversion in either direction, ever. The only two functions that cross the wall are str.encode and bytes.decode, and each one demands that you name a codec. That refusal is the feature.

The thing that fools everyone is the repr. When Python shows you b'caf\xc3\xa9', that leading b is a signpost: this is a sequence of integers. It renders a byte as ASCII when the byte happens to be a printable ASCII character, and as \xNN when it isn't. It's a display convenience, not text. list(b'caf\xc3\xa9') unmasks it as [99, 97, 102, 195, 169] — five integers, nothing more. The c, a, f you see are just bytes 99, 97, 102 that happen to fall in the printable ASCII range. The machine has no idea they "spell" anything.

You now have two clearly separated shores. Next: what actually lives on the left shore — a code point isn't a letter and it isn't a byte, it's a third thing the Unicode standard invented to end the chaos. →

02Unicode decouples character from storage

Here is the chaos the last section left you with. The character ë takes 1 byte in one file and 2 bytes in another. Two é's that look pixel-identical fail an == test. You paste a name from a web form and its length comes back "wrong." It feels like the machine is improvising. It isn't. There is one organising idea that dissolves all of it, and once you have it the next three sections become mere storage trivia. The idea is this: every character has a single, fixed, storage-independent identity, assigned once by the Unicode standard — and everything else is a downstream choice.

At its core, Unicode is a giant numbered lookup table. It defines a code space of exactly 1,114,112 slots — the code points U+0000 through U+10FFFF — carved into 17 planes of 65,536 each. Plane 0, the Basic Multilingual Plane, holds almost every character of every living language. The astral planes 1 through 16 hold emoji, historic scripts, and math alphanumerics. A code point is an abstract integer identity, permanently bound to a character NAME and a set of properties. U+00EB is, forever and everywhere, LATIN SMALL LETTER E WITH DIAERESIS. That binding says nothing at all about bytes. It is pure identity — a museum accession number for the character.

Python hands you this exact map through three built-ins. ord('ë') goes character → code point and returns 235 (that's 0x00EB). chr(235) goes the other way, code point → character, returning 'ë'. And unicodedata.name('ë') returns the official standard name, 'LATIN SMALL LETTER E WITH DIAERESIS'. Try it:

import unicodedata
for ch in "ë€🌍":                 # e-diaeresis, euro sign, globe emoji
    print(ch, hex(ord(ch)), unicodedata.name(ch))

# ë  0xeb    LATIN SMALL LETTER E WITH DIAERESIS
# €  0x20ac  EURO SIGN
# 🌍  0x1f30d  EARTH GLOBE EUROPE-AFRICA   ← above U+FFFF: an astral character

Now the decoupling, which runs along two axes. The first axis is the one this whole chapter builds toward: the same code point can be serialised into different byte sequences by different encodings. U+00EB is [EB] in Latin-1, [C3 AB] in UTF-8, and [EB 00] in UTF-16LE. One identity, three byte spellings. That's section 3's business — identity versus storage.

The second axis is subtler, and it's the one that produces the failed ==. The same visible character can be built from different sequences of code points. Your ë might be the single precomposed U+00EB, or it might be two code points glued together: U+0065 (plain e) followed by U+0308 (COMBINING DIAERESIS, the two dots). Both render as an identical ë on screen, but they are different strings — one code point long versus two — so 'ë' == 'ë' can be False. Here identity (code points) diverges from appearance (glyph).

import unicodedata
a = "ë"              # precomposed: one code point
b = "ë"             # decomposed: 'e' + combining diaeresis
print(len(a), len(b), a == b)          # 1 2 False  — same glyph, different identity
print(unicodedata.normalize("NFC", b) == a)   # True   — NFC folds them together

unicodedata.normalize('NFC', s) folds a string to its composed canonical form; 'NFD' gives the decomposed form. This is why you normalize first any time you compare or store user-entered text. Otherwise two names that look identical to a human become two different keys in your database. The rule to carry: code point ≠ byte sequence ≠ glyph. Three layers, and the standard pins down only the top one.

Identity is settled: ë is U+00EB, everywhere, forever. But how does U+00EB become the bytes C3 AB on disk? That number appears nowhere in 0xEB. Time to open the black box and derive UTF-8 one bit at a time. →

03UTF-8, decoded bit by bit

You accept that ë is U+00EB. But 'ë'.encode('utf-8') handed you b'\xc3\xab', and 0xEB is neither of those numbers. Where did C3 and AB come from? Until you can produce those two bytes from 0xEB with a pencil, UTF-8 stays magic — and you'll never be able to look at a truncated multibyte sequence and know what broke. So we're going to turn the black box into an algorithm you run on paper. Give me a code point, and I'll give you the exact bytes and tell you why every single bit sits where it does.

UTF-8 is a variable-width encoding: it packs a code point's bits into 1, 2, 3, or 4 bytes, using self-describing leading bytes. The only question it asks is: how many significant bits does this code point need? The four ranges map to four templates, where the x's are payload slots you fill with the code point's bits:

U+0000 .. U+007F   (≤ 7 bits)   0xxxxxxx                            # 1 byte  — identical to ASCII
U+0080 .. U+07FF   (≤11 bits)   110xxxxx 10xxxxxx                   # 2 bytes
U+0800 .. U+FFFF   (≤16 bits)   1110xxxx 10xxxxxx 10xxxxxx          # 3 bytes
U+10000..U+10FFFF  (≤21 bits)   11110xxx 10xxxxxx 10xxxxxx 10xxxxxx # 4 bytes

Two invariants make this robust. First: a leading byte's high bits announce the total length — 0 means 1 byte, 110 means 2, 1110 means 3, 11110 means 4. Second: every continuation byte matches 10xxxxxx, and nothing else does. A continuation byte can never be mistaken for a start byte, and a start byte can never be mistaken for a continuation. Hold onto that — it's the whole reason UTF-8 won.

Now let's derive ë by hand. ë = U+00EB = 235 = binary 1110 1011, which is 8 significant bits. Eight is more than 7, so the 1-byte form won't hold it. We move to the 2-byte form, which offers 11 payload slots. Left-pad our value to 11 bits: 000 1110 1011. Split it 5 + 6: 00011 and 101011. Drop those into the two templates:

U+00EB = 235 = 000 1110 1011        # 11 payload bits: 00011 | 101011

 byte 1:  110 00011  = 1100 0011  = 0xC3     # 110 marker + first 5 payload bits
 byte 2:  10  101011 = 1010 1011  = 0xAB     # 10 marker  + last 6 payload bits

 result:  b'\xc3\xab'                        # exactly what Python produced ✓

There it is: C3 AB, derived from 0xEB with nothing but bit-shuffling. The C3 and AB aren't stored anywhere in the number — they're the number's bits re-packaged into a self-describing frame. Run 'ë'.encode('utf-8').hex() and Python confirms 'c3ab'.

Those two invariants buy you UTF-8's killer property: self-synchronization. Drop into a UTF-8 stream at any random byte and you can find a character boundary instantly. If the byte starts with 10, you're mid-character — step backward until you hit a byte that doesn't. If it starts with 0 or 11, you're already at a character start. Because no continuation byte can pose as a start, a single corrupted byte damages exactly one character. The rest of the file stays perfectly readable. UTF-16 has no such property: one lost byte shifts every following pair and shreds the entire remainder. Self-sync is why UTF-8 became the encoding of the web.

The 1-byte branch hides a gift. For U+0000 through U+007F the template is 0xxxxxxx — the top bit is zero and the low 7 bits are the ASCII value. So UTF-8 is byte-for-byte identical to ASCII for the first 128 characters. Every ASCII file ever written is already valid UTF-8. That backward compatibility is the second reason UTF-8 won: the entire existing corpus came along for free.

You can now turn any code point into bytes by hand. But you rarely call .encode directly — you call open(), print(), socket.recv(). Next: every one of those is encode or decode wearing a costume. →

04encode and decode are the only doors

You've read files, printed API responses, and written logs for years without once consciously choosing an encoding. So when I claim that every read is a decode and every write is an encode, it sounds like an abstraction you can safely skip. You can't. It means those two functions from section 1 aren't niche utilities. They are the only hinges the entire text/binary boundary swings on. open(), print(), sockets, JSON over HTTP, subprocess output — all of it is str.encode and bytes.decode in costume. Once you can point at the door in any I/O situation, encoding bugs stop being spooky and collapse into one question: which of my two doors used the wrong key?

There are exactly two functions that cross the wall, and each takes a codec name. str.encode(codec, errors) walks the code points and emits a bytes. bytes.decode(codec, errors) walks the bytes and reconstructs the code points. Everything else is a wrapper around these two. Consider open:

with open("note.txt", "w", encoding="utf-8") as f:
    f.write("héllo")          # encode door: str → bytes → write() syscall

with open("note.txt", "rb") as f:
    print(f.read())         # b'h\xc3\xa9llo'  — raw bytes, no codec at all

open(path, 'r', encoding='utf-8') returns an io.TextIOWrapper layered over a raw binary buffer. On .read() it pulls bytes from the OS via the read() syscall and feeds them through the codec's incremental decoder. On .write() it runs the incremental encoder and hands the resulting bytes to write(). The wrapper is nothing but an automated decode/encode door with a buffer bolted on. Open the same file 'rb' and you skip the wrapper entirely: you get raw bytes, no codec, no interpretation. Sockets and pipes are always bytes. sock.recv() returns bytes and you decode; sock.sendall() demands bytes so you encode. There is no text on a wire, ever.

Why incremental? Because a multibyte character can straddle a buffer boundary. Suppose a chunked read splits ï = C3 AF across two 4096-byte reads: C3 at the end of one chunk, AF at the start of the next. A naive decoder would choke on the lone C3. The incremental decoder instead holds the partial byte, emits nothing, and completes the character when AF arrives. That's why streaming text is a codec object with state, not a one-shot function call.

The codec name is itself a key into the codecs registry: codecs.lookup('utf-8') resolves to an encoder/decoder pair. 'utf-8', 'latin-1', 'ascii', 'utf-16', and 'cp1252' each name a different pair. The errors parameter governs what happens when the codec meets input it can't map. This matters more than it looks:

b = b"\xff\xfehi"                              # \xff is not valid UTF-8
print(b.decode("utf-8", errors="replace"))    # '��hi'  — U+FFFD for each bad byte
print(b.decode("utf-8", errors="ignore"))     # 'hi'    — bad bytes dropped
print(b.decode("utf-8", errors="backslashreplace"))  # '\xff\xfehi'
# errors="strict" (the default) would raise UnicodeDecodeError here

'strict' (the default) raises UnicodeDecodeError / UnicodeEncodeError. 'replace' substitutes U+FFFD (the � replacement character) on decode, or '?' on encode. 'ignore' silently drops the offender. 'backslashreplace' emits \xNN escapes. 'surrogateescape' is the clever one: it smuggles each undecodable byte into the reserved surrogate range U+DC80–U+DCFF, so a later encode with the same handler can round-trip it back to the exact original byte. That's how os.fsdecode survives a filename on disk that isn't valid UTF-8: the raw bytes ride through your program hidden in surrogates and come out untouched.

The law that ties it together: for any str s and any codec that covers it, s.encode(codec).decode(codec) == s. The doors are exact inverses when you use the same key on both sides. Encode with UTF-8, decode with UTF-8 — you get s back, guaranteed. Encode with one key and decode with another, and you get section 5's nightmare.

Same key both sides = round-trip. Different keys = the most infamous bug in all of text handling — the one that works on your laptop and corrupts on your coworker's. Let's dissect it. →

05Mojibake and 'works on my machine'

This is the bug that gets junior engineers blamed. Your script reads a CSV of customer names flawlessly on your Mac. You ship it. On your coworker's Windows box every ë is now ë — or worse, the program dies with UnicodeDecodeError on line 4,213. Not one character of your code changed. It feels like randomness, or a Python bug, or cosmic punishment. It is none of those. It is a decoder reading correct bytes against the wrong table, and it differs by machine because open() silently picked a locale-dependent default encoding. Let's take it apart mechanically, then kill the entire bug class with one habit.

Mojibake is a decode-side type confusion: correct bytes, wrong codec. Take ë stored as UTF-8, which section 3 taught you is C3 AB. Now decode those two bytes as Latin-1, where every byte is its own character by definition — byte N maps to U+00NN. You get U+00C3 + U+00AB = 'Ã' + '«' = 'ë'. The bytes were never damaged. They were read against the wrong table, so two bytes that meant one character got misread as two separate one-byte characters:

s = "café — 30€"
b = s.encode("utf-8")        # the correct bytes on disk
print(b.decode("utf-8"))      # 'café — 30€'   — right key, perfect
print(b.decode("latin-1"))    # 'café — 30€' — wrong key, mojibake

The reverse direction fails harder, and seeing why is the key to the whole asymmetry. Many byte sequences are simply illegal UTF-8. A lone continuation byte like 0xAB (which is 10101011, a 10xxxxxx fragment) with no leading byte in front of it matches no valid template from section 3. So the strict UTF-8 decoder can't guess. It raises UnicodeDecodeError: invalid start byte at a precise position. That's the asymmetry: Latin-1 can decode any byte (all 256 are legal), UTF-8 cannot. So UTF-8 text mistakenly read as Latin-1 tends to silently garble, while Latin-1 text mistakenly read as UTF-8 tends to loudly crash:

bad = "ñ".encode("latin-1")     # b'\xf1'  — a lone high byte
try:
    bad.decode("utf-8")
except UnicodeDecodeError as e:
    print(e.reason, e.start, e.object)  # 'invalid start byte' 0 b'\xf1'

Now the "works on my machine" part. When you call open(path) in text mode with no encoding= argument, Python historically used locale.getpreferredencoding(False) to pick the codec. On modern macOS and Linux that resolves to UTF-8. On Western-configured Windows it was cp1252 (a Latin-1 superset); on a Japanese Windows box it might be cp932. Same bytes, same code, different default table per machine — so a different result. Your laptop and your coworker's laptop literally ran two different decoders behind the identical line open('names.csv'). Nothing was random. The codec was just invisible, and its value came from the operating system's locale.

Python surfaced this landmine with EncodingWarning (PEP 597). Run with python -X warn_default_encoding and the interpreter flags every open() and .decode() that leaned on the locale default — a lint for exactly this bug. But the fix isn't a library or a flag. It's a discipline: name the codec on every boundary crossing. Write open(path, encoding='utf-8') and b.decode('utf-8') everywhere. The moment the codec is explicit, the table no longer depends on whose machine runs the code, and the entire bug class evaporates.

You can now diagnose the byte-level bug. One last trap lives at the other end — not in the bytes, but in counting. Ask a string "how long are you?" and it answers a question you probably didn't mean to ask. →

06len() measures code points, not width

You build a 280-character tweet limiter. Or you truncate a name to fit a VARCHAR(20) column. Or you draw a box around a string in the terminal. And it's subtly, maddeningly wrong. len('🇫🇷') returns 2 for a single flag. A comment with a family emoji counts as 7 when the user typed one "character." Truncating with s[:20] slices an emoji into rubble �. The root cause is that len() answers exactly one question — how many code points — and that number is almost never the same as "how many bytes on disk" or "how many symbols a human sees." Knowing which of three counts your task needs is what prevents a whole family of off-by-one, overflow, and truncation bugs.

len(s) on a str returns the number of code points — internally PyUnicode_GET_LENGTH, an O(1) field read of the code-point array's length. Not a byte count. Not a glyph count. There are three genuinely different measurements, and different text tasks demand different ones:

(1) Byte count = len(s.encode('utf-8')) — what a file, a socket, or a byte-sized database column actually consumes. A single ё is 1 code point but 2 UTF-8 bytes; 🌍 is 1 code point but 4 bytes. (2) Code point count = len(s) — the string's own element count, and what indexing and slicing operate on. (3) Grapheme count = the number of user-perceived characters, defined by the Unicode text-segmentation algorithm UAX #29. The standard library does not compute this; you need a grapheme library that implements extended grapheme clusters.

for s in ["A", "ё", "🌍", "🇫🇷", "👨‍👩‍👧"]:
    print(len(s.encode("utf-8")), len(s), "→", s)

# bytes  code points  glyph
#   1        1        A          plain ASCII
#   2        1        ё          one code point, two bytes
#   4        1        🌍          astral: one code point, four bytes
#   8        2        🇫🇷         one flag = TWO regional-indicator code points
#  18        5        👨‍👩‍👧        three people joined by two ZWJ

Look at how badly the three diverge, because several code points routinely render as one grapheme. A combining sequence 'e' + U+0308 is 2 code points and 1 grapheme. A country flag like 🇫🇷 is two Regional Indicator code points (U+1F1EB + U+1F1F7): 1 grapheme, 8 UTF-8 bytes. A ZWJ emoji like 👨‍👩‍👧 is 5 code points — three people joined by two U+200D zero-width joiners — making 1 grapheme and 18 bytes. One symbol on screen, five elements in the array, eighteen bytes on the wire. Three completely different numbers for one "character."

The danger is that str indexing and slicing work in code points, so s[:n] can cut straight through the middle of a grapheme. Truncate '🇫🇷' at [:1] and you keep only the first Regional Indicator, which renders as a bare regional letter, not a flag. Slice a database field at a byte budget and you can leave a half-encoded multibyte fragment that decodes to �. That's why a naive "characters remaining" counter, or a naive truncation, is a bug factory:

t = "Bonjour 🇫🇷"
print(len(t))        # 10 code points  (8 letters + space + 2 indicators)
print(t[:9])       # 'Bonjour 🇫'  — flag split, collapses to a lone letter
print([hex(ord(c)) for c in "🇫🇷"])   # ['0x1f1eb', '0x1f1f7'] — two code points, one flag

One clarification that ties this back to section 2: a single astral code point is one Python str element. len('🌍') == 1, because Python's str is code-point-based, not UTF-16-surrogate-based. JavaScript reports 2 for the same globe because its strings are UTF-16 and it counts the two surrogate halves. Python spares you that particular trap. But the flag and ZWJ traps remain, because those are about multiple code points forming one grapheme, which no code-point count can collapse.

Text is now fully demystified: identity is a code point, storage is an encoding, the two doors are encode and decode, and length is three different questions. Next chapter carries the same wall to a bigger boundary — the file system itself, where those bytes finally hit the disk. →