10Conveyors & generators

In Chapter 9 we turned functions into values you can pass around. Now we put that to work: we pour a whole playlist down a conveyor and let machines do the walking. Here's the plan. We map every track through a function, filter for the long ones, and reduce a column of runtimes down to a single total. Then we write that same work as one readable comprehension instead of a hand-cranked loop. All the way through, we keep asking the one question that matters — when does the work really happen? — because the answer is almost never the moment you'd guess. That question leads straight into generators: a function that pauses, a sequence that doesn't exist until you ask for the next item, a million songs modelled in a few hundred bytes. By the end, it's clear why sorted() hands you a fresh list while .sort() quietly rewrites the old one. And adding "Blinding Lights" at 200s and "Titanium" at 245s, then totalling their runtime, is two plain lines.

01map and filter: two machines on a conveyor

Let's start with the loop you'd write by hand. You open an empty list, walk the playlist, and append each reshaped item. You did exactly that in Chapter 9, and there's nothing wrong with it. But Python ships two machines that do the walking for you. The mental model is to picture them bolted over a conveyor belt: items roll past, a machine acts on each one, and a new stream flows out the far end. map(fn, stream) takes a function reference — the bare name, no parentheses, Chapter 9's rule — and runs every item that rolls past through it, emitting the reshaped stream. filter(pred, stream) takes a predicate, which is just a function that answers True or False about one item, and keeps only the ones that answer True. Everything else falls off the belt.

Now point them at our playlist and watch what they do. map with an upper-casing function turns every title LOUD. filter with a "longer than five minutes" test keeps only the marathon tracks and lets the radio-length ones drop away. Same conveyor, two different jobs: one machine reshapes every item, the other waves items through or off. And notice that neither one touches the songs on disk. Each spins up a fresh stream flowing out the far end, and the original list just sits there, untouched.

# a stream of songs — the belt everything rides on
playlist = [
    {"title": "Blinding Lights", "artist": "The Weeknd",           "seconds": 200},
    {"title": "Titanium",        "artist": "David Guetta ft. Sia", "seconds": 245},
    {"title": "One More Time",   "artist": "Daft Punk",            "seconds": 320},
    {"title": "Strobe",          "artist": "deadmau5",             "seconds": 634},
]

# map: a function REFERENCE (no parens!) + the belt to feed it
loud = map(lambda s: s["title"].upper(), playlist)
print(loud)              # <map object at 0x...>  — an IOU, not the titles
print(list(loud))        # ['BLINDING LIGHTS', 'TITANIUM', 'ONE MORE TIME', 'STROBE']

# filter: a predicate (True/False for ONE item) + the same belt
long = filter(lambda s: s["seconds"] > 300, playlist)
for s in long:
    print(s["title"])    # One More Time / Strobe  — the rest fell off

One catch, and it's the whole personality of this chapter: neither machine has done a scrap of work yet. Both return a lazy object. print(loud) doesn't show your titles — it shows <map object at 0x...>, a machine sitting idle. Nothing has been computed. It's a promise, not a result. The work happens only when something consumes the stream — list() around it, a for-loop over it, a sum() pulling on it — and even then it pays up one item at a time, computing each one only at the instant it's asked for. That sounds like a mere quirk until you meet the job that needs it: a stream you could never hold all at once, like a 50 GB log on an 8 GB laptop, or a sequence with no end. An eager list dies on both. A machine that computes one item on demand handles both with ease. That gap is the felt need this whole chapter is built to answer.

map has one more trick. Hand it several streams and it walks them in lockstep, plucking one item from each per step and handing the whole bunch to your function. map(min, radio, extended) pairs each track's radio-edit length with its extended-mix length and keeps the shorter — [200, 210, 190, 200]. It stops the instant the shortest stream runs dry, so feeding it streams of mismatched lengths never throws. One honesty note on filter: it keeps every item whose predicate comes back truthy, not literally True. Truthy is simply what you get when Python runs bool() on a value. A short, fixed list of things count as falsy — 0, 0.0, "", None, and any empty container ([], {}) — and everything else is truthy. That's why the barest predicate of all, filter(None, stream), needs no function at all. With None in the predicate slot, filter tests each item's own truthiness, drops every falsy one, and keeps the rest.

map turned four songs into four titles; filter kept two survivors. But the playlist's total runtime is one number squeezed out of all four lengths at once — and neither machine folds a stream down to a single value. So which one does? →

02reduce: fold many values into one

map hands back one new item for every item you gave it — many to many. filter keeps a subset — many to some. reduce does the third thing, and the most radical: it takes a whole sequence and folds it down to a single value — many to one. The total runtime of a playlist. The longest track. A merged index. This works one altitude down at the metal, too. A three-minute track is a stream of raw audio samples — millions of little integers, each one a byte pattern of the kind chapter 0 first switched on — and reduce is exactly what folds that whole wall of numbers down to the single loudest one, the peak. Any time a pile of things has to collapse into one answer, you are looking at a fold.

It's the only one of the three that Python doesn't hand you for free. You import it: from functools import reduce. It works by carrying a single running result called the accumulator. Your function takes exactly two parameters — the accumulator always first, the next item second — and returns the new accumulator. reduce seeds that accumulator with a starting value (here 0, for "no seconds counted yet"). Then it feeds the playlist through one song at a time, each step rolling the last result forward: 0+200→200, 200+245→445, 445+203→648, 648+215→863. The last accumulator standing is your answer.

from functools import reduce

total = reduce(lambda acc, song: acc + song["seconds"], playlist, 0)
print(total)          # 863
# fold: 0+200 → 200 | +245 → 445 | +203 → 648 | +215 → 863

Five moving parts — reduce, map, filter, lambda, and a list() to catch it all — is a lot of machinery for a one-line job. Python looked at map-then-filter and decided most of it was ceremony. Next: the comprehension, which folds a map and a filter into one line you can read like a sentence →

03Comprehensions: map + filter in one readable line

You have a list of songs and you want a new list from it — just the long ones, or every title upper-cased, or each length converted to minutes. In most languages that is a little ritual: set up an empty accumulator, loop, test, append, then read the accumulator back out. Python has a purpose-built spelling that states the whole intent on one line you can read aloud — the comprehension. Its shape is [expr for x in xs if cond]. It fuses the two oldest moves in all of programming — map (turn each item into something else) and filter (throw some items away) — into a single expression that evaluates to a finished container.

Read it in three slots, front to back. The expression up front says what each kept item becomes. The for clause names which items you visit. The trailing if is the gate: an item only survives into the result if that condition is True. Drop the if entirely and nothing is filtered — you keep every item, just reshaped by the expression. That is the whole grammar. Everything else is a variation on those three slots.

Two placements trip up everyone, and they are worth learning now, because both use the word if. A filtering condition goes after the for: [x for x in xs if cond]. But a transforming ternary — the one-line a if cond else b that reshapes each item — goes before the for: [a if cond else b for x in xs]. Same keyword, two entirely different jobs. After the for, the if decides which items get in. Before it, sitting inside the result slot, the if/else decides what shape each kept item takes. The tell is the else. A filter never has one — there is no "else, keep it out", the item simply isn't produced — but a transforming ternary always does.

# playlist: five dicts, each with title / artist / seconds
# "Blinding Lights" 200 · "Titanium" 245 · "Save Your Tears" 215
# "One More Time" 320 · "Strobe" 607

# FILTER — the if goes AFTER the for: keep only the long ones
long_titles = [s["title"] for s in playlist if s["seconds"] > 300]
# ['One More Time', 'Strobe']

# TRANSFORM — the ternary goes BEFORE the for: reshape every song
tags = ["long" if s["seconds"] > 300 else "short" for s in playlist]
# ['short', 'short', 'short', 'long', 'long']

# swap the brackets, swap the container — same three slots
durations = {s["title"]: s["seconds"] for s in playlist}   # dict comp
artists   = {s["artist"] for s in playlist}                # set comp
print(len(artists))   # 4 — The Weeknd appears twice but a set keeps one

The last block is the quietly brilliant part: the brackets on the outside pick the container, and swapping them costs you nothing. Square brackets build a list. Curly braces build a dict when the expression is a key: value pair, or a set when it is a lone value. And a set, being a set, silently drops repeats. That's exactly why pulling artist out of five songs hands you back four unique names, not five. Slide the gate below and watch a filtering comprehension rebuild its list in real time.

There is one property of every comprehension you have written so far that is easy to miss precisely because it is invisible: a comprehension is eager. The instant the line finishes running, the entire container exists in memory, every element computed and stored, all at once. For five songs that is nothing. But ask for [n*n for n in range(1_000_000)] and Python parks about 8 megabytes of finished list in RAM before you so much as glance at the first element. The round-bracket version from the curio weighs roughly 200 bytes, because it has built nothing. Hold on to that gap. It's what leads into the next section.

A comprehension of a million items builds all million before you touch the first — and throws the whole thing away if you only needed one. What if the sequence could stay a promise, handing you elements one at a time, only ever computing the next when you ask? →

04Generators: sequences that don't exist yet

Every collection you've met so far is already there. A list of song lengths is a million boxes filled and sitting in RAM before you read a single one. And a song's length is only its human label. Drop one altitude and Blinding Lights is its actual bytes: a run of roughly 8.8 million signed 16-bit samples, each one a speaker-cone position. A generator flips that on its head. It's a function that says yield where an ordinary function would say return, and that one word changes what "calling" even means. Call it and none of the body runs. Not the first line, not the loop, nothing. What you get back isn't a value. It's a generator object holding one thing: a suspended frame. A running function always has a frame — its private scratch space of local variables plus a pointer to the exact line it is on (chapter 9 stacked one such frame on every call). A generator simply freezes that frame instead of discarding it when the function pauses. The frozen locals, and a bookmark on the line to resume at, sit parked at the top and wait.

The values arrive one at a time, only when something asks. A for loop asks. next() asks. sum() asks. Each request wakes the body, runs it forward until it evaluates the next yield expression, hands over that single value, and then freezes on the spot, with every local variable still standing exactly where it was. Ask again and it thaws from that same line, not from the beginning. This is lazy evaluation: nothing is computed until the instant it's needed. The body advances by exactly one yield per knock. However many statements sit between two yields, one next() runs precisely that far and no further. The sequence doesn't exist in memory. It exists as a promise to produce the next item when you knock.

Remember the round brackets from the last section? (expr for x in xs) is a generator expression, a comprehension that builds nothing. Swap a list comprehension's square brackets for round ones and the memory story changes from brutal to free. A list of a million squares materialises a million boxed int objects plus a million-slot pointer array. On this machine that's ≈ 40 MB, every byte of it paid up front, before you so much as glance at the first square. The generator version is a single small object — exactly 200 bytes on this Python — and it stays 200 bytes whether it will eventually yield ten values or ten trillion. It stores the recipe, so its size has nothing to do with how much meal it can cook.

def song_lengths(songs):
    for song in songs:
        yield song["seconds"]

gen = song_lengths(playlist)   # nothing runs yet — you hold a paused recipe
print(next(gen))   # 200  — runs to the first yield, then freezes
print(next(gen))   # 245  — wakes exactly where it paused

squares_list = [x*x for x in range(1_000_000)]   # ~40 MB, built NOW
squares_gen  = (x*x for x in range(1_000_000))   # 200 bytes, builds nothing
print(sum(squares_gen))   # 333332833333500000 — one square at a time

To see the freeze-and-resume for yourself, drive it by hand. Below, each nudge of the slider is one next(gen) on our four-song playlist. Watch the bookmark walk down to the yield, hand you a value, and stop, with the locals still in place. Then it picks up from the very same line on the next call. Push it one step past the end and you'll meet the signal that ends every loop.

Now slide the count of items to a million and watch who pays. The list's cost climbs in a straight line. The generator's cost never moves off its 200-byte floor.

Lazy or eager, a playlist eventually wants to be in ORDER — and Python ships two ways to sort it that look almost identical yet behave in opposite ways. One returns a new list; the other quietly rewrites yours and hands back nothing. →

05sorted() vs .sort(): the method/function rule

There are two ways to put your playlist in order. They're spelled almost identically, and they do opposite things to your data. Learn the difference once and you've learned a rule that runs through the whole language, far past sorting: a dot-method reaches into the object and edits that very object, then hands you back None; a wrapping function reads the object, leaves it untouched, and hands you back a brand-new value. Say it out loud with the playlist in hand. playlist.sort() grabs the actual list and reorders the references right where they sit. It never builds a second list object, so the old order is gone for good. (This isn't literally zero extra memory: the sort borrows a small temporary buffer, up to about half the list, to merge through — but nothing the size of a whole second list.) sorted(playlist) does build that second list. It makes a fresh copy, arranges it, and gives it to you. Your original never so much as twitches.

There's a second split hiding in the punctuation. .sort() is a method: it lives on lists, and on nothing else. Hand it a tuple or a set and Python refuses: 'tuple' object has no attribute 'sort'. But sorted() is a free-standing function that will happily work through any iterable — a tuple, a set, a dict, even a one-shot generator that hasn't produced its values yet — and it always answers with a list. One is a specialist bolted to a single type. The other is a generalist that meets your data wherever it lives.

Both accept the same control, key=: you hand it a function reference, the name with no parentheses. Each item is then judged not by itself but by whatever that function returns for it — a stand-in, a proxy value. Want the playlist ordered by length? Pass key=lambda s: s["seconds"], and every song is compared by its number of seconds instead of by the whole dict. That's chapter 9's first-class functions cashing in: a function you can pass around like any other value. Add reverse=True and the same ordering runs longest-first. Miss the "reference, not call" part and you get an error: key=len() tries to run len right now, with no argument, and dies, while key=len passes the function along to be called later, once per item.

# the function: reads the playlist, builds a NEW list, original untouched
by_length = sorted(playlist, key=lambda s: s["seconds"])
print(by_length[0]["title"])    # Old Town Road — shortest first
print(playlist[0]["title"])     # Blinding Lights — the original never moved

# the method: reorders the actual list, hands back nothing
result = playlist.sort(key=lambda s: s["seconds"], reverse=True)
print(result)                   # None — .sort() returns None, always
print(playlist[0]["title"])     # One More Time — the list ITSELF was reordered

Reading about it is one thing; watching it happen under your hand is another. Below is one playlist of four songs, drawn as bars, where taller is longer. Slide through the four calls you might write. Watch two questions answer themselves at once: did the original playlist change? and what did the call hand back? Each run starts from the same fresh playlist, so you can compare them cleanly.

# key returns a PROXY: sort by whole minutes, not exact seconds
for s in sorted(playlist, key=lambda s: s["seconds"] // 60):
    print(s["title"], s["seconds"] // 60)
# Blinding Lights 3   ← both land in minute-bucket 3,
# Levels 3            ← and keep their ORIGINAL order (stable sort)
# Titanium 4

The mirror image lives one verb over. reversed(order) is a function: it leaves order untouched and hands back a new reversed iterator. order.reverse() is a method: it flips the list in place and returns None. That's exactly the same split as sorted versus .sort, which is the whole point. It was never a one-off rule about ordering. It's the shape of the language.

You just measured a sorted copy costing eight million bytes while the original sat right beside it. But where does that number come from — what does a list, an int, a whole object actually weigh, and how do you make it lighter? Next chapter opens the hood: weighing anything with getsizeof, slimming objects with __slots__, and caches that trade memory for speed. Memory, mastered →