Producing Values One at a Time

for item in collection: looks like the language just knows how to step through anything you hand it. It does not — it is calling two methods you have never had to type yourself, and every object that works in a for loop has agreed to support them the same way.

$ >>> numbers = [10, 20, 30]$ >>> iterator = iter(numbers)$ >>> next(iterator)$ 10$ >>> next(iterator)$ 20$ >>> next(iterator)$ 30$ >>> next(iterator)$ Traceback (most recent call last):  ...$ StopIteration

iter(numbers) asks the list for a fresh iterator — an object that remembers position and knows how to produce the next value. next(iterator) asks it for exactly one value, and the fourth call, with nothing left, raises StopIteration. A for loop is this same pair of calls, written by the language for you: it calls iter() once, calls next() repeatedly, and quietly catches StopIteration as its signal to stop rather than let it crash the program.

[n for n in range(1_000_000)] computes all one million values immediately and holds every one in memory before the line even finishes. If the program only ever needed the first three, the other 999,997 were wasted work.

$ >>> import sys$ >>> sys.getsizeof([n for n in range(1_000_000)])$ 8448728$ >>> sys.getsizeof((n for n in range(1_000_000)))$ 112

That is not a rounding difference — it is the difference between a container that already holds a million pointers and an object that holds only the instruction for how to make the next number, plus a note on where it stopped. The list has to exist in full before the line finishes. The generator expression on the right never builds anything until something asks it to.

$ >>> def fibonacci_below(limit):$ ...     a, b = 1, 1$ ...     while a < limit:$ ...         yield a$ ...         a, b = b, a + b$ ...$ >>> gen = fibonacci_below(20)$ >>> gen$ <generator object fibonacci_below at 0x1046a3...>

Calling fibonacci_below(20) runs none of the function's body. It returns a generator — an object that remembers where to resume, but has not computed a single value yet.

$ >>> next(gen)$ 1$ >>> next(gen)$ 1$ >>> next(gen)$ 2

Every next(gen) call resumes the function exactly where yield last paused it, runs until the next yield, and hands back that one value. return would end the function for good — yield only pauses it.

Everything local to the function survives the pause. a and b keep their exact values between calls, the same way they would if the function had never stopped running — the only thing that actually stopped is your access to the rest of the body, until you ask for more.

A generator that never finishes, like one counting upward forever, would be impossible as a list — there would be no last value to stop at. As a generator it works fine, because nothing is computed until something actually asks next() for it.

An open file is an iterator, the same as the list example above — for line in f: calls next() on it once per line, and each call reads only as far as the next newline character before handing that one line back.

def count_errors(path):    total = 0    with open(path) as f:        for line in f:            if "ERROR" in line:                total += 1    return total

This runs in the same, small, constant amount of memory whether path points at a ten-line log or a ten-gigabyte one, because at any moment only one line is ever actually in memory — the one just read, being checked, about to be discarded. The counterexample is open(path).readlines(), which reads the entire file into a list of lines before your code sees a single one of them, and on a ten-gigabyte file tries to hold roughly ten gigabytes in RAM to do it.

Swap the square brackets of a list comprehension for round parentheses and you get a generator expression — the same filtering and transforming syntax from the comprehensions chapter, but producing values lazily instead of building the whole list up front.

sum(n * n for n in range(1_000_000)) never needs the intermediate list at all — the parentheses can even be dropped when the generator expression is the sole argument to a function call, which is why you will see it written with no extra punctuation around it.

$ >>> squares = (n * n for n in range(5))$ >>> list(squares)$ [0, 1, 4, 9, 16]$ >>> list(squares)$ []

A generator does not reset once it is exhausted. The second list(squares) does not error and does not repeat the sequence — it simply has nothing left to give, because every value was already pulled out and thrown away the first time.