Generators, Laziness and itertools
A generator produces values one at a time, on demand, keeping its own place between calls. That buys two things that a list cannot: constant memory over an arbitrarily long sequence, and the ability to represent something infinite.
A function that pauses
def countdown(start):
print("starting")
while start > 0:
yield start
start -= 1
print("finished")
gen = countdown(3)
print(type(gen)) # <class 'generator'> - nothing has run yet
print(next(gen)) # 'starting' then 3
print(next(gen)) # 2
print(next(gen)) # 1
try:
next(gen) # 'finished' then StopIteration
except StopIteration:
print("done")
Open this example in the editor and change it. Nothing you type is saved.
Calling the function does not run it. It builds a generator object. The body runs up to the first yield, hands out a value and freezes - local variables, the loop counter, the position in the code - until next() is called again.
That frozen state is what makes a generator different from a function returning a list. There is no list.
Laziness is the point
import sys
numbers = list(range(1000000))
generator = (n for n in range(1000000))
print(sys.getsizeof(numbers)) # about 8 megabytes
print(sys.getsizeof(generator)) # about 200 bytes
Open this example in the editor and change it. Nothing you type is saved.
A pipeline where nothing is materialised until the last step:
def read_lines(text):
for line in text.splitlines():
yield line
def non_empty(lines):
for line in lines:
if line.strip():
yield line
def parsed(lines):
for line in lines:
parts = line.split(",")
if len(parts) == 3:
yield {"name": parts[0], "team": parts[1], "score": int(parts[2])}
TEXT = """
aisha,red,91
noshad,blue,78
sam,red,64
bad line
leah,blue,55
"""
pipeline = parsed(non_empty(read_lines(TEXT)))
print(type(pipeline)) # a generator; no work done yet
for record in pipeline:
print(record["name"], record["score"])
Open this example in the editor and change it. Nothing you type is saved.
Each stage holds one line at a time. The same pipeline works unchanged over a four-gigabyte file, because at no point does anything hold more than one record.
Infinite sequences, made safe by laziness
from itertools import count, islice, takewhile
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a + b
print(list(islice(fibonacci(), 10)))
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
print(list(takewhile(lambda n: n < 100, fibonacci())))
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89]
print(next(n for n in fibonacci() if n > 1000)) # 1597
Open this example in the editor and change it. Nothing you type is saved.
fibonacci() never ends and the program terminates, because the consumer decides when to stop. That inversion of control is the thing generators give you that a list cannot.
Sending values in, and yield from
yield is an expression, so a generator can receive as well as produce:
def running_total():
total = 0
while True:
value = yield total
if value is None:
break
total += value
return total
acc = running_total()
print(next(acc)) # 0 - advance to the first yield
print(acc.send(10)) # 10
print(acc.send(5)) # 15
print(acc.send(100)) # 115
Open this example in the editor and change it. Nothing you type is saved.
The first next() is required: there is no suspended yield to receive a value until the generator has started.
yield from delegates to another iterable, flattening the plumbing away:
def chain_manually(*iterables):
for it in iterables:
for item in it:
yield item
def chain(*iterables):
for it in iterables:
yield from it
print(list(chain([1, 2], "ab", range(2)))) # [1, 2, 'a', 'b', 0, 1]
def flatten(nested):
for item in nested:
if isinstance(item, list):
yield from flatten(item) # recursion, readably
else:
yield item
print(list(flatten([1, [2, [3, [4, 5]], 6], 7]))) # [1, 2, 3, 4, 5, 6, 7]
Open this example in the editor and change it. Nothing you type is saved.
yield from also forwards send, throw and the return value, which the manual loop does not.
itertools
The standard library has the pieces you would otherwise write badly.
from itertools import (accumulate, chain, combinations, count, cycle,
groupby, islice, pairwise, product, repeat, tee)
print(list(islice(count(10, 5), 4))) # [10, 15, 20, 25]
print(list(islice(cycle("abc"), 7))) # ['a','b','c','a','b','c','a']
print(list(repeat("x", 3))) # ['x', 'x', 'x']
print(list(accumulate([1, 2, 3, 4]))) # [1, 3, 6, 10] - running sum
print(list(accumulate([1, 2, 3, 4], max))) # [1, 2, 3, 4]
print(list(pairwise([1, 2, 3, 4]))) # [(1,2), (2,3), (3,4)]
print(list(combinations("abc", 2))) # [('a','b'), ('a','c'), ('b','c')]
print(list(product([0, 1], repeat=2))) # [(0,0), (0,1), (1,0), (1,1)]
Open this example in the editor and change it. Nothing you type is saved.
pairwise is the sliding window of two, which is how you compute differences between consecutive readings without an index:
from itertools import pairwise
readings = [21.5, 22.1, 23.8, 22.0]
print([round(b - a, 2) for a, b in pairwise(readings)]) # [0.6, 1.7, -1.8]
Open this example in the editor and change it. Nothing you type is saved.
groupby groups consecutive equal keys, so sort first if you want all of them:
from itertools import groupby
rows = [("red", 1), ("blue", 2), ("red", 3), ("red", 4)]
print({k: [v for _, v in g] for k, g in groupby(rows, key=lambda r: r[0])})
# {'red': [3, 4], 'blue': [2]} - WRONG: red appears twice, as two
# separate consecutive runs, and the second overwrote the first
ordered = sorted(rows, key=lambda r: r[0])
print({k: [v for _, v in g] for k, g in groupby(ordered, key=lambda r: r[0])})
# {'blue': [2], 'red': [1, 3, 4]} - right
Open this example in the editor and change it. Nothing you type is saved.
That first result is the classic groupby bug. It is not a defect: groupby is designed to work over a stream it cannot sort, which is exactly why it only looks at neighbours.
tee splits one iterator into several, at the cost of buffering:
from itertools import tee
source = (n * n for n in range(5))
a, b = tee(source, 2)
print(list(a), list(b)) # [0,1,4,9,16] [0,1,4,9,16]
Open this example in the editor and change it. Nothing you type is saved.
Do not use the original after tee-ing it; tee consumes it on demand and the buffer is its business.
Cleaning up
A generator's finally runs when it is closed - whether by exhaustion, by close(), or by the consumer breaking out of the loop.
def tracked(items):
print("opening")
try:
for item in items:
yield item
finally:
print("closing")
for n in tracked([1, 2, 3, 4]):
print(n)
if n == 2:
break
# opening / 1 / 2 / closing
Open this example in the editor and change it. Nothing you type is saved.
That is what makes a generator safe for resources: a with open(...) around the loop inside the generator is released even if the consumer stops early.
The one case it does not cover is a generator that is never garbage collected - at interpreter shutdown, or held in a reference cycle. For anything holding a scarce resource, prefer a context manager and let the generator yield from inside it.
A worked example
from itertools import groupby, islice
RAW = """
2026-10-01,kitchen,21.5
2026-10-01,loft,17.2
2026-10-01,kitchen,22.1
2026-10-02,loft,16.9
2026-10-02,kitchen,23.8
bad,row
2026-10-02,kitchen,22.0
2026-10-03,loft,18.4
"""
def lines(text):
"""Stage one: strip and drop blanks. One line in memory at a time."""
for line in text.splitlines():
line = line.strip()
if line:
yield line
def records(source):
"""Stage two: parse, skipping anything malformed."""
for line in source:
parts = line.split(",")
if len(parts) != 3:
continue
try:
yield {"day": parts[0], "sensor": parts[1], "celsius": float(parts[2])}
except ValueError:
continue
def warmer_than(source, threshold):
"""Stage three: a filter, written as a generator so it composes."""
for record in source:
if record["celsius"] > threshold:
yield record
# Three stages, composed, and still nothing has been read.
pipeline = warmer_than(records(lines(RAW)), 17.0)
for record in pipeline:
print(f"{record['day']} {record['sensor']:<8} {record['celsius']:>5.1f}")
# Grouping needs a sort first, because groupby only looks at neighbours.
parsed = sorted(records(lines(RAW)), key=lambda r: (r["sensor"], r["day"]))
for sensor, group in groupby(parsed, key=lambda r: r["sensor"]):
temperatures = [r["celsius"] for r in group]
print(f"{sensor:<8} n={len(temperatures)} "
f"min={min(temperatures):.1f} max={max(temperatures):.1f} "
f"mean={sum(temperatures) / len(temperatures):.1f}")
# kitchen n=4 min=21.5 max=23.8 mean=22.4
# loft n=3 min=16.9 max=18.4 mean=17.5
def batched(source, size):
"""Chunking a stream, which is how you write to a database in
batches without ever holding the whole stream."""
iterator = iter(source)
while True:
batch = list(islice(iterator, size))
if not batch:
return
yield batch
for batch in batched(records(lines(RAW)), 2):
print([r["celsius"] for r in batch])
# [21.5, 17.2]
# [22.1, 16.9]
# [23.8, 22.0]
# [18.4]
Open this example in the editor and change it. Nothing you type is saved.
batched is worth keeping. It takes any iterable, including an infinite one, and yields fixed-size lists without ever materialising the source - which is exactly the shape of "insert 1,000 rows at a time from a stream of unknown length".
When not to use a generator
- The collection is small and used more than once. A list is simpler, and re-iterating works.
- You need
len(), indexing or sorting. None of those exist on a generator. - The consumer is the same function. If nothing else sees the sequence, a loop with an accumulator is clearer.
When it goes wrong
| Symptom | Cause |
|---|---|
| The second pass over a generator was empty | Generators are consumed once |
groupby produced duplicate keys |
It groups consecutive items; sort first |
TypeError: object of type generator has no len() |
Materialise with list(), or count by summing |
send() raised TypeError on the first call |
Advance with next() first |
| Cleanup never ran | The generator was neither exhausted nor closed |
| A pipeline did nothing | Nothing consumed it; a generator is inert until iterated |
| Memory grew anyway | A list() somewhere in the middle of the pipeline |
tee used more memory than the list would have |
The consumers are far apart; just build the list |
A check you can run
def noisy(n):
for i in range(n):
print("producing", i)
yield i
print("built")
gen = noisy(3)
print("about to loop")
for value in gen:
print("got", value)
Open this example in the editor and change it. Nothing you type is saved.
Nothing is produced between "built" and "about to loop" - and then production and consumption interleave, one at a time, rather than all the producing happening first. Watching that alternation once is what makes laziness stop being an abstract word.