Comprehensions and the Iterator Protocol
A comprehension builds a collection from an iterable in one expression. The iterator protocol is what makes for work on anything at all. They are the same subject, because a comprehension is a for loop with the bookkeeping removed.
Build, filter, and transform
numbers = [1, 2, 3, 4, 5, 6]
print([n * n for n in numbers]) # [1, 4, 9, 16, 25, 36]
print([n for n in numbers if n % 2 == 0]) # [2, 4, 6]
print([n * n for n in numbers if n % 2 == 0]) # [4, 16, 36]
print([n if n % 2 else 0 for n in numbers]) # [1, 0, 3, 0, 5, 0]
Open this example in the editor and change it. Nothing you type is saved.
Note where the conditions go. A filter (if with no else) goes at the end, after the for. A conditional expression (if/else) goes at the front, because it is part of the thing being built rather than a decision about whether to build it. Getting these the wrong way round is a syntax error, which is merciful.
The same syntax builds dicts and sets:
words = ["apple", "banana", "cherry"]
print({w: len(w) for w in words}) # {'apple': 5, 'banana': 6, 'cherry': 6}
print({len(w) for w in words}) # {5, 6}
print({w[0] for w in words}) # {'a', 'b', 'c'}, in some order
Open this example in the editor and change it. Nothing you type is saved.
And a generator expression, which looks the same with round brackets and builds nothing until asked:
squares = (n * n for n in range(1000000))
print(type(squares)) # <class 'generator'>
print(sum(squares)) # computed one at a time
Open this example in the editor and change it. Nothing you type is saved.
When a generator expression is the only argument to a function, the brackets can be dropped: sum(n * n for n in numbers).
Two for clauses nest, outer first - the same order you would write the loops in:
pairs = [(x, y) for x in "ab" for y in [1, 2]]
print(pairs) # [('a', 1), ('a', 2), ('b', 1), ('b', 2)]
grid = [[0] * 3 for _ in range(2)] # a new row each time
print(grid) # [[0, 0, 0], [0, 0, 0]]
matrix = [[1, 2], [3, 4], [5, 6]]
print([n for row in matrix for n in row]) # [1, 2, 3, 4, 5, 6] - flattened
print([list(r) for r in zip(*matrix)]) # [[1, 3, 5], [2, 4, 6]] - transposed
Open this example in the editor and change it. Nothing you type is saved.
The flatten reads left to right exactly like the nested loop it replaces: for row in matrix then for n in row. The common mistake is writing the clauses the other way round, which raises NameError because row has not been bound yet.
When not to use one
A comprehension is for building a collection from an iterable. It is not a general-purpose loop.
numbers = [1, 2, 3]
# No. A comprehension built purely for side effects, discarding the list.
# [print(n) for n in numbers]
# Yes.
for n in numbers:
print(n)
Open this example in the editor and change it. Nothing you type is saved.
And once it needs two conditions, a transformation and a nested loop, it has stopped being readable:
# Hard to read.
result = [transform(x) for row in data for x in row if x > 0 if x % 2 == 0]
# Easier, and no slower in any way that matters.
result = []
for row in data:
for x in row:
if x > 0 and x % 2 == 0:
result.append(transform(x))
Open this example in the editor and change it. Nothing you type is saved.
The rule of thumb: if it does not fit on one line at a comfortable width, or if you had to read it twice, write the loop.
What iteration actually is
for x in thing does three things: it calls iter(thing) to get an iterator, it calls next() on that iterator repeatedly, and it stops when next() raises StopIteration.
numbers = [10, 20, 30]
iterator = iter(numbers)
print(next(iterator)) # 10
print(next(iterator)) # 20
print(next(iterator)) # 30
try:
next(iterator)
except StopIteration:
print("exhausted")
print(next(iter(numbers), "default")) # 10 - a default instead of raising
print(next(iter([]), "default")) # 'default'
Open this example in the editor and change it. Nothing you type is saved.
Implementing the protocol makes your own type work everywhere a list does:
class Countdown:
def __init__(self, start):
self.start = start
def __iter__(self):
# Returning a fresh generator each time means this object can be
# iterated more than once - which an iterator cannot be.
current = self.start
while current > 0:
yield current
current -= 1
c = Countdown(4)
print(list(c)) # [4, 3, 2, 1]
print(list(c)) # [4, 3, 2, 1] - again, because __iter__
# returns a NEW generator
print(max(c), sum(c)) # 4 10
print(3 in c) # True - `in` falls back to iteration
a, b, *rest = c
print(a, b, rest) # 4 3 [2, 1]
Open this example in the editor and change it. Nothing you type is saved.
One __iter__ method and the object works with list, max, sum, in, unpacking, sorted, zip, and every comprehension.
An iterator is consumed once
numbers = [1, 2, 3]
squares = (n * n for n in numbers)
print(list(squares)) # [1, 4, 9]
print(list(squares)) # [] - already exhausted
print(sum(squares)) # 0
Open this example in the editor and change it. Nothing you type is saved.
That empty list on the second pass is silent, which is what makes it dangerous. zip, map, filter, enumerate, reversed and every generator behave the same way, for the same reason:
pairs = zip([1, 2], "ab")
print(list(pairs)) # [(1, 'a'), (2, 'b')]
print(list(pairs)) # []
Open this example in the editor and change it. Nothing you type is saved.
If you need the values twice, materialise them once with list().
Iterating while mutating
numbers = [1, 2, 3, 4, 5, 6]
for n in numbers:
if n % 2 == 0:
numbers.remove(n) # the iterator's position does not shift back
print(numbers) # [1, 3, 5] here, but it skipped elements
Open this example in the editor and change it. Nothing you type is saved.
That one happens to come out right; with a different pattern it does not:
numbers = [1, 2, 2, 3]
for n in numbers:
if n == 2:
numbers.remove(n)
print(numbers) # [1, 2, 3] - one 2 survived
Open this example in the editor and change it. Nothing you type is saved.
Build a new one instead:
numbers = [1, 2, 2, 3]
numbers = [n for n in numbers if n != 2]
print(numbers) # [1, 3]
scores = {"a": 1, "b": 0, "c": 2}
scores = {k: v for k, v in scores.items() if v}
print(scores) # {'a': 1, 'c': 2}
Open this example in the editor and change it. Nothing you type is saved.
Mutating a dict or set during iteration is worse: it raises RuntimeError: dictionary changed size during iteration rather than silently skipping. That is an improvement.
A worked example
LOG = """
2026-10-01 12:00:01 GET /courses 200 0.031
2026-10-01 12:00:02 GET /courses/css 200 0.112
2026-10-01 12:00:04 POST /enrol 500 1.902
2026-10-01 12:00:05 GET /about 200 0.019
malformed line
2026-10-01 12:00:09 GET /courses/js 404 0.008
2026-10-01 12:00:11 POST /enrol 500 2.455
2026-10-01 12:00:14 GET /courses 200 0.027
"""
def parse(line):
parts = line.split()
if len(parts) != 6:
return None
date, time, method, path, status, seconds = parts
return {
"at": date + " " + time,
"method": method,
"path": path,
"status": int(status),
"seconds": float(seconds),
}
# One generator expression feeding another: nothing is materialised
# until the list() at the end, and the malformed line is dropped once.
parsed = (parse(line) for line in LOG.strip().splitlines())
requests = [r for r in parsed if r is not None]
print(f"{len(requests)} of {len(LOG.strip().splitlines())} lines parsed") # 7 of 8
# Filtering and transforming, each a single readable clause.
errors = [r for r in requests if r["status"] >= 500]
print(f"{len(errors)} server errors") # 2
slow = {r["path"] for r in requests if r["seconds"] > 0.1}
print("slow paths:", sorted(slow)) # ['/courses/css', '/enrol']
# A dict comprehension, with the worst time per path.
worst = {}
for r in requests:
worst[r["path"]] = max(worst.get(r["path"], 0), r["seconds"])
print({k: v for k, v in sorted(worst.items()) if v > 0.05})
# {'/courses/css': 0.112, '/enrol': 2.455}
# Aggregates over a generator, which never builds an intermediate list.
total = sum(r["seconds"] for r in requests)
print(f"total {total:.3f}s, mean {total / len(requests):.3f}s")
print("any 404s?", any(r["status"] == 404 for r in requests)) # True
print("all under 3s?", all(r["seconds"] < 3 for r in requests)) # True
# any() and all() short-circuit, so this stops at the first match
# rather than testing all seven.
first_error = next((r for r in requests if r["status"] >= 500), None)
print("first error at", first_error["at"]) # 2026-10-01 12:00:04
# And the trap: `parsed` was a generator, so it is now empty.
print(list(parsed)) # []
Open this example in the editor and change it. Nothing you type is saved.
The last line is the lesson repeated as a punchline. parsed was consumed by the list comprehension on line two of the analysis, and every later use of it would silently see nothing.
When it goes wrong
| Symptom | Cause |
|---|---|
| The second loop over a generator saw nothing | Iterators are consumed once |
NameError in a nested comprehension |
The for clauses are in the wrong order |
A comprehension returned a list of None |
It was built for side effects; use a loop |
| Items were skipped while removing in a loop | Mutating a list while iterating it |
RuntimeError: dictionary changed size |
Mutating a dict while iterating it |
SyntaxError on if/else in a comprehension |
Conditional expressions go at the front |
| A huge comprehension exhausted memory | Use a generator expression and aggregate |
len() failed on a generator |
Generators have no length; sum(1 for _ in it) |
A check you can run
rows = [[1, 2], [3, 4]]
doubled = ([n * 2 for n in row] for row in rows)
print(list(doubled))
print(list(doubled))
doubled = [[n * 2 for n in row] for row in rows]
print(list(doubled))
print(list(doubled))
Open this example in the editor and change it. Nothing you type is saved.
The only difference is round brackets versus square. The first pair prints the data and then an empty list; the second prints it twice. One character, and the difference between a pipeline that streams and a result you can use more than once - which is the whole trade that generators offer.