Generators and Iterators
A generator is a function that produces values one at a time, keeping its place between them. It is how you process a file larger than memory, represent an infinite sequence, or build a pipeline that does no work until someone asks.
A function that pauses
<?php
declare(strict_types=1);
function countdown(int $from): Generator
{
echo 'starting', PHP_EOL;
while ($from > 0) {
yield $from;
$from--;
}
echo 'finished', PHP_EOL;
return 'done'; // the return value, not a yield
}
$generator = countdown(3);
echo get_class($generator), PHP_EOL; // Generator - nothing has run yet
foreach ($generator as $value) {
echo $value, PHP_EOL;
}
echo $generator->getReturn(), PHP_EOL; // done
Open this example in the editor and change it. Nothing you type is saved.
Calling the function does not run the body. It builds a Generator. The body runs up to the first yield, hands out a value and freezes - local variables, the loop counter, the position - until the next value is asked for.
Any function containing yield is a generator function, even if the yield is unreachable. That is occasionally surprising and always worth checking when a function mysteriously returns a Generator.
Laziness is the point
<?php
declare(strict_types=1);
function eager(int $n): array
{
$out = [];
for ($i = 0; $i < $n; $i++) {
$out[] = $i * $i;
}
return $out;
}
function lazy(int $n): Generator
{
for ($i = 0; $i < $n; $i++) {
yield $i * $i;
}
}
$before = memory_get_usage();
$array = eager(200000);
$arrayCost = memory_get_usage() - $before;
$before = memory_get_usage();
$generator = lazy(200000);
$generatorCost = memory_get_usage() - $before;
printf("array: %s bytes%s", number_format($arrayCost), PHP_EOL);
printf("generator: %s bytes%s", number_format($generatorCost), PHP_EOL);
var_dump($generatorCost < $arrayCost / 100);
Open this example in the editor and change it. Nothing you type is saved.
The array holds two hundred thousand integers. The generator holds a position. For a database cursor, a log file or an API that pages, that difference is the difference between working and running out of memory.
A pipeline where each stage holds one item:
<?php
declare(strict_types=1);
function lines(string $text): Generator
{
foreach (explode(PHP_EOL, $text) as $line) {
$line = trim($line);
if ($line !== '') {
yield $line;
}
}
}
function parsed(iterable $source): Generator
{
foreach ($source as $line) {
$parts = array_map('trim', explode(',', $line));
if (count($parts) === 3) {
yield ['name' => $parts[0], 'team' => $parts[1], 'score' => (int) $parts[2]];
}
}
}
function above(iterable $source, int $threshold): Generator
{
foreach ($source as $record) {
if ($record['score'] > $threshold) {
yield $record;
}
}
}
$text = <<<'TEXT'
aisha,red,91
noshad,blue,78
sam,red,64
not a row
leah,blue,55
TEXT;
$pipeline = above(parsed(lines($text)), 70);
foreach ($pipeline as $record) {
printf("%-8s %-5s %d%s", $record['name'], $record['team'], $record['score'], PHP_EOL);
}
Open this example in the editor and change it. Nothing you type is saved.
Note the iterable type on the parameters. It accepts an array or any Traversable, which is what lets these compose without caring whether the source is a generator, a file handle wrapper or a plain array.
Infinite sequences, made safe by laziness
<?php
declare(strict_types=1);
function naturals(): Generator
{
$n = 1;
while (true) {
yield $n++;
}
}
function take(iterable $source, int $count): Generator
{
foreach ($source as $value) {
if ($count-- <= 0) {
return;
}
yield $value;
}
}
function mapped(iterable $source, callable $fn): Generator
{
foreach ($source as $value) {
yield $fn($value);
}
}
$squares = take(mapped(naturals(), fn (int $n): int => $n * $n), 5);
echo implode(', ', iterator_to_array($squares)), PHP_EOL; // 1, 4, 9, 16, 25
Open this example in the editor and change it. Nothing you type is saved.
naturals() never ends and the program terminates, because the consumer stops asking. That inversion - the consumer deciding how much work the producer does - is the thing a returned array cannot give you.
Keys, sending and delegation
A generator can yield keys as well as values, and the keys need not be integers:
<?php
declare(strict_types=1);
function settings(): Generator
{
yield 'host' => 'localhost';
yield 'port' => 3306;
yield 'tls' => false;
}
foreach (settings() as $key => $value) {
printf("%-5s %s%s", $key, var_export($value, true), PHP_EOL);
}
print_r(iterator_to_array(settings()));
Open this example in the editor and change it. Nothing you type is saved.
yield is also an expression, so a generator can receive values through send():
<?php
declare(strict_types=1);
function runningTotal(): Generator
{
$total = 0;
while (true) {
$value = yield $total;
if ($value === null) {
return $total;
}
$total += $value;
}
}
$accumulator = runningTotal();
echo $accumulator->current(), PHP_EOL; // 0 - run to the first yield
echo $accumulator->send(10), PHP_EOL; // 10
echo $accumulator->send(5), PHP_EOL; // 15
echo $accumulator->send(100), PHP_EOL; // 115
Open this example in the editor and change it. Nothing you type is saved.
current() advances to the first yield; the first send() would otherwise skip a value.
yield from delegates to another iterable, which flattens the plumbing away and preserves keys:
<?php
declare(strict_types=1);
function chained(iterable ...$sources): Generator
{
foreach ($sources as $source) {
yield from $source;
}
}
function flatten(array $nested): Generator
{
foreach ($nested as $item) {
if (is_array($item)) {
yield from flatten($item); // recursion, readably
} else {
yield $item;
}
}
}
echo implode(',', iterator_to_array(chained([1, 2], [3, 4]), false)), PHP_EOL;
echo implode(',', iterator_to_array(flatten([1, [2, [3, [4, 5]], 6], 7]), false)), PHP_EOL;
Open this example in the editor and change it. Nothing you type is saved.
The false second argument to iterator_to_array discards the keys. Without it, yield from preserves the inner sequences' keys and later ones overwrite earlier ones - which is the commonest yield from bug and produces a result shorter than the input.
Implementing the interfaces directly
Sometimes you want an object that can be iterated more than once, which a generator cannot do.
<?php
declare(strict_types=1);
final class Range implements IteratorAggregate, Countable
{
public function __construct(
private readonly int $from,
private readonly int $to,
private readonly int $step = 1,
) {}
// A FRESH generator each time, so this object re-iterates.
public function getIterator(): Generator
{
for ($n = $this->from; $n <= $this->to; $n += $this->step) {
yield $n;
}
}
public function count(): int
{
return intdiv($this->to - $this->from, $this->step) + 1;
}
}
$range = new Range(1, 9, 2);
echo implode(',', iterator_to_array($range)), PHP_EOL; // 1,3,5,7,9
echo implode(',', iterator_to_array($range)), PHP_EOL; // again - works
echo count($range), PHP_EOL; // 5
$generator = (function (): Generator { yield 1; yield 2; })();
echo implode(',', iterator_to_array($generator)), PHP_EOL; // 1,2
try {
iterator_to_array($generator); // a second pass
} catch (Exception $e) {
echo $e->getMessage(), PHP_EOL; // Cannot traverse an already closed generator
}
Open this example in the editor and change it. Nothing you type is saved.
IteratorAggregate needs one method and is almost always the right choice. The full Iterator interface - current, key, next, rewind, valid - is five methods of bookkeeping you rarely need to write by hand.
Cleaning up
A generator's finally runs when it is exhausted, closed, or garbage collected after an early break:
<?php
declare(strict_types=1);
function tracked(array $items): Generator
{
echo 'opening', PHP_EOL;
try {
foreach ($items as $item) {
yield $item;
}
} finally {
echo 'closing', PHP_EOL;
}
}
foreach (tracked([1, 2, 3, 4]) as $value) {
echo $value, PHP_EOL;
if ($value === 2) {
break;
}
}
echo 'after the loop', PHP_EOL;
Open this example in the editor and change it. Nothing you type is saved.
That is what makes generators safe for resources: a fopen before the loop and an fclose in the finally is released even when the consumer stops early.
A worked example
<?php
declare(strict_types=1);
/**
* Reading a large delimited source in batches, without ever holding
* more than one batch in memory - the shape of every "import a file
* into the database" job.
*/
function sourceLines(string $text): Generator
{
$handle = fopen('php://memory', 'r+');
try {
fwrite($handle, $text);
rewind($handle);
while (($line = fgets($handle)) !== false) {
$line = rtrim($line);
if ($line !== '') {
yield $line;
}
}
} finally {
fclose($handle);
echo '[handle closed]', PHP_EOL;
}
}
function records(iterable $lines): Generator
{
$lineNumber = 0;
foreach ($lines as $line) {
$lineNumber++;
$parts = array_map('trim', explode('|', $line));
if (count($parts) !== 3) {
continue;
}
[$sku, $name, $pence] = $parts;
if (!ctype_digit($pence)) {
continue;
}
yield $lineNumber => ['sku' => $sku, 'name' => $name, 'pence' => (int) $pence];
}
}
function batched(iterable $source, int $size): Generator
{
$batch = [];
foreach ($source as $item) {
$batch[] = $item;
if (count($batch) === $size) {
yield $batch;
$batch = [];
}
}
if ($batch !== []) {
yield $batch;
}
}
$catalogue = <<<'TEXT'
CSS-01 | Selectors and the cascade | 0
CSS-02 | The box model | 0
BAD ROW
CSS-03 | Colour, units and type | 0
JS-01 | Values and types | 1999
JS-02 | Closures | 2499
PY-01 | Values and names | not a number
PY-02 | Lists and tuples | 1299
TEXT;
$inserted = 0;
$batches = 0;
foreach (batched(records(sourceLines($catalogue)), 3) as $batch) {
$batches++;
$skus = array_column($batch, 'sku');
printf("batch %d: %d row(s) - %s%s", $batches, count($batch), implode(', ', $skus), PHP_EOL);
$inserted += count($batch);
}
printf("%s%d row(s) in %d batch(es)%s", PHP_EOL, $inserted, $batches, PHP_EOL);
// Early exit: the finally still runs, so the handle is closed even
// though the consumer stopped after the first batch.
echo PHP_EOL, 'stopping early:', PHP_EOL;
foreach (batched(records(sourceLines($catalogue)), 3) as $batch) {
printf(" first batch has %d row(s)%s", count($batch), PHP_EOL);
break;
}
// And keys survive the pipeline, so an error can name the line.
echo PHP_EOL, 'line numbers of the paid items:', PHP_EOL;
foreach (records(sourceLines($catalogue)) as $lineNumber => $record) {
if ($record['pence'] > 0) {
printf(" line %d: %s at %s%s",
$lineNumber, $record['sku'],
number_format($record['pence'] / 100, 2), PHP_EOL);
}
}
Open this example in the editor and change it. Nothing you type is saved.
batched is the piece worth keeping. It takes any iterable, including an infinite one, and yields fixed-size arrays without materialising the source - which is exactly "insert a thousand rows at a time from a stream of unknown length". And because records yields keys, an error message can name the line it came from, which an array of values could not.
When not to use one
- The collection is small and used twice. A generator is consumed once; a second
foreachsees nothing. - You need
count(), sorting or random access. None of those exist on a generator; build the array. - The consumer is the same function. If nothing else sees the sequence, a loop with an accumulator is clearer.
When it goes wrong
| Symptom | Cause |
|---|---|
| The second loop over a generator was empty | Generators are consumed once |
iterator_to_array lost rows |
Duplicate keys; pass false as the second argument |
count() on a generator is a fatal error |
No Countable; use iterator_count or an array |
A function unexpectedly returned a Generator |
Any yield in the body makes it one |
The first send() skipped a value |
Call current() first |
| A file handle stayed open | Nothing closed the generator; put it in finally |
| A pipeline did nothing | Nothing consumed it; a generator is inert |
| Memory grew anyway | An iterator_to_array in the middle of the pipeline |
A check you can run
<?php
function noisy(int $n): Generator {
for ($i = 0; $i < $n; $i++) {
echo "producing $i", PHP_EOL;
yield $i;
}
}
echo 'built', PHP_EOL;
$gen = noisy(3);
echo 'about to loop', PHP_EOL;
foreach ($gen as $value) {
echo "got $value", PHP_EOL;
}
Open this example in the editor and change it. Nothing you type is saved.
Nothing is produced between "built" and "about to loop", and then production and consumption alternate one at a time rather than all the producing happening first.
Watching that interleaving once is what makes laziness stop being an abstract word, and it is the moment the memory numbers above stop being surprising.