Building Enumerables and Going Lazy

Article · 13:00

Enumerable is a module with about sixty methods, all built on one you supply. Write each, include the module, and your own type gains the entire vocabulary - then make the chain lazy so it never builds the intermediate arrays at all.

This lesson has 7 runnable examples. Press Try yourself! under any of them to open it in the editor.

Building Enumerables and Going Lazy

Enumerable is a module with about sixty methods, all built on one you supply. Write each, include the module, and your own type gains the entire vocabulary.

RB Example 1 25 lines
class Playlist
  include Enumerable

  def initialize(tracks) = @tracks = tracks

  # This is the only method Enumerable needs.
  def each(&block)
    @tracks.each(&block)
    self
  end
end

playlist = Playlist.new([
  { title: "Aria", seconds: 210 },
  { title: "Bloom", seconds: 185 },
  { title: "Cinder", seconds: 320 },
])

puts playlist.map { |t| t[:title] }.inspect
puts playlist.select { |t| t[:seconds] > 200 }.map { |t| t[:title] }.inspect
puts playlist.sum { |t| t[:seconds] }
puts playlist.min_by { |t| t[:seconds] }[:title]
puts playlist.sort_by { |t| -t[:seconds] }.first[:title]
puts playlist.each_slice(2).to_a.length
puts playlist.include?({ title: "Aria", seconds: 210 })

Open this example in the editor and change it. Nothing you type is saved.

Sixty methods for the price of one. Add <=> and include Comparable on the elements and sort, min and max work without a block as well.

Returning an Enumerator when no block is given

The convention every core method follows: with a block, do the work; without one, hand back an Enumerator so the caller can chain.

RB Example 2 23 lines
class Countdown
  include Enumerable

  def initialize(from) = @from = from

  def each
    # This one line is what makes countdown.each.with_index work.
    return to_enum(:each) { @from } unless block_given?

    @from.downto(1) { |n| yield n }
  end
end

countdown = Countdown.new(5)
countdown.each { |n| print n, " " }
puts

enumerator = countdown.each
puts enumerator.class
puts enumerator.size            # answered by the block passed to to_enum
puts enumerator.next
puts enumerator.next
puts countdown.each.with_index(1).map { |n, i| "#{i}:#{n}" }.inspect

Open this example in the editor and change it. Nothing you type is saved.

Enumerator, and generating values on demand

Enumerator.new takes a block that is handed a yielder. Nothing inside runs until a value is asked for, so an infinite sequence is perfectly safe.

RB Example 3 25 lines
naturals = Enumerator.new do |yielder|
  n = 1
  loop { yielder << (n += 1) - 1 }
end

puts naturals.take(5).inspect
puts naturals.first(3).inspect

fibonacci = Enumerator.new do |y|
  a, b = 0, 1
  loop do
    y << a
    a, b = b, a + b
  end
end

puts fibonacci.take(10).inspect
puts fibonacci.lazy.select(&:even?).first(5).inspect

# External iteration: next pulls one value at a time and remembers where it is.
pages = fibonacci.each_entry
puts pages.next
puts pages.next
puts pages.peek
puts pages.next

Open this example in the editor and change it. Nothing you type is saved.

Lazy, and why the order of the chain matters

A normal chain builds a full intermediate array at every step. lazy builds none: each element travels the whole chain before the next one starts.

RB Example 4 21 lines
# Without lazy this never finishes: map would try to build an infinite array.
result = (1..Float::INFINITY)
  .lazy
  .map { |n| n * n }
  .select { |n| n % 3 == 0 }
  .first(4)
puts result.inspect

# The saving is visible when you count the work.
seen = 0
answer = (1..1_000_000)
  .lazy
  .map { |n| seen += 1; n * 2 }
  .select { |n| n % 7 == 0 }
  .first(3)
puts answer.inspect
puts "examined #{seen} of a million"

eager_seen = 0
(1..1_000_000).map { |n| eager_seen += 1; n * 2 }.select { |n| n % 7 == 0 }.first(3)
puts "eager examined #{eager_seen}"

Open this example in the editor and change it. Nothing you type is saved.

force or first(n) ends a lazy chain. Until one of those is called, nothing has happened at all.

RB Example 5 3 lines
chain = (1..20).lazy.map { |n| puts "  computing #{n}"; n * 10 }.select(&:positive?)
puts "nothing has run yet"
puts chain.first(2).inspect

Open this example in the editor and change it. Nothing you type is saved.

Generators that free the memory

The real use is a source too big to hold: a file, a paginated API, a long query. Here is the shape, with an array standing in for the remote source.

RB Example 6 27 lines
PAGES = {
  1 => %w[alpha bravo charlie],
  2 => %w[delta echo foxtrot],
  3 => [],
}

def fetch_page(number)
  puts "  fetching page #{number}"
  PAGES.fetch(number, [])
end

def all_records
  Enumerator.new do |y|
    page = 1
    loop do
      rows = fetch_page(page)
      break if rows.empty?
      rows.each { |row| y << row }
      page += 1
    end
  end
end

# Only page one is ever fetched, because two matches were enough.
puts all_records.lazy.select { |r| r.length == 5 }.first(2).inspect
puts "---"
puts all_records.to_a.length

Open this example in the editor and change it. Nothing you type is saved.

each_with_object, and the reduce trap

RB Example 7 12 lines
words = %w[apple avocado banana blueberry cherry]

# reduce must return the accumulator every time. Forget that and you get nil.
puts words.reduce(Hash.new(0)) { |acc, w| acc[w[0]] += 1; acc }.inspect

# each_with_object hands the accumulator back for you, so it cannot be dropped.
puts words.each_with_object(Hash.new(0)) { |w, acc| acc[w[0]] += 1 }.inspect

# Note the argument order is reversed between the two. It is the most common
# reason one of them "does not work".
puts words.each_with_object([]) { |word, acc| acc << word.length }.inspect
puts words.reduce([]) { |acc, word| acc << word.length }.inspect

Open this example in the editor and change it. Nothing you type is saved.

What to take away

  • Define each, include Enumerable, and sixty methods arrive.
  • Return to_enum(:each) when no block was given, so callers can chain.
  • Enumerator.new produces values on demand; infinite sequences are fine.
  • lazy passes one element through the whole chain before starting the next.
  • A lazy chain does nothing until first or force.
  • each_with_object takes (element, accumulator); reduce takes the reverse.