Iterators and the Iterable Protocol

In the genetics laboratory of Jurassic Park, Dr. Henry Wu is reviewing enormous collections of dinosaur DNA data. "We need a way to process genetic sequences element by element, without loading the entire database into memory," he says. A for...of loop handles an array out of the box, but pass a plain park object with sectors through it and you get a TypeError saying the object is not iterable. Fortunately, JavaScript offers iterators and the iterable protocol - mechanisms that let you control how collections of data are traversed.

What Is an Iterator?

An iterator is an object that knows how to access the elements of a collection one at a time, tracking its current position. An iterator must implement a next() method that returns an object with two properties:

• value - the current value
• done - true if there are no more elements, false otherwise

1// Simple iterator - created manually
2function createDinoIterator(dinosaurs) {
3  let index = 0;
4
5  return {
6    next() {
7      if (index < dinosaurs.length) {
8        return { value: dinosaurs[index++], done: false };
9      }
10      return { value: undefined, done: true };
11    }
12  };
13}
14
15const raptorIterator = createDinoIterator(["Blue", "Charlie", "Delta", "Echo"]);
16
17console.log(raptorIterator.next()); // { value: "Blue", done: false }
18console.log(raptorIterator.next()); // { value: "Charlie", done: false }
19console.log(raptorIterator.next()); // { value: "Delta", done: false }
20console.log(raptorIterator.next()); // { value: "Echo", done: false }
21console.log(raptorIterator.next()); // { value: undefined, done: true }

The index variable lives in a closure and remembers the position between next() calls. After the last raptor the iterator reports done: true and keeps answering that way forever - it does not start over. The array it received has not changed one bit.

The Iterable Protocol (Symbol.iterator)

For an object to be usable in a for...of loop, it must implement the iterable protocol - meaning it must have a [Symbol.iterator]() method that returns an iterator. Many built-in types already implement this: Array, String, Map, Set.

1// Arrays are iterable
2const species = ["T-Rex", "Velociraptor", "Triceratops"];
3
4for (const dino of species) {
5  console.log(dino);
6}
7// T-Rex
8// Velociraptor
9// Triceratops
10
11// Strings are also iterable
12for (const char of "DINO") {
13  console.log(char); // D, I, N, O
14}
15
16// We can manually retrieve an iterator from an array
17const iterator = species[Symbol.iterator]();
18console.log(iterator.next()); // { value: "T-Rex", done: false }
19console.log(iterator.next()); // { value: "Velociraptor", done: false }

Behind the scenes, for...of does what you see in the last lines: it gets an iterator through Symbol.iterator and calls next() until it sees done: true. Symbol.iterator is a built-in symbol, a unique key you cannot confuse with an ordinary property name. An object literal has no such method, hence the error from the introduction.

Creating a Custom Iterable Object

We can make any object iterable - all we need to do is define a [Symbol.iterator]() method. Our park will walk through all its sectors, handing out one dinosaur at a time:

1// Jurassic Park as an iterable object
2const jurassicPark = {
3  name: "Jurassic Park",
4  sectors: [
5    { id: "A", dinosaurs: ["Rexy", "Blue"] },
6    { id: "B", dinosaurs: ["Spike", "Trike"] },
7    { id: "C", dinosaurs: ["Pteranodon", "Mosasaurus"] }
8  ],
9
10  // Implementation of the iterable protocol
11  [Symbol.iterator]() {
12    let sectorIndex = 0;
13    let dinoIndex = 0;
14    const sectors = this.sectors;
15
16    return {
17      next() {
18        // Traverse all dinosaurs in all sectors
19        while (sectorIndex < sectors.length) {
20          const sector = sectors[sectorIndex];
21          if (dinoIndex < sector.dinosaurs.length) {
22            const value = {
23              sector: sector.id,
24              dinosaur: sector.dinosaurs[dinoIndex]
25            };
26            dinoIndex++;
27            return { value, done: false };
28          }
29          sectorIndex++;
30          dinoIndex = 0;
31        }
32        return { value: undefined, done: true };
33      }
34    };
35  }
36};
37
38// Now we can iterate over the park!
39for (const entry of jurassicPark) {
40  console.log(`Sector ${entry.sector}: ${entry.dinosaur}`);
41}
42// Sector A: Rexy
43// Sector A: Blue
44// Sector B: Spike
45// Sector B: Trike
46// Sector C: Pteranodon
47// Sector C: Mosasaurus
48
49// Spread operator and destructuring also work
50const allDinos = [...jurassicPark];
51console.log(allDinos.length); // 6

The square brackets around the method name mean that the key is a symbol, not a string. The while loop in next() jumps to the next sector when the current one runs out of dinosaurs, so from the outside you see one flat list. The data in sectors stays untouched - the iterator only reads it.

Generators as Iterators

Writing next() by hand is tedious. Generators are special functions marked with an asterisk (function*) that simplify creating iterators. They use the yield keyword to "produce" successive values:

1// DNA sequence generator
2function* dnaSequenceGenerator(sequence) {
3  for (const nucleotide of sequence) {
4    yield nucleotide;
5  }
6}
7
8const dna = dnaSequenceGenerator("ATCGATCG");
9console.log(dna.next()); // { value: "A", done: false }
10console.log(dna.next()); // { value: "T", done: false }
11console.log(dna.next()); // { value: "C", done: false }
12
13// Generators are iterable - they work with for...of
14for (const nucleotide of dnaSequenceGenerator("GCTA")) {
15  console.log(nucleotide); // G, C, T, A
16}

Calling a generator does not run its body yet - it returns a generator object. Each next() runs the code up to the nearest yield and pauses there, remembering its state. The asterisk goes next to the word function, not the name - writing function myGen*() is a syntax error.

Generator with Logic

A generator can produce values endlessly, because it computes each one only when someone asks for it. This is how an ID-issuing machine works:

1// Dinosaur ID generator
2function* dinoIdGenerator(prefix, startFrom = 1) {
3  let id = startFrom;
4  while (true) {
5    yield `${prefix}-${String(id).padStart(3, "0")}`;
6    id++;
7  }
8}
9
10const raptorIds = dinoIdGenerator("RAPTOR");
11console.log(raptorIds.next().value); // "RAPTOR-001"
12console.log(raptorIds.next().value); // "RAPTOR-002"
13console.log(raptorIds.next().value); // "RAPTOR-003"
14
15// Infinite generator - but we only take what we need
16const trexIds = dinoIdGenerator("TREX", 100);
17const firstFive = [];
18for (let i = 0; i < 5; i++) {
19  firstFive.push(trexIds.next().value);
20}
21console.log(firstFive);
22// ["TREX-100", "TREX-101", "TREX-102", "TREX-103", "TREX-104"]

The while (true) loop does not freeze the program, because the generator works lazily: it stops at yield and waits for the next next(). Never expand an infinite generator with spread, though - such a call never ends. The padStart(3, "0") method pads the number with zeros to three digits.

Practical Example: Data Pagination

Iterators are great for paginating large datasets, because each page is created only on demand:

1// Page generator for dinosaur data
2function* paginateDinosaurs(dinosaurs, pageSize) {
3  for (let i = 0; i < dinosaurs.length; i += pageSize) {
4    yield {
5      page: Math.floor(i / pageSize) + 1,
6      data: dinosaurs.slice(i, i + pageSize),
7      hasMore: i + pageSize < dinosaurs.length
8    };
9  }
10}
11
12const allDinosaurs = [
13  "T-Rex", "Velociraptor", "Triceratops", "Stegosaurus",
14  "Brachiosaurus", "Pteranodon", "Mosasaurus", "Dilophosaurus",
15  "Gallimimus", "Parasaurolophus"
16];
17
18const pages = paginateDinosaurs(allDinosaurs, 3);
19
20console.log(pages.next().value);
21// { page: 1, data: ["T-Rex", "Velociraptor", "Triceratops"], hasMore: true }
22
23console.log(pages.next().value);
24// { page: 2, data: ["Stegosaurus", "Brachiosaurus", "Pteranodon"], hasMore: true }
25
26console.log(pages.next().value);
27// { page: 3, data: ["Mosasaurus", "Dilophosaurus", "Gallimimus"], hasMore: true }
28
29console.log(pages.next().value);
30// { page: 4, data: ["Parasaurolophus"], hasMore: false }

Each next() hands over a portion of three species and a hasMore flag that tells you whether to ask for another. The last page has a single element, because slice() never goes past the end of the array.

Iterable Object with a Generator

Generators simplify creating iterable objects. Instead of assembling an object with a next() method by hand, you just mark the [Symbol.iterator] method with an asterisk:

1class DinosaurEnclosure {
2  constructor(name) {
3    this.name = name;
4    this.dinosaurs = [];
5  }
6
7  add(dinosaur) {
8    this.dinosaurs.push(dinosaur);
9  }
10
11  // Generator as Symbol.iterator - much simpler!
12  *[Symbol.iterator]() {
13    for (const dino of this.dinosaurs) {
14      yield dino;
15    }
16  }
17}
18
19const paddock = new DinosaurEnclosure("Raptor Paddock");
20paddock.add({ name: "Blue", species: "Velociraptor" });
21paddock.add({ name: "Charlie", species: "Velociraptor" });
22paddock.add({ name: "Delta", species: "Velociraptor" });
23
24for (const raptor of paddock) {
25  console.log(`Raptor: ${raptor.name}`);
26}
27// Raptor: Blue
28// Raptor: Charlie
29// Raptor: Delta
30
31// Spread operator works automatically
32const names = [...paddock].map(r => r.name);
33console.log(names); // ["Blue", "Charlie", "Delta"]

Compare this with the park from the earlier section: a dozen lines with indexes there, three lines here. A generator object satisfies both protocols at once, so the class works with for...of and with spread straight away.

Destructuring and Spread

Spread and array destructuring use the iterable protocol too - both arrived in ES6 (ES2015), as did object destructuring:

1const raptors = ["Blue", "Charlie", "Delta", "Echo"];
2
3// Array destructuring - successive values from the iterator
4const [first, second, third] = raptors;
5console.log(first, second, third); // Blue Charlie Delta
6
7// Spread - merging arrays
8const newcomers = ["Fox", "Ghost"];
9const allRaptors = [...raptors, ...newcomers];
10console.log(allRaptors.length); // 6
11
12// Object destructuring and spread - no iterators involved
13const raptor = { name: "Blue", age: 4, isAlpha: true };
14const { name, age, isAlpha } = raptor;
15const olderBlue = { ...raptor, age: 5 };
16console.log(name, age, isAlpha, olderBlue.age); // Blue 4 true 5

Array destructuring pulls values one by one from an iterator, which is why it also works with our park or a generator. Objects are a different story: object destructuring reads properties by name, and object spread, added in ES2018, copies an object's own fields - neither of them uses Symbol.iterator. The copy is shallow, and the original raptor is still 4 years old.

Summary

Dr. Wu sums up: "Iterators are like specialized genetic probes - they let you scan data sequentially, step by step:"

1. Iterator - an object with a next() method returning { value, done }
2. Symbol.iterator - a method that makes an object iterable (works with for...of)
3. Generators (function* + yield) - simplify creating iterators
4. Iterable protocol - a contract that an object can be iterated, used by for...of, spread (...), and destructuring

In everyday code I recommend a generator over a hand-written next() - it is shorter and harder to get wrong. Spread returns later in this location in the Builder pattern, where the build() method hands back a copy of an object. In the lab below you will build a DinosaurEnclosure class with a generator as its Symbol.iterator.

Remember: an iterator is a probe that reads DNA samples one at a time, and Symbol.iterator is the socket every for...of loop plugs into.

1// Iterators and the iterable protocol
2// Jurassic Park - Genetics laboratory
3
4console.log("=== MANUAL ITERATOR ===");
5
6// Creating a simple iterator
7function createDinoIterator(dinosaurs) {
8  let index = 0;
9  return {
10    next() {
11      if (index < dinosaurs.length) {
12        return { value: dinosaurs[index++], done: false };
13      }
14      return { value: undefined, done: true };
15    }
16  };
17}
18
19const raptorIterator = createDinoIterator(["Blue", "Charlie", "Delta", "Echo"]);
20console.log(raptorIterator.next()); // { value: "Blue", done: false }
21console.log(raptorIterator.next()); // { value: "Charlie", done: false }
22console.log(raptorIterator.next()); // { value: "Delta", done: false }
23console.log(raptorIterator.next()); // { value: "Echo", done: false }
24console.log(raptorIterator.next()); // { value: undefined, done: true }
25
26console.log("\n=== SYMBOL.ITERATOR ===");
27
28// An iterable object with Symbol.iterator
29const jurassicPark = {
30  sectors: [
31    { id: "A", dinosaurs: ["Rexy", "Blue"] },
32    { id: "B", dinosaurs: ["Spike", "Trike"] },
33  ],
34
35  [Symbol.iterator]() {
36    let sectorIndex = 0;
37    let dinoIndex = 0;
38    const sectors = this.sectors;
39
40    return {
41      next() {
42        while (sectorIndex < sectors.length) {
43          const sector = sectors[sectorIndex];
44          if (dinoIndex < sector.dinosaurs.length) {
45            const value = {
46              sector: sector.id,
47              dinosaur: sector.dinosaurs[dinoIndex]
48            };
49            dinoIndex++;
50            return { value, done: false };
51          }
52          sectorIndex++;
53          dinoIndex = 0;
54        }
55        return { value: undefined, done: true };
56      }
57    };
58  }
59};
60
61// for...of works thanks to Symbol.iterator
62for (const entry of jurassicPark) {
63  console.log(`Sector ${entry.sector}: ${entry.dinosaur}`);
64}
65
66// The spread operator works too
67const allDinos = [...jurassicPark];
68console.log("All dinosaurs:", allDinos);
69
70console.log("\n=== GENERATORS ===");
71
72// Generator - a simpler way to create iterators
73function* dinoIdGenerator(prefix, startFrom = 1) {
74  let id = startFrom;
75  while (true) {
76    yield `${prefix}-${String(id).padStart(3, "0")}`;
77    id++;
78  }
79}
80
81const raptorIds = dinoIdGenerator("RAPTOR");
82console.log(raptorIds.next().value); // "RAPTOR-001"
83console.log(raptorIds.next().value); // "RAPTOR-002"
84console.log(raptorIds.next().value); // "RAPTOR-003"
85
86// Pagination generator
87function* paginateDinosaurs(dinosaurs, pageSize) {
88  for (let i = 0; i < dinosaurs.length; i += pageSize) {
89    yield {
90      page: Math.floor(i / pageSize) + 1,
91      data: dinosaurs.slice(i, i + pageSize),
92      hasMore: i + pageSize < dinosaurs.length
93    };
94  }
95}
96
97const allSpecies = ["T-Rex", "Velociraptor", "Triceratops", "Stegosaurus",
98  "Brachiosaurus", "Pteranodon", "Mosasaurus", "Dilophosaurus"];
99
100const pages = paginateDinosaurs(allSpecies, 3);
101console.log("\nPagination:");
102console.log(pages.next().value);
103console.log(pages.next().value);
104console.log(pages.next().value);
105
106// TODO: Create a class DinosaurEnclosure with a generator as Symbol.iterator
107// The class should have an add() method and be iterable
108console.log("\n=== Exercise ===");
109console.log("Create the DinosaurEnclosure class - see the TODO in the code");