skip to content

The Iterator Protocol

13 roadmaps10 questions

The contract under every for-loop and comprehension: __iter__ hands back an iterator, __next__ produces values, StopIteration ends the loop. 'Iterable vs iterator' is the standard probe.

part ofPythonoverview, primer and where to startread it →
on this pageshow

explore

questions

10

How do you make a custom Python class iterable in a `for` loop?

level: juniorimportance: must knowfreq 55%

answer

  1. One method is all a for loop needs
  2. The method must hand back a cursor
  3. yield inside the method does the work
  4. A generator is already an iterator
  5. Returning a list raises TypeError

basics

~20 s

Give the class an iter method. The simplest implementation is to write iter as a generator function that yields each element: Python then gets a fresh iterator on every loop, and you never write next or raise StopIteration yourself.

solid answer

~50 s

A `for` statement calls `iter(obj)` once and then `__next__()` on the result until `StopIteration`, so the only method your class must provide is `__iter__`, and its one obligation is to **return an iterator** -- not a list, not the underlying collection. The idiomatic implementation is a generator function: put `yield` in the body of `__iter__` and calling it hands back a brand-new generator object, which is already a complete iterator. Two cheaper variants exist: `return iter(self._items)` when the class just wraps one stored collection, and a separate iterator class when the cursor needs its own identity. All three create a fresh iterator per call, which is what lets the object be looped twice, nested inside itself, and passed to `sorted`, `sum`, `list` or a comprehension. Returning a list from `__iter__` fails immediately with `TypeError: iter() returned non-iterator of type 'list'`.

code

python · 13 lines
python
class Deck:
    def __init__(self, cards):
        self._cards = list(cards)

    def __iter__(self):
        for card in self._cards:
            yield card


d = Deck(["A", "K", "Q"])
print([c for c in d])
print([c for c in d])
print(sorted(d))

go deeper

for a junior

Be ready to write the four-line class from memory: __iter__ with a yield in it, nothing else. Know that the method must return an iterator and that a generator already is one.

for a middle

Explain the mechanics: for calls iter() once, __next__() until StopIteration, and each __iter__() call must hand back a new cursor. Name the TypeError you get when it returns a list.

for a senior

Show judgement about which of the three implementations fits: generator for traversal, iter(self._items) for a thin wrapper, a named iterator class when the cursor is passed around. Mention lazy body execution and resource cleanup inside the generator.

for a principal

Own the API contract this sets for callers: whether your type is documented as re-iterable, whether it should inherit a collections.abc base so the guarantee is checkable, and what it costs downstream when a supposedly re-iterable type quietly becomes single-pass.

A `for` statement in Python is defined entirely by two builtin calls. `for card in deck:` evaluates `iter(deck)` exactly once, then calls `__next__()` on whatever came back, over and over, until `StopIteration` is raised. Everything your own class has to supply sits in the first half of that sentence: give it an `__iter__` method and the loop machinery does the rest. ### The one rule `__iter__` must obey It must return an **iterator**: an object with a `__next__` method and an `__iter__` method that returns itself. It may not return a list, a tuple, or the underlying collection -- those are iterables, not iterators. Write `def __iter__(self): return self._items` where `_items` is a list and the first loop over your object dies with `TypeError: iter() returned non-iterator of type 'list'`. That message is the most common first bug when hand-rolling an iterable, and recognising it on sight is worth as much in an interview as reciting the protocol. ### Three implementations, in the order you should reach for them **1. `__iter__` as a generator function.** Put `yield` in the body. Calling `__iter__()` now returns a brand-new generator object, and a generator is already a fully formed iterator: CPython supplies `__next__`, supplies `__iter__` returning self, and raises `StopIteration` when the function body falls off the end. You write the traversal as ordinary straight-line code -- loops, recursion, conditionals, early `return` -- and never touch `__next__` or `StopIteration` by hand. This is the default answer. **2. Delegate with `return iter(self._items)`.** When the class is a thin wrapper over one stored collection this is a one-liner, and it is marginally faster than a generator because there is no extra Python frame per element. `iter()` on a list hands back a fresh list iterator each call, so repeated loops still work. **3. A separate iterator class.** Write a second class that holds the cursor, defines `__next__`, and returns `self` from its own `__iter__`; the container's `__iter__` returns a new instance of it. Reach for this only when the iterator needs an identity of its own -- extra methods, picklability, state you want to assert on in a test. It is roughly three times the code of the generator form for the same behaviour. ### A fresh object per call is the whole point Each of those three creates a *new* iterator on every `__iter__()` call, so the object can be looped repeatedly and nested inside itself. That single property is what makes `sorted(obj)`, `sum(obj)`, `max(obj)`, `list(obj)`, `", ".join(obj)`, tuple unpacking, comprehensions and `x in obj` all work -- and work more than once. Every one of them calls `iter()` internally, and some call it twice. Membership deserves a note: with no `__contains__` defined, Python falls back to iterating and comparing, so implementing `__iter__` gives you `in` for free at linear cost. ### What you have not built Your class is now an *iterable*, not an *iterator*. `next(deck)` still raises `TypeError: 'Deck' object is not an iterator`, because `next()` looks for `__next__` on the object itself. State that distinction deliberately: a container is re-iterable and holds no cursor, an iterator holds a cursor and is consumed once. ### Recognition by the abstract base classes `isinstance(deck, collections.abc.Iterable)` returns `True` the moment the class defines `__iter__`, with no registration and no subclassing, because `Iterable` implements `__subclasshook__` and simply looks for the method on the type. It checks only that the attribute *exists* -- a broken `__iter__` that returns a list still passes `isinstance` and still fails at loop time. For a stronger contract, inherit from `collections.abc.Iterable`, which forces you to define `__iter__`, or from `collections.abc.Sequence`, which builds `__contains__`, `__iter__`, `__reversed__`, `index` and `count` on top of the `__len__` and `__getitem__` you supply. ### Practical notes Keep `__iter__` cheap at call time. A generator function's body does not run until the first `__next__`, so validation you place at the top of it is deferred, sometimes surprisingly -- the exception surfaces at the loop, not at the call. If the traversal owns a resource, open it inside the generator body under a `with` block so it closes when the pass ends. And never store one iterator in `__init__` and return it from `__iter__`: that smuggles one-shot behaviour back in, because every caller shares the same cursor and the second loop sees nothing. The recipe fits in four lines, and it is what an interviewer wants first, before any protocol discussion: define `__iter__`, make it a generator, and every part of Python that consumes iterables now consumes your object.

  • Your class defines `__iter__` but not `__next__`. What does `next(obj)` do?
    It raises `TypeError: 'X' object is not an iterator`. `next()` looks for `__next__` on the object you hand it, and does not call `__iter__` first. The object is an iterable, so `for` and `iter(obj)` work fine; only the direct `next()` call fails. If you want a single element, call `next(iter(obj))`.
  • When would you write a separate iterator class instead of making `__iter__` a generator?
    When the cursor needs an identity beyond stepping: extra methods on the iterator, picklability (generators cannot be pickled), or state a test needs to inspect mid-pass. Also when the traversal is simple but the iterator is handed around and stored, so a named class documents it. For ordinary traversal the generator form is shorter and less error-prone, and it supplies `__next__`, `__iter__` and `StopIteration` for you.
  • Does defining `__iter__` also make `x in obj` work?
    Yes. With no `__contains__`, the `in` operator falls back to iterating the object and comparing each item with `==`, so membership testing comes for free at O(n). Define `__contains__` yourself when the class can answer faster -- for example by consulting an underlying set or dict -- or when it should stop short of a full pass.

iter is a ticket machine, not a ticket: each caller presses the button and walks away with their own numbered stub, which is why two people can queue at once.

saying these in an interview costs you the question

  • Says __iter__ can return a list of the elements
  • Thinks a class needs __next__ to be usable in a for loop
  • Stores one iterator in __init__ and returns it from __iter__
  • Claims for calls __iter__ once per element
  • Confuses making the class iterable with making it an iterator
  • Believes __len__ is required for iteration

context

resolving links…

open as a page

What is the difference between an iterable and an iterator in Python?

level: juniorimportance: must knowfreq 85%

basics

~20 s

An iterable can hand out a fresh iterator via iter. An iterator is the cursor itself: it defines next to produce one item at a time, and its own iter returns self, so every iterator is also an iterable.

open as a page

What does a Python for loop do under the hood with iter() and next()?

level: juniorimportance: must knowfreq 70%

basics

~20 s

A for loop calls iter() on the object once to get an iterator, then calls next() on that iterator repeatedly, binding each result to the loop variable. When next() raises StopIteration, the loop catches it and ends normally.

open as a page

When should a Python class's `__iter__` return `self` instead of a fresh iterator?

level: middleimportance: must knowfreq 48%

basics

~20 s

Return self only when the object is itself a cursor over a source that cannot be replayed, such as a stream reader; it is then one-shot. A container that must survive repeated and nested loops builds a new iterator per call.

open as a page

Why can a list be looped over repeatedly while a generator object is exhausted after one pass?

level: middleimportance: should knowfreq 70%

basics

~20 s

Each loop over a list calls iter() and gets a brand-new cursor starting at the first element. A generator object is already the cursor: its iter returns self, so a second loop resumes at the end and finishes without running the body.

open as a page

Why does next(obj) ignore a __next__ attribute set on the instance?

level: middleimportance: should knowfreq 30%

basics

~10 s

Implicitly invoked special methods are looked up on the object's type, not on the instance. next(obj) resolves next through type(obj), so a function stored in the instance dict is skipped and next() raises TypeError.

open as a page

A nightly report job's custom iterable caches every row so callers can loop it twice, and memory grows unbounded. How would you redesign the class?

level: seniorimportance: should knowfreq 32%

basics

~20 s

Make it a restartable iterable: store only how to reach the source, and have iter open a fresh handle inside a with block and yield rows. Nothing is retained between passes, so memory stays flat.

open as a page

When should you replay a one-shot iterator with itertools.tee() rather than list()?

level: seniorimportance: should knowfreq 45%

basics

~20 s

Use itertools.tee() only when the branches are consumed roughly in step, so its internal buffer stays small. If one branch is drained before the other starts, tee buffers the entire stream anyway and list() is simpler and faster.

open as a page

A collector's __next__ delegates to an inner iterator and a 6,800-row telemetry batch stops at row 900 with no error. What happened, and how do you fix it?

level: seniorimportance: should knowfreq 35%

basics

~20 s

A StopIteration raised inside next by the delegated call, or by a helper it uses, is indistinguishable from a deliberate end-of-stream signal. The consuming for loop catches it and ends normally, so the short batch looks like a successful one.

open as a page

Why can a Python class with only `__getitem__` and no `__iter__` be looped over?

level: middleimportance: nice to knowfreq 18%

basics

~20 s

It hits the legacy sequence-iteration fallback: with no iter, iter() builds an iterator that calls obj[0], obj[1], obj[2] and so on until IndexError is raised. The class works in a for loop yet isinstance against collections.abc.Iterable still reports False.

open as a page