A few years ago, I was reviewing a PR from a junior dev who was trying to process a 12GB system log file. Their approach was straightforward: lines = open('system.log').readlines() followed by a for line in lines: loop. The problem, of course, is that readlines() pulls the entire file into RAM. The staging server crashed instantly. When I asked why they didn't just iterate over the file object itself, they looked at me like I was speaking a foreign language. They understood for loops, but they didn't actually understand what Python was doing under the hood to make those loops work.

That's where the Iterator Protocol comes in. It's the "secret handshake" that allows Python to loop over almost anything—be it a list, a file, or a database cursor—without needing to know the internal structure of that object. If you want to create your own custom objects that can be used in a for loop or passed into sum() and max(), you have to implement this protocol.

The Handshake of __iter__ and __next__

At its core, the iterator protocol consists of two magic methods: __iter__() and __next__(). I like to think of __iter__ as the setup phase and __next__ as the execution phase.

When you write for item in my_object:, Python doesn't actually just start looping. First, it calls iter(my_object), which triggers the __iter__ method. This method is expected to return an iterator object. Once Python has that iterator, it calls next() on it repeatedly, which triggers the __next__ method. Each time __next__ is called, it returns the next value in the sequence.

But how does the loop know when to stop? It doesn't check the length of the collection. Instead, it waits for the iterator to raise a specific exception: StopIteration. This is one of those Python quirks that feels weird at first—using an exception to signal the end of a normal process—but it's incredibly efficient because it removes the need for the loop to constantly check a counter or a boundary.

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        # The iterator protocol requires __iter__ to return the iterator itself
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        
        self.current -= 1
        return self.current + 1

In the example above, I've created a simple countdown. Notice that __iter__ just returns self. This is a common pattern when the object itself is designed to be the iterator. If I were to run for num in Countdown(3): print(num), Python would call __iter__, then __next__ three times, and finally stop when StopIteration is raised.

Iterables vs. Iterators: The Crucial Distinction

This is where most developers get tripped up. An Iterable is something you can get an iterator from (it has an __iter__ method). A Iterator is the actual object that keeps track of the state and produces the values (it has both __iter__ and __next__).

Think of a book as an Iterable. The book itself isn't "at" a specific page; it's just a collection of pages. Your finger, however, is the Iterator. Your finger knows exactly which page you're on and how to move to the next one. You can have three different fingers (iterators) pointing to three different pages in the same book (iterable) simultaneously.

This distinction is why you can loop over a list multiple times. Every time you start a for loop on a list, Python calls __iter__, which creates a fresh iterator starting at index 0. If a list were its own iterator, you'd exhaust it after the first loop and it would be empty forever. I've seen people try to "reset" an iterator by calling __next__ again, but once StopIteration is raised, that specific iterator instance is spent. You need a new one.

📋 Practical Task

Build a Custom "Power-of-Two" Sequence Iterator

Your task is to create a class called PowerOfTwo that implements the iterator protocol. The class should take a limit argument in its constructor, representing the maximum value the sequence can reach.

Requirements:

• Implement __iter__ so the object can be used in a for loop.
• Implement __next__ to return powers of two (2, 4, 8, 16, etc.) sequentially.
• The iterator must raise StopIteration as soon as the next power of two would exceed the limit.
• The state (the current power) must be tracked internally within the instance.

Test your code:

# This should print 2, 4, 8, 16, 32, 64
pow2 = PowerOfTwo(100)
for val in pow2:
    print(val)