🎓 Advanced · Topic 02
Generators & Iterators
Process massive datasets lazily — one item at a time — without loading everything into memory. Master the iterator protocol and the power of yield.
⏱ ~55 min
🔴 Advanced
💾 Memory efficiency
The Iterator Protocol
Any object that implements __iter__ and __next__ is an iterator. Python's for loop calls these methods under the hood.
custom_iterator.py
class Countdown:
def __init__(self, start):
self.current = start
def __iter__(self):
return self
def __next__(self):
if self.current <= 0:
raise StopIteration
self.current -= 1
return self.current + 1
for n in Countdown(3):
print(n) # 3, 2, 1
Generator Functions with yield
Any function containing yield becomes a generator. It pauses execution at each yield and resumes when the next value is requested.
generators.py
def fibonacci():
a, b = 0, 1
while True: # infinite sequence!
yield a
a, b = b, a + b
fib = fibonacci()
print([next(fib) for _ in range(8)])
# [0, 1, 1, 2, 3, 5, 8, 13]
# Read a 10 GB file line by line — zero memory cost
def read_large_file(path):
with open(path) as f:
for line in f:
yield line.strip()
⚡ vs list — memory comparison [x**2 for x in range(1_000_000)] allocates ~8 MB. (x**2 for x in range(1_000_000)) allocates ~120 bytes. Same values, 66,000× less memory.
Generator Expressions
Like list comprehensions but with parentheses — lazy by default, perfect for chaining transformations.
gen_expressions.py
# Sum of squares without building a list
total = sum(x**2 for x in range(1_000_000))
# Chain generators — each step is lazy
lines = (line.strip() for line in open("log.txt"))
errors = (l for l in lines if "ERROR" in l)
for err in errors:
print(err) # processes one line at a time
yield from — Delegating to Sub-generators
yield from delegates to another iterable, transparently forwarding values and exceptions.
yield_from.py
def flatten(nested):
for item in nested:
if isinstance(item, list):
yield from flatten(item) # recurse lazily
else:
yield item
data = [1, [2, [3, 4]], [5, 6]]
print(list(flatten(data))) # [1, 2, 3, 4, 5, 6]
