Generator Expressions

A generator expression looks like a list comprehension but never builds the list — it produces values one at a time, on demand, which can turn a memory-hungry computation into a nearly free one.

A generator expression uses comprehension-like syntax — (expression for item in iterable if condition) — but instead of eagerly building a container, it returns a generator object that produces values lazily, one at a time, only as they're requested. This makes generator expressions the natural choice when you need to iterate over a computed sequence once, especially a large or unbounded one, without materializing it all in memory at once.

Syntax: Parentheses Instead of Brackets

Swap a list comprehension's square brackets for parentheses and you have a generator expression: (expr for item in iterable if condition). The for/if clauses and expression work exactly as they do in a list comprehension — only the result type and evaluation timing differ.

numbers = [1, 2, 3, 4, 5]

squares_list = [n ** 2 for n in numbers]        # a list, built immediately
squares_gen = (n ** 2 for n in numbers)          # a generator, built lazily

print(type(squares_list))   # <class 'list'>
print(type(squares_gen))     # <class 'generator'>

Lazy Evaluation: Nothing Happens Until You Ask

Creating a generator expression does no work beyond capturing the iterable and expression — it does not evaluate the expression for any items yet. Values are computed one at a time, only when next() is called on the generator (directly, or implicitly by a for loop, list(), sum(), and similar consumers).

def noisy_square(n):
    print(f"computing square of {n}")
    return n ** 2

gen = (noisy_square(n) for n in [1, 2, 3])
print("generator created, nothing printed yet")

first = next(gen)   # NOW 'computing square of 1' prints
print(first)          # 1

Memory Benefits

A list comprehension over a large or huge iterable holds every result in memory simultaneously. A generator expression holds only the current item's worth of state, regardless of how many items will ultimately be produced — the memory footprint is effectively constant, not proportional to the number of items. This matters enormously when the source data is large, or when only some prefix of the results is ever actually needed.

import sys

list_version = [n for n in range(1_000_000)]
gen_version = (n for n in range(1_000_000))

print(sys.getsizeof(list_version))   # roughly 8+ million bytes
print(sys.getsizeof(gen_version))     # a small, fixed number of bytes
# (exact byte counts are a CPython implementation detail and vary by version)

One-Time Iteration: Generators Are Exhausted After Use

A generator is a one-shot iterator: once it has been fully consumed (by a for loop, list(), or repeated next() calls until StopIteration), it is exhausted and yields nothing further — even if you try to iterate it again. This is a fundamental difference from a list, which can be iterated as many times as you like.

If you need to iterate the same computed sequence more than once, either build a list ([n for n in ...]) or create a fresh generator expression each time you need to iterate — a single generator object cannot be 'rewound'.

gen = (n for n in range(3))
print(list(gen))   # [0, 1, 2] -- fully consumed here
print(list(gen))    # [] -- already exhausted, nothing left

Passing Directly to Functions: Omitting the Parentheses

When a generator expression is the sole argument to a function call, the parentheses of the call itself double as the generator expression's parentheses — you can drop the inner pair. This is idiomatic for calls like sum(), any(), all(), max(), and sorted().

If the function takes more than one argument, the extra parentheses ARE required: sorted((n for n in numbers), key=abs) -- otherwise Python cannot tell where the generator expression ends and the next argument begins.

numbers = range(10)

total = sum(n ** 2 for n in numbers)      # no extra parentheses needed
has_negative = any(n < 0 for n in numbers)
all_positive = all(n >= 0 for n in numbers)

print(total, has_negative, all_positive)   # 285 False True

Generator Expressions vs Generator Functions

A generator expression is a compact, one-line way to create a generator; a generator function (a function containing yield, covered as its own topic) is the more general, multi-statement way. Both produce the same kind of object — an iterator implementing the lazy generator protocol — and both are exhausted after one full pass. Reach for a generator expression when the logic fits in a single expression; reach for a generator function when you need branching, multiple yield points, loops with complex state, or a name you can call multiple times to get fresh generators.

# Generator expression -- fine for a simple transform/filter
evens = (n for n in range(20) if n % 2 == 0)

# Generator function -- needed once the logic doesn't fit one expression
def fibonacci_up_to(limit):
    a, b = 0, 1
    while a < limit:
        yield a
        a, b = b, a + b

print(list(fibonacci_up_to(20)))   # [0, 1, 1, 2, 3, 5, 8, 13]

Chaining Generator Expressions

Generator expressions can be chained, with one feeding into another, and Python evaluates them lazily end-to-end — a value only flows through the whole chain when it's actually requested by the final consumer. This lets you compose multi-step pipelines without ever materializing an intermediate list.

words = ["apple", "Banana", "cherry", "Date"]

lowered = (w.lower() for w in words)
capitalized = (w[0].upper() + w[1:] for w in lowered)
long_only = (w for w in capitalized if len(w) > 5)

print(list(long_only))   # ['Apple', 'Banana', 'Cherry']

Nested for Clauses in Generator Expressions

Just like list comprehensions, generator expressions support multiple for clauses for flattening or Cartesian-product-style iteration — the same left-to-right nesting rules apply, just evaluated lazily.

matrix = [[1, 2], [3, 4], [5, 6]]
flat_gen = (x for row in matrix for x in row)
print(sum(flat_gen))   # 21 -- summed lazily, no intermediate flat list built

When to Use a Generator Expression vs a List Comprehension

Use a generator expression when you'll consume the values exactly once, in order (a single for loop, a single pass to sum()/any()/all(), or feeding another iterator-consuming function) — especially when the source is large or the full result is never needed all at once (e.g. any(...) can stop at the first True without computing the rest). Use a list comprehension when you need to index into the result, iterate it more than once, check its length, or otherwise need an actual, reusable container.

# Good use of a generator: any() can short-circuit, no need to build a full list
has_error = any("ERROR" in line for line in open("log.txt"))

# Needs a list: indexing, length, and multiple passes are all required
rows = [line.strip() for line in open("data.csv")]
print(rows[0], len(rows))

Key Takeaways

(expr for item in iterable if condition) creates a generator object; unlike a list comprehension, no values are computed until they're requested.

Generator expressions use constant memory relative to the number of items, since only the current item's state is held at a time.

A generator is exhausted after one full pass — it cannot be reused or rewound; create a new one (or use a list) if you need to iterate again.

Parentheses can be omitted when a generator expression is the sole argument to a function call, e.g. sum(n for n in items).

Prefer a generator expression for single-pass consumption of potentially large data; prefer a list comprehension when you need indexing, length, or multiple iterations.

gen = (n for n in range(5)) first_pass = list(gen) second_pass = list(gen) # [] -- silently empty, not an error def make_gen(): return (n for n in range(5)) first_pass = list(make_gen()) second_pass = list(make_gen())

A generator is exhausted after being fully consumed once; reusing the same generator object for a second pass silently produces nothing rather than raising an error. Create a fresh generator (or store a list) if more than one pass is needed.

total = sum((n for n in numbers), start=0) # unnecessary extra parens, easy to get wrong with keyword args total = sum(n for n in numbers) # bare parens of sum() double as the generator's parens, since it's the sole positional argument

When the generator expression is the only argument, the call's own parentheses suffice -- adding an inner pair is not wrong here but is unnecessary noise; it becomes necessary only when another argument follows.

squares = (n 2 for n in range(5)) print(len(squares)) # TypeError: generator has no len() squares = [n 2 for n in range(5)] print(len(squares))

A generator does not know its own length in advance (it hasn't computed anything yet) and doesn't support len(), indexing, or repeated iteration -- use a list comprehension when you need any of those.

What is the key difference in behavior between [n2 for n in range(5)] and (n2 for n in range(5))?

The list comprehension immediately computes all five values and returns a list holding them. The generator expression returns a generator object with no values computed yet -- each value is computed lazily, one at a time, only as it's requested by iteration.

Why does a generator expression use less memory than the equivalent list comprehension for a large iterable?

A generator only needs to keep enough state to produce the next value on demand -- it never holds the full sequence of results in memory at once. A list comprehension, by contrast, holds every computed result simultaneously, so its memory use grows with the number of items.

What happens if you try to iterate the same generator object a second time after fully consuming it once?

It yields nothing -- an already-exhausted generator produces an empty sequence on further iteration, without raising an error. A fresh generator expression (or a stored list) is needed for a second pass.

When can you omit the parentheses around a generator expression?

When it is the sole argument to a function call -- the call's own parentheses double as the generator expression's parentheses, e.g. sum(n for n in items). If the function takes additional arguments, the generator expression needs its own explicit parentheses.

Why might any(cond(x) for x in huge_iterable) be more efficient than any([cond(x) for x in huge_iterable])?

any() with a generator expression can short-circuit and stop consuming as soon as it finds a True value, computing cond(x) for only as many items as needed. The list comprehension version computes cond(x) for every single item up front, building the whole list, before any() even starts checking it.

What is the difference between a generator expression and a generator function (one containing yield)?

A generator expression is a compact, single-expression way to define lazy iteration; a generator function is a full function (with yield statements) offering more flexibility -- branching, loops, multiple yield points, complex state -- and can be called repeatedly to produce fresh generator objects each time. Both produce generator objects that follow the same lazy, one-shot iterator protocol.

Why does len() raise a TypeError when called on a generator object?

A generator has not computed its values (or even necessarily determined how many there will be) at the time len() would be called -- it only knows how to produce the next value on request. Since it doesn't know its total length in advance without fully consuming itself, len() is not supported; use a list comprehension instead if the count is needed.

Write a function sum_of_squares(n) that computes the sum of the squares of integers from 1 to n using a generator expression fed directly to sum(), without building an intermediate list.

def sum_of_squares(n: int) -> int: return sum(i 2 for i in range(1, n + 1)) print(sum_of_squares(5)) # 1 + 4 + 9 + 16 + 25 = 55 sum(i 2 for i in range(1, n + 1)) passes a generator expression directly as sum()'s sole argument (parentheses omitted since it's the only argument), so squares are computed and added one at a time without ever materializing a full list.

What does this code do, and what happens on the final line?

It prints 0, then 2, then [4] (the remaining value collected by list()), and then the final next(gen) raises StopIteration because the generator is now fully exhausted -- there are no values left to produce.

Write a function first_matching(iterable, predicate) that returns the first item in iterable for which predicate(item) is True, using a generator expression and next(), or None if no item matches.

def first_matching(iterable, predicate): return next((item for item in iterable if predicate(item)), None) print(first_matching([1, 3, 4, 7, 8], lambda x: x % 2 == 0)) # 4 print(first_matching([1, 3, 5], lambda x: x % 2 == 0)) # None

A generator expression filters iterable lazily, and next(generator, default) requests just the first value, returning the given default (None) instead of raising StopIteration if the generator produces nothing -- this stops after finding one match, without scanning the rest of iterable.

What does this print, and why does g2 still produce values even after g1 was consumed?

It prints [0, 1, 4] then [0, 1, 4] again. Because make_squares() creates and returns a brand-new generator object on each call, g1 and g2 are two entirely separate generators -- exhausting g1 has no effect on g2, which has its own independent state.

gen = (x * 2 for x in range(3))
print(next(gen))
print(next(gen))
print(list(gen))
print(next(gen))

def make_squares():
    return (n ** 2 for n in range(3))

g1 = make_squares()
g2 = make_squares()
print(list(g1))
print(list(g2))