yield

The yield keyword in Python turns a function into a generator. Instead of returning a single value and stopping, a generator function can pause execution and yield multiple values over time. This enables lazy evaluation — producing values on-demand rather than building entire lists in memory.

Syntax

def generator_function():
    # Generator body
    yield value1
    yield value2
    # ...

This syntactic structure is deceptively simple. The real power comes from what happens at runtime: each call to a generator function returns a generator object rather than executing the function body. The function code only begins executing when the first value is requested through iteration or next. This separation of definition from execution is the foundation that makes generators suitable for streaming data, coroutines, and cooperative multitasking patterns in Python.

Basic Examples

A Simple Counter Generator

def count_up_to(n):
    i = 1
    while i <= n:
        yield i
        i += 1

for num in count_up_to(5):
    print(num)
# 1
# 2
# 3
# 4
# 5

The counter example shows a generator that produces its own values from scratch, but more often you will want to transform values that come from another iterable source. This is where generators truly excel — they let you wrap any existing sequence, iterator, or even another generator with additional processing logic, all while preserving the lazy evaluation property that keeps memory usage low.

Yielding from an Iterable

def double_values(items):
    for item in items:
        yield item * 2

for doubled in double_values([1, 2, 3]):
    print(doubled)
# 2
# 4
# 6

These basic examples illustrate the surface-level behavior of generators, but understanding what happens internally is essential for using them effectively. A generator function does not run to completion like a normal function. Instead, it maintains a frozen execution context — local variables, instruction pointer, and exception state — between yields. This mechanism is what enables the memory and performance benefits that follow.

How Generators Work

Lazy Evaluation

Generators produce values one at a time, only when requested:

def lazy_range(n):
    print("Starting")
    for i in range(n):
        yield i
    print("Done")

gen = lazy_range(3)
# Nothing printed yet — generator is idle

print(next(gen))  # "Starting" then 0
print(next(gen))  # 1
print(next(gen))  # 2
print(next(gen))  # StopIteration exception

The lazy evaluation example reveals the stop-and-resume nature of generators, but the practical consequence of this behavior is far more important. Because generators hold only the current state rather than the entire dataset, they make it possible to work with data sources that exceed available RAM. This property is not merely convenient — it is often the difference between a script that runs and one that crashes with a MemoryError.

Memory Efficiency

Generators shine when working with large datasets:

# Bad — builds entire list in memory
def get_millions_bad():
    return [i for i in range(1_000_000)]

# Good — yields one at a time
def get_millions_good():
    for i in range(1_000_000):
        yield i

# Both can be iterated, but generator uses constant memory

Beyond simple iteration, Python generators expose a richer protocol through three special methods that most developers overlook. These methods turn generators from one-way data producers into bidirectional communication channels. Understanding them is key to using generators for coroutines, cooperative scheduling, and complex control flow patterns that go well beyond basic looping.

Generator Methods

next()

Get the next value from a generator:

def numbers():
    yield 1
    yield 2
    yield 3

gen = numbers()
print(next(gen))  # 1
print(next(gen))  # 2
print(next(gen))  # 3
# print(next(gen))  # StopIteration

The next function is the simplest way to pull values out of a generator, treating it as a pure data source. However, generators can also accept values pushed into them during execution. This capability forms the basis of coroutines, where a generator can yield intermediate results, receive updated parameters, and continue processing. The send method is the entry point for this bidirectional communication pattern.

send()

Send a value into a generator:

def accumulator():
    total = 0
    while True:
        value = yield total
        if value is not None:
            total += value

gen = accumulator()
print(next(gen))          # 0 — prime the generator
print(gen.send(10))       # 10
print(gen.send(5))        # 15

The accumulator example demonstrates sending numeric values into a running generator, but there are situations where you need to signal an error condition rather than a data value. Perhaps a downstream consumer has determined that the current computation should be abandoned, or an external timeout has triggered. Rather than building a custom sentinel protocol, generators support injecting exceptions directly, which the generator can catch and handle like any other exception in its normal control flow.

throw()

Inject an exception into a generator:

def fragile():
    try:
        yield 1
    except ValueError:
        yield "Caught!"

gen = fragile()
print(next(gen))           # 1
print(gen.throw(ValueError))  # "Caught!"

The throw method lets the caller raise exceptions inside the generator for graceful error handling, but sometimes you simply need to shut down a running generator entirely. This is common when processing stops early — for instance, a search algorithm finds its match and no longer needs the remaining values. Rather than letting the generator hang indefinitely or waiting for it to exhaust naturally, close provides a clean termination mechanism that triggers any cleanup logic the generator may have registered in finally blocks.

close()

Terminate a generator:

def infinite():
    while True:
        yield 1

gen = infinite()
print(next(gen))  # 1
gen.close()        # Generator closed
# next(gen) raises StopIteration

The individual generator methods give you fine-grained control over single generators, but real programs often compose multiple generators together. Before Python 3.3, delegating from one generator to another required manual loops that were verbose and error-prone, especially when bidirectional communication was needed. The yield from syntax, introduced in PEP 380, simplifies these patterns dramatically by handling all the plumbing automatically.

yield from

Delegating to Sub-generators

yield from delegates to another iterable:

def chain(*iterables):
    for it in iterables:
        yield from it

for item in chain([1, 2], [3, 4], [5]):
    print(item)
# 1
# 2
# 3
# 4
# 5

The chain example shows yield from flattening iteration across multiple sequences, but it is worth understanding exactly what the syntax expands to under the hood. For simple iteration without send or throw, yield from behaves identically to a manual for loop. Recognizing this equivalence helps you decide when the syntactic sugar is appropriate and when an explicit loop might be clearer for readers unfamiliar with the construct.

Equivalent to a Loop

# These are equivalent:
def gen1():
    yield from [1, 2, 3]

def gen2():
    for item in [1, 2, 3]:
        yield item

With the mechanics of basic generators and yield from established, the real value emerges when these tools are combined in practical programming patterns. Python developers encounter generators most often in three recurring scenarios: processing files that are too large to fit in memory, producing infinite or very long mathematical sequences, and constructing data transformation pipelines where each stage operates lazily on the output of the previous stage.

Common Patterns

Processing Large Files

def read_lines(filepath):
    with open(filepath) as f:
        for line in f:
            yield line.strip()

# Process a million-line file without loading it all
for line in read_lines("huge_file.txt"):
    process(line)  # One line at a time

File processing shows generators handling bounded data that happens to be too large for memory, but generators can also represent sequences that have no natural endpoint at all. Mathematical sequences like Fibonacci numbers or prime numbers are conceptually infinite, and a generator can model them cleanly without artificial limits. The caller decides how many terms to consume, while the generator expresses the recurrence relation in its purest form.

Infinite Sequences

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

fib = fibonacci()
for _ in range(10):
    print(next(fib))
# 0
# 1
# 1
# 2
# 3
# 5
# 8
# 13
# 21
# 34

Standalone generators that produce or transform a single data stream are useful, but the functional programming style really shines when you chain multiple generators into a processing pipeline. Each generator performs one narrow transformation — filtering, mapping, aggregating — and passes its output to the next stage. This compositional approach keeps each piece testable and reusable while avoiding the nested loops that would otherwise be required.

Pipeline Processing

Chain generators for data pipelines:

def numbers():
    yield from range(10)

def double(n):
    return n * 2

def is_even(n):
    return n % 2 == 0

# Chain: numbers -> double -> filter -> collect
result = [double(n) for n in numbers() if is_even(double(n))]
print(result)  # [0, 4, 8, 12, 16]

The pipeline example chains named generator functions for clarity, but Python also provides a lightweight syntax for simple transformations: generator expressions. These look like list comprehensions wrapped in parentheses and produce values lazily rather than building a complete list. They occupy a middle ground between full generator functions and inline comprehensions, suitable for one-off transformations that do not warrant a separate named function.

Generator Expressions

For simple transformations, use generator expressions:

# Generator expression (lazy)
gen = (x * 2 for x in range(10))

# List comprehension (eager)
lst = [x * 2 for x in range(10)]

The preceding patterns focus on generators as one-way data producers where values flow outward to the caller. However, the combination of send and yield from unlocks genuine coroutine patterns where data flows in both directions through nested generators. This is an advanced technique most commonly seen in asynchronous frameworks before native async and await were introduced, but it remains part of the language and appears in certain networking and event-driven libraries.

Send Values with yield from

Two-Way Communication

def transformer(source):
    while True:
        received = yield
        for item in source:
            if item > received:
                yield item

gen = transformer([1, 5, 3, 8, 2])
next(gen)  # Prime
print(gen.send(4))  # 5, 8
print(gen.send(6))  # 8

Having covered the full range of generator capabilities — from simple value production through bidirectional communication with nested delegation — the remaining question is how to apply these tools effectively in production code. Generators are powerful but can be misused in ways that produce confusing control flow or subtle bugs. The following guidelines distill community experience into actionable rules that keep generator-heavy code maintainable and correct.

Best Practices

Use Generators for Large Data

# Bad — loads everything
data = [process(item) for item in huge_list]

# Good — processes one at a time
def process_all(items):
    for item in items:
        yield process(item)

The memory argument for generators is the most commonly cited benefit, but it is equally important to write generators in a style that is readable and maintainable. A common anti-pattern is manually iterating over nested structures with multiple for loops when a single yield from would express the same intent more clearly. The syntactic choice matters because it signals to readers whether the generator is simply delegating or performing custom logic at each level of iteration.

Prefer yield from Over Manual Loops

# Verbose
def flatten(list_of_lists):
    for sublist in list_of_lists:
        for item in sublist:
            yield item

# Clean
def flatten(list_of_lists):
    yield from list_of_lists

Choosing the right syntactic form matters for clarity, but there is also a behavioral constraint that catches many newcomers: generators are single-use iterators. Once a generator has been fully consumed — whether by a for loop, a call to list, or any other iteration protocol — it cannot be restarted or rewound. This is fundamentally different from lists or other collections that can be traversed repeatedly.

Remember Generators Are One-Way

def gen():
    yield 1
    yield 2

g = gen()
list(g)      # [1, 2]
list(g)      # [] — exhausted!

The single-use nature of generators means you must either recreate them for each pass or materialize them into a collection if repeated access is required. Beyond the core yield syntax, Python’s standard library provides the itertools module, which contains a rich set of pre-built generator tools for common iteration patterns. Knowing when to reach for itertools rather than writing custom generator logic can save significant development time while producing code that other Python developers will recognize immediately.

Use itertools for Complex Operations

from itertools import islice, takewhile

def numbers():
    n = 0
    while True:
        yield n
        n += 1

# Take first 10
print(list(islice(numbers(), 10)))  # [0, 1, 2, ..., 9]

# Take while condition is true
print(list(takewhile(lambda x: x < 5, numbers())))  # [0, 1, 2, 3, 4]

See Also

• return keyword — returning from functions
• def keyword — function definitions
• async keyword — async generators with async def
• lambda keyword — anonymous functions and generator expressions