Generators

Generators implement a practical form of lazy evaluation: values are produced as they are requested rather than calculated and stored together in advance. A generator retains enough execution state to continue after each yielded value, including local variables and its current position. This makes generators useful for data pipelines whose inputs are large, expensive to retrieve, or potentially unbounded.

In Python, a function containing yield creates a generator iterator when called. Its body starts executing when iteration begins, not when the function is called. Each yield supplies a value and suspends execution; the next request resumes execution after that yield. Normal completion ends iteration through StopIteration. A generator expression, such as (transform(row) for row in rows), also produces values lazily, whereas a list comprehension constructs a list eagerly.

A practical ingestion example is processing newline-delimited JSON from object storage. An application can iterate over an object's streaming response body, decode each line, parse the JSON, and yield validated records. The consumer can collect a bounded batch and write it to a destination before requesting more records. Reading the entire response body before creating the generator defeats much of the memory benefit. Memory consumption still depends on record size, parser behavior, retained state, and output buffering.

Generators also compose well. One stage yields source records, another filters invalid records, and another normalizes timestamps or extracts fields. If every stage consumes incrementally, the pipeline avoids storing full intermediate collections. API pagination follows a similar consumption pattern: an SDK paginator can supply pages incrementally, while an application generator yields individual records from those pages. A paginator is not necessarily a Python generator, even though both support incremental iteration.

There are important trade-offs. A generator normally supports only one traversal, so an exhausted instance does not restart automatically. Consumers that need another pass must recreate the generator and its input or store the results. Converting a generator to a list materializes its remaining values. Global sorting and other operations that require the complete dataset can also eliminate the memory advantage unless an external-memory or distributed algorithm is used.

Laziness changes error handling and resource management. Network failures, parsing errors, and transformation exceptions may occur during iteration rather than during generator creation. Keep files, response bodies, and database cursors open for the required consumption period, and close them deterministically when processing finishes or stops early. For restartable ingestion, store durable source positions or checkpoints separately; a generator's in-memory execution state is not a recovery mechanism.

Generators do not automatically provide parallelism, asynchronous execution, durable buffering, or exactly-once processing. Synchronous iteration lets the consumer control when it requests another value, but does not establish end-to-end backpressure across external systems. In Spark, a Python generator can yield results within mapPartitions, while Spark remains responsible for partition scheduling and task retries. External writes must therefore account for possible repeated execution.

For programming questions, distinguish generators from reusable collections and from the broader iterator abstraction. Every Python generator is an iterator, but not every iterator is a generator. Java streams, Scala iterators, SQL execution pipelines, and shell pipelines can support related incremental-processing patterns, but they do not all implement Python's yield-based generator semantics. Choose the language and framework abstraction that preserves incremental consumption throughout the actual workload.