Generator (computer programming)

Uses

Generators are usually invoked inside loops. The first time that a generator invocation is reached in a loop, an iterator object is created that encapsulates the state of the generator routine at its beginning, with arguments bound to the corresponding parameters. The generator's body is then executed in the context of that iterator until a special yield action is encountered; at that time, the value provided with the yield action is used as the value of the invocation expression. The next time the same generator invocation is reached in a subsequent iteration, the execution of the generator's body is resumed after the yield action, until yet another yield action is encountered. In addition to the yield action, execution of the generator body can also be terminated by a finish action, at which time the innermost loop enclosing the generator invocation is terminated. In more complicated situations, a generator may be used manually outside of a loop to create an iterator, which can then be used in various ways.

Because generators compute their yielded values only on demand, they are useful for representing streams, such as sequences that would be expensive or impossible to compute at once. These include e.g. infinite sequences and live data streams.

When eager evaluation is desirable (primarily when the sequence is finite, as otherwise evaluation will never terminate), one can either convert to a list, or use a parallel construction that creates a list instead of a generator. For example, in Python a generator g can be evaluated to a list l via l = list(g), while in F# the sequence expression seq { ... } evaluates lazily (a generator or sequence) but [ ... ] evaluates eagerly (a list).

In the presence of generators, loop constructs of a language – such as for and while – can be reduced into a single loop ... end loop construct; all the usual loop constructs can then be comfortably simulated by using suitable generators in the right way. For example, a ranged loop like for x = 1 to 10 can be implemented as iteration through a generator, as in Python's for x in xrange(10, 1). Further, break can be implemented as sending finish to the generator and then using continue in the loop.

Timeline

Generators first appeared in CLU (1975), were a prominent feature in the string manipulation language Icon (1977) and are now available in Python, C#, Ruby, the later versions of ECMAScript (as of ES6/ES2015) and other languages. In CLU and C#, generators are called iterators, and in Ruby, enumerators.

Lisp

The final Common Lisp standard does not natively provide generators, yet various library implementations exist, such as SERIES documented in CLtL2 or pygen.

CLU

A yield statement is used to implement iterators over user-defined data abstractions.

Icon

Every expression (including loops) is a generator. The language has many generators built-in and even implements some of the logic semantics using the generator mechanism (logical disjunction or "OR" is done this way).

Printing squares from 0 to 20 can be achieved using a co-routine by writing:

local squares, j squares := create (seq(0) ^ 2) every j := |@squares do if j <= 20 then write(j) else break

However, most of the time custom generators are implemented with the "suspend" keyword which functions exactly like the "yield" keyword in CLU.

C++

It is possible to introduce generators into C++ using pre-processor macros. The resulting code might have aspects very different from native C++. but the generator syntax can be very uncluttered. A very good example can be found at. The set of pre-processor macros defined in this source allow generators defined with the syntax as in the following example:

This can then be iterated using:

Moreover, C++11 allows foreach loops to be applied to any class that provides the begin and end functions. It's then possible to write generator-like classes by defining both the iterable methods (begin and end) and the iterator methods (operator!=, operator++ and operator*) in the same class. For example, it is possible to write the following program:

A basic range implementation would look like that:

Perl

Perl does not natively provide generators, but support is provided by the Coro::Generator module which uses the Coro co-routine framework. Example usage:

Tcl

In Tcl 8.6, the generator mechanism is founded on named coroutines.

Haskell

In Haskell, with its lazy evaluation model, everything is a generator - every datum created with a non-strict data constructor is generated on demand. For example,

where (:) is a non-strict list constructor, cons, and $ is just a "called-with" operator, used for parenthesization. This uses the standard adaptor function,

which re-fetches values agreeable with a predicate, and stops requesting new values as soon as a non-agreeable one is encountered. The shared storage access is used as a universal mediator in Haskell. List comprehensions can be freely used:

Racket

Racket provides several related facilities for generators. First, its for-loop forms work with sequences, which are a kind of a producer:

and these sequences are also first-class values:

Some sequences are implemented imperatively (with private state variables) and some are implemented as (possibly infinite) lazy lists. Also, new struct definitions can have a property that specifies how they can be used as sequences.

But more directly, Racket comes with a generator library for a more traditional generator specification. For example,

Note that the Racket core implements powerful continuation features, providing general (re-entrant) continuations that are composable, and also delimited continuations. Using this, the generator library is implemented in Racket.

PHP

The community of PHP implemented generators in PHP 5.5. Details can be found in the original RFC about Generator.

Ruby

Ruby supports generators (starting from version 1.9) in the form of the built-in Enumerator class.

Java

Java has had a standard interface for implementing iterators since its early days, and since Java 5, the "foreach" construction makes it easy to loop over objects that provide the java.lang.Iterable interface. (The Java collections framework and other collections frameworks, typically provide iterators for all collections.)

However, Java does not have generators built into the language. This means that creating iterators is often much trickier than in languages with built-in generators, especially when the generation logic is complex. Because all state must be saved and restored every time an item is to be yielded from an iterator, it is not possible to store state in local variables or use built-in looping routines, as when generators are available; instead, all of this must be manually simulated, using object fields to hold local state and loop counters.

Even simple iterators built this way tend to be significantly bulkier than those using generators, with a lot of boilerplate code.

The original example above could be written in Java 7 as:

An infinite Fibonacci sequence could also be written int Java 7 as an Iterator:

Also an infinite Fibonacci sequence could also be written using java 8 Stream interface:

Or get an Iterator from the Java 8 super-interface BaseStream of Stream interface.

C#

An example C# 2.0 generator (the yield is available since C# version 2.0): Both of these examples utilise Generics, but this is not required.

It is possible to use multiple yield return statements and they are applied in sequence on each iteration:

XL

In XL, iterators are the basis of 'for' loops:

F#

F# provides generators via sequence expressions, since version 1.9.1. These can define a sequence (lazily evaluated, sequential access) via seq { ... }, a list (eagerly evaluated, sequential access) via [ ... ] or an array (eagerly evaluated, indexed access) via [| ... |] that contain code that generates values. For example,

forms a sequence of squares of numbers from 0 to 14 by filtering out numbers from the range of numbers from 0 to 25.

Python

Generators were added to Python in version 2.2. An example generator:

In Python, a generator can be thought of as an iterator that contains a frozen stack frame. Whenever the iterator's next() method is called, Python resumes the frozen frame, which executes normally until the next yield statement is reached. The generator's frame is then frozen again, and the yielded value is returned to the caller.

PEP 380 (implemented in Python 3.3) adds the yield from expression, allowing a generator to delegate part of its operations to another generator.

Generator expressions

Python has a syntax modeled on that of list comprehensions, called a generator expression that aids in the creation of generators. The following extends the example above by using a generator expression to compute squares from the countfrom generator function:

ECMAScript

ECMAScript 6 (AKA Harmony) will feature generator functions.

An infinite Fibonacci sequence can be written using a function generator:

R

The iterators package can be used for this purpose.