Generating the World !

— Learn about Python Generator

Photo by Kelvin Yup on Unsplash

If you are like me who have a few months experience of Python, then you might be in the same position as me — know something about Python, but not enough. So I spent sometime to gather bits and pieces of scattered knowledge to dig deeper into Python. In this blog, the piece will be generators.

But first, remember about the Iterator we mentioned earlier? Generator closely links with the iterator concept:

generator

A function which returns a generator iterator. It looks like a normal function except that it contains yield expressions for producing a series of values usable in a for-loop or that can be retrieved one at a time with the next() function.

Usually refers to a generator function, but may refer to a generator iterator in some contexts. In cases where the intended meaning isn’t clear, using the full terms avoids ambiguity.

generator iterator

An object created by a generator function.

Each yield temporarily suspends processing, remembering the location execution state. When the generator iterator resumes, it picks up where it left off (in contrast to functions which start fresh on every invocation).

generator expression

An expression that returns an iterator. It looks like a normal expression followed by a for clause defining a loop variable, range, and an optional if clause. The combined expression generates values for an enclosing function:

Generators at a glance

So first, Generators. Why they are in use? Well, just like the same terminology in JavaScript, Generators are implemented so that we don’t need to write counterintuitive Iterator protocol everytime with __iter__ and __next__ .

The difference between a Generator and a normal functions are mainly in that:

• It contains one or more yield statements that can remember the state and in pause, will return the control to the outer scope;
• It complies with the iteration protocol naturally so no manual implementation of __iter__() and __next__() is required;

Let’s see below how a generator function range_3_gen_func returns a generator iterator:

>>> def range_3_gen_func():
...     x = 0
...     while x < 3:
...       print(f"{x} is yielded")
...       yield x
...       x = x + 1
... result = range_3_gen_func()
... next(result)
... next(result)
... next(result)
... next(result)
==========================
0 is yielded
1 is yielded
2 is yielded
Traceback (most recent call last):
  File "main.py", line 13, in <module>
    next(result)
StopIteration

You can see from above that the local variable x is remembered throughout the lifetime. Also, note that the generator object can be iterated only once. To start the cycle again we need to invoke range_3_gen_func and assign the returned new generator to another variable.

Generator expressions

Sometimes you want to have a more lightweight version of generator just like list comprehension to list. This is where Generator Expression comes in(short form: “listcomps” and “genexps”).

In short, Generator expressions are like list comprehensions except it creates anonymous generator functions. (The syntax is only different in the () vs. [] )

Also noted that the list comprehension returns the entire list while the generator expression return one element at a time, and more memory efficient as a result.

>>> line_list = ['  line 1\n', 'line 2\n','line 3\n' ]
>>> stripped_iter = (line.strip() for line in line_list)
>>> stripped_list = [line.strip() for line in line_list]
>>> print(stripped_list)
['line 1', 'line 2', 'line 3']
>>> print(stripped_iter)
<generator object <genexpr> at 0x10c502ed0>
>>> for _ in stripped_iter:
...   print(_)
...
line 1
line 2
line 3

You can see that stripped_iter returns an iterator that computes the values as necessary, not needing to materialise all the values at once, and lazy evaluates the return.

Just like list comprehension, we can add logic to the genexp:

( expression for expr in sequence1
             if condition1
             for expr2 in sequence2
             if condition2
             for expr3 in sequence3 ...
             if condition3
             for exprN in sequenceN
             if conditionN )

Also, when used as function argument, the outside parenthesis can be ignored:

obj_total = sum(obj.count for obj in list_all_objects())

That’s so much of it!

Happy Reading!