Generating the World !

— Learn about Python Generator

Photo by Kelvin Yup on Unsplash

If you are like me who have a few months experience of Python, then you might be in the same position as me — know something about Python, but not enough. So I spent sometime to gather bits and pieces of scattered knowledge to dig deeper into Python. In this blog, the piece will be generators.

But first, remember about the Iterator we mentioned earlier? Generator closely links with the iterator concept:

generator

generator iterator

generator expression

Generators at a glance

So first, Generators. Why they are in use? Well, just like the same terminology in JavaScript, Generators are implemented so that we don’t need to write counterintuitive Iterator protocol everytime with __iter__ and __next__ .

The difference between a Generator and a normal functions are mainly in that:

  • It contains one or more yield statements that can remember the state and in pause, will return the control to the outer scope;
  • It complies with the iteration protocol naturally so no manual implementation of __iter__() and __next__() is required;

Let’s see below how a generator function range_3_gen_func returns a generator iterator:

>>> def range_3_gen_func():
... x = 0
... while x < 3:
... print(f"{x} is yielded")
... yield x
... x = x + 1
... result = range_3_gen_func()
... next(result)
... next(result)
... next(result)
... next(result)
==========================
0 is yielded
1 is yielded
2 is yielded
Traceback (most recent call last):
File "main.py", line 13, in <module>
next(result)
StopIteration

You can see from above that the local variable x is remembered throughout the lifetime. Also, note that the generator object can be iterated only once. To start the cycle again we need to invoke range_3_gen_func and assign the returned new generator to another variable.

Generator expressions

Sometimes you want to have a more lightweight version of generator just like list comprehension to list. This is where Generator Expression comes in(short form: “listcomps” and “genexps”).

In short, Generator expressions are like list comprehensions except it creates anonymous generator functions. (The syntax is only different in the () vs. [] )

Also noted that the list comprehension returns the entire list while the generator expression return one element at a time, and more memory efficient as a result.

>>> line_list = ['  line 1\n', 'line 2\n','line 3\n' ]
>>> stripped_iter = (line.strip() for line in line_list)
>>> stripped_list = [line.strip() for line in line_list]
>>> print(stripped_list)
['line 1', 'line 2', 'line 3']>>> print(stripped_iter)<generator object <genexpr> at 0x10c502ed0>>>> for _ in stripped_iter:
... print(_)
...
line 1
line 2
line 3

You can see that stripped_iter returns an iterator that computes the values as necessary, not needing to materialise all the values at once, and lazy evaluates the return.

Just like list comprehension, we can add logic to the genexp:

( expression for expr in sequence1
if condition1
for expr2 in sequence2
if condition2
for expr3 in sequence3 ...
if condition3
for exprN in sequenceN
if conditionN )

Also, when used as function argument, the outside parenthesis can be ignored:

obj_total = sum(obj.count for obj in list_all_objects())

That’s so much of it!

Happy Reading!

Hi :)