Python understands parsed expressions

Python Understanding Comprehensions List comprehensions are one of my favorite Python features. They may seem a bit mysterious at first, but once you fully understand them, you’ll find they’re actually quite simple.

The key to understanding is that list comprehensions are a more streamlined and compact version of the for loops used for various containers.

This is sometimes called syntactic sugar, which is used to quickly accomplish common functions, thus reducing the burden on Python programmers. Consider the following list comprehension:

>>> squares = [x * x for x in range(10)]

This comprehension generates a list containing the squares of all integers from 0 to 9:

>>> squares
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

If we wanted to construct the same list using a pure for loop, we might write:

>>> squares = []
>>> for x in range(10):
... squares.append(x * x)

That’s a pretty simple loop, right? If you look back at the for loop version and the list comprehension version, you’ll notice some commonalities and patterns. By summarizing the common structures, you end up with a template like this:

values = [expression for item in collection]

The list comprehension “template” above is equivalent to the following for loop:

values = []
for item in collection:
values.append(expression)

Here, a new list instance is set up to receive the output value, then iterates over all the elements in the collection, applying any expression to each element, and then appending each result to the output list.

This is a consistent pattern that can be used to convert many for loops to list comprehensions, and vice versa. Now let’s add a more useful feature to this template, namely filtering elements using a condition.

List comprehensions can filter elements based on some condition and add the values that meet the condition to the output list. Let’s take a look at an example:

>>> even_squares = [x * x for x in range(10)
if x % 2 == 0]

This list comprehension will produce a list of the squares of all even integers from 0 to 9. It uses the modulo (%) operator to return the remainder after dividing two numbers, which in this example is used to test whether a number is even. This comprehension produces the expected result:

>>> even_squares
[0, 4, 16, 36, 64]

Similar to the first example, this new list comprehension can be converted into an equivalent for loop:

even_squares = []
for x in range(10):
if x % 2 == 0:
even_squares.append(x * x)

Now let’s try to infer a transformation pattern from this list comprehension and the corresponding for loop again. This time, we need to add a filter to the template to determine the values that the output list will contain. Here’s the modified list comprehension template:

values = [expression
for item in collection
if condition]

Similarly, this list comprehension can be converted to a for loop of the following pattern:

values = []
for item in collection:
if condition:
values.append(expression)

This conversion is also simple, requiring only a slight refinement of the previous fixed pattern. Hopefully, this approach will help demystify list comprehensions. List comprehensions are a useful tool that every Python programmer should master.
Before continuing, it’s important to note that Python not only supports list comprehensions but also has similar syntactic sugar for collections and dictionaries.
The following is a set comprehension:

>>> { x * x for x in range(-9, 10) }
set([64, 1, 36, 0, 49, 9, 16, 81, 25, 4])

Lists preserve the order of their elements, but Python sets are unordered. Therefore, when elements are added to a set container, the order is random.
Here are dictionary comprehensions:

>>> { x: x * x for x in range(5) }
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

Both comprehensions are useful in practice. However, there’s a caveat with Python comprehensions: once you become familiar with them, it’s easy to end up with code that’s hard to read. If you’re not careful, you might end up with a lot of hard-to-understand list, set, and dictionary comprehensions. Too much of a good thing is often counterproductive.

After much frustration, I’ve limited my comprehensions to one level of nesting. In most cases, multiple levels of nesting are better handled directly with for loops, which are much easier to read and maintain.

Key Points

• Comprehensions are a key feature of Python. Understanding and applying comprehensions makes your code feel more Pythonic.

• Comprehensions are just fancy syntactic sugar for the simple for loop pattern. Once you understand the pattern, you’ll have an intuitive understanding of comprehensions.

• In addition to list comprehensions, there are also set comprehensions and dictionary comprehensions.