Functional Programming in Python

Functional programming provides many techniques for creating concise and clear software. While Python is not a purely functional language, it is still possible to use it for functional programming.

Python incorporates many core features of functional programming, allowing us to borrow design patterns and programming techniques from other functional languages to write concise and elegant code. Particularly noteworthy is Python’s generator expressions, which avoid creating large data structures in memory, thus improving execution speed by reducing resource consumption.

Python lacks some language features necessary for creating purely functional programs, such as infinite recursion, lazy evaluation of all expressions, and an optimizing compiler.

Many core elements of functional programming are embodied in Python, such as first-class functions. Python also provides many typical higher-order functions, such as the widely used built-in functions map(), filter(), and functools.reduce(), as well as less obvious functions such as sorted(), min(), and max().

This tutorial uses Python to explain the core concepts of functional programming, aiming to leverage the advantages of functional programming and help you write concise and clear Python programs.

Table of Contents

Functional Programming Overview
Functional programming performs computations by defining and evaluating expressions within functions. It minimizes the complexity introduced by state changes and the use of mutable objects, making programs concise and clear. This chapter introduces some basic functional programming techniques and how to apply them in Python. Finally, we’ll discuss the benefits of functional programming when building Python applications using these design patterns.

Python Object-Oriented Programming
Python Functional Programming Example

Functional Programming Features
Python has most of the functional programming features built in. Writing functional Python code requires avoiding imperative (including procedural and object-oriented) programming techniques as much as possible.

The sample code in this chapter will use Python 3’s type hint syntax. Type hints help developers clarify the core purpose of a function definition. Here, we use the mypy tool to analyze type hints. Similar to pylint, which provides unit testing and static code analysis, mypy is an essential component of building a high-quality software toolchain.
Python Pure Functions
Python Immutable Objects
Python Lazy Evaluation
Python Recursion Instead of a Loop
Python Functional Type System
Functions, Iterators, and Generators
The core of functional programming is the use of pure functions to map values in a domain to a range. Pure functions have no side effects and are easy to implement in Python.
Avoiding side effects means reducing reliance on maintaining computational state through variable assignments. While it’s impossible to completely eliminate assignment statements from Python, we can reduce our reliance on stateful objects. This means choosing data types from Python’s built-in data structures that don’t rely on stateful operations.
Writing Pure Functions in Python
Python Function First-Class Objects
Using Strings in Python
Using Tuples and Named Tuples in Python
Cleaning Raw Data in Python
Using Sets
Python provides many functions for working with collections. These functions can be used on sequences (lists and tuples), sets, mappings, and iterables returned by generator expressions. This chapter examines Python’s collection functions from a functional programming perspective.

Python Function Classification
Python Iterable Object
Python Parsing XML Files
Python Advanced Methods for Parsing Files
Python Grouping sequence elements
Python using the iter() function
Python extended simple loop
Python generator expression
Python any() and all() reduction
Python len() and sum()

Python Structured One-Dimensional Sequence 2
Python reversed() to change the order

Higher-Order Functions

A key feature of the functional programming paradigm is higher-order functions. Higher-order functions are functions that take functions as arguments or return functions as values. This chapter introduces the higher-order functions provided by Python and provides some logical extensions.

Python max() and min() find extreme values
Python anonymous functions
Python map() applies to sets
Python map() processes multiple sequences
Python Higher-Order Mapping and Filtering Functions
Python Unpacking and Mapping Data
Python Packing Multiple Pieces of Data and Mapping
Python Tiling Data and Mapping
Filtering and structuring data in Python
Writing generator functions in Python
Building higher-order functions in Python

Recursion and reduction

Compilers in functional languages optimize recursive functions by converting tail-recursive calls into loops, significantly improving performance. In Python, pure recursion is limited, and tail-call optimization must be performed manually, explicitly converting recursive calls into for loops.

This chapter introduces the reduction algorithms sum(), count(), max(), and min(). It also introduces the collections.Counter() function and the related groupby() function. Parsing (and text scanning) is a type of reduction because they transform sequences of tokens (or characters) into higher-order collections with complex properties.

Python Simple Numerical Recursion
Python Implement Tail Call Optimization
Python Keep Recursive Form
Python Complex Tail Call Optimization
Python Using Recursion to Process Collections
Tail Call Optimization of Python Collections
Reduction and Folding of Python Collections
Python Group-by Reduction
Python Using Counters for Mapping
Tuple Processing Techniques

One advantage of object-oriented programming is that it allows you to incrementally create complex data structures. Sometimes objects are used as caches to store function calculation results, which is very consistent with the idea of functional design patterns. Sometimes object property methods define complex calculations based on object properties to obtain data, which can also be easily converted to functional design.

This chapter will introduce (1) how to create and use named tuples. (2) Using immutable named tuples to replace stateful object class methods. (3) Techniques for writing abstract functions without relying on polymorphic classes. Of course, you can create a polymorphic class hierarchy based on the Callable class, but in some cases, using functional design can avoid this unnecessary overhead.

Python Collecting Data Using Tuples
Python Collecting Data Using Named Tuples
Python Creating Named Tuples
Python Tuple Structures Replace State Classes
Python Assigning Rank Values
Python Packaging Instead of State Changes
Python Multiple Packaging Instead of State Changes
Python Spearman Rank Order Correlation
Python Polymorphism and Type Matching

itertools Module

Functional programming emphasizes the use of stateless objects. In Python, this goal can be achieved by using generator expressions, generator functions, and iterable objects instead of monolithic mutable objects. This chapter introduces how to use the functions in the itertools library to work with iterable collections. This library provides many functions that help you work with iterable sequence objects and collection objects.

Python counting with count()
Python counting with real number arguments
Python iterating with cycle()
Python repeating a single value with repeat()
Python Using Finite Iterators
Python enumerate() to add serial numbers
Python accumulate() to calculate summary values
Python chain() to combine multiple iterators
Python groupby() to shard iterators
Python using compress() to filter
Python using islice() to select a subset
Python dropwhile() and takewhile() filtering states
Python filterfalse() and filter() filtering methods
Python starmap() and map() apply to data
Python tee() clone iterator
Python itertools module code example

Advanced itertools techniques

Functional programming emphasizes statelessness. In Python, this means using generator expressions. This chapter continues with the itertools library functions for working with iterable collections.

Python Cartesian Product
Python Reduce the Product
Python Calculate Distance
Python Get Pixels and Colors
Python Performance Analysis
Python Reconfiguration Problem
Python Merge Two Transformations
Python Arrange Set Elements
Python Generate All Combinations
Python Code Examples

functools Module

Functional programming emphasizes treating functions as first-class objects. Previously, we introduced higher-order functions that take functions as parameters and return values. This chapter introduces several higher-order functions in the functools library for creating and modifying functions.

Python lru_cache saves calculated results
Python total_ordering definition class
Python defines number class
Python uses the partial() function
Python Reduce a dataset with reduce()
Merge map() and reduce() in Python
Use reduce() and partial() in Python
Clean data with Python map() and reduce()
Groupby() and reduce() in Python

Decorator Design Techniques

Use decorator functions to easily create composite functions. A composite function is a single function that represents the functionality of multiple source functions. For a complex algorithm, a composite function f o g(x) is clearer than f(g(x)). It is often helpful for developers to have multiple syntactic alternatives for expressing complex operations.

This chapter discusses the following topics: (1) Using decorators to build functions based on other functions; (2) The wraps() function in the functools module, which can be used to build decorators; and (3) The update_wrapper() function is also useful.

Python Decorators for Higher-Order Functions
Python Crosscutting Concerns
Python Composite Design
Python Adding Parameters to Decorators
Python Complex Decorators
Considerations for Complex Design in Python

Multiprocessing and Threading Modules

By eliminating complex shared state and designing around non-strict execution, you can exploit parallelism to improve performance. This chapter introduces multiprocessing and multithreading techniques. Python libraries are particularly useful for algorithms that allow lazy evaluation.

Python Functional Programming and Concurrency
The Meaning of Python Concurrency
Python Uses Multi-Process Pools and Tasks
Python Handles a Large Number of Large Files
Python Parsing Log Files: Collecting Line Data
Python Parsing Log Files: Named Tuples
Python Parsing Access Objects
Python Filtering Access Details
Python Analyze Access Details
Python Complete Analysis Process
Python Multi-Process Pools for Concurrent Processing

Conditional Expressions and the Operator Module

Python’s if, elif, and else statements impose a strict order on the evaluation of their conditions. This chapter shows how to break free from this strict order and, within a limited scope, write non-strict conditionals. It’s not clear whether this approach will actually help, but it demonstrates how to express algorithms in a more functional way.

Python conditional expression evaluation
Python use non-strict dictionary rules
Python filter True conditional expression
Python looking for matching patterns
PyMonad Library
Monads allow us to specify the evaluation order of expressions in a relaxed manner. For example, we can use monads to require that the expression a+b+c be evaluated strictly from left to right. This can interfere with the compiler’s ability to optimize expression evaluation, but sometimes it’s necessary, such as when reading and writing files in a specific order. In such cases, monads can ensure that the read() and write() functions execute in the specified order.

PyMonad Library Download and Installation
Python Functional Composition and Currying
Python Use of Curried Higher-Order Functions
Python Currying: Easy to Avoid Is Difficult
Python Functional Composition and PyMonad* Operators
Python Functors and Application Functors
Python Monad’s bind() Function and >> Operator
Python Simulated Implementation of Monads

Python HTTP Request-Response Model
Python Injecting State Through Cookies
Python WSGI Standard
Python Throws an Exception During WSGI Processing
Practical WSGI Application in Python
Define a Web Service as a Function in Python
Create a WSGI Application in Python
Get Raw Data in Python
Operating Filters in Python
Python serialized data in JSON or CSV format
Python serialized data in XML
Python serialized data in HTML
Python Tracking Usage
Optimizations and Improvements
In most cases, small changes to a program will only slightly improve performance. Replacing a function with an anonymous function will have minimal impact. If a program is unacceptably slow, you usually need to find a completely new algorithm or data structure. Replacing an O(n^2) algorithm with an O(n log n) algorithm is the best way to improve performance.

Python Memoization and Caching
Python Specified Memoization
Python Tail Recursive Optimization
Python Optimize Storage and Accuracy
Python Chi-square decision
Python Counter filters and counts raw data
Python Read Summary Information
Sum calculation of Python Counter objects
Probability calculation of Python Counter objects
Python calculates expected value
Python Calculates Chi-square-value
Python calculates chi-square-threshold
Python incomplete gamma function
Python Calculate the complete gamma function
Python calculates the probability of random distribution
Python functional programming design pattern