From PEP 484 to PEP 698 - tracing the evolution of Python's typing features

Direct answer

Python typing matured from optional hints into a production reliability tool for large SaaS codebases. Use it to reduce runtime defects and speed onboarding. For Python/Django delivery, see Python / Django expertise.

Introduction

Python, a dynamically typed language, has undergone notable changes in its typing syntax in recent versions, with the introduction of PEP 484 and the subsequent enhancements in PEP 585 and PEP 586. One of the key features that emerged from these changes is the support for generics, allowing developers to write more expressive and type-safe code. PEP 695 and PEP 698, introduced with the Python 3.12 release, provide new changes to the typing syntax.

This article will delve into these advancements, exploring topics such as generic functions, generics with classes and bounds, constraints, variance, TypeVarTuple, ParamSpec, and addressing incompatibilities introduced by these updates. Let's navigate through the evolving landscape of the Python typing system.

Generic functions

A generic function in Python allows for creating versatile and reusable code by accommodating various data types. Unlike traditional functions designed for specific types, generics, introduced through PEP 484 and further refined in subsequent PEPs, enable the creation of functions that can work with diverse data types. This flexibility reduces redundancy and promotes code abstraction, making it particularly useful when applying the same logic to different data types.

In the reverse_list function example, the function reverses a list of any type T, maintaining the type information in the return value as well.

from typing import TypeVar

T = TypeVar('T')

def reverse_list(input_list: list[T]) -> list[T]:
    return input_list[::-1]

string_list = ["apple", "banana", "orange"]
reversed_string_list = reverse_list(string_list)

New syntax:

def reverse_list[T](input_list: list[T]) -> list[T]:
    return input_list[::-1]

string_list = ["apple", "banana", "orange"]
reversed_string_list = reverse_list(string_list)

The old syntax employed a TypeVar to introduce the generic type T. In the new syntax, we directly specify the generic type T within square brackets. In the context of a function, [T] denotes a type variable within the function's scope, marking a departure from the previous module-scoped behavior of TypeVars.

Generics with Classes and Bounds

Generic classes in Python provide a powerful mechanism for creating versatile and reusable code that can operate on various data types. Similarly to generic functions, generic classes allow developers to define classes that can work with a range of data types. This flexibility is especially valuable when building components that need to maintain type information while providing a generic interface.

In this illustration, we've crafted a generic repository, BaseRepository, tailored for database operations. The repository showcases the application of bounded generics (MODEL), imposing type constraints on valid model classes. This bounded generics approach bolsters type safety by explicitly specifying valid model schemas.

Old syntax:

import abc
from typing import Type, Generic, TypeVar

from sqlmodel import SQLModel, select

from sqlalchemy.ext.asyncio import AsyncSession



MODEL = TypeVar("MODEL", bound=SQLModel)


class BaseRepository(Generic[MODEL], metaclass=abc.ABCMeta):
    def __init__(self, db_session: AsyncSession):
        self._db_session = db_session

    @property
    @abc.abstractmethod
     def _model(self) -> Type[MODEL]:
         raise NotImplementedError

    async def get_all(self) -> list[MODEL]:
        result = await self._db_session.execute(select(self._model))
        return result.scalars().all()

New syntax:‍‍

import abc
from typing import Type, Generic, TypeVar

from sqlmodel import SQLModel, select

from sqlalchemy.ext.asyncio import AsyncSession



class BaseRepository[MODEL: SQLModel](metaclass=abc.ABCMeta):
    def __init__(self, db_session: AsyncSession):
        self._db_session = db_session

    @property
    @abc.abstractmethod
    def _model(self) -> Type[MODEL]:
        raise NotImplementedError

    async def get_all(self) -> list[MODEL]:
        result = await self._db_session.execute(select(self._model))
        return result.scalars().all()

Type aliases now undergo lazy evaluation, eliminating the need for quotes and allowing for the use of forward references.

Crucially, [T] type variables within classes and functions are distinct, a deliberate change from the past. This intentional separation ensures that type variables are specific to the functions or classes in which they are defined.

Constraints

There has been a shift in how constraints are expressed, offering a more streamlined syntax while preserving the fundamental concept of defining constraints as choices. In the old syntax, TypeVar was employed to declare constraints explicitly:‍

T = TypeVar('T', str, bytes)

def process_data(t: T) -> T:
    # Example processing logic
    return t

This example stipulates that the type variable T must be either a string or bytes. The new syntax simplifies this with a more concise form:

def process_data[T: (str, bytes)](t: T) -> T:
    # Example processing logic
    return t

Here, the constraints (str, bytes) are enclosed within square brackets, illustrating the choices available for the type variable T. It's important to note that the tuple-defining constraints must be inside the square brackets and cannot reference a tuple-defined outside.

Variance

Variance refers to how the subtyping relationship between types is preserved in the presence of generic types. In simpler terms, it defines how the type of a container (like a list or a class) behaves when its elements or attributes have subtyping relationships. There are two main types of variance:

Covariance: it allows the use of a more specific type in place of a more general type. For example, if Dog is a subtype of Animal, a covariant container of Dog can be used where a container of Animal is expected.
Contravariance: it allows the use of a more general type in place of a more specific type. In other words, a contravariant container of Animal can be used where a container of Dog is expected.

Variance in Python's typing system has undergone changes, particularly in the syntax for specifying covariant or contravariant behavior for type variables. In the old syntax, you could explicitly define variance using TypeVar('U', covariant=True).

However, there's no direct way to specify variance in the new syntax. Instead, the type checker now infers variance by examining the entire scope in which the type variable is defined. The benefit of this approach lies in the strict scoping, allowing the type checker to automatically determine the variance. This enhancement also comes with backward compatibility, as it has been backported to the old syntax. You can use TypeVar('U', infer_variance=True) to instruct the type checker to deduce the variance automatically.

TypeVarTuple

TypeVarTuples in Python provide a concise and expressive syntax for functions that need to handle a variable number of type arguments, offering flexibility in scenarios where the exact number of types is uncertain or varies. The old syntax, utilizing TypeVarTuple, is now replaced. Previously, you would define a TypeVar tuple like this:

from typing import TypeVarTuple

Ts = TypeVarTuple('Ts')

def concatenate(*args: Ts) -> str:
    return “”.join(map(str, args))

result = concatenate('hello', 42, [1, 2, 3])
print(result)

However, in the new syntax, the entire tuple syntax is simplified, making it more straightforward:

def concatenate[*Ts](*args: Ts) -> str:
    return “”.join(map(str, args))

result = concatenate('hello', 42, [1, 2, 3])
print(result)

In this example, the concatenate function now uses the new syntax for TypeVarTuple, represented as [*Ts], to accept a variable number of arguments of any types. The function then concatenates these arguments into a string.

ParamSpec

ParamSpec is a feature particularly useful for describing the parameters of decorators, wrappers, or functions that call other functions. The old syntax involving ParamSpec and TypeVar has been replaced with a more concise and expressive representation. In this scenario, let's consider a decorator that logs information about function calls.

Old syntax:

from typing import ParamSpec, TypeVar, Callable
import functools

P = ParamSpec('P')
R = TypeVar('R')

def log_call(f: Callable[P, R]) -> Callable[P, R]:
    @functools.wraps(f)
    def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
        result = f(*args, **kwargs)
        print(f"Function {f.__name__} called with args: {args}, kwargs: {kwargs}, and returned: {result}")
        return result
    return wrapper

# Example usage:
@log_call
def add_numbers(a: int, b: int) -> int:
    return a + b

result = add_numbers(3, 5)

New syntax:

def log_call[R, **P](f: Callable[P, R]) -> Callable[P, R]:
    @functools.wraps(f)
    def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
        result = f(*args, **kwargs)
        print(f"Function {f.__name__} called with args: {args}, kwargs: {kwargs}, and returned: {result}")
        return result
    return wrapper

# Example usage:
@log_call
def add_numbers2(a: int, b: int) -> int:
    return a + b

result = add_numbers2(3, 5)

In this comparison, the old syntax utilizes ParamSpec and TypeVar for capturing the parameter specifications of a callable f, while the new syntax integrates ParamSpec directly into the function signature.

Incompatibilities

One notable incompatibility arises from the shift to lazy evaluation, where PEP 695 ensures that type parameters and aliases are always lazily evaluated. Type variables no longer introduce names globally, resulting in the intentional removal of the shareability of type variables. Each type variable is now locally scoped, enhancing clarity and preventing unintended sharing.

Another significant change involves the behavior of type aliases. In the old syntax, type aliases could be directly used with isinstance, providing a convenient way to check types. However, the new type aliases exhibit a different behavior. Attempting to use them with isinstance leads to a TypeError, signaling a departure from the previous compatibility:

Old syntax:

from typing import TypeAlias

OldNumberType: TypeAlias = int | float
print(isinstance(1, OldNumberType)) # True
print(isinstance('str', OldNumberType)) # False

New syntax:

type NewNumberTypes = int | float
print(isinstance(1, NewNumberTypes)) # TypeError: isinstance() arg 2 must be a type, a tuple of types, or a union

Summary

Python's typing system has evolved through PEP 484, PEP 585, and PEP 586, introducing generics for expressive and type-safe code. The recent changes in Python 3.12, with PEP 695 and PEP 698, further refine typing syntax.

Key highlights include a streamlined syntax for generic functions and classes, emphasizing versatility and reusability. Bounded generics enforce type constraints, enhancing type safety.

Constraints, variance, TypeVar tuples, and ParamSpec bring concise and expressive syntax for handling various typing scenarios. Notable incompatibilities, like lazy evaluation of type parameters and changes in type alias behavior, ensure local scoping and clarity.

In summary, these updates contribute to a more flexible, concise, and type-safe Python, aligning with its commitment to modern development practices.

Typed Python in production SaaS

We ship multi-tenant products with senior typed Python. See Python and Django expertise and our open-source SaaS Boilerplate.

Contact

Company Information

Follow Us

From PEP 484 to PEP 698: Tracing the evolution of Python's typing features

Direct answer

Introduction

Generic functions

Generics with Classes and Bounds

Constraints

Variance

TypeVarTuple

ParamSpec

Incompatibilities

Summary

Typed Python in production SaaS

Typed Python that survives production

Ship with senior Python engineers