NEP 52 — Python API cleanup for NumPy 2.0#

Abstract#

We propose to clean up NumPy’s Python API for the NumPy 2.0 release. This includes a more clearly defined split between what is public and what is private, and reducing the size of the main namespace by removing aliases and functions that have better alternatives. Furthermore, each function is meant to be accessible from only one place, so all duplicates also need to be dropped.

Motivation and scope#

NumPy has a large API surface that evolved organically over many years:

>>> objects_in_api = [s for s in dir(np) if not s.startswith('_')]
>>> len(objects_in_api)
562
>>> modules = [s for s in objects_in_api if inspect.ismodule(eval(f'np.{s}'))]
>>> modules
['char', 'compat', 'ctypeslib', 'emath', 'fft', 'lib', 'linalg', 'ma', 'math', 'polynomial', 'random', 'rec', 'testing', 'version']
>>> len(modules)
14

The above doesn’t even include items that are public but have been hidden from __dir__. A particularly problematic example of that is np.core, which is technically private but heavily used in practice. For a full overview of what’s considered public, private or a bit in between, see numpy/numpy.

The size of the API and the lacking definition of its boundaries incur significant costs:

• Users find it hard to disambiguate between similarly named functions.

  Looking for functions with tab completion in IPython, a notebook, or an IDE is a challenge. E.g., type np.<TAB> and look at the first six items offered: two ufuncs (abs, add), one alias (absolute), and three functions that are not intended for end-users (add_docstring, add_newdoc, add_newdoc_ufunc). As a result, the learning curve for NumPy is steeper than it has to be.

• Libraries that mimic the NumPy API face significant implementation barriers.

  For maintainers of NumPy API-compatible array libraries (Dask, CuPy, JAX, PyTorch, TensorFlow, cuNumeric, etc.) and compilers/transpilers (Numba, Pythran, Cython, etc.) there is an implementation cost to each object in the namespace. In practice, no other library has full support for the entire NumPy API, partly because it is so hard to know what to include when faced with a slew of aliases and legacy objects.

• Teaching NumPy is more complicated than it needs to be.

  Similarly, a larger API is confusing to learners, who not only have to find functions but have to choose which functions to use.

• Developers are hesitant to grow the API surface.

  This happens even when the changes are warranted, because they are aware of the above concerns.

The scope of this NEP includes:

• Deprecating or removing functionality that is too niche for NumPy, not well-designed, superseded by better alternatives, an unnecessary alias, or otherwise a candidate for removal.

• Clearly separating public from private NumPy API by use of underscores.

• Restructuring the NumPy namespaces to be easier to understand and navigate.

Out of scope for this NEP are:

• Introducing new functionality or performance enhancements.

Usage and impact#

A key principle of this API refactor is to ensure that, when code has been adapted to the changes and is 2.0-compatible, that code then also works with NumPy 1.2x.x. This keeps the burden on users and downstream library maintainers low by not having to carry duplicate code which switches on the NumPy major version number.

Backward compatibility#

As mentioned above, while the new (or cleaned up, NumPy 2.0) API should be backward compatible, there is no guarantee of forward compatibility from 1.25.X to 2.0. Code will have to be updated to account for deprecated, moved, or removed functions/classes, as well as for more strictly enforced private APIs.

In order to make it easier to adopt the changes in this NEP, we will:

1. Provide a transition guide that lists each API change and its replacement.

2. Explicitly flag all expired attributes with a meaningful AttributeError that points out to the new place or recommends an alternative.

3. Provide a script to automate the migration wherever possible. This will be similar to tools/replace_old_macros.sed (which adapts code for a previous C API naming scheme change). This will be sed (or equivalent) based rather than attempting AST analysis, so it won’t cover everything.

Detailed description#

# Regular/recommended user-facing namespaces for general use. Present these
# as the primary set of namespaces to the users.
numpy
numpy.exceptions
numpy.fft
numpy.linalg
numpy.polynomial
numpy.random
numpy.testing
numpy.typing

# Special-purpose namespaces. Keep these, but document them in a separate
# grouping in the reference guide and explain their purpose.
numpy.array_api
numpy.ctypeslib
numpy.emath
numpy.f2py  # only a couple of public functions, like `compile` and `get_include`
numpy.lib.stride_tricks
numpy.lib.npyio
numpy.rec
numpy.dtypes
numpy.array_utils

# Legacy (prefer not to use, there are better alternatives and/or this code
# is deprecated or isn't reliable). This will be a third grouping in the
# reference guide; it's still there, but de-emphasized and the problems
# with it or better alternatives are explained in the docs.
numpy.char
numpy.distutils
numpy.ma
numpy.matlib

# To remove
numpy.compat
numpy.core  # rename to _core
numpy.doc
numpy.math
numpy.version  # rename to _version
numpy.matrixlib

# To clean out or somehow deal with: everything in `numpy.lib`

>>> # np.intp is different, but compares equal too
>>> np.int64 == np.int_ == np.dtype('i8') == np.sctypeDict['i8']
True
>>> np.float64 == np.double == np.float_ == np.dtype('f8') == np.sctypeDict['f8']
True
### Really?
>>> np.clongdouble == np.clongfloat == np.longcomplex == np.complex256
True

API changes considered and rejected#

For some functions and submodules it turned out that removing them would cause too much disruption or would require an amount of work disproportional to the actual gain. We arrived at this conclusion for such items:

• Removing business day functions: np.busday_count, np.busday_offset, np.busdaycalendar.

• Removing np.nan* functions and introducing new nan_mode argument to the related base functions.

• Hiding histogram functions in the np.histograms submodule.

• Hiding c_, r_ and s_ in the np.lib.index_tricks submodule.

• Functions that looked niche but are present in the Array API (for example np.can_cast).

• Removing .repeat and .ctypes from ndarray object.

Related work#

A clear split between public and private API was recently established as part of SciPy 1.8.0 (2021), see tracking issue scipy#14360. The results were beneficial, and the impact on users relatively modest.

Implementation#

The implementation has been split over many different PRs, each touching on a single API or a set of related APIs. Here’s a sample of the most impactful PRs:

• gh-24634: Rename numpy/core to numpy/_core

• gh-24357: Cleaning numpy/__init__.py and main namespace - Part 2

• gh-24376: Cleaning numpy/__init__.py and main namespace - Part 3

The complete list of cleanup work done in the 2.0 release can be found by searching a dedicated label:

• Numpy 2.0 API Changes:

Some PRs has already been merged and shipped with the 1.25.0 release. For example, deprecating non-preferred aliases:

• gh-23302: deprecate np.round_; add round/min/max to the docs

• gh-23314: deprecate product/cumproduct/sometrue/alltrue

Hiding or removing objects that are accidentally made public or not even NumPy objects at all:

• gh-21403: remove some names from main numpy namespace

Creation of new namespaces to make it easier to navigate the module structure:

• gh-22644: Add new np.exceptions namespace for errors and warnings

Alternatives#

Discussion#

• gh-23999: Tracking issue for the NEP 52

• gh-24306: Overhaul of the main namespace

• gh-24507: Overhaul of the np.lib namespace

References and footnotes#

Copyright#

This document has been placed in the public domain.