NEP 53 — Evolving the NumPy C-API for NumPy 2.0#

Adding state is out of scope#

New developments such as CPython’s support for subinterpreters and the HPy API may require the NumPy C-API to evolve in a way that may require (or at least prefer) state to be passed in.

As of now, we do not aim to include changes for this here. We cannot expect users to do large code updates to pass in for example an HPy context to many NumPy functions.

While we could introduce a second API for this purpose in NumPy 2.0, we expect that this is unnecessary and that the provisions introduced here:

• the ability to compile with the newest NumPy version but be compatible with older versions,

• and the possibility of updating a numpy2_compat package.

should allow to add such an API also in a minor release.

Backwards compatible builds#

Backwards compatible builds will be described in more details in the documentation. Briefly, we will allow users to use a definition like:

#define NPY_TARGET_VERSION NPY_1_22_API_VERSION

to select the version they wish to compile for (lowest version to be compatible with). By default the backwards compatibility will be such that the resulting binary is compatible with the oldest NumPy version which supports the same version of Python: NumPy 1.19.x was the first to support Python 3.9 and NumPy 1.25 supports Python 3.9 or greater, so NumPy 1.25 defaults to 1.19 compatibility. Thus, users of new API may be required to add the define, but users of who want to be compatible with older versions need not do anything unless they wish to have exceptionally long compatibility.

The API additions in the past years were so limited that such a change should be necessary at most for a hand-full of users worldwide.

This mechanism is much the same as the Python limited API since NumPy’s C-API has a similar need for ABI stability.

Breaking the C-API and changing the ABI#

NumPy has too many functions, many of which are aliases. The following lists examples of things we plan to remove and users will have to adapt to be compatible with NumPy 2.0:

• PyArray_Mean and PyArray_Std are untested implementation similar to arr.mean() and arr.std(). We are planning on removing these as they can be replaced with method calls relatively easily.

• The MapIter set of API functions (and struct) allows to implement advanced indexing like semantics downstream. There was a single historic known user of this (theano) and the use-case would be faster and easier to implement in a different way. The API is complicated, requires reaching deep into NumPy to be useful and its exposure makes the implementation more difficult. Unless new important use cases are found, we propose to remove it.

An example for an ABI change is to change the layout of PyArray_Descr (the struct of np.dtype instances) to allow a larger maximum itemsize and new flags (useful for future custom user DTypes). For this specific change, users who access the structs fields directly will have to change their code. A downstream search shows that this should not be very common, the main impacts are:

• Access of the descr->elsize field (and others) would have to be replaced with a macro’s like PyDataType_ITEMSIZE(descr) (NumPy may include a version check when needed).

• Implementers of user defined dtypes, will have to change a few lines of code and luckily, there are very few of such user defined dtypes. (The details are that we rename the struct to PyArray_DescrProto for the static definition and fetch the actual instance from NumPy explicitly.)

A last example is increasing NPY_MAXDIMS to 64. NPY_MAXDIMS is mainly used to statically allocate scratch space:

func(PyArrayObject *arr) {
    npy_intp shape[NPY_MAXDIMS];
    /* Work with a shape or strides from the array */
}

If NumPy changed it to 64 in a minor release, this would lead to undefined behavior if the code was compiled with NPY_MAXDIMS=32 but a 40 dimensional array is passed in. But the larger value is also a correct maximum on previous versions of NumPy making it generally safe for NumPy 2.0 change. (One can imagine code that wants to know the actual runtime value. We have not seen such code in practice, but it would need to be adjusted.)

Impact on Cython users#

Cython users may use the NumPy C-API via cimport numpy as cnp. Due to the uncertainty of Cython development, there are two scenarios for impact on Cython users.

If Cython 3 can be relied on, Cython users would be impacted less than C-API users, because Cython 3 allows us to hide struct layout changes (i.e. changes to PyArray_Descr). If this is not the case and we must support Cython 0.29.x (which is the historic branch before Cython 3), then Cython users will also have to use a function/macro like PyDataType_ITEMSIZE() (or use the Python object). This is unfortunately less typical in Cython code, but also unlikely to be a common pattern for dtype struct fields/attributes.

A further impact is that some future API additions such as new classes may need to placed in a distinct .pyd file to avoid Cython generating code that would fail on older NumPy versions.

End-user and packaging impact#

Packaging in a way that is compatible with NumPy 2.0 will require a recompilation of downstream libraries that rely on the NumPy C-API. This may take some time, although hopefully the process will start before NumPy 2.0 is itself released.

Further, to allow bigger changes more easily in NumPy 2.0, we expect to create a numpy2_compat package. When a library is build with NumPy 2.0 but wants to support NumPy 1.x it will have to depend on numpy2_compat. End-users should not need to be aware of this dependency and an informative error can be raised when the module is missing.

Some new API can be backported#

One large advantage of allowing users to compile with the newest version of NumPy is that in some cases we will be able to backport new API. Some new API functions can be written in terms of old ones or included directly.

This means that at some new API additions could be made available to downstreams users faster. They would require a new NumPy version for compilation but their wheels can be backwards compatible with earlier versions.

NumPy 2 compatibility and API table changes#

To allow changing the API table, NumPy 2.0 would ship a different table than NumPy 1.x (a table is a list of functions and symbols).

For compatibility we would need to translate the 1.x table to the 2.0 table. This could be done in headers only in theory, but this seems unwieldy. We thus propose to add a numpy2_compat package. This package’s main purpose would be to provide a translation of the 1.x table to the 2.x one in a single place (filling in any necessary blanks).

Introducing this package solves the “transition” issue because it allows a user to:

• Install a SciPy version that is compatible with 2.0 and 1.x

• and keep using NumPy 1.x because of other packages they are using are not yet compatible.

The import of numpy2_compat (and an error when it is missing) will be inserted by the NumPy eaders as part of the import_array() call.