NEP 22 — Duck typing for NumPy arrays – high level overview#

Principle 1: focus on “full” duck arrays, but don’t rule out “partial” duck arrays#

We can distinguish between two classes:

• “full” duck arrays, which aspire to fully implement np.ndarray’s Python-level APIs and work essentially anywhere that np.ndarray works

• “partial” duck arrays, which intentionally implement only a subset of np.ndarray’s API.

Full duck arrays are, well, kind of boring. They have exactly the same semantics as ndarray, with differences being restricted to under-the-hood decisions about how the data is actually stored. The kind of people that are excited about making numpy more extensible are also, unsurprisingly, excited about changing or extending numpy’s semantics. So there’s been a lot of discussion of how to best support partial duck arrays. We’ve been guilty of this ourself.

At this point though, we think the best general strategy is to focus our efforts primarily on supporting full duck arrays, and only worry about partial duck arrays as much as we need to make sure we don’t accidentally rule them out for no reason.

Why focus on full duck arrays? Several reasons:

First, there are lots of very clear use cases. Potential consumers of the full duck array interface include almost every package that uses numpy (scipy, sklearn, astropy, …), and in particular packages that provide array-wrapping-classes that handle multiple types of arrays, such as xarray and dask.array. Potential implementers of the full duck array interface include: distributed arrays, sparse arrays, masked arrays, arrays with units (unless they switch to using dtypes), labeled arrays, and so forth. Clear use cases lead to good and relevant APIs.

Second, the Anna Karenina principle applies here: full duck arrays are all alike, but every partial duck array is partial in its own way:

• xarray.DataArray is mostly a duck array, but has incompatible broadcasting semantics.

• xarray.Dataset wraps multiple arrays in one object; it still implements some array interfaces like __array_ufunc__, but certainly not all of them.

• pandas.Series has methods with similar behavior to numpy, but unique null-skipping behavior.

• scipy’s LinearOperators support matrix multiplication and nothing else

• h5py and similar libraries for accessing array storage have objects that support numpy-like slicing and conversion into a full array, but not computation.

• Some classes may be similar to ndarray, but without supporting the full indexing semantics.

And so forth.

Despite our best attempts, we haven’t found any clear, unique way of slicing up the ndarray API into a hierarchy of related types that captures these distinctions; in fact, it’s unlikely that any single person even understands all the distinctions. And this is important, because we have a lot of APIs that we need to add duck array support to (both in numpy and in all the projects that depend on numpy!). By definition, these already work for ndarray, so hopefully getting them to work for full duck arrays shouldn’t be so hard, since by definition full duck arrays act like ndarray. It’d be very cumbersome to have to go through each function and identify the exact subset of the ndarray API that it needs, then figure out which partial array types can/should support it. Once we have things working for full duck arrays, we can go back later and refine the APIs needed further as needed. Focusing on full duck arrays allows us to start making progress immediately.

In the future, it might be useful to identify specific use cases for duck arrays and standardize narrower interfaces targeted just at those use cases. For example, it might make sense to have a standard “array loader” interface that file access libraries like h5py, netcdf, pydap, zarr, … all implement, to make it easy to switch between these libraries. But that’s something that we can do as we go, and it doesn’t necessarily have to involve the NumPy devs at all. For an example of what this might look like, see the documentation for dask.array.from_array.

Principle 2: take advantage of duck typing#

ndarray has a very large API surface area:

In [1]: len(set(dir(np.ndarray)) - set(dir(object)))
Out[1]: 138

And this is a huge underestimate, because there are also many free-standing functions in NumPy and other libraries which currently use the NumPy C API and thus only work on ndarray objects. In type theory, a type is defined by the operations you can perform on an object; thus, the actual type of ndarray includes not just its methods and attributes, but all of these functions. For duck arrays to be successful, they’ll need to implement a large proportion of the ndarray API – but not all of it. (For example, dask.array.Array does not provide an equivalent to the ndarray.ptp method, presumably because no-one has ever noticed or cared about its absence. But this doesn’t seem to have stopped people from using dask.)

This means that realistically, we can’t hope to define the whole duck array API up front, or that anyone will be able to implement it all in one go; this will be an incremental process. It also means that even the so-called “full” duck array interface is somewhat fuzzily defined at the borders; there are parts of the np.ndarray API that duck arrays won’t have to implement, but we aren’t entirely sure what those are.

And ultimately, it isn’t really up to the NumPy developers to define what does or doesn’t qualify as a duck array. If we want scikit-learn functions to work on dask arrays (for example), then that’s going to require negotiation between those two projects to discover incompatibilities, and when an incompatibility is discovered it will be up to them to negotiate who should change and how. The NumPy project can provide technical tools and general advice to help resolve these disagreements, but we can’t force one group or another to take responsibility for any given bug.

Therefore, even though we’re focusing on “full” duck arrays, we don’t attempt to define a normative “array ABC” – maybe this will be useful someday, but right now, it’s not. And as a convenient side-effect, the lack of a normative definition leaves partial duck arrays room to experiment.

But, we do provide some more detailed advice for duck array implementers and consumers below.

Principle 3: focus on protocols#

Historically, numpy has had lots of success at interoperating with third-party objects by defining protocols, like __array__ (asks an arbitrary object to convert itself into an array), __array_interface__ (a precursor to Python’s buffer protocol), and __array_ufunc__ (allows third-party objects to support ufuncs like np.exp).

NEP 16 took a different approach: we need a duck-array equivalent of asarray, and it proposed to do this by defining a version of asarray that would let through objects which implemented a new AbstractArray ABC. As noted above, we now think that trying to define an ABC is a bad idea for other reasons. But when this NEP was discussed on the mailing list, we realized that even on its own merits, this idea is not so great. A better approach is to define a method that can be called on an arbitrary object to ask it to convert itself into a duck array, and then define a version of asarray that calls this method.

This is strictly more powerful: if an object is already a duck array, it can simply return self. It allows more correct semantics: NEP 16 assumed that asarray(obj, dtype=X) is the same as asarray(obj).astype(X), but this isn’t true. And it supports more use cases: if h5py supported sparse arrays, it might want to provide an object which is not itself a sparse array, but which can be automatically converted into a sparse array. See NEP <XX, to be written> for full details.

The protocol approach is also more consistent with core Python conventions: for example, see the __iter__ method for coercing objects to iterators, or the __index__ protocol for safe integer coercion. And finally, focusing on protocols leaves the door open for partial duck arrays, which can pick and choose which subset of the protocols they want to participate in, each of which have well-defined semantics.

Conclusion: protocols are one honking great idea – let’s do more of those.

Principle 4: reuse existing methods when possible#

It’s tempting to try to define cleaned up versions of ndarray methods with a more minimal interface to allow for easier implementation. For example, __array_reshape__ could drop some of the strange arguments accepted by reshape and __array_basic_getitem__ could drop all the strange edge cases of NumPy’s advanced indexing.

But as discussed above, we don’t really know what APIs we need for duck-typing ndarray. We would inevitably end up with a very long list of new special methods. In contrast, existing methods like reshape and __getitem__ have the advantage of already being widely used/exercised by libraries that use duck arrays, and in practice, any serious duck array type is going to have to implement them anyway.

Principle 5: make it easy to do the right thing#

Making duck arrays work well is going to be a community effort. Documentation helps, but only goes so far. We want to make it easy to implement duck arrays that do the right thing.

One way NumPy can help is by providing mixin classes for implementing large groups of related functionality at once. NDArrayOperatorsMixin is a good example: it allows for implementing arithmetic operators implicitly via the __array_ufunc__ method. It’s not complete, and we’ll want more helpers like that (e.g. for reductions).

(We initially thought that the importance of these mixins might be an argument for providing an array ABC, since that’s the standard way to do mixins in modern Python. But in discussion around NEP 16 we realized that partial duck arrays also wanted to take advantage of these mixins in some cases, so even if we did have an array ABC then the mixins would still need some sort of separate existence. So never mind that argument.)