NEP 26 — Summary of missing data NEPs and discussion#

The NumPy developers’ problem#

For this discussion, “missing data” means array elements which can be indexed (e.g. A[3] in an array A with shape (5,)), but have, in some sense, no value.

It does not refer to compressed or sparse storage techniques where the value for A[3] is not actually stored in memory, but still has a well-defined value like 0.

This is still vague, and to create an actual implementation, it is necessary to answer such questions as:

• What values are computed when doing element-wise ufuncs.

• What values are computed when doing reductions.

• Whether the storage for an element gets overwritten when marking that value missing.

• Whether computations resulting in NaN automatically treat in the same way as a missing value.

• Whether one interacts with missing values using a placeholder object (e.g. called “NA” or “masked”), or through a separate boolean array.

• Whether there is such a thing as an array object that cannot hold missing array elements.

• How the (C and Python) API is expressed, in terms of dtypes, masks, and other constructs.

• If we decide to answer some of these questions in multiple ways, then that creates the question of whether that requires multiple systems, and if so how they should interact.

There’s clearly a very large space of missing-data APIs that could be implemented. There is likely at least one user, somewhere, who would find any possible implementation to be just the thing they need to solve some problem. On the other hand, much of NumPy’s power and clarity comes from having a small number of orthogonal concepts, such as strided arrays, flexible indexing, broadcasting, and ufuncs, and we’d like to preserve that simplicity.

There has been dissatisfaction among several major groups of NumPy users about the existing status quo of missing data support. In particular, neither the numpy.ma component nor use of floating-point NaNs as a missing data signal fully satisfy the performance requirements and ease of use for these users. The example of R, where missing data is treated via an NA placeholder and is deeply integrated into all computation, is where many of these users point to indicate what functionality they would like. Doing a deep integration of missing data like in R must be considered carefully, it must be clear it is not being done in a way which sacrifices existing performance or functionality.

Our problem is, how can we choose some incremental additions to NumPy that will make a large class of users happy, be reasonably elegant, complement the existing design, and that we’re comfortable we won’t regret being stuck with in the long term.

Prior art#

So a major (maybe the major) problem is figuring out how ambitious the project to add missing data support to NumPy should be, and which kinds of problems are in scope. Let’s start with the best understood situation where “missing data” comes into play:

Person | Height | Age | Income
------------------------------
   1   |   63   | 25  | 15000
   2   |   58   | 32  | <missing>
   3   |   71   | 45  | 30000

Semantics, storage, API, oh my!#

We think it’s useful to draw a clear line between use cases, semantics, and storage. Use cases are situations that users encounter, regardless of what NumPy does; they’re the focus of the previous section. When we say semantics, we mean the result of different operations as viewed from the Python level without regard to the underlying implementation.

NA semantics are the ones described above and used by R:

1 + NA = NA
sum([1, 2, NA]) = NA
NA | False = NA
NA | True = True

With na.rm=TRUE or skipNA=True, this switches to:

1 + NA = illegal # in R, only reductions take na.rm argument
sum([1, 2, NA], skipNA=True) = 3

There’s also been discussion of what we’ll call ignore semantics. These are somewhat underdefined:

sum([1, 2, IGNORED]) = 3
# Several options here:
1 + IGNORED = 1
#  or
1 + IGNORED = <leaves output array untouched>
#  or
1 + IGNORED = IGNORED

The numpy.ma semantics are:

sum([1, 2, masked]) = 3
1 + masked = masked

If either NA or ignore semantics are implemented with masks, then there is a choice of what should be done to the value in the storage for an array element which gets assigned a missing value. Three possibilities are:

• Leave that memory untouched (the choice made in the NEP).

• Do the calculation with the values independently of the mask (perhaps the most useful option for Paul Hobson’s use-case above).

• Copy whatever value is stored behind the input missing value into the output (this is what numpy.ma does. Even that is ambiguous in the case of masked + masked – in this case numpy.ma copies the value stored behind the leftmost masked value).

When we talk about storage, we mean the debate about whether missing values should be represented by designating a particular value of the underlying data-type (the bitpattern dtype option, as used in R), or by using a separate mask stored alongside the data itself.

For mask-based storage, there is also an important question about what the API looks like for accessing the mask, modifying the mask, and “peeking behind” the mask.

Designs that have been proposed#

One option is to just copy R, by implementing a mechanism whereby dtypes can arrange for certain bitpatterns to be given NA semantics.

One option is to copy numpy.ma closely, but with a more optimized implementation. (Or to simply optimize the existing implementation.)

One option is that described in NEP12, for which an implementation of mask-based missing data exists. This system is roughly:

• There is both bitpattern and mask-based missing data, and both have identical interoperable NA semantics.

• Masks are modified by assigning np.NA or values to array elements. The way to peek behind the mask or to unmask values is to keep a view of the array that shares the data pointer but not the mask pointer.

• Mark would like to add a way to access and manipulate the mask more directly, to be used in addition to this view-based API.

• If an array has both a bitpattern dtype and a mask, then assigning np.NA writes to the mask, rather than to the array itself. Writing a bitpattern NA to an array which supports both requires accessing the data by “peeking under the mask”.

Another option is that described in NEP24, which is to implement bitpattern dtypes with NA semantics for the “statistical missing data” use case, and to also implement a totally independent API for masked arrays with ignore semantics and all mask manipulation done explicitly through a .mask attribute.

Another option would be to define a minimalist aligned array container that holds multiple arrays and that can be used to pass them around together. It would support indexing (to help with the common problem of wanting to subset several arrays together without their becoming unaligned), but all arithmetic etc. would be done by accessing the underlying arrays directly via attributes. The “prior art” discussion above suggests that something like this holding a .data and a .mask array might actually be solve a number of people’s problems without requiring any major architectural changes to NumPy. This is similar to a structured array, but with each field in a separately stored array instead of packed together.

Several people have suggested that there should be a single system that has multiple missing values that each have different semantics, e.g., a MISSING value that has NA semantics, and a separate IGNORED value that has ignored semantics.

None of these options are necessarily exclusive.

The debate#

We both are dubious of using ignored semantics as a default missing data behavior. Nathaniel likes NA semantics because he is most interested in the “statistical missing data” use case, and NA semantics are exactly right for that. Mark isn’t as interested in that use case in particular, but he likes the NA computational abstraction because it is unambiguous and well-defined in all cases, and has a lot of existing experience to draw from.

What Nathaniel thinks, overall:

• The “statistical missing data” use case is clear and compelling; the other use cases certainly deserve our attention, but it’s hard to say what they are exactly yet, or even if the best way to support them is by extending the ndarray object.

• The “statistical missing data” use case is best served by an R-style system that uses bitpattern storage to implement NA semantics. The main advantage of bitpattern storage for this use case is that it avoids the extra memory and speed overhead of storing and checking a mask (especially for the common case of floating point data, where some tricks with NaNs allow us to effectively hardware-accelerate most NA operations). These concerns alone appears to make a mask-based implementation unacceptable to many NA users, particularly in areas like neuroscience (where memory is tight) or financial modeling (where milliseconds are critical). In addition, the bit-pattern approach is less confusing conceptually (e.g., assignment really is just assignment, no magic going on behind the curtain), and it’s possible to have in-memory compatibility with R for inter-language calls via rpy2. The main disadvantage of the bitpattern approach is the need to give up a value to represent NA, but this is not an issue for the most important data types (float, bool, strings, enums, objects); really, only integers are affected. And even for integers, giving up a value doesn’t really matter for statistical problems. (Occupy Wall Street notwithstanding, no-one’s income is 2**63 - 1. And if it were, we’d be switching to floats anyway to avoid overflow.)

• Adding new dtypes requires some cooperation with the ufunc and casting machinery, but doesn’t require any architectural changes or violations of NumPy’s current orthogonality.

• His impression from the mailing list discussion, esp. the “what can we agree on?” thread, is that many numpy.ma users specifically like the combination of masked storage, the mask being easily accessible through the API, and ignored semantics. He could be wrong, of course. But he cannot remember seeing anybody besides Mark advocate for the specific combination of masked storage and NA semantics, which makes him nervous.

• Also, he personally is not very happy with the idea of having two storage implementations that are almost-but-not-quite identical at the Python level. While there likely are people who would like to temporarily pretend that certain data is “statistically missing data” without making a copy of their array, it’s not at all clear that they outnumber the people who would like to use bitpatterns and masks simultaneously for distinct purposes. And honestly he’d like to be able to just ignore masks if he wants and stick to bitpatterns, which isn’t possible if they’re coupled together tightly in the API. So he would say the jury is still very much out on whether this aspect of the NEP design is an advantage or a disadvantage. (Certainly he’s never heard of any R users complaining that they really wish they had an option of making a different trade-off here.)

• R’s NA support is a headline feature and its target audience consider it a compelling advantage over other platforms like Matlab or Python. Working with statistical missing data is very painful without platform support.

• By comparison, we clearly have much more uncertainty about the use cases that require a mask-based implementation, and it doesn’t seem like people will suffer too badly if they are forced for now to settle for using NumPy’s excellent mask-based indexing, the new where= support, and even numpy.ma.

• Therefore, bitpatterns with NA semantics seem to meet the criteria of making a large class of users happy, in an elegant way, that fits into the original design, and where we can have reasonable certainty that we understand the problem and use cases well enough that we’ll be happy with them in the long run. But no mask-based storage proposal does, yet.

What Mark thinks, overall:

• The idea of using NA semantics by default for missing data, inspired by the “statistical missing data” problem, is better than all the other default behaviors which were considered. This applies equally to the bitpattern and the masked approach.

• For NA-style functionality to get proper support by all NumPy features and eventually all third-party libraries, it needs to be in the core. How to correctly and efficiently handle missing data differs by algorithm, and if thinking about it is required to fully support NumPy, NA support will be broader and higher quality.

• At the same time, providing two different missing data interfaces, one for masks and one for bitpatterns, requires NumPy developers and third-party NumPy plugin developers to separately consider the question of what to do in either case, and do two additional implementations of their code. This complicates their job, and could lead to inconsistent support for missing data.

• Providing the ability to work with both masks and bitpatterns through the same C and Python programming interface makes missing data support cleanly orthogonal with all other NumPy features.

• There are many trade-offs of memory usage, performance, correctness, and flexibility between masks and bitpatterns. Providing support for both approaches allows users of NumPy to choose the approach which is most compatible with their way of thinking, or has characteristics which best match their use-case. Providing them through the same interface further allows them to try both with minimal effort, and choose the one which performs better or uses the least memory for their programs.

• Memory Usage

  • With bitpatterns, less memory is used for storing a single array containing some NAs.

  • With masks, less memory is used for storing multiple arrays that are identical except for the location of their NAs. (In this case a single data array can be re-used with multiple mask arrays; bitpattern NAs would need to copy the whole data array.)

• Performance

  • With bitpatterns, the floating point type can use native hardware operations, with nearly correct behavior. For fully correct floating point behavior and with other types, code must be written which specially tests for equality with the missing-data bitpattern.

  • With masks, there is always the overhead of accessing mask memory and testing its truth value. The implementation that currently exists has no performance tuning, so it is only good to judge a minimum performance level. Optimal mask-based code is in general going to be slower than optimal bitpattern-based code.

• Correctness

  • Bitpattern integer types must sacrifice a valid value to represent NA. For larger integer types, there are arguments that this is ok, but for 8-bit types there is no reasonable choice. In the floating point case, if the performance of native floating point operations is chosen, there is a small inconsistency that NaN+NA and NA+NaN are different.

  • With masks, it works correctly in all cases.

• Generality

  • The bitpattern approach can work in a fully general way only when there is a specific value which can be given up from the data type. For IEEE floating point, a NaN is an obvious choice, and for booleans represented as a byte, there are plenty of choices. For integers, a valid value must be sacrificed to use this approach. Third-party dtypes which plug into NumPy will also have to make a bitpattern choice to support this system, something which may not always be possible.

  • The mask approach works universally with all data types.

Recommendations for moving forward#

Nathaniel thinks we should:

• Go ahead and implement bitpattern NAs.

• Don’t implement masked arrays in the core – or at least, not yet. Instead, we should focus on figuring out how to implement them out-of-core, so that people can try out different approaches without us committing to any one approach. And so new prototypes can be released more quickly than the NumPy release cycle. And anyway, we’re going to have to figure out how to experiment with such changes out-of-core if NumPy is to continue to evolve without forking – might as well do it now. The existing code can live in the main branch, be disabled, or live its own branch – it’ll still be there once we know what we’re doing.

Mark thinks we should:

• The existing code should remain as is, with a global run-time experimental flag added which disables NA support by default.

A more detailed rationale for this recommendation is:

• A solid preliminary NA-mask implementation is currently in NumPy main. This implementation has been extensively tested against scipy and other third-party packages, and has been in main in a stable state for a significant amount of time.

• This implementation integrates deeply with the core, providing an interface which is usable in the same way R’s NA support is. It provides a compelling, user-friendly answer to R’s NA support.

• The missing data NEP provides a plan for adding bitpattern-based dtype support of NAs, which will operate through the same interface but allow for the same performance/correctness tradeoffs that R has made.

• Making it very easy for users to try out this implementation, which has reasonable feature coverage and performance characteristics, is the best way to get more concrete feedback about how NumPy’s missing data support should look.

Because of its preliminary state, the existing implementation is marked as experimental in the NumPy documentation. It would be good for this to remain marked as experimental until it is more fleshed out, for example supporting struct and array dtypes and with a fuller set of NumPy operations.

I think the code should stay as it is, except to add a run-time global NumPy flag, perhaps numpy.experimental.maskna, which defaults to False and can be toggled to True. In its default state, any NA feature usage would raise an “ExperimentalError” exception, a measure which would prevent it from being accidentally used and communicate its experimental status very clearly.

The ABI issues seem very tricky to deal with effectively in the 1.x series of releases, but I believe that with proper implementation-hiding in a 2.0 release, evolving the software to support various other ABI ideas that have been discussed is feasible. This is the approach I like best.

Nathaniel notes in response that he doesn’t really have any objection to shipping experimental APIs in the main numpy distribution if we’re careful to make sure that they don’t “leak out” in a way that leaves us stuck with them. And in principle some sort of “this violates your warranty” global flag could be a way to do that. (In fact, this might also be a useful strategy for the kinds of changes that he favors, of adding minimal hooks to enable us to build prototypes more easily – we could have some “rapid prototyping only” hooks that let prototype hacks get deeper access to NumPy’s internals than we were otherwise ready to support.)

But, he wants to point out two things. First, it seems like we still have fundamental questions to answer about the NEP design, like whether masks should have NA semantics or ignore semantics, and there are already plans to majorly change how NEP masks are exposed and accessed. So he isn’t sure what we’ll learn by asking for feedback on the NEP code in its current state.

And second, given the concerns about their causing (minor) ABI issues, it’s not clear that we could really prevent them from leaking out. (He looks forward to 2.0 too, but we’re not there yet.) So maybe it would be better if they weren’t present in the C API at all, and the hoops required for testers were instead something like, ‘we have included a hacky pure-Python prototype accessible by typing “import numpy.experimental.donttrythisathome.NEP” and would welcome feedback’?

If so, then he should mention that he did implement a horribly klugy, pure Python implementation of the NEP API that works with NumPy 1.6.1. This was mostly as an experiment to see how possible such prototyping was and to test out a possible ufunc override mechanism, but if there’s interest, the module is available here: njsmith/numpyNEP

It passes the maskna test-suite, with some minor issues described in a big comment at the top.

Mark responds:

I agree that it’s important to be careful when adding new features to NumPy, but I also believe it is essential that the project have forward development momentum. A project like NumPy requires developers to write code for advancement to occur, and obstacles that impede the writing of code discourage existing developers from contributing more, and potentially scare away developers who are thinking about joining in.

All software projects, both open source and closed source, must balance between short-term practicality and long-term planning. In the case of the missing data development, there was a short-term resource commitment to tackle this problem, which is quite immense in scope. If there isn’t a high likelihood of getting a contribution into NumPy that concretely advances towards a solution, I expect that individuals and companies interested in doing such work will have a much harder time justifying a commitment of their resources. For a project which is core to so many other libraries, only relying on the good will of selfless volunteers would mean that NumPy could more easily be overtaken by another project.

In the case of the existing NA contribution at issue, how we resolve this disagreement represents a decision about how NumPy’s developers, contributors, and users should interact. If we create a document describing a dispute resolution process, how do we design it so that it doesn’t introduce a large burden and excessive uncertainty on developers that could prevent them from productively contributing code?

If we go this route of writing up a decision process which includes such a dispute resolution mechanism, I think the meat of it should be a roadmap that potential contributors and developers can follow to gain influence over NumPy. NumPy development needs broad support beyond code contributions, and tying influence in the project to contributions seems to me like it would be a good way to encourage people to take on tasks like bug triaging/management, continuous integration/build server administration, and the myriad other tasks that help satisfy the project’s needs. No specific meritocratic, democratic, consensus-striving system will satisfy everyone, but the vigour of the discussions around governance and process indicate that something at least a little bit more formal than the current status quo is necessary.

In conclusion, I would like the NumPy project to prioritize movement towards a more flexible and modular ABI/API, balanced with strong backwards-compatibility constraints and feature additions that individuals, universities, and companies want to contribute. I do not believe keeping the NA code in 1.7 as it is, with the small additional measure of requiring it to be enabled by an experimental flag, poses a risk of long-term ABI troubles. The greater risk I see is a continuing lack of developers contributing to the project, and I believe backing out this code because these worries would create a risk of reducing developer contribution.