standOff. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote shown on the same row, so the borrowing can be checked against the thing borrowed from without leaving this page. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.| relation to other work | claim_relations | how this paper stands to another: agrees, contradicts, extends, qualifies |
| claim type | claim_type | the shape of the claim: existence, mechanism, method, critique, prediction, null |
| concept edge | concept_edges | an asserted relation between two concepts, with the sentence asserting it |
| concept | concepts | an idea the paper works with, quoted where the paper uses it |
| content tag | content_tags | a finding, definition, quantity, statistic or variable named in the paper |
| contribution | contribution | what the paper claims to add |
| control | controls | an experiment run to eliminate a rival explanation |
| limitation | limitations | what the paper says its evidence cannot support |
| mechanism | mechanisms | how the reported effect is said to come about |
| method | method | how the paper produced its evidence |
| object of study | object_of_study | what the paper is about — the thing it investigates |
| phenomenon | phenomenon | the effect or regularity under study |
| research question | research_question | the question the paper sets itself |
| sense | senses | a distinct use of a term, quoted where the paper fixes it |
| unrecordable relation | unhoused_relations | a relation the paper asserts that this vocabulary cannot express, with the reason |
Array programming with NumPy
Abstract
object_of_studycontributionclaim_typephenomenonconcepts↗concepts↗concepts↗concepts↗concepts↗concepts↗unhoused_relationssensescontent_tagscontent_tags↗Array programming provides a powerful, compact and expressive syntax for accessing, manipulating and operating on data in vectors, matrices and higher-dimensional arrays. NumPy is the primary array programming library for the Python language. It has an essential role in research analysis pipelines in fields as diverse as physics, chemistry, astronomy, geoscience, biology, psychology, materials science, engineering, finance and economics. For example, in astronomy, NumPy was an important part of the software stack used in the discovery of gravitational waves 1 and in the first imaging of a black hole 2 . Here we review how a few fundamental array concepts lead to a simple and powerful programming paradigm for organizing, exploring and analysing scientific data. NumPy is the foundation upon which the scientific Python ecosystem is constructed. It is so pervasive that several projects, targeting audiences with specialized needs, have developed their own NumPy-like interfaces and array objects. Owing to its central position in the ecosystem, NumPy increasingly acts as an interoperability layer between such array computation libraries and, together with its application programming interface (API), provides a flexible framework to support the next decade of scientific and industrial analysis.
Subject terms: Computational neuroscience, Solar physics, Computational science, Computer science, Software
Main
concepts↗concepts↗concepts↗unhoused_relationsunhoused_relationsunhoused_relationsclaim_relationsclaim_relationsconcept_edgescontent_tagsTwo Python array packages existed before NumPy. The Numeric package was developed in the mid-1990s and provided array objects and array-aware functions in Python. It was written in C and linked to standard fast implementations of linear algebra 3 , 4 . One of its earliest uses was to steer C++ applications for inertial confinement fusion research at Lawrence Livermore National Laboratory 5 . To handle large astronomical images coming from the Hubble Space Telescope, a reimplementation of Numeric, called Numarray, added support for structured arrays, flexible indexing, memory mapping, byte-order variants, more efficient memory use, flexible IEEE 754-standard error-handling capabilities, and better type-casting rules 6 . Although Numarray was highly compatible with Numeric, the two packages had enough differences that it divided the community; however, in 2005 NumPy emerged as a ‘best of both worlds’ unification 7 —combining the features of Numarray with the small-array performance of Numeric and its rich C API.
unhoused_relationssensescontent_tagscontent_tagsNow, 15 years later, NumPy underpins almost every Python library that does scientific or numerical computation 8 – 11 , including SciPy 12 , Matplotlib 13 , pandas 14 , scikit-learn 15 and scikit-image 16 . NumPy is a community-developed, open-source library, which provides a multidimensional Python array object along with array-aware functions that operate on it. Because of its inherent simplicity, the NumPy array is the de facto exchange format for array data in Python.
NumPy operates on in-memory arrays using the central processing unit (CPU). To utilize modern, specialized storage and hardware, there has been a recent proliferation of Python array packages. Unlike with the Numarray–Numeric divide, it is now much harder for these new libraries to fracture the user community—given how much work is already built on top of NumPy. However, to provide the community with access to new and exploratory technologies, NumPy is transitioning into a central coordinating mechanism that specifies a well defined array programming API and dispatches it, as appropriate, to specialized array implementations.
NumPy arrays
mechanisms↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗sensesconcept_edgesconcept_edgesconcept_edgesconcept_edgescontent_tags↗The NumPy array is a data structure that efficiently stores and accesses multidimensional arrays 17 (also known as tensors), and enables a wide variety of scientific computation. It consists of a pointer to memory, along with metadata used to interpret the data stored there, notably ‘data type’, ‘shape’ and ‘strides’ (Fig. 1a ).
The NumPy array incorporates several fundamental array concepts. a , The NumPy array data structure and its associated metadata fields. b , Indexing an array with slices and steps. These operations return a ‘view’ of the original data. c , Indexing an array with masks, scalar coordinates or other arrays, so that it returns a ‘copy’ of the original data. In the bottom example, an array is indexed with other arrays; this broadcasts the indexing arguments before performing the lookup. d , Vectorization efficiently applies operations to groups of elements. e , Broadcasting in the multiplication of two-dimensional arrays. f , Reduction operations act along one or more axes. In this example, an array is summed along select axes to produce a vector, or along two axes consecutively to produce a scalar. g , Example NumPy code, illustrating some of these concepts.
concepts↗content_tags↗content_tags↗The data type describes the nature of elements stored in an array. An array has a single data type, and each element of an array occupies the same number of bytes in memory. Examples of data types include real and complex numbers (of lower and higher precision), strings, timestamps and pointers to Python objects.
concepts↗concepts↗content_tags↗content_tagscontent_tagscontent_tags↗The shape of an array determines the number of elements along each axis, and the number of axes is the dimensionality of the array. For example, a vector of numbers can be stored as a one-dimensional array of shape N , whereas colour videos are four-dimensional arrays of shape ( T , M , N , 3).
mechanisms↗concepts↗concepts↗unhoused_relationscontent_tags↗content_tagscontent_tags↗content_tags↗Strides are necessary to interpret computer memory, which stores elements linearly, as multidimensional arrays. They describe the number of bytes to move forward in memory to jump from row to row, column to column, and so forth. Consider, for example, a two-dimensional array of floating-point numbers with shape (4, 3), where each element occupies 8 bytes in memory. To move between consecutive columns, we need to jump forward 8 bytes in memory, and to access the next row, 3 × 8 = 24 bytes. The strides of that array are therefore (24, 8). NumPy can store arrays in either C or Fortran memory order, iterating first over either rows or columns. This allows external libraries written in those languages to access NumPy array data in memory directly.
method↗concepts↗concepts↗concepts↗Users interact with NumPy arrays using ‘indexing’ (to access subarrays or individual elements), ‘operators’ (for example, +, − and × for vectorized operations and @ for matrix multiplication), as well as ‘array-aware functions’; together, these provide an easily readable, expressive, high-level API for array programming while NumPy deals with the underlying mechanics of making operations fast.
mechanisms↗concepts↗unhoused_relationssensescontent_tagscontent_tags↗Indexing an array returns single elements, subarrays or elements that satisfy a specific condition (Fig. 1b ). Arrays can even be indexed using other arrays (Fig. 1c ). Wherever possible, indexing that retrieves a subarray returns a ‘view’ on the original array such that data are shared between the two arrays. This provides a powerful way to operate on subsets of array data while limiting memory usage.
mechanisms↗concepts↗unhoused_relationssensescontent_tagscontent_tags↗To complement the array syntax, NumPy includes functions that perform vectorized calculations on arrays, including arithmetic, statistics and trigonometry (Fig. 1d ). Vectorization—operating on entire arrays rather than their individual elements—is essential to array programming. This means that operations that would take many tens of lines to express in languages such as C can often be implemented as a single, clear Python expression. This results in concise code and frees users to focus on the details of their analysis, while NumPy handles looping over array elements near-optimally—for example, taking strides into consideration to best utilize the computer’s fast cache memory.
mechanisms↗concepts↗unhoused_relationssensescontent_tagscontent_tags↗When performing a vectorized operation (such as addition) on two arrays with the same shape, it is clear what should happen. Through ‘broadcasting’ NumPy allows the dimensions to differ, and produces results that appeal to intuition. A trivial example is the addition of a scalar value to an array, but broadcasting also generalizes to more complex examples such as scaling each column of an array or generating a grid of coordinates. In broadcasting, one or both arrays are virtually duplicated (that is, without copying any data in memory), so that the shapes of the operands match (Fig. 1d ). Broadcasting is also applied when an array is indexed using arrays of indices (Fig. 1c ).
mechanisms↗concepts↗sensescontent_tags↗content_tags↗Other array-aware functions, such as sum, mean and maximum, perform element-by-element ‘reductions’, aggregating results across one, multiple or all axes of a single array. For example, summing an n -dimensional array over d axes results in an array of dimension n − d (Fig. 1f ).
mechanisms↗concepts↗concepts↗concepts↗unhoused_relationsunhoused_relationsNumPy also includes array-aware functions for creating, reshaping, concatenating and padding arrays; searching, sorting and counting data; and reading and writing files. It provides extensive support for generating pseudorandom numbers, includes an assortment of probability distributions, and performs accelerated linear algebra, using one of several backends such as OpenBLAS 18 , 19 or Intel MKL optimized for the CPUs at hand (see Supplementary Methods for more details).
Altogether, the combination of a simple in-memory array representation, a syntax that closely mimics mathematics, and a variety of array-aware utility functions forms a productive and powerfully expressive array programming language.
Scientific Python ecosystem
concepts↗claim_relationsPython is an open-source, general-purpose interpreted programming language well suited to standard programming tasks such as cleaning data, interacting with web resources and parsing text. Adding fast array operations and linear algebra enables scientists to do all their work within a single programming language—one that has the advantage of being famously easy to learn and teach, as witnessed by its adoption as a primary learning language in many universities.
concept_edgesEven though NumPy is not part of Python’s standard library, it benefits from a good relationship with the Python developers. Over the years, the Python language has added new features and special syntax so that NumPy would have a more succinct and easier-to-read array notation. However, because it is not part of the standard library, NumPy is able to dictate its own release policies and development patterns.
concepts↗concepts↗concepts↗concepts↗sensesclaim_relationscontent_tagsSciPy and Matplotlib are tightly coupled with NumPy in terms of history, development and use. SciPy provides fundamental algorithms for scientific computing, including mathematical, scientific and engineering routines. Matplotlib generates publication-ready figures and visualizations. The combination of NumPy, SciPy and Matplotlib, together with an advanced interactive environment such as IPython 20 or Jupyter 21 , provides a solid foundation for array programming in Python. The scientific Python ecosystem (Fig. 2 ) builds on top of this foundation to provide several, widely used technique-specific libraries 15 , 16 , 22 , that in turn underlie numerous domain-specific projects 23 – 28 . NumPy, at the base of the ecosystem of array-aware libraries, sets documentation standards, provides array testing infrastructure and adds build support for Fortran and other compilers.
NumPy is the base of the scientific Python ecosystem. Essential libraries and projects that depend on NumPy’s API gain access to new array implementations that support NumPy’s array protocols (Fig. 3 ).
Many research groups have designed large, complex scientific libraries that add application-specific functionality to the ecosystem. For example, the eht-imaging library 29 , developed by the Event Horizon Telescope collaboration for radio interferometry imaging, analysis and simulation, relies on many lower-level components of the scientific Python ecosystem. In particular, the EHT collaboration used this library for the first imaging of a black hole. Within eht-imaging, NumPy arrays are used to store and manipulate numerical data at every step in the processing chain: from raw data through calibration and image reconstruction. SciPy supplies tools for general image-processing tasks such as filtering and image alignment, and scikit-image, an image-processing library that extends SciPy, provides higher-level functionality such as edge filters and Hough transforms. The ‘scipy.optimize’ module performs mathematical optimization. NetworkX 22 , a package for complex network analysis, is used to verify image comparison consistency. Astropy 23 , 24 handles standard astronomical file formats and computes time–coordinate transformations. Matplotlib is used to visualize data and to generate the final image of the black hole.
concepts↗The interactive environment created by the array programming foundation and the surrounding ecosystem of tools—inside of IPython or Jupyter—is ideally suited to exploratory data analysis. Users can fluidly inspect, manipulate and visualize their data, and rapidly iterate to refine programming statements. These statements are then stitched together into imperative or functional programs, or notebooks containing both computation and narrative. Scientific computing beyond exploratory work is often done in a text editor or an integrated development environment (IDE) such as Spyder. This rich and productive environment has made Python popular for scientific research.
mechanisms↗controlscontrolsconcepts↗concepts↗concepts↗content_tagsTo complement this facility for exploratory work and rapid prototyping, NumPy has developed a culture of using time-tested software engineering practices to improve collaboration and reduce error 30 . This culture is not only adopted by leaders in the project but also enthusiastically taught to newcomers. The NumPy team was early to adopt distributed revision control and code review to improve collaboration on code, and continuous testing that runs an extensive battery of automated tests for every proposed change to NumPy. The project also has comprehensive, high-quality documentation, integrated with the source code 31 – 33 .
content_tagsThis culture of using best practices for producing reliable scientific software has been adopted by the ecosystem of libraries that build on NumPy. For example, in a recent award given by the Royal Astronomical Society to Astropy, they state: “The Astropy Project has provided hundreds of junior scientists with experience in professional-standard software development practices including use of version control, unit testing, code review and issue tracking procedures. This is a vital skill set for modern researchers that is often missing from formal university education in physics or astronomy” 34 . Community members explicitly work to address this lack of formal education through courses and workshops 35 – 37 .
The recent rapid growth of data science, machine learning and artificial intelligence has further and dramatically boosted the scientific use of Python. Examples of its important applications, such as the eht-imaging library, now exist in almost every discipline in the natural and social sciences. These tools have become the primary software environment in many fields. NumPy and its ecosystem are commonly taught in university courses, boot camps and summer schools, and are the focus of community conferences and workshops worldwide. NumPy and its API have become truly ubiquitous.
Array proliferation and interoperability
NumPy provides in-memory, multidimensional, homogeneously typed (that is, single-pointer and strided) arrays on CPUs. It runs on machines ranging from embedded devices to the world’s largest supercomputers, with performance approaching that of compiled languages. For most its existence, NumPy addressed the vast majority of array computation use cases.
limitationslimitationsconcepts↗concepts↗concepts↗concepts↗content_tagscontent_tags↗However, scientific datasets now routinely exceed the memory capacity of a single machine and may be stored on multiple machines or in the cloud. In addition, the recent need to accelerate deep-learning and artificial intelligence applications has led to the emergence of specialized accelerator hardware, including graphics processing units (GPUs), tensor processing units (TPUs) and field-programmable gate arrays (FPGAs). Owing to its in-memory data model, NumPy is currently unable to directly utilize such storage and specialized hardware. However, both distributed data and also the parallel execution of GPUs, TPUs and FPGAs map well to the paradigm of array programming: therefore leading to a gap between available modern hardware architectures and the tools necessary to leverage their computational power.
concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗unhoused_relationscontent_tags↗content_tags↗content_tags↗The community’s efforts to fill this gap led to a proliferation of new array implementations. For example, each deep-learning framework created its own arrays; the PyTorch 38 , Tensorflow 39 , Apache MXNet 40 and JAX arrays all have the capability to run on CPUs and GPUs in a distributed fashion, using lazy evaluation to allow for additional performance optimizations. SciPy and PyData/Sparse both provide sparse arrays, which typically contain few non-zero values and store only those in memory for efficiency. In addition, there are projects that build on NumPy arrays as data containers, and extend its capabilities. Distributed arrays are made possible that way by Dask, and labelled arrays—referring to dimensions of an array by name rather than by index for clarity, compare x[:, 1] versus x.loc[:, 'time']—by xarray 41 .
limitationslimitationsconcepts↗unhoused_relationsunhoused_relationscontent_tagsSuch libraries often mimic the NumPy API, because this lowers the barrier to entry for newcomers and provides the wider community with a stable array programming interface. This, in turn, prevents disruptive schisms such as the divergence between Numeric and Numarray. But exploring new ways of working with arrays is experimental by nature and, in fact, several promising libraries (such as Theano and Caffe) have already ceased development. And each time that a user decides to try a new technology, they must change import statements and ensure that the new library implements all the parts of the NumPy API they currently use.
research_questionIdeally, operating on specialized arrays using NumPy functions or semantics would simply work, so that users could write code once, and would then benefit from switching between NumPy arrays, GPU arrays, distributed arrays and so forth as appropriate. To support array operations between external array objects, NumPy therefore added the capability to act as a central coordination mechanism with a well specified API (Fig. 2 ).
mechanisms↗mechanisms↗concepts↗concepts↗concepts↗concepts↗concepts↗unhoused_relationsunhoused_relationssensesconcept_edgescontent_tagscontent_tagscontent_tags↗content_tagscontent_tagsTo facilitate this interoperability, NumPy provides ‘protocols’ (or contracts of operation), that allow for specialized arrays to be passed to NumPy functions (Fig. 3 ). NumPy, in turn, dispatches operations to the originating library, as required. Over four hundred of the most popular NumPy functions are supported. The protocols are implemented by widely used libraries such as Dask, CuPy, xarray and PyData/Sparse. Thanks to these developments, users can now, for example, scale their computation from a single machine to distributed systems using Dask. The protocols also compose well, allowing users to redeploy NumPy code at scale on distributed, multi-GPU systems via, for instance, CuPy arrays embedded in Dask arrays. Using NumPy’s high-level API, users can leverage highly parallel code execution on multiple systems with millions of cores, all with minimal code changes 42 .
NumPy’s API and array protocols expose new arrays to the ecosystem. In this example, NumPy’s ‘mean’ function is called on a Dask array. The call succeeds by dispatching to the appropriate library implementation (in this case, Dask) and results in a new Dask array. Compare this code to the example code in Fig. 1g .
These array protocols are now a key feature of NumPy, and are expected to only increase in importance. The NumPy developers—many of whom are authors of this Review—iteratively refine and add protocol designs to improve utility and simplify adoption.
Discussion
NumPy combines the expressive power of array programming, the performance of C, and the readability, usability and versatility of Python in a mature, well tested, well documented and community-developed library. Libraries in the scientific Python ecosystem provide fast implementations of most important algorithms. Where extreme optimization is warranted, compiled languages can be used, such as Cython 43 , Numba 44 and Pythran 45 ; these languages extend Python and transparently accelerate bottlenecks. Owing to NumPy’s simple memory model, it is easy to write low-level, hand-optimized code, usually in C or Fortran, to manipulate NumPy arrays and pass them back to Python. Furthermore, using array protocols, it is possible to utilize the full spectrum of specialized hardware acceleration with minimal changes to existing code.
NumPy was initially developed by students, faculty and researchers to provide an advanced, open-source array programming library for Python, which was free to use and unencumbered by license servers and software protection dongles. There was a sense of building something consequential together for the benefit of many others. Participating in such an endeavour, within a welcoming community of like-minded individuals, held a powerful attraction for many early contributors.
These user–developers frequently had to write code from scratch to solve their own or their colleagues’ problems—often in low-level languages that preceded Python, such as Fortran 46 and C. To them, the advantages of an interactive, high-level array library were evident. The design of this new tool was informed by other powerful interactive programming languages for scientific computing such as Basis 47 – 50 , Yorick 51 , R 52 and APL 53 , as well as commercial languages and environments such as IDL (Interactive Data Language) and MATLAB.
concepts↗What began as an attempt to add an array object to Python became the foundation of a vibrant ecosystem of tools. Now, a large amount of scientific work depends on NumPy being correct, fast and stable. It is no longer a small community project, but core scientific infrastructure.
limitationsconcepts↗The developer culture has matured: although initial development was highly informal, NumPy now has a roadmap and a process for proposing and discussing large changes. The project has formal governance structures and is fiscally sponsored by NumFOCUS, a nonprofit that promotes open practices in research, data and scientific computing. Over the past few years, the project attracted its first funded development, sponsored by the Moore and Sloan Foundations, and received an award as part of the Chan Zuckerberg Initiative’s Essentials of Open Source Software programme. With this funding, the project was (and is) able to have sustained focus over multiple months to implement substantial new features and improvements. That said, the development of NumPy still depends heavily on contributions made by graduate students and researchers in their free time (see Supplementary Methods for more details).
concepts↗concepts↗unhoused_relationsunhoused_relationssensescontent_tagsNumPy is no longer merely the foundational array library underlying the scientific Python ecosystem, but it has become the standard API for tensor computation and a central coordinating mechanism between array types and technologies in Python. Work continues to expand on and improve these interoperability features.
limitationsOver the next decade, NumPy developers will face several challenges. New devices will be developed, and existing specialized hardware will evolve to meet diminishing returns on Moore’s law. There will be more, and a wider variety of, data science practitioners, a large proportion of whom will use NumPy. The scale of scientific data gathering will continue to increase, with the adoption of devices and instruments such as light-sheet microscopes and the Large Synoptic Survey Telescope (LSST) 54 . New generation languages, interpreters and compilers, such as Rust 55 , Julia 56 and LLVM 57 , will create new concepts and data structures, and determine their viability.
limitationslimitationsconcepts↗concepts↗unhoused_relationsconcept_edgescontent_tagscontent_tagsThrough the mechanisms described in this Review, NumPy is poised to embrace such a changing landscape, and to continue playing a leading part in interactive scientific computation, although to do so will require sustained funding from government, academia and industry. But, importantly, for NumPy to meet the needs of the next decade of data science, it will also need a new generation of graduate students and community contributors to drive it forward.
Supplementary information
This file contains Supplementary Methods, including Supplementary Figure 1 and additional references.
Acknowledgements
We thank R. Barnowski, P. Dubois, M. Eickenberg, and P. Greenfield, who suggested text and provided helpful feedback on the manuscript. K.J.M. and S.J.v.d.W. were funded in part by the Gordon and Betty Moore Foundation through grant GBMF3834 and by the Alfred P. Sloan Foundation through grant 2013-10-27 to the University of California, Berkeley. S.J.v.d.W., S.B., M.P. and W.W. were funded in part by the Gordon and Betty Moore Foundation through grant GBMF5447 and by the Alfred P. Sloan Foundation through grant G-2017-9960 to the University of California, Berkeley.
Author contributions
K.J.M. and S.J.v.d.W. composed the manuscript with input from others. S.B., R.G., K.S., W.W., M.B. and T.R. contributed text. All authors contributed substantial code, documentation and/or expertise to the NumPy project. All authors reviewed the manuscript.
Competing interests
The authors declare no competing interests.
Footnotes
Contributor Information
K. Jarrod Millman, Email: millman@berkeley.edu.
Stéfan J. van der Walt, Email: stefanv@berkeley.edu
Ralf Gommers, Email: ralf.gommers@gmail.com.
Supplementary information
is available for this paper at 10.1038/s41586-020-2649-2.
References
Associated Data
Supplementary Materials
This file contains Supplementary Methods, including Supplementary Figure 1 and additional references.
Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; 8 of 8 relations here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.
| facet | tag | field | conf. | whose words |
|---|---|---|---|---|
| object_of_study | NumPy's array concepts, scientific-Python role, interoperability protocols, development model, and future constraints | object_of_study | the paper's words | |
| research_question | how fundamental array concepts and a shared API organize scientific computation while allowing specialized array implementations to interoperate | research_question | the paper's words | |
| contribution | a review of NumPy's array-programming model, ecosystem position, and transition into the standard coordinating API for Python array technologies | contribution | the paper's words | |
| claim_type | existence | claim_type | the paper's words | |
| phenomenon | array programming and interoperable scientific computation in Python | phenomenon | the paper's words | |
| method | system-or-tool-building also in: system-or-tool-building | method | the paper's words | |
| mechanisms | A memory pointer stores the data while type, shape, and strides determine its interpretation. also in: memory pointer and metadata | mechanisms[0] | 0.95 | the tagger's words |
| mechanisms | Byte-step metadata maps linear computer memory onto multidimensional rows and columns. also in: strides | mechanisms[1] | 0.95 | the tagger's words |
| mechanisms | Subarray indexing shares data with the original array to constrain memory use. also in: views | mechanisms[2] | 0.95 | the tagger's words |
| mechanisms | Whole-array expressions delegate element loops and cache-aware execution to NumPy. also in: vectorization | mechanisms[3] | 0.95 | the tagger's words |
| mechanisms | Operands are virtually duplicated until their shapes agree, without copying data. also in: broadcasting | mechanisms[4] | 0.95 | the tagger's words |
| mechanisms | Element-wise operations aggregate across selected axes and lower dimensionality. also in: reductions | mechanisms[5] | 0.95 | the tagger's words |
| mechanisms | OpenBLAS or Intel MKL executes accelerated routines for the available CPU. also in: optimized linear-algebra backends | mechanisms[6] | 0.95 | the tagger's words |
| mechanisms | NumPy accepts specialized arrays and dispatches each operation to the originating library. also in: array protocols | mechanisms[7] | 0.95 | the tagger's words |
| mechanisms | CuPy arrays embedded in Dask arrays combine GPU and distributed execution. also in: protocol composition | mechanisms[8] | 0.95 | the tagger's words |
| mechanisms | Every proposed change runs through an extensive automated test battery. also in: continuous testing | mechanisms[9] | 0.95 | the tagger's words |
| limitations | NumPy's in-memory data model cannot directly use distributed storage, GPUs, TPUs, or FPGAs. | limitations[0] | 0.95 | the tagger's words |
| limitations | Scientific datasets can exceed a single machine's memory and reside across machines or in the cloud. | limitations[1] | 0.95 | the tagger's words |
| limitations | Trying a new array library can require import changes and verification that the needed NumPy API is implemented. | limitations[2] | 0.95 | the tagger's words |
| limitations | Promising specialized array libraries can cease development. | limitations[3] | 0.95 | the tagger's words |
| limitations | NumPy still depends heavily on free-time contributions by graduate students and researchers. | limitations[4] | 0.95 | the tagger's words |
| limitations | New hardware, languages, interpreters, compilers, and expanding data scale will challenge current interfaces. | limitations[5] | 0.95 | the tagger's words |
| limitations | Continuing NumPy's leading role requires sustained government, academic, and industry funding. | limitations[6] | 0.95 | the tagger's words |
| limitations | Meeting the next decade's needs also requires a new generation of graduate students and community contributors. | limitations[7] | 0.95 | the tagger's words |
| controls | extensive automated test battery eliminates: regressions introduced by proposed NumPy changes · outcome: held | controls[0] | 0.95 | the tagger's words |
| controls | distributed revision control and code review eliminates: unreviewed or poorly coordinated project changes · outcome: held | controls[1] | 0.95 | the tagger's words |
| concepts | Array programming also in: Array programming | concepts[0] | 0.95 | the paper's words |
| concepts | NumPy also in: NumPy | concepts[1] | 0.95 | the paper's words |
| concepts | scientific Python ecosystem also in: scientific Python ecosystem | concepts[2] | 0.95 | the paper's words |
| concepts | interoperability layer also in: interoperability layer | concepts[3] | 0.95 | the paper's words |
| concepts | application programming interface also in: application programming interface | concepts[4] | 0.95 | the paper's words |
| concepts | API also in: API | concepts[5] | 0.95 | the paper's words |
| concepts | Numeric also in: Numeric | concepts[6] | 0.95 | the paper's words |
| concepts | Numarray also in: Numarray | concepts[7] | 0.95 | the paper's words |
| concepts | best of both worlds also in: best of both worlds | concepts[8] | 0.95 | the paper's words |
| concepts | NumPy array also in: NumPy array | concepts[9] | 0.95 | the paper's words |
| concepts | data structure also in: data structure | concepts[10] | 0.95 | the paper's words |
| concepts | multidimensional arrays also in: multidimensional arrays | concepts[11] | 0.95 | the paper's words |
| concepts | tensors also in: tensors | concepts[12] | 0.95 | the paper's words |
| concepts | pointer to memory also in: pointer to memory | concepts[13] | 0.95 | the paper's words |
| concepts | metadata also in: metadata | concepts[14] | 0.95 | the paper's words |
| concepts | data type also in: data type | concepts[15] | 0.95 | the paper's words |
| concepts | shape also in: shape | concepts[16] | 0.95 | the paper's words |
| concepts | dimensionality also in: dimensionality | concepts[17] | 0.95 | the paper's words |
| concepts | strides also in: strides | concepts[18] | 0.95 | the paper's words |
| concepts | C or Fortran memory order also in: C or Fortran memory order | concepts[19] | 0.95 | the paper's words |
| concepts | indexing also in: indexing | concepts[20] | 0.95 | the paper's words |
| concepts | operators also in: operators | concepts[21] | 0.95 | the paper's words |
| concepts | array-aware functions also in: array-aware functions | concepts[22] | 0.95 | the paper's words |
| concepts | view also in: view | concepts[23] | 0.95 | the paper's words |
| concepts | vectorization also in: vectorization | concepts[24] | 0.95 | the paper's words |
| concepts | broadcasting also in: broadcasting | concepts[25] | 0.95 | the paper's words |
| concepts | reductions also in: reductions | concepts[26] | 0.95 | the paper's words |
| concepts | pseudorandom numbers also in: pseudorandom numbers | concepts[27] | 0.95 | the paper's words |
| concepts | OpenBLAS also in: OpenBLAS | concepts[28] | 0.95 | the paper's words |
| concepts | Intel MKL also in: Intel MKL | concepts[29] | 0.95 | the paper's words |
| concepts | Python also in: Python | concepts[30] | 0.95 | the paper's words |
| concepts | SciPy also in: SciPy | concepts[31] | 0.95 | the paper's words |
| concepts | Matplotlib also in: Matplotlib | concepts[32] | 0.95 | the paper's words |
| concepts | IPython also in: IPython | concepts[33] | 0.95 | the paper's words |
| concepts | Jupyter also in: Jupyter | concepts[34] | 0.95 | the paper's words |
| concepts | exploratory data analysis also in: exploratory data analysis | concepts[35] | 0.95 | the paper's words |
| concepts | distributed revision control also in: distributed revision control | concepts[36] | 0.95 | the paper's words |
| concepts | code review also in: code review | concepts[37] | 0.95 | the paper's words |
| concepts | continuous testing also in: continuous testing | concepts[38] | 0.95 | the paper's words |
| concepts | in-memory data model also in: in-memory data model | concepts[39] | 0.95 | the paper's words |
| concepts | graphics processing units also in: graphics processing units | concepts[40] | 0.95 | the paper's words |
| concepts | tensor processing units also in: tensor processing units | concepts[41] | 0.95 | the paper's words |
| concepts | field-programmable gate arrays also in: field-programmable gate arrays | concepts[42] | 0.95 | the paper's words |
| concepts | PyTorch also in: PyTorch | concepts[43] | 0.95 | the paper's words |
| concepts | Tensorflow also in: Tensorflow | concepts[44] | 0.95 | the paper's words |
| concepts | JAX also in: JAX | concepts[45] | 0.95 | the paper's words |
| concepts | sparse arrays also in: sparse arrays | concepts[46] | 0.95 | the paper's words |
| concepts | non-zero values also in: non-zero values | concepts[47] | 0.95 | the paper's words |
| concepts | distributed arrays also in: distributed arrays | concepts[48] | 0.95 | the paper's words |
| concepts | labelled arrays also in: labelled arrays | concepts[49] | 0.95 | the paper's words |
| concepts | Dask also in: Dask | concepts[50] | 0.95 | the paper's words |
| concepts | xarray also in: xarray | concepts[51] | 0.95 | the paper's words |
| concepts | stable array programming interface also in: stable array programming interface | concepts[52] | 0.95 | the paper's words |
| concepts | protocols also in: protocols | concepts[53] | 0.95 | the paper's words |
| concepts | contracts of operation also in: contracts of operation | concepts[54] | 0.95 | the paper's words |
| concepts | specialized arrays also in: specialized arrays | concepts[55] | 0.95 | the paper's words |
| concepts | dispatches operations also in: dispatches operations | concepts[56] | 0.95 | the paper's words |
| concepts | CuPy arrays also in: CuPy arrays | concepts[57] | 0.95 | the paper's words |
| concepts | standard API for tensor computation also in: standard API for tensor computation | concepts[58] | 0.95 | the paper's words |
| concepts | central coordinating mechanism also in: central coordinating mechanism | concepts[59] | 0.95 | the paper's words |
| concepts | core scientific infrastructure also in: core scientific infrastructure | concepts[60] | 0.95 | the paper's words |
| concepts | formal governance structures also in: formal governance structures | concepts[61] | 0.95 | the paper's words |
| concepts | sustained funding also in: sustained funding | concepts[62] | 0.95 | the paper's words |
| concepts | community contributors also in: community contributors | concepts[63] | 0.95 | the paper's words |
| unhoused_relations | NumPy → scientific Python ecosystem NumPy underpins the scientific Python ecosystem. | unhoused_relations[0] | 0.95 | the tagger's words |
| unhoused_relations | NumPy → best of both worlds NumPy emerged as a best-of-both-worlds unification of Numarray and Numeric. | unhoused_relations[1] | 0.95 | the tagger's words |
| unhoused_relations | NumPy → Numeric NumPy combines Numeric's small-array performance and rich C API. | unhoused_relations[2] | 0.95 | the tagger's words |
| unhoused_relations | NumPy → Numarray NumPy combines features inherited from Numarray. | unhoused_relations[3] | 0.95 | the tagger's words |
| unhoused_relations | NumPy → Python library NumPy underpins almost every Python library doing scientific or numerical computation. | unhoused_relations[4] | 0.95 | the tagger's words |
| unhoused_relations | view → memory usage Views share source data and limit memory usage. | unhoused_relations[5] | 0.95 | the tagger's words |
| unhoused_relations | vectorization → concise code Vectorization permits concise array expressions while NumPy handles loops. | unhoused_relations[6] | 0.95 | the tagger's words |
| unhoused_relations | broadcasting → data copies Broadcasting matches operand shapes without copying array data. | unhoused_relations[7] | 0.95 | the tagger's words |
| unhoused_relations | NumPy → OpenBLAS NumPy uses OpenBLAS as one accelerated linear-algebra backend. | unhoused_relations[8] | 0.95 | the tagger's words |
| unhoused_relations | NumPy → Intel MKL NumPy uses Intel MKL as one accelerated linear-algebra backend. | unhoused_relations[9] | 0.95 | the tagger's words |
| unhoused_relations | NumPy array → C or Fortran memory order NumPy arrays can use either C or Fortran memory order. | unhoused_relations[10] | 0.95 | the tagger's words |
| unhoused_relations | sparse arrays → non-zero values Sparse arrays typically contain few non-zero values and store only those values for efficiency. | unhoused_relations[11] | 0.95 | the tagger's words |
| unhoused_relations | NumPy API → barrier to entry Mimicking the NumPy API lowers the barrier to entry for specialized libraries. | unhoused_relations[12] | 0.95 | the tagger's words |
| unhoused_relations | stable array programming interface → disruptive schisms A stable NumPy-like interface prevents disruptive community schisms. | unhoused_relations[13] | 0.95 | the tagger's words |
| unhoused_relations | protocols → dispatches operations Protocols cause NumPy to dispatch operations to the originating library. | unhoused_relations[14] | 0.95 | the tagger's words |
| unhoused_relations | protocols → specialized arrays Protocols allow specialized arrays to be passed to NumPy functions. | unhoused_relations[15] | 0.95 | the tagger's words |
| unhoused_relations | NumPy → standard API for tensor computation NumPy has become the standard API for tensor computation. | unhoused_relations[16] | 0.95 | the tagger's words |
| unhoused_relations | NumPy → central coordinating mechanism NumPy has become a central coordinating mechanism between Python array technologies. | unhoused_relations[17] | 0.95 | the tagger's words |
| unhoused_relations | NumPy → community contributors NumPy's future depends on a new generation of community contributors. | unhoused_relations[18] | 0.95 | the tagger's words |
| senses | array programming as compact operations on vectors, matrices, and higher-dimensional arrays | senses[0] | 0.95 | the tagger's words |
| senses | NumPy as both a concrete in-memory array library and the foundation of scientific Python | senses[1] | 0.95 | the tagger's words |
| senses | NumPy array as pointer-backed homogeneous data interpreted by type, shape, and strides | senses[2] | 0.95 | the tagger's words |
| senses | view as a subarray sharing storage with its source rather than copying it | senses[3] | 0.95 | the tagger's words |
| senses | vectorization as whole-array expression with optimized implicit element loops | senses[4] | 0.95 | the tagger's words |
| senses | broadcasting as virtual operand expansion without copying memory | senses[5] | 0.95 | the tagger's words |
| senses | reduction as aggregation across selected axes that lowers array dimensionality | senses[6] | 0.95 | the tagger's words |
| senses | scientific Python ecosystem as layers of foundation, technique-specific, and domain-specific libraries | senses[7] | 0.95 | the tagger's words |
| senses | protocol as a contract that hands specialized-array operations back to their own library | senses[8] | 0.95 | the tagger's words |
| senses | NumPy as the standard tensor API and coordination layer across Python array technologies | senses[9] | 0.95 | the tagger's words |
| claim_relations | builds-on → Numeric note: NumPy retained Numeric's small-array performance and rich C API. | claim_relations[0] | 0.95 | the tagger's words |
| claim_relations | builds-on → Numarray note: NumPy combined Numarray's feature set with Numeric's strengths. | claim_relations[1] | 0.95 | the tagger's words |
| claim_relations | complements → SciPy and Matplotlib note: The three libraries jointly form the array-programming foundation. | claim_relations[2] | 0.95 | the tagger's words |
| claim_relations | extends → Python note: Fast array operations and linear algebra let scientific work remain in Python. | claim_relations[3] | 0.95 | the tagger's words |
| concept_edges | Numeric —distinct-from→ Numarray the relation itself is separately evidenced · strength of the relation: 0.95 · note: Direct stated difference. · the relation stated: Numeric and Numarray had enough differences to divide the community. | concept_edges[0] | 0.95 | the tagger's words |
| concept_edges | NumPy array —is-a→ data structure the relation itself is separately evidenced · strength of the relation: 1 · note: Explicit categorical definition. · the relation stated: The NumPy array is a data structure. also in: NumPy array · data structure | concept_edges[1] | 0.95 | the tagger's words |
| concept_edges | multidimensional arrays —same-as→ tensors the relation itself is separately evidenced · strength of the relation: 1 · note: Explicit naming equivalence. · the relation stated: Multidimensional arrays are also known as tensors. also in: multidimensional arrays · tensors | concept_edges[2] | 0.95 | the tagger's words |
| concept_edges | pointer to memory —part-of→ NumPy array the relation itself is separately evidenced · strength of the relation: 1 · note: The array explicitly consists of this pointer plus metadata. · the relation stated: A pointer to memory is one component of the NumPy array. also in: pointer to memory · NumPy array | concept_edges[3] | 0.95 | the tagger's words |
| concept_edges | metadata —part-of→ NumPy array the relation itself is separately evidenced · strength of the relation: 1 · note: The array explicitly consists of a pointer plus metadata. · the relation stated: Metadata interpreting stored data is part of the NumPy array. also in: metadata · NumPy array | concept_edges[4] | 0.95 | the tagger's words |
| concept_edges | NumPy —distinct-from→ Python the relation itself is separately evidenced · strength of the relation: 0.95 · note: Explicit non-membership and independent policy difference. · the relation stated: NumPy is not part of Python's standard library and sets its own release policies. | concept_edges[5] | 0.95 | the tagger's words |
| concept_edges | protocols —same-as→ contracts of operation the relation itself is separately evidenced · strength of the relation: 1 · note: Explicit parenthetical equivalence. · the relation stated: Protocols are described parenthetically as contracts of operation. also in: protocols · contracts of operation | concept_edges[6] | 0.95 | the tagger's words |
| concept_edges | sustained funding —causes→ NumPy direction on the measure: increases · the relation itself is separately evidenced · strength of the relation: 0.9 · note: Necessary-resource claim rather than a measured deterministic effect. · the relation stated: Sustained funding is required for NumPy to continue its leading role.
Audit 2026-09-28: DOUBTFUL - read against its span and kept, but NumPy's future REQUIRES sustained funding; causes 'NumPy' drops the object. also in: sustained funding · NumPy | concept_edges[7] | 0.95 | the tagger's words |
| content_tags | NumPy is the primary array-programming library for Python and a foundation of the scientific-Python ecosystem. | content_tags.findings[0] | 0.95 | the tagger's words |
| content_tags | NumPy unified Numarray's features with Numeric's small-array performance and C API. | content_tags.findings[1] | 0.95 | the tagger's words |
| content_tags | The NumPy array has become Python's de facto exchange format for array data. | content_tags.findings[2] | 0.95 | the tagger's words |
| content_tags | Views share data with their source arrays and therefore limit memory usage. | content_tags.findings[3] | 0.95 | the tagger's words |
| content_tags | Vectorization expresses multi-line element loops as concise Python expressions while NumPy performs near-optimal looping. | content_tags.findings[4] | 0.95 | the tagger's words |
| content_tags | Broadcasting virtually duplicates operands so differently shaped arrays can participate in one operation without memory copies. | content_tags.findings[5] | 0.95 | the tagger's words |
| content_tags | NumPy, SciPy, Matplotlib, and an interactive environment form the foundation of array programming in Python. | content_tags.findings[6] | 0.95 | the tagger's words |
| content_tags | Continuous testing, revision control, and code review support collaboration and error reduction. | content_tags.findings[7] | 0.95 | the tagger's words |
| content_tags | NumPy's in-memory CPU model cannot directly use distributed storage or specialized accelerator hardware. | content_tags.findings[8] | 0.95 | the tagger's words |
| content_tags | Specialized libraries often imitate NumPy's API to lower adoption barriers and preserve a stable interface. | content_tags.findings[9] | 0.95 | the tagger's words |
| content_tags | More than four hundred popular NumPy functions participate in protocol-based dispatch. | content_tags.findings[10] | 0.95 | the tagger's words |
| content_tags | Protocol composition permits NumPy code to run at scale on distributed multi-GPU systems with minimal changes. | content_tags.findings[11] | 0.95 | the tagger's words |
| content_tags | NumPy has become both the standard tensor-computation API and a coordinating mechanism among Python array technologies. | content_tags.findings[12] | 0.95 | the tagger's words |
| content_tags | Future viability requires sustained funding and a new generation of contributors. | content_tags.findings[13] | 0.95 | the tagger's words |
| content_tags | compact array-level syntax for accessing, manipulating, and operating on vectors, matrices, and higher-dimensional arrays also in: array programming | content_tags.definitions[0] | 0.95 | the tagger's words |
| content_tags | a data structure that stores and accesses multidimensional arrays through a memory pointer plus data-type, shape, and stride metadata also in: NumPy array | content_tags.definitions[1] | 0.95 | the tagger's words |
| content_tags | metadata describing the nature and fixed byte width of array elements also in: data type | content_tags.definitions[2] | 0.95 | the tagger's words |
| content_tags | the number of elements on each axis, whose axis count determines dimensionality also in: shape | content_tags.definitions[3] | 0.95 | the tagger's words |
| content_tags | byte offsets for moving among rows, columns, and other dimensions in linear memory also in: strides | content_tags.definitions[4] | 0.95 | the tagger's words |
| content_tags | an indexed subarray that shares its source array's data also in: view | content_tags.definitions[5] | 0.95 | the tagger's words |
| content_tags | operating on complete arrays instead of individual elements also in: vectorization | content_tags.definitions[6] | 0.95 | the tagger's words |
| content_tags | virtual duplication that makes operand shapes match without copying data also in: broadcasting | content_tags.definitions[7] | 0.95 | the tagger's words |
| content_tags | element-by-element aggregation across one, several, or all array axes also in: reductions | content_tags.definitions[8] | 0.95 | the tagger's words |
| content_tags | arrays that typically store only their few non-zero values also in: sparse arrays | content_tags.definitions[9] | 0.95 | the tagger's words |
| content_tags | arrays whose dimensions are referred to by name rather than index also in: labelled arrays | content_tags.definitions[10] | 0.95 | the tagger's words |
| content_tags | contracts that pass specialized arrays through NumPy functions and dispatch operations to their originating libraries also in: array protocols | content_tags.definitions[11] | 0.95 | the tagger's words |
| content_tags | 15 years — interval between NumPy's 2005 emergence and this 2020 review | content_tags.quantities[0] | 0.95 | the tagger's words |
| content_tags | one-dimensional — dimensionality of the vector example | content_tags.quantities[1] | 0.95 | the tagger's words |
| content_tags | four-dimensional — dimensionality of the colour-video example | content_tags.quantities[2] | 0.95 | the tagger's words |
| content_tags | (24, 8) — strides for a 4-by-3 array of eight-byte floating-point values | content_tags.quantities[3] | 0.95 | the tagger's words |
| content_tags | over four hundred — popular NumPy functions supported by interoperability protocols | content_tags.quantities[4] | 0.95 | the tagger's words |
| content_tags | millions of cores — scale claimed for protocol-composed parallel execution | content_tags.quantities[5] | 0.95 | the tagger's words |
| content_tags | hundreds of junior scientists — people given professional software-development experience by Astropy | content_tags.quantities[6] | 0.95 | the tagger's words |
| content_tags | next decade — planning horizon for NumPy's changing technical landscape | content_tags.quantities[7] | 0.95 | the tagger's words |
| content_tags | data type also in: data type | content_tags.variables[0] | 0.95 | the tagger's words |
| content_tags | shape also in: shape | content_tags.variables[1] | 0.95 | the tagger's words |
| content_tags | strides also in: strides | content_tags.variables[2] | 0.95 | the tagger's words |
| content_tags | memory order also in: memory order | content_tags.variables[3] | 0.95 | the tagger's words |
| content_tags | axes also in: axes | content_tags.variables[5] | 0.95 | the tagger's words |
| content_tags | non-zero values also in: non-zero values | content_tags.variables[6] | 0.95 | the tagger's words |
| content_tags | specialized hardware also in: specialized hardware | content_tags.variables[7] | 0.95 | the tagger's words |
Tags that could not be placed in the text:
What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.