Stephan HoyerpersName, Joe HammanpersName (2017date). “xarray: N-D labeled Arrays and Datasets in Python.” Journal of Open Research Softwaretitle·j.
Object of study — xarrayterm's data model, architecture, capabilities, quality controls, availability, and reuse potential
Research question — how to make labeled N-dimensional scientific arrays support metadata-aware analysis, interoperability, serialization, and out-of-core computationterm
Contribution — the xarrayterm Python packageterm, combining a pandas-inspired API with the self-describing Common Data Modelterm for arbitrary-rank labeled arrays
Cross-disciplinary analysis
content-tags & facets:
phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | metadata-aware manipulation of labeled multidimensional scientific data — “Scientific data is inherently labeled. For example, time series data includes timestamps that label individual periods or points in time, spatial data has coordinates (e.g. longitude, latitude, elevation), and model or laboratory experiments are often identified by unique identifiers. Figure 1 provides an example of a labeled dataset. In this case the data is a map of global air temperature from a numeric weather model. The labels on this particular dataset are time (e.g. “2016-05-01”), longitude (x-axis), and latitude (y-axis).” | |
| claim-type | method — “xarray is an open source project and Python package that provides a toolkit and data structures for N-dimensional labeled arrays. Our approach combines an application programing interface (API) inspired by pandas with the Common Data Model for self-described scientific data. Key features of the xarray package include label-based indexing and arithmetic, interoperability with the core scientific Python packages (e.g., pandas, NumPy, Matplotlib), out-of-core computation on datasets that don’t fit into memory, a wide range of serialization and input/output (I/O) options, and advanced multi-dimensional data manipulation tools such as group-by and resampling. xarray, as a data model and analytics toolkit, has been widely adopted in the geoscience community but is also used more broadly for multi-dimensional data analysis in physics, machine learning and finance.” | |
| method — evidence | — “NetCDF forms the basis of the xarray data model and provides a natural and portable serialization format. Building on netCDF, xarray features two main data structures: the DataArray and the Dataset. The API for these data structures is summarized in the following sections and in Figure 2.” | |
| object of study — evidence | — “xarray is an open source project and Python package that provides a toolkit and data structures for N-dimensional labeled arrays. Our approach combines an application programing interface (API) inspired by pandas with the Common Data Model for self-described scientific data. Key features of the xarray package include label-based indexing and arithmetic, interoperability with the core scientific Python packages (e.g., pandas, NumPy, Matplotlib), out-of-core computation on datasets that don’t fit into memory, a wide range of serialization and input/output (I/O) options, and advanced multi-dimensional data manipulation tools such as group-by and resampling. xarray, as a data model and analytics toolkit, has been widely adopted in the geoscience community but is also used more broadly for multi-dimensional data analysis in physics, machine learning and finance.” | |
| research question — evidence | — “Unlabeled, N-dimensional arrays of numbers (e.g., NumPy’s ndarray) are the most widely used data structure in scientific computing. However, they lack a meaningful representation of the metadata associated with their data. Implementing such functionality is left to individual users and domain-specific packages. As a result, programmers frequently encounter pitfalls in the form of questions like “is the time axis of my array in the first or third index position?” or “does my array of timestamps still align with my data after resampling?”.” | |
| contribution — evidence | — “xarray is an open source project and Python package that provides a toolkit and data structures for N-dimensional labeled arrays. Our approach combines an application programing interface (API) inspired by pandas with the Common Data Model for self-described scientific data. Key features of the xarray package include label-based indexing and arithmetic, interoperability with the core scientific Python packages (e.g., pandas, NumPy, Matplotlib), out-of-core computation on datasets that don’t fit into memory, a wide range of serialization and input/output (I/O) options, and advanced multi-dimensional data manipulation tools such as group-by and resampling. xarray, as a data model and analytics toolkit, has been widely adopted in the geoscience community but is also used more broadly for multi-dimensional data analysis in physics, machine learning and finance.” | |
| mechanism | label consistency Every operation consumes and preserves labels so data and metadata stay aligned. — “The core motivation for developing xarray was to provide labeled data tools for N-dimensional arrays that render such questions moot. Every operation in xarray both relies on and maintains the consistency of labels.” | mechanisms[0] |
| mechanism | named dimensions Dimension names replace positional axes for indexing and aggregation. — “Label-based indexing: Similarly to pandas objects, xarray objects support both integer- and label-based lookups along each dimension. However, xarray objects also have named dimensions, so you can optionally use dimension names instead of relying on the positional ordering of dimensions.” | mechanisms[1] |
| mechanism | coordinate indexing Ancillary coordinate variables build on pandas Index for fast lookup and alignment. — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | mechanisms[2] |
| mechanism | dimension-name broadcasting Arithmetic loops over distinct named dimensions without dummy size-one axes. — “Arithmetic: arithmetic between xarray objects vectorizes based on dimension names, automatically looping (broadcasting) over each distinct dimension. This eliminates the need to insert dummy dimensions of size one to facilitate broadcasting, a common pattern with NumPy.” | mechanisms[3] |
| mechanism | coordinate alignment Database-like joins combine xarray objects along coordinates shared between them. — “Alignment: xarray supports database-like join operations for combining xarray objects along common coordinates.” | mechanisms[4] |
| mechanism | third-party I/O integration Multiple serialization formats are exposed by integrating their libraries. — “Serialization and I/O: xarray supports direct serialization and I/O to several file formats including pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF by integrating with third-party libraries. Additional serialization formats for 1-dimensional data are available through pandas.” | mechanisms[5] |
| mechanism | dask backing Dask-backed structures execute parallel and streaming computations outside memory. — “Out-of-core computation: xarray’s data structures can be backed by dask [21] instead of NumPy to support parallel and streaming computation on data that does not fit into memory, up to 100s of GB or TBs in size. Such large datasets (“big data”) are increasingly prevalent in science.” | mechanisms[6] |
| mechanism | continuous integration The unit suite is run automatically on Linux and Windows services. — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” | mechanisms[7] |
| mechanism | composition and accessors Composition, subclassing, and accessor interfaces add domain methods without putting them in the core. — “xarray was written in a modular, objected-oriented way, to build upon and extend the core scientific Python libraries in a domain-agnostic fashion. The xarray documentation is complete with a wide range of examples and a number of tutorials that use real-world datasets that are available in the xarray repository. We have intentionally avoided including domain-specific functionality in the library, leaving that to third party libraries. It has been widely adopted in the geoscience community [e.g. 6, 10, 9], but has also been used in physics [e.g. 3], time series analytics [8], and finance. The core xarray data structures (the DataArray and the Dataset) are extensible through subclassing or the preferred approach of composition. We also provide an extensible high-level accessor interface to allow users to implement domain specific methods on xarray data objects.” | mechanisms[8] |
| builds-on | pandas — “xarray implements data structures and an analytics toolkit for multi-dimensional labeled arrays strongly inspired by pandas. While pandas includes a data structure called the Panel for three dimensional data, its fixed rank design make it unsuitable for applications that require arbitrary rank arrays. Additionally, many of the features that make the pandas DataFrame and Series objects so useful, are not fully available on the Panel. Our approach with xarray adopts Unidata’s self-describing Common Data Model on which the network Common Data Form (netCDF) is built [20, 7]. NetCDF provides a well-defined data model for labeled N-dimensional array-oriented scientific data analysis.” | claim_relations[0] |
| builds-on | netCDF — “NetCDF forms the basis of the xarray data model and provides a natural and portable serialization format. Building on netCDF, xarray features two main data structures: the DataArray and the Dataset. The API for these data structures is summarized in the following sections and in Figure 2.” | claim_relations[1] |
| extends | core scientific Python libraries — “xarray was written in a modular, objected-oriented way, to build upon and extend the core scientific Python libraries in a domain-agnostic fashion. The xarray documentation is complete with a wide range of examples and a number of tutorials that use real-world datasets that are available in the xarray repository. We have intentionally avoided including domain-specific functionality in the library, leaving that to third party libraries. It has been widely adopted in the geoscience community [e.g. 6, 10, 9], but has also been used in physics [e.g. 3], time series analytics [8], and finance. The core xarray data structures (the DataArray and the Dataset) are extensible through subclassing or the preferred approach of composition. We also provide an extensible high-level accessor interface to allow users to implement domain specific methods on xarray data objects.” | claim_relations[2] |
| complements | domain-specific packages — “xarray builds on top of, and seamlessly interoperates with, the core scientific Python packages, such as NumPy, SciPy, Matplotlib [13], and pandas. xarray provides a range of backends for serialization and input/output (IO), including the Pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF file formats. Leveraging the dask parallel computing library [21], xarray can optionally perform efficient parallel, out-of-core analysis on datasets that are too large to fit into memory. Finally, xarray interfaces with existing domain-specific packages such as UV-CDAT [25], Iris [17], and Cartopy [18].” | claim_relations[3] |
| sense | xarray as a pandas-inspired, netCDF-grounded labeled-array toolkit — “xarray is an open source project and Python package that provides a toolkit and data structures for N-dimensional labeled arrays. Our approach combines an application programing interface (API) inspired by pandas with the Common Data Model for self-described scientific data. Key features of the xarray package include label-based indexing and arithmetic, interoperability with the core scientific Python packages (e.g., pandas, NumPy, Matplotlib), out-of-core computation on datasets that don’t fit into memory, a wide range of serialization and input/output (I/O) options, and advanced multi-dimensional data manipulation tools such as group-by and resampling. xarray, as a data model and analytics toolkit, has been widely adopted in the geoscience community but is also used more broadly for multi-dimensional data analysis in physics, machine learning and finance.” | senses[0] |
| sense | scientific labels as timestamps, coordinates, or experiment identifiers — “Scientific data is inherently labeled. For example, time series data includes timestamps that label individual periods or points in time, spatial data has coordinates (e.g. longitude, latitude, elevation), and model or laboratory experiments are often identified by unique identifiers. Figure 1 provides an example of a labeled dataset. In this case the data is a map of global air temperature from a numeric weather model. The labels on this particular dataset are time (e.g. “2016-05-01”), longitude (x-axis), and latitude (y-axis).” | senses[1] |
| sense | netCDF as both a self-describing data model and portable serialization basis — “NetCDF forms the basis of the xarray data model and provides a natural and portable serialization format. Building on netCDF, xarray features two main data structures: the DataArray and the Dataset. The API for these data structures is summarized in the following sections and in Figure 2.” | senses[2] |
| sense | DataArray as one labeled multidimensional array — “The DataArray is xarray’s implementation of a labeled, multi-dimensional array. It has several key properties:” | senses[3] |
| sense | Dataset as aligned DataArrays sharing a coordinate system — “The Dataset is xarray’s multi-dimensional equivalent of a DataFrame. It is a dict-like container of labeled arrays (DataArrays) with aligned dimensions. It is designed as an in-memory representation of a netCDF dataset. In addition to the dict-like interface of the dataset itself, which can be used to access any DataArray in a Dataset, datasets have four key properties:” | senses[4] |
| sense | dimensions as named axes and coordinates as ancillary index variables — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | senses[5] |
| sense | alignment as database-like joining on common coordinates — “Alignment: xarray supports database-like join operations for combining xarray objects along common coordinates.” | senses[6] |
| sense | out-of-core as dask-backed parallel or streaming work beyond memory — “Out-of-core computation: xarray’s data structures can be backed by dask [21] instead of NumPy to support parallel and streaming computation on data that does not fit into memory, up to 100s of GB or TBs in size. Such large datasets (“big data”) are increasingly prevalent in science.” | senses[7] |
| sense | quality control as cross-platform automated unit testing plus reproducible sample data — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” | senses[8] |
| sense | reuse as a domain-agnostic core extended through third-party composition and accessors — “xarray was written in a modular, objected-oriented way, to build upon and extend the core scientific Python libraries in a domain-agnostic fashion. The xarray documentation is complete with a wide range of examples and a number of tutorials that use real-world datasets that are available in the xarray repository. We have intentionally avoided including domain-specific functionality in the library, leaving that to third party libraries. It has been widely adopted in the geoscience community [e.g. 6, 10, 9], but has also been used in physics [e.g. 3], time series analytics [8], and finance. The core xarray data structures (the DataArray and the Dataset) are extensible through subclassing or the preferred approach of composition. We also provide an extensible high-level accessor interface to allow users to implement domain specific methods on xarray data objects.” | senses[9] |
| control | over 1,500 unit tests — held — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” — rules out regressions in core and optional-dependency functionality — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” | controls[0] |
| control | TravisCI and Appveyor — held — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” — rules out platform-specific failures missed by a single operating system — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” | controls[1] |
| control | sample data distributed with the source code — held — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” — rules out examples that users cannot reproduce — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” | controls[2] |
| note | pandas — The labeled-array structures and analytics are strongly inspired by pandas. — “xarray implements data structures and an analytics toolkit for multi-dimensional labeled arrays strongly inspired by pandas. While pandas includes a data structure called the Panel for three dimensional data, its fixed rank design make it unsuitable for applications that require arbitrary rank arrays. Additionally, many of the features that make the pandas DataFrame and Series objects so useful, are not fully available on the Panel. Our approach with xarray adopts Unidata’s self-describing Common Data Model on which the network Common Data Form (netCDF) is built [20, 7]. NetCDF provides a well-defined data model for labeled N-dimensional array-oriented scientific data analysis.” | claim_relations[0] |
| weight | pandas — 0.95 — “xarray implements data structures and an analytics toolkit for multi-dimensional labeled arrays strongly inspired by pandas. While pandas includes a data structure called the Panel for three dimensional data, its fixed rank design make it unsuitable for applications that require arbitrary rank arrays. Additionally, many of the features that make the pandas DataFrame and Series objects so useful, are not fully available on the Panel. Our approach with xarray adopts Unidata’s self-describing Common Data Model on which the network Common Data Form (netCDF) is built [20, 7]. NetCDF provides a well-defined data model for labeled N-dimensional array-oriented scientific data analysis.” | claim_relations[0] |
| note | netCDF — NetCDF explicitly forms the basis of the xarray model. — “NetCDF forms the basis of the xarray data model and provides a natural and portable serialization format. Building on netCDF, xarray features two main data structures: the DataArray and the Dataset. The API for these data structures is summarized in the following sections and in Figure 2.” | claim_relations[1] |
| note | core scientific Python libraries — The modular design builds upon and extends core libraries without embedding domains. — “xarray was written in a modular, objected-oriented way, to build upon and extend the core scientific Python libraries in a domain-agnostic fashion. The xarray documentation is complete with a wide range of examples and a number of tutorials that use real-world datasets that are available in the xarray repository. We have intentionally avoided including domain-specific functionality in the library, leaving that to third party libraries. It has been widely adopted in the geoscience community [e.g. 6, 10, 9], but has also been used in physics [e.g. 3], time series analytics [8], and finance. The core xarray data structures (the DataArray and the Dataset) are extensible through subclassing or the preferred approach of composition. We also provide an extensible high-level accessor interface to allow users to implement domain specific methods on xarray data objects.” | claim_relations[2] |
| weight | core scientific Python libraries — 0.95 — “xarray was written in a modular, objected-oriented way, to build upon and extend the core scientific Python libraries in a domain-agnostic fashion. The xarray documentation is complete with a wide range of examples and a number of tutorials that use real-world datasets that are available in the xarray repository. We have intentionally avoided including domain-specific functionality in the library, leaving that to third party libraries. It has been widely adopted in the geoscience community [e.g. 6, 10, 9], but has also been used in physics [e.g. 3], time series analytics [8], and finance. The core xarray data structures (the DataArray and the Dataset) are extensible through subclassing or the preferred approach of composition. We also provide an extensible high-level accessor interface to allow users to implement domain specific methods on xarray data objects.” | claim_relations[2] |
| note | domain-specific packages — xarray interfaces with domain-specific packages while retaining a domain-agnostic core. — “xarray builds on top of, and seamlessly interoperates with, the core scientific Python packages, such as NumPy, SciPy, Matplotlib [13], and pandas. xarray provides a range of backends for serialization and input/output (IO), including the Pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF file formats. Leveraging the dask parallel computing library [21], xarray can optionally perform efficient parallel, out-of-core analysis on datasets that are too large to fit into memory. Finally, xarray interfaces with existing domain-specific packages such as UV-CDAT [25], Iris [17], and Cartopy [18].” | claim_relations[3] |
| weight | domain-specific packages — 0.95 — “xarray builds on top of, and seamlessly interoperates with, the core scientific Python packages, such as NumPy, SciPy, Matplotlib [13], and pandas. xarray provides a range of backends for serialization and input/output (IO), including the Pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF file formats. Leveraging the dask parallel computing library [21], xarray can optionally perform efficient parallel, out-of-core analysis on datasets that are too large to fit into memory. Finally, xarray interfaces with existing domain-specific packages such as UV-CDAT [25], Iris [17], and Cartopy [18].” | claim_relations[3] |
| note | xarray —is-a→ Python package — Flat categorical classification. — “xarray is an open source project and Python package that provides a toolkit and data structures for N-dimensional labeled arrays. Our approach combines an application programing interface (API) inspired by pandas with the Common Data Model for self-described scientific data. Key features of the xarray package include label-based indexing and arithmetic, interoperability with the core scientific Python packages (e.g., pandas, NumPy, Matplotlib), out-of-core computation on datasets that don’t fit into memory, a wide range of serialization and input/output (I/O) options, and advanced multi-dimensional data manipulation tools such as group-by and resampling. xarray, as a data model and analytics toolkit, has been widely adopted in the geoscience community but is also used more broadly for multi-dimensional data analysis in physics, machine learning and finance.” | concept_edges[0] |
| relation | xarray —is-a→ Python package — xarray is a Python package. — “xarray is an open source project and Python package that provides a toolkit and data structures for N-dimensional labeled arrays. Our approach combines an application programing interface (API) inspired by pandas with the Common Data Model for self-described scientific data. Key features of the xarray package include label-based indexing and arithmetic, interoperability with the core scientific Python packages (e.g., pandas, NumPy, Matplotlib), out-of-core computation on datasets that don’t fit into memory, a wide range of serialization and input/output (I/O) options, and advanced multi-dimensional data manipulation tools such as group-by and resampling. xarray, as a data model and analytics toolkit, has been widely adopted in the geoscience community but is also used more broadly for multi-dimensional data analysis in physics, machine learning and finance.” | concept_edges[0] |
| note | xarray —contrasted-with→ Panel — Direct while-contrast and stated suitability difference. — “xarray implements data structures and an analytics toolkit for multi-dimensional labeled arrays strongly inspired by pandas. While pandas includes a data structure called the Panel for three dimensional data, its fixed rank design make it unsuitable for applications that require arbitrary rank arrays. Additionally, many of the features that make the pandas DataFrame and Series objects so useful, are not fully available on the Panel. Our approach with xarray adopts Unidata’s self-describing Common Data Model on which the network Common Data Form (netCDF) is built [20, 7]. NetCDF provides a well-defined data model for labeled N-dimensional array-oriented scientific data analysis.” | concept_edges[1] |
| relation | xarray —contrasted-with→ Panel — xarray's arbitrary-rank labeled arrays contrast with pandas Panel's fixed three-dimensional rank. — “xarray implements data structures and an analytics toolkit for multi-dimensional labeled arrays strongly inspired by pandas. While pandas includes a data structure called the Panel for three dimensional data, its fixed rank design make it unsuitable for applications that require arbitrary rank arrays. Additionally, many of the features that make the pandas DataFrame and Series objects so useful, are not fully available on the Panel. Our approach with xarray adopts Unidata’s self-describing Common Data Model on which the network Common Data Form (netCDF) is built [20, 7]. NetCDF provides a well-defined data model for labeled N-dimensional array-oriented scientific data analysis.” | concept_edges[1] |
| weight | xarray —contrasted-with→ Panel — 0.95 — “xarray implements data structures and an analytics toolkit for multi-dimensional labeled arrays strongly inspired by pandas. While pandas includes a data structure called the Panel for three dimensional data, its fixed rank design make it unsuitable for applications that require arbitrary rank arrays. Additionally, many of the features that make the pandas DataFrame and Series objects so useful, are not fully available on the Panel. Our approach with xarray adopts Unidata’s self-describing Common Data Model on which the network Common Data Form (netCDF) is built [20, 7]. NetCDF provides a well-defined data model for labeled N-dimensional array-oriented scientific data analysis.” | concept_edges[1] |
| note | network Common Data Form —is-a→ collection of self-describing, machine-independent binary data formats and software tools — Explicit categorical definition. — “The network Common Data Form is a collection of self-describing, machine-independent binary data formats and software tools. These data formats and tools facilitate the creation, access, and sharing of scientific data stored in N-dimensional arrays, along with metadata describing the contents of each array [20]. NetCDF has become very popular in the geoscience community, and there are existing libraries for reading and writing netCDF in many programming languages, including C, Fortran, Python, Java, Matlab, and Julia.” | concept_edges[2] |
| relation | network Common Data Form —is-a→ collection of self-describing, machine-independent binary data formats and software tools — The network Common Data Form is a collection of self-describing binary formats and software tools. — “The network Common Data Form is a collection of self-describing, machine-independent binary data formats and software tools. These data formats and tools facilitate the creation, access, and sharing of scientific data stored in N-dimensional arrays, along with metadata describing the contents of each array [20]. NetCDF has become very popular in the geoscience community, and there are existing libraries for reading and writing netCDF in many programming languages, including C, Fortran, Python, Java, Matlab, and Julia.” | concept_edges[2] |
| note | dimensions, variables, and attributes —part-of→ netCDF dataset — Categorical containment stated by 'contains'. — “The principal data structure in the netCDF data model is the dataset. Each netCDF dataset contains dimensions, variables, and attributes, each of which are identified by a hierarchy of unique names. The dataset and variable objects may contain attributes that describe the contents, units, history, or other metadata of the object. Standardized conventions, such as the Climate and Forecast (CF) Conventions [12], allow for the associations of coordinate variables with dimensions.” | concept_edges[3] |
| relation | dimensions, variables, and attributes —part-of→ netCDF dataset — Dimensions, variables, and attributes are contained in each netCDF dataset. — “The principal data structure in the netCDF data model is the dataset. Each netCDF dataset contains dimensions, variables, and attributes, each of which are identified by a hierarchy of unique names. The dataset and variable objects may contain attributes that describe the contents, units, history, or other metadata of the object. Standardized conventions, such as the Climate and Forecast (CF) Conventions [12], allow for the associations of coordinate variables with dimensions.” | concept_edges[3] |
| audit | Dataset —same-as→ DataFrame — Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the Dataset is the multi-dimensional EQUIVALENT of a DataFrame; same-as overstates. — “The Dataset is xarray’s multi-dimensional equivalent of a DataFrame. It is a dict-like container of labeled arrays (DataArrays) with aligned dimensions. It is designed as an in-memory representation of a netCDF dataset. In addition to the dict-like interface of the dataset itself, which can be used to access any DataArray in a Dataset, datasets have four key properties:” | concept_edges[4] |
| note | Dataset —same-as→ DataFrame — Scoped multidimensional equivalence, so below full identity strength. — “The Dataset is xarray’s multi-dimensional equivalent of a DataFrame. It is a dict-like container of labeled arrays (DataArrays) with aligned dimensions. It is designed as an in-memory representation of a netCDF dataset. In addition to the dict-like interface of the dataset itself, which can be used to access any DataArray in a Dataset, datasets have four key properties:” | concept_edges[4] |
| relation | Dataset —same-as→ DataFrame — Dataset is xarray's multidimensional equivalent of a DataFrame. — “The Dataset is xarray’s multi-dimensional equivalent of a DataFrame. It is a dict-like container of labeled arrays (DataArrays) with aligned dimensions. It is designed as an in-memory representation of a netCDF dataset. In addition to the dict-like interface of the dataset itself, which can be used to access any DataArray in a Dataset, datasets have four key properties:” | concept_edges[4] |
| weight | Dataset —same-as→ DataFrame — 0.9 — “The Dataset is xarray’s multi-dimensional equivalent of a DataFrame. It is a dict-like container of labeled arrays (DataArrays) with aligned dimensions. It is designed as an in-memory representation of a netCDF dataset. In addition to the dict-like interface of the dataset itself, which can be used to access any DataArray in a Dataset, datasets have four key properties:” | concept_edges[4] |
| note | Dataset —is-a→ dict-like container — Explicit categorical classification. — “The Dataset is xarray’s multi-dimensional equivalent of a DataFrame. It is a dict-like container of labeled arrays (DataArrays) with aligned dimensions. It is designed as an in-memory representation of a netCDF dataset. In addition to the dict-like interface of the dataset itself, which can be used to access any DataArray in a Dataset, datasets have four key properties:” | concept_edges[5] |
| relation | Dataset —is-a→ dict-like container — Dataset is a dict-like container of aligned DataArrays. — “The Dataset is xarray’s multi-dimensional equivalent of a DataFrame. It is a dict-like container of labeled arrays (DataArrays) with aligned dimensions. It is designed as an in-memory representation of a netCDF dataset. In addition to the dict-like interface of the dataset itself, which can be used to access any DataArray in a Dataset, datasets have four key properties:” | concept_edges[5] |
| evidence status note | Names and attributes —contrasted-with→ coordinates — DOUBTFUL, found by an audit that read the edge against its span: weak contrast. The edge is kept live; a reader should weigh it with this in mind. — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | concept_edges[6] |
| note | Names and attributes —contrasted-with→ coordinates — Explicit in-contrast comparison. — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | concept_edges[6] |
| relation | Names and attributes —contrasted-with→ coordinates — Names and attributes are not interpreted generally, in contrast to coordinates persisted through transformations. — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | concept_edges[6] |
| weight | Names and attributes —contrasted-with→ coordinates — 0.95 — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | concept_edges[6] |
| note | coordinates —distinct-from→ varying/dependent quantities — Explicit unlike distinction. — “DataArray objects inside a Dataset may have any number of dimensions but are presumed to share a common coordinate system. Coordinates can also have any number of dimensions but denote constant/independent quantities, unlike the varying/dependent quantities that belong in data. Figure 3 illustrates these concepts for an example Dataset containing meteorological data.” | concept_edges[7] |
| relation | coordinates —distinct-from→ varying/dependent quantities — Coordinates denote constant or independent quantities, unlike varying or dependent data quantities. — “DataArray objects inside a Dataset may have any number of dimensions but are presumed to share a common coordinate system. Coordinates can also have any number of dimensions but denote constant/independent quantities, unlike the varying/dependent quantities that belong in data. Figure 3 illustrates these concepts for an example Dataset containing meteorological data.” | concept_edges[7] |
| weight | coordinates —distinct-from→ varying/dependent quantities — 0.95 — “DataArray objects inside a Dataset may have any number of dimensions but are presumed to share a common coordinate system. Coordinates can also have any number of dimensions but denote constant/independent quantities, unlike the varying/dependent quantities that belong in data. Figure 3 illustrates these concepts for an example Dataset containing meteorological data.” | concept_edges[7] |
| note | named dimensions —contrasted-with→ positional ordering of dimensions — Direct instead-of contrast. — “Label-based indexing: Similarly to pandas objects, xarray objects support both integer- and label-based lookups along each dimension. However, xarray objects also have named dimensions, so you can optionally use dimension names instead of relying on the positional ordering of dimensions.” | concept_edges[8] |
| relation | named dimensions —contrasted-with→ positional ordering of dimensions — Named dimensions are used instead of relying on positional dimension ordering. — “Label-based indexing: Similarly to pandas objects, xarray objects support both integer- and label-based lookups along each dimension. However, xarray objects also have named dimensions, so you can optionally use dimension names instead of relying on the positional ordering of dimensions.” | concept_edges[8] |
| weight | named dimensions —contrasted-with→ positional ordering of dimensions — 0.95 — “Label-based indexing: Similarly to pandas objects, xarray objects support both integer- and label-based lookups along each dimension. However, xarray objects also have named dimensions, so you can optionally use dimension names instead of relying on the positional ordering of dimensions.” | concept_edges[8] |
| note | xarray plotting functionality —is-a→ thin wrapper — Explicit categorical classification. — “Plotting: xarray plotting functionality is a thin wrapper around the popular Matplotlib library. xarray uses the syntax and function names from Matplotlib whenever possible, resulting in a seamless transition between the two.” | concept_edges[9] |
| relation | xarray plotting functionality —is-a→ thin wrapper — xarray plotting functionality is a thin wrapper around Matplotlib. — “Plotting: xarray plotting functionality is a thin wrapper around the popular Matplotlib library. xarray uses the syntax and function names from Matplotlib whenever possible, resulting in a seamless transition between the two.” | concept_edges[9] |
| outcome | control — held — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” | controls[0] |
| outcome | control — held — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” | controls[1] |
| outcome | control — held — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” | controls[2] |
| relation | xarray → core scientific Python packages — xarray builds on and interoperates with core scientific Python packages. — “xarray builds on top of, and seamlessly interoperates with, the core scientific Python packages, such as NumPy, SciPy, Matplotlib [13], and pandas. xarray provides a range of backends for serialization and input/output (IO), including the Pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF file formats. Leveraging the dask parallel computing library [21], xarray can optionally perform efficient parallel, out-of-core analysis on datasets that are too large to fit into memory. Finally, xarray interfaces with existing domain-specific packages such as UV-CDAT [25], Iris [17], and Cartopy [18].” | unhoused_relations[0] |
| weight | xarray → core scientific Python packages — 0.95 — “xarray builds on top of, and seamlessly interoperates with, the core scientific Python packages, such as NumPy, SciPy, Matplotlib [13], and pandas. xarray provides a range of backends for serialization and input/output (IO), including the Pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF file formats. Leveraging the dask parallel computing library [21], xarray can optionally perform efficient parallel, out-of-core analysis on datasets that are too large to fit into memory. Finally, xarray interfaces with existing domain-specific packages such as UV-CDAT [25], Iris [17], and Cartopy [18].” | unhoused_relations[0] |
| relation | xarray → domain-specific packages — xarray interfaces with domain-specific packages. — “xarray builds on top of, and seamlessly interoperates with, the core scientific Python packages, such as NumPy, SciPy, Matplotlib [13], and pandas. xarray provides a range of backends for serialization and input/output (IO), including the Pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF file formats. Leveraging the dask parallel computing library [21], xarray can optionally perform efficient parallel, out-of-core analysis on datasets that are too large to fit into memory. Finally, xarray interfaces with existing domain-specific packages such as UV-CDAT [25], Iris [17], and Cartopy [18].” | unhoused_relations[1] |
| weight | xarray → domain-specific packages — 0.95 — “xarray builds on top of, and seamlessly interoperates with, the core scientific Python packages, such as NumPy, SciPy, Matplotlib [13], and pandas. xarray provides a range of backends for serialization and input/output (IO), including the Pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF file formats. Leveraging the dask parallel computing library [21], xarray can optionally perform efficient parallel, out-of-core analysis on datasets that are too large to fit into memory. Finally, xarray interfaces with existing domain-specific packages such as UV-CDAT [25], Iris [17], and Cartopy [18].” | unhoused_relations[1] |
| relation | xarray → consistency of labels — Every xarray operation relies on and maintains label consistency. — “The core motivation for developing xarray was to provide labeled data tools for N-dimensional arrays that render such questions moot. Every operation in xarray both relies on and maintains the consistency of labels.” | unhoused_relations[2] |
| relation | DataArray → labeled, multi-dimensional array — DataArray implements a labeled multidimensional array. — “The DataArray is xarray’s implementation of a labeled, multi-dimensional array. It has several key properties:” | unhoused_relations[3] |
| relation | Dimensions → core metadata-aware operations — Dimensions enable xarray's metadata-aware operations. — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | unhoused_relations[4] |
| weight | Dimensions → core metadata-aware operations — 0.95 — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | unhoused_relations[4] |
| relation | dimension names → dummy dimensions of size one — Dimension-name broadcasting eliminates the need for dummy size-one dimensions. — “Arithmetic: arithmetic between xarray objects vectorizes based on dimension names, automatically looping (broadcasting) over each distinct dimension. This eliminates the need to insert dummy dimensions of size one to facilitate broadcasting, a common pattern with NumPy.” | unhoused_relations[5] |
| relation | dask → parallel and streaming computation — Dask backing supports parallel and streaming computation beyond memory. — “Out-of-core computation: xarray’s data structures can be backed by dask [21] instead of NumPy to support parallel and streaming computation on data that does not fit into memory, up to 100s of GB or TBs in size. Such large datasets (“big data”) are increasingly prevalent in science.” | unhoused_relations[6] |
| weight | dask → parallel and streaming computation — 0.95 — “Out-of-core computation: xarray’s data structures can be backed by dask [21] instead of NumPy to support parallel and streaming computation on data that does not fit into memory, up to 100s of GB or TBs in size. Such large datasets (“big data”) are increasingly prevalent in science.” | unhoused_relations[6] |
| relation | serialization and I/O → third-party libraries — xarray serialization and I/O integrate third-party libraries. — “Serialization and I/O: xarray supports direct serialization and I/O to several file formats including pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF by integrating with third-party libraries. Additional serialization formats for 1-dimensional data are available through pandas.” | unhoused_relations[7] |
| weight | serialization and I/O → third-party libraries — 0.95 — “Serialization and I/O: xarray supports direct serialization and I/O to several file formats including pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF by integrating with third-party libraries. Additional serialization formats for 1-dimensional data are available through pandas.” | unhoused_relations[7] |
| relation | large test suite → TravisCI — The unit-test suite is executed automatically on TravisCI. — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” | unhoused_relations[8] |
| relation | xarray → compiled dependencies — The pure-Python xarray implementation relies on compiled dependencies for speed. — “xarray is implemented in pure Python and relies on compiled dependencies for speed.” | unhoused_relations[9] |
| relation | high-level accessor interface → domain-specific functionality — The accessor interface lets users implement domain-specific methods on xarray objects. — “xarray was written in a modular, objected-oriented way, to build upon and extend the core scientific Python libraries in a domain-agnostic fashion. The xarray documentation is complete with a wide range of examples and a number of tutorials that use real-world datasets that are available in the xarray repository. We have intentionally avoided including domain-specific functionality in the library, leaving that to third party libraries. It has been widely adopted in the geoscience community [e.g. 6, 10, 9], but has also been used in physics [e.g. 3], time series analytics [8], and finance. The core xarray data structures (the DataArray and the Dataset) are extensible through subclassing or the preferred approach of composition. We also provide an extensible high-level accessor interface to allow users to implement domain specific methods on xarray data objects.” | unhoused_relations[10] |
| weight | high-level accessor interface → domain-specific functionality — 0.95 — “xarray was written in a modular, objected-oriented way, to build upon and extend the core scientific Python libraries in a domain-agnostic fashion. The xarray documentation is complete with a wide range of examples and a number of tutorials that use real-world datasets that are available in the xarray repository. We have intentionally avoided including domain-specific functionality in the library, leaving that to third party libraries. It has been widely adopted in the geoscience community [e.g. 6, 10, 9], but has also been used in physics [e.g. 3], time series analytics [8], and finance. The core xarray data structures (the DataArray and the Dataset) are extensible through subclassing or the preferred approach of composition. We also provide an extensible high-level accessor interface to allow users to implement domain specific methods on xarray data objects.” | unhoused_relations[10] |
| no type for this | xarray → core scientific Python packages — xarray builds on and interoperates with core scientific Python packages. — “xarray builds on top of, and seamlessly interoperates with, the core scientific Python packages, such as NumPy, SciPy, Matplotlib [13], and pandas. xarray provides a range of backends for serialization and input/output (IO), including the Pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF file formats. Leveraging the dask parallel computing library [21], xarray can optionally perform efficient parallel, out-of-core analysis on datasets that are too large to fit into memory. Finally, xarray interfaces with existing domain-specific packages such as UV-CDAT [25], Iris [17], and Cartopy [18].” | unhoused_relations[0] |
| no type for this | xarray → domain-specific packages — xarray interfaces with domain-specific packages. — “xarray builds on top of, and seamlessly interoperates with, the core scientific Python packages, such as NumPy, SciPy, Matplotlib [13], and pandas. xarray provides a range of backends for serialization and input/output (IO), including the Pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF file formats. Leveraging the dask parallel computing library [21], xarray can optionally perform efficient parallel, out-of-core analysis on datasets that are too large to fit into memory. Finally, xarray interfaces with existing domain-specific packages such as UV-CDAT [25], Iris [17], and Cartopy [18].” | unhoused_relations[1] |
| no type for this | xarray → consistency of labels — Every xarray operation relies on and maintains label consistency. — “The core motivation for developing xarray was to provide labeled data tools for N-dimensional arrays that render such questions moot. Every operation in xarray both relies on and maintains the consistency of labels.” | unhoused_relations[2] |
| no type for this | DataArray → labeled, multi-dimensional array — DataArray implements a labeled multidimensional array. — “The DataArray is xarray’s implementation of a labeled, multi-dimensional array. It has several key properties:” | unhoused_relations[3] |
| no type for this | Dimensions → core metadata-aware operations — Dimensions enable xarray's metadata-aware operations. — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | unhoused_relations[4] |
| no type for this | dimension names → dummy dimensions of size one — Dimension-name broadcasting eliminates the need for dummy size-one dimensions. — “Arithmetic: arithmetic between xarray objects vectorizes based on dimension names, automatically looping (broadcasting) over each distinct dimension. This eliminates the need to insert dummy dimensions of size one to facilitate broadcasting, a common pattern with NumPy.” | unhoused_relations[5] |
| no type for this | dask → parallel and streaming computation — Dask backing supports parallel and streaming computation beyond memory. — “Out-of-core computation: xarray’s data structures can be backed by dask [21] instead of NumPy to support parallel and streaming computation on data that does not fit into memory, up to 100s of GB or TBs in size. Such large datasets (“big data”) are increasingly prevalent in science.” | unhoused_relations[6] |
| no type for this | serialization and I/O → third-party libraries — xarray serialization and I/O integrate third-party libraries. — “Serialization and I/O: xarray supports direct serialization and I/O to several file formats including pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF by integrating with third-party libraries. Additional serialization formats for 1-dimensional data are available through pandas.” | unhoused_relations[7] |
| no type for this | large test suite → TravisCI — The unit-test suite is executed automatically on TravisCI. — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” | unhoused_relations[8] |
| no type for this | xarray → compiled dependencies — The pure-Python xarray implementation relies on compiled dependencies for speed. — “xarray is implemented in pure Python and relies on compiled dependencies for speed.” | unhoused_relations[9] |
| no type for this | high-level accessor interface → domain-specific functionality — The accessor interface lets users implement domain-specific methods on xarray objects. — “xarray was written in a modular, objected-oriented way, to build upon and extend the core scientific Python libraries in a domain-agnostic fashion. The xarray documentation is complete with a wide range of examples and a number of tutorials that use real-world datasets that are available in the xarray repository. We have intentionally avoided including domain-specific functionality in the library, leaving that to third party libraries. It has been widely adopted in the geoscience community [e.g. 6, 10, 9], but has also been used in physics [e.g. 3], time series analytics [8], and finance. The core xarray data structures (the DataArray and the Dataset) are extensible through subclassing or the preferred approach of composition. We also provide an extensible high-level accessor interface to allow users to implement domain specific methods on xarray data objects.” | unhoused_relations[10] |
| limitation | Pandas Panel has a fixed three-dimensional rank and lacks features available to DataFrame and Series. — “xarray implements data structures and an analytics toolkit for multi-dimensional labeled arrays strongly inspired by pandas. While pandas includes a data structure called the Panel for three dimensional data, its fixed rank design make it unsuitable for applications that require arbitrary rank arrays. Additionally, many of the features that make the pandas DataFrame and Series objects so useful, are not fully available on the Panel. Our approach with xarray adopts Unidata’s self-describing Common Data Model on which the network Common Data Form (netCDF) is built [20, 7]. NetCDF provides a well-defined data model for labeled N-dimensional array-oriented scientific data analysis.” | limitations[0] |
| limitation | Unlabeled arrays do not meaningfully represent associated metadata, leaving alignment to users or domain packages. — “Unlabeled, N-dimensional arrays of numbers (e.g., NumPy’s ndarray) are the most widely used data structure in scientific computing. However, they lack a meaningful representation of the metadata associated with their data. Implementing such functionality is left to individual users and domain-specific packages. As a result, programmers frequently encounter pitfalls in the form of questions like “is the time axis of my array in the first or third index position?” or “does my array of timestamps still align with my data after resampling?”.” | limitations[1] |
| limitation | Names and attributes are not interpreted by xarray and are propagated only in unambiguous cases. — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | limitations[2] |
| limitation | DataArrays in a Dataset are presumed to share a common coordinate system. — “DataArray objects inside a Dataset may have any number of dimensions but are presumed to share a common coordinate system. Coordinates can also have any number of dimensions but denote constant/independent quantities, unlike the varying/dependent quantities that belong in data. Figure 3 illustrates these concepts for an example Dataset containing meteorological data.” | limitations[3] |
| limitation | OPeNDAP and GRIB serialization backends are read-only. — “Serialization and I/O: xarray supports direct serialization and I/O to several file formats including pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF by integrating with third-party libraries. Additional serialization formats for 1-dimensional data are available through pandas.” | limitations[4] |
| limitation | The pure-Python implementation relies on compiled dependencies for speed. — “xarray is implemented in pure Python and relies on compiled dependencies for speed.” | limitations[5] |
| limitation | Domain-specific functionality is intentionally left to third-party libraries. — “xarray was written in a modular, objected-oriented way, to build upon and extend the core scientific Python libraries in a domain-agnostic fashion. The xarray documentation is complete with a wide range of examples and a number of tutorials that use real-world datasets that are available in the xarray repository. We have intentionally avoided including domain-specific functionality in the library, leaving that to third party libraries. It has been widely adopted in the geoscience community [e.g. 6, 10, 9], but has also been used in physics [e.g. 3], time series analytics [8], and finance. The core xarray data structures (the DataArray and the Dataset) are extensible through subclassing or the preferred approach of composition. We also provide an extensible high-level accessor interface to allow users to implement domain specific methods on xarray data objects.” | limitations[6] |
| limitation | Development and support depend on a volunteer team. — “xarray is developed and supported by a team of volunteers. The primary avenue for user support is StackOverflow [4], with the “xarray-python” tag. Additionally, we use GitHub for a bug tracker (https://github.com/pydata/xarray/issues) and maintain the “xarray” mailing list on Google Groups (https://groups.google.com/forum/#!forum/xarray).” | limitations[7] |
Content tags
content-observations:
finding · quantity · variable · definition · key-term — grounded in the full text| finding | xarray combines a pandas-inspired API with a self-describing scientific-data model. — “xarray is an open source project and Python package that provides a toolkit and data structures for N-dimensional labeled arrays. Our approach combines an application programing interface (API) inspired by pandas with the Common Data Model for self-described scientific data. Key features of the xarray package include label-based indexing and arithmetic, interoperability with the core scientific Python packages (e.g., pandas, NumPy, Matplotlib), out-of-core computation on datasets that don’t fit into memory, a wide range of serialization and input/output (I/O) options, and advanced multi-dimensional data manipulation tools such as group-by and resampling. xarray, as a data model and analytics toolkit, has been widely adopted in the geoscience community but is also used more broadly for multi-dimensional data analysis in physics, machine learning and finance.” | content_tags.findings[0] |
| finding | Pandas Panel's fixed rank is unsuitable for arbitrary-rank array applications. — “xarray implements data structures and an analytics toolkit for multi-dimensional labeled arrays strongly inspired by pandas. While pandas includes a data structure called the Panel for three dimensional data, its fixed rank design make it unsuitable for applications that require arbitrary rank arrays. Additionally, many of the features that make the pandas DataFrame and Series objects so useful, are not fully available on the Panel. Our approach with xarray adopts Unidata’s self-describing Common Data Model on which the network Common Data Form (netCDF) is built [20, 7]. NetCDF provides a well-defined data model for labeled N-dimensional array-oriented scientific data analysis.” | content_tags.findings[1] |
| finding | Every xarray operation relies on and maintains label consistency. — “The core motivation for developing xarray was to provide labeled data tools for N-dimensional arrays that render such questions moot. Every operation in xarray both relies on and maintains the consistency of labels.” | content_tags.findings[2] |
| finding | NetCDF supplies xarray's data-model basis and portable serialization format. — “NetCDF forms the basis of the xarray data model and provides a natural and portable serialization format. Building on netCDF, xarray features two main data structures: the DataArray and the Dataset. The API for these data structures is summarized in the following sections and in Figure 2.” | content_tags.findings[3] |
| finding | DataArray represents a labeled multidimensional array. — “The DataArray is xarray’s implementation of a labeled, multi-dimensional array. It has several key properties:” | content_tags.findings[4] |
| finding | Dataset is a dictionary-like container of aligned DataArrays and an in-memory netCDF representation. — “The Dataset is xarray’s multi-dimensional equivalent of a DataFrame. It is a dict-like container of labeled arrays (DataArrays) with aligned dimensions. It is designed as an in-memory representation of a netCDF dataset. In addition to the dict-like interface of the dataset itself, which can be used to access any DataArray in a Dataset, datasets have four key properties:” | content_tags.findings[5] |
| finding | Named dimensions replace positional axis arguments for xarray indexing. — “Label-based indexing: Similarly to pandas objects, xarray objects support both integer- and label-based lookups along each dimension. However, xarray objects also have named dimensions, so you can optionally use dimension names instead of relying on the positional ordering of dimensions.” | content_tags.findings[6] |
| finding | Dimension-name broadcasting removes the need to insert dummy size-one dimensions. — “Arithmetic: arithmetic between xarray objects vectorizes based on dimension names, automatically looping (broadcasting) over each distinct dimension. This eliminates the need to insert dummy dimensions of size one to facilitate broadcasting, a common pattern with NumPy.” | content_tags.findings[7] |
| finding | Dask backing supports parallel and streaming computation on data too large for memory. — “Out-of-core computation: xarray’s data structures can be backed by dask [21] instead of NumPy to support parallel and streaming computation on data that does not fit into memory, up to 100s of GB or TBs in size. Such large datasets (“big data”) are increasingly prevalent in science.” | content_tags.findings[8] |
| finding | The test suite contains more than 1,500 unit tests run on Linux and Windows continuous integration. — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” | content_tags.findings[9] |
| finding | The core data structures are extensible through composition, subclassing, and high-level accessors. — “xarray was written in a modular, objected-oriented way, to build upon and extend the core scientific Python libraries in a domain-agnostic fashion. The xarray documentation is complete with a wide range of examples and a number of tutorials that use real-world datasets that are available in the xarray repository. We have intentionally avoided including domain-specific functionality in the library, leaving that to third party libraries. It has been widely adopted in the geoscience community [e.g. 6, 10, 9], but has also been used in physics [e.g. 3], time series analytics [8], and finance. The core xarray data structures (the DataArray and the Dataset) are extensible through subclassing or the preferred approach of composition. We also provide an extensible high-level accessor interface to allow users to implement domain specific methods on xarray data objects.” | content_tags.findings[10] |
| quantity | 100s of GB or TBs — scale of out-of-memory datasets supported through dask — “Out-of-core computation: xarray’s data structures can be backed by dask [21] instead of NumPy to support parallel and streaming computation on data that does not fit into memory, up to 100s of GB or TBs in size. Such large datasets (“big data”) are increasingly prevalent in science.” | content_tags.quantities[0] |
| quantity | over 1,500 unit tests — size of the xarray test suite — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” | content_tags.quantities[1] |
| quantity | two main data structures — DataArray and Dataset in the xarray model — “NetCDF forms the basis of the xarray data model and provides a natural and portable serialization format. Building on netCDF, xarray features two main data structures: the DataArray and the Dataset. The API for these data structures is summarized in the following sections and in Figure 2.” | content_tags.quantities[2] |
| quantity | any number of dimensions — rank permitted for DataArrays and coordinates — “DataArray objects inside a Dataset may have any number of dimensions but are presumed to share a common coordinate system. Coordinates can also have any number of dimensions but denote constant/independent quantities, unlike the varying/dependent quantities that belong in data. Figure 3 illustrates these concepts for an example Dataset containing meteorological data.” | content_tags.quantities[3] |
| variable | timestamps (labels for time-series periods or points) — “Scientific data is inherently labeled. For example, time series data includes timestamps that label individual periods or points in time, spatial data has coordinates (e.g. longitude, latitude, elevation), and model or laboratory experiments are often identified by unique identifiers. Figure 1 provides an example of a labeled dataset. In this case the data is a map of global air temperature from a numeric weather model. The labels on this particular dataset are time (e.g. “2016-05-01”), longitude (x-axis), and latitude (y-axis).” | content_tags.variables[0] |
| variable | coordinates (spatial and ancillary labels for data points) — “Scientific data is inherently labeled. For example, time series data includes timestamps that label individual periods or points in time, spatial data has coordinates (e.g. longitude, latitude, elevation), and model or laboratory experiments are often identified by unique identifiers. Figure 1 provides an example of a labeled dataset. In this case the data is a map of global air temperature from a numeric weather model. The labels on this particular dataset are time (e.g. “2016-05-01”), longitude (x-axis), and latitude (y-axis).” | content_tags.variables[1] |
| variable | dimension names (axis identifiers used for metadata-aware operations) — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | content_tags.variables[2] |
| variable | attrs (arbitrary user metadata such as units) — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | content_tags.variables[3] |
| variable | data variables (varying or dependent quantities inside a Dataset) — “DataArray objects inside a Dataset may have any number of dimensions but are presumed to share a common coordinate system. Coordinates can also have any number of dimensions but denote constant/independent quantities, unlike the varying/dependent quantities that belong in data. Figure 3 illustrates these concepts for an example Dataset containing meteorological data.” | content_tags.variables[4] |
| variable | time, y, and x (dimensions in the weather-forecast example) — “An example of how a dataset (netCDF or xarray) for a weather forecast might be structured. This dataset has three dimensions, time, y, and x, each of which is also a one-dimensional coordinate. Temperature and precipitation are three-dimensional data variables. Also included in the dataset are two-dimensional coordinates latitude and longitude, having dimensions y and x, and reference time, a zero-dimensional (scalar) coordinate.” | content_tags.variables[5] |
| variable | Temperature and precipitation (three-dimensional example data variables) — “An example of how a dataset (netCDF or xarray) for a weather forecast might be structured. This dataset has three dimensions, time, y, and x, each of which is also a one-dimensional coordinate. Temperature and precipitation are three-dimensional data variables. Also included in the dataset are two-dimensional coordinates latitude and longitude, having dimensions y and x, and reference time, a zero-dimensional (scalar) coordinate.” | content_tags.variables[6] |
| variable | latitude and longitude (two-dimensional coordinates in the example) — “An example of how a dataset (netCDF or xarray) for a weather forecast might be structured. This dataset has three dimensions, time, y, and x, each of which is also a one-dimensional coordinate. Temperature and precipitation are three-dimensional data variables. Also included in the dataset are two-dimensional coordinates latitude and longitude, having dimensions y and x, and reference time, a zero-dimensional (scalar) coordinate.” | content_tags.variables[7] |
| definition | xarray — “xarray is an open source project and Python package that provides a toolkit and data structures for N-dimensional labeled arrays. Our approach combines an application programing interface (API) inspired by pandas with the Common Data Model for self-described scientific data. Key features of the xarray package include label-based indexing and arithmetic, interoperability with the core scientific Python packages (e.g., pandas, NumPy, Matplotlib), out-of-core computation on datasets that don’t fit into memory, a wide range of serialization and input/output (I/O) options, and advanced multi-dimensional data manipulation tools such as group-by and resampling. xarray, as a data model and analytics toolkit, has been widely adopted in the geoscience community but is also used more broadly for multi-dimensional data analysis in physics, machine learning and finance.” | content_tags.definitions[0] |
| definition | network Common Data Form — “The network Common Data Form is a collection of self-describing, machine-independent binary data formats and software tools. These data formats and tools facilitate the creation, access, and sharing of scientific data stored in N-dimensional arrays, along with metadata describing the contents of each array [20]. NetCDF has become very popular in the geoscience community, and there are existing libraries for reading and writing netCDF in many programming languages, including C, Fortran, Python, Java, Matlab, and Julia.” | content_tags.definitions[1] |
| definition | dataset — “The principal data structure in the netCDF data model is the dataset. Each netCDF dataset contains dimensions, variables, and attributes, each of which are identified by a hierarchy of unique names. The dataset and variable objects may contain attributes that describe the contents, units, history, or other metadata of the object. Standardized conventions, such as the Climate and Forecast (CF) Conventions [12], allow for the associations of coordinate variables with dimensions.” | content_tags.definitions[2] |
| definition | DataArray — “The DataArray is xarray’s implementation of a labeled, multi-dimensional array. It has several key properties:” | content_tags.definitions[3] |
| definition | Dataset — “The Dataset is xarray’s multi-dimensional equivalent of a DataFrame. It is a dict-like container of labeled arrays (DataArrays) with aligned dimensions. It is designed as an in-memory representation of a netCDF dataset. In addition to the dict-like interface of the dataset itself, which can be used to access any DataArray in a Dataset, datasets have four key properties:” | content_tags.definitions[4] |
| definition | Dimensions — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | content_tags.definitions[5] |
| definition | Coordinates — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | content_tags.definitions[6] |
| definition | Alignment — “Alignment: xarray supports database-like join operations for combining xarray objects along common coordinates.” | content_tags.definitions[7] |
| definition | Split-apply-combine — “Split-apply-combine: xarray includes N-dimensional grouped operations implementing the split-apply-combine strategy [24].” | content_tags.definitions[8] |
| definition | Out-of-core computation — “Out-of-core computation: xarray’s data structures can be backed by dask [21] instead of NumPy to support parallel and streaming computation on data that does not fit into memory, up to 100s of GB or TBs in size. Such large datasets (“big data”) are increasingly prevalent in science.” | content_tags.definitions[9] |
Concepts
terms this paper carries, each with the sentence it was cut from — the layer that joins one paper to another. Terms with no quote yet are on CONCEPTS.html.
| concept | xarray — not defined in this paper's own words; open-source toolkit for labeled N-dimensional arrays — “xarray is an open source project and Python package that provides a toolkit and data structures for N-dimensional labeled arrays. Our approach combines an application programing interface (API) inspired by pandas with the Common Data Model for self-described scientific data. Key features of the xarray package include label-based indexing and arithmetic, interoperability with the core scientific Python packages (e.g., pandas, NumPy, Matplotlib), out-of-core computation on datasets that don’t fit into memory, a wide range of serialization and input/output (I/O) options, and advanced multi-dimensional data manipulation tools such as group-by and resampling. xarray, as a data model and analytics toolkit, has been widely adopted in the geoscience community but is also used more broadly for multi-dimensional data analysis in physics, machine learning and finance.” | concepts[0] |
| concept | Python package — not defined in this paper's own words; software class assigned to xarray — “xarray is an open source project and Python package that provides a toolkit and data structures for N-dimensional labeled arrays. Our approach combines an application programing interface (API) inspired by pandas with the Common Data Model for self-described scientific data. Key features of the xarray package include label-based indexing and arithmetic, interoperability with the core scientific Python packages (e.g., pandas, NumPy, Matplotlib), out-of-core computation on datasets that don’t fit into memory, a wide range of serialization and input/output (I/O) options, and advanced multi-dimensional data manipulation tools such as group-by and resampling. xarray, as a data model and analytics toolkit, has been widely adopted in the geoscience community but is also used more broadly for multi-dimensional data analysis in physics, machine learning and finance.” | concepts[1] |
| concept | N-dimensional labeled arrays — not defined in this paper's own words; arrays carrying dimension and coordinate labels — “xarray is an open source project and Python package that provides a toolkit and data structures for N-dimensional labeled arrays. Our approach combines an application programing interface (API) inspired by pandas with the Common Data Model for self-described scientific data. Key features of the xarray package include label-based indexing and arithmetic, interoperability with the core scientific Python packages (e.g., pandas, NumPy, Matplotlib), out-of-core computation on datasets that don’t fit into memory, a wide range of serialization and input/output (I/O) options, and advanced multi-dimensional data manipulation tools such as group-by and resampling. xarray, as a data model and analytics toolkit, has been widely adopted in the geoscience community but is also used more broadly for multi-dimensional data analysis in physics, machine learning and finance.” | concepts[2] |
| concept | Common Data Model — not defined in this paper's own words; self-describing scientific-data model combined with the API — “xarray is an open source project and Python package that provides a toolkit and data structures for N-dimensional labeled arrays. Our approach combines an application programing interface (API) inspired by pandas with the Common Data Model for self-described scientific data. Key features of the xarray package include label-based indexing and arithmetic, interoperability with the core scientific Python packages (e.g., pandas, NumPy, Matplotlib), out-of-core computation on datasets that don’t fit into memory, a wide range of serialization and input/output (I/O) options, and advanced multi-dimensional data manipulation tools such as group-by and resampling. xarray, as a data model and analytics toolkit, has been widely adopted in the geoscience community but is also used more broadly for multi-dimensional data analysis in physics, machine learning and finance.” | concepts[3] |
| concept | pandas — not defined in this paper's own words; tabular-analysis library inspiring xarray's API — “xarray implements data structures and an analytics toolkit for multi-dimensional labeled arrays strongly inspired by pandas. While pandas includes a data structure called the Panel for three dimensional data, its fixed rank design make it unsuitable for applications that require arbitrary rank arrays. Additionally, many of the features that make the pandas DataFrame and Series objects so useful, are not fully available on the Panel. Our approach with xarray adopts Unidata’s self-describing Common Data Model on which the network Common Data Form (netCDF) is built [20, 7]. NetCDF provides a well-defined data model for labeled N-dimensional array-oriented scientific data analysis.” | concepts[4] |
| concept | Panel — not defined in this paper's own words; fixed-rank pandas structure contrasted with xarray — “xarray implements data structures and an analytics toolkit for multi-dimensional labeled arrays strongly inspired by pandas. While pandas includes a data structure called the Panel for three dimensional data, its fixed rank design make it unsuitable for applications that require arbitrary rank arrays. Additionally, many of the features that make the pandas DataFrame and Series objects so useful, are not fully available on the Panel. Our approach with xarray adopts Unidata’s self-describing Common Data Model on which the network Common Data Form (netCDF) is built [20, 7]. NetCDF provides a well-defined data model for labeled N-dimensional array-oriented scientific data analysis.” | concepts[5] |
| concept | Scientific data — not defined in this paper's own words; inherently labeled observations or experiments — “Scientific data is inherently labeled. For example, time series data includes timestamps that label individual periods or points in time, spatial data has coordinates (e.g. longitude, latitude, elevation), and model or laboratory experiments are often identified by unique identifiers. Figure 1 provides an example of a labeled dataset. In this case the data is a map of global air temperature from a numeric weather model. The labels on this particular dataset are time (e.g. “2016-05-01”), longitude (x-axis), and latitude (y-axis).” | concepts[6] |
| concept | Unlabeled, N-dimensional arrays — not defined in this paper's own words; scientific-computing arrays lacking associated metadata — “Unlabeled, N-dimensional arrays of numbers (e.g., NumPy’s ndarray) are the most widely used data structure in scientific computing. However, they lack a meaningful representation of the metadata associated with their data. Implementing such functionality is left to individual users and domain-specific packages. As a result, programmers frequently encounter pitfalls in the form of questions like “is the time axis of my array in the first or third index position?” or “does my array of timestamps still align with my data after resampling?”.” | concepts[7] |
| concept | metadata — not defined in this paper's own words; labels and descriptions associated with data — “Unlabeled, N-dimensional arrays of numbers (e.g., NumPy’s ndarray) are the most widely used data structure in scientific computing. However, they lack a meaningful representation of the metadata associated with their data. Implementing such functionality is left to individual users and domain-specific packages. As a result, programmers frequently encounter pitfalls in the form of questions like “is the time axis of my array in the first or third index position?” or “does my array of timestamps still align with my data after resampling?”.” | concepts[8] |
| concept | network Common Data Form — not defined in this paper's own words; self-describing formats and tools for scientific arrays — “The network Common Data Form is a collection of self-describing, machine-independent binary data formats and software tools. These data formats and tools facilitate the creation, access, and sharing of scientific data stored in N-dimensional arrays, along with metadata describing the contents of each array [20]. NetCDF has become very popular in the geoscience community, and there are existing libraries for reading and writing netCDF in many programming languages, including C, Fortran, Python, Java, Matlab, and Julia.” | concepts[9] |
| concept | collection of self-describing, machine-independent binary data formats and software tools — not defined in this paper's own words; class assigned to network Common Data Form — “The network Common Data Form is a collection of self-describing, machine-independent binary data formats and software tools. These data formats and tools facilitate the creation, access, and sharing of scientific data stored in N-dimensional arrays, along with metadata describing the contents of each array [20]. NetCDF has become very popular in the geoscience community, and there are existing libraries for reading and writing netCDF in many programming languages, including C, Fortran, Python, Java, Matlab, and Julia.” | concepts[10] |
| concept | dimensions, variables, and attributes — not defined in this paper's own words; named elements contained in a netCDF dataset — “The principal data structure in the netCDF data model is the dataset. Each netCDF dataset contains dimensions, variables, and attributes, each of which are identified by a hierarchy of unique names. The dataset and variable objects may contain attributes that describe the contents, units, history, or other metadata of the object. Standardized conventions, such as the Climate and Forecast (CF) Conventions [12], allow for the associations of coordinate variables with dimensions.” | concepts[11] |
| concept | netCDF dataset — not defined in this paper's own words; dataset object containing named dimensions, variables, and attributes — “The principal data structure in the netCDF data model is the dataset. Each netCDF dataset contains dimensions, variables, and attributes, each of which are identified by a hierarchy of unique names. The dataset and variable objects may contain attributes that describe the contents, units, history, or other metadata of the object. Standardized conventions, such as the Climate and Forecast (CF) Conventions [12], allow for the associations of coordinate variables with dimensions.” | concepts[12] |
| concept | DataArray — not defined in this paper's own words; labeled multidimensional array implementation — “The DataArray is xarray’s implementation of a labeled, multi-dimensional array. It has several key properties:” | concepts[13] |
| concept | Dataset — not defined in this paper's own words; aligned DataArray container and in-memory netCDF representation — “The Dataset is xarray’s multi-dimensional equivalent of a DataFrame. It is a dict-like container of labeled arrays (DataArrays) with aligned dimensions. It is designed as an in-memory representation of a netCDF dataset. In addition to the dict-like interface of the dataset itself, which can be used to access any DataArray in a Dataset, datasets have four key properties:” | concepts[14] |
| concept | DataFrame — not defined in this paper's own words; tabular structure for which Dataset is the multidimensional equivalent — “The Dataset is xarray’s multi-dimensional equivalent of a DataFrame. It is a dict-like container of labeled arrays (DataArrays) with aligned dimensions. It is designed as an in-memory representation of a netCDF dataset. In addition to the dict-like interface of the dataset itself, which can be used to access any DataArray in a Dataset, datasets have four key properties:” | concepts[15] |
| concept | dict-like container — not defined in this paper's own words; container class assigned to Dataset — “The Dataset is xarray’s multi-dimensional equivalent of a DataFrame. It is a dict-like container of labeled arrays (DataArrays) with aligned dimensions. It is designed as an in-memory representation of a netCDF dataset. In addition to the dict-like interface of the dataset itself, which can be used to access any DataArray in a Dataset, datasets have four key properties:” | concepts[16] |
| concept | Dimensions — not defined in this paper's own words; names replacing numeric axes in operations — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | concepts[17] |
| concept | ancillary variables — not defined in this paper's own words; role assigned to coordinates — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | concepts[18] |
| concept | Names and attributes — not defined in this paper's own words; user metadata not generally interpreted by xarray — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | concepts[19] |
| concept | coordinates — not defined in this paper's own words; operation-persistent labels interpreted by xarray — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | concepts[20] |
| concept | varying/dependent quantities — not defined in this paper's own words; Dataset data contrasted with constant coordinate quantities — “DataArray objects inside a Dataset may have any number of dimensions but are presumed to share a common coordinate system. Coordinates can also have any number of dimensions but denote constant/independent quantities, unlike the varying/dependent quantities that belong in data. Figure 3 illustrates these concepts for an example Dataset containing meteorological data.” | concepts[21] |
| concept | named dimensions — not defined in this paper's own words; dimensions identified by name rather than position — “Label-based indexing: Similarly to pandas objects, xarray objects support both integer- and label-based lookups along each dimension. However, xarray objects also have named dimensions, so you can optionally use dimension names instead of relying on the positional ordering of dimensions.” | concepts[22] |
| concept | positional ordering of dimensions — not defined in this paper's own words; axis-position convention avoided by named indexing — “Label-based indexing: Similarly to pandas objects, xarray objects support both integer- and label-based lookups along each dimension. However, xarray objects also have named dimensions, so you can optionally use dimension names instead of relying on the positional ordering of dimensions.” | concepts[23] |
| concept | dimension names — not defined in this paper's own words; identifiers governing arithmetic broadcasting — “Arithmetic: arithmetic between xarray objects vectorizes based on dimension names, automatically looping (broadcasting) over each distinct dimension. This eliminates the need to insert dummy dimensions of size one to facilitate broadcasting, a common pattern with NumPy.” | concepts[24] |
| concept | dummy dimensions of size one — not defined in this paper's own words; broadcasting scaffolds made unnecessary by named arithmetic — “Arithmetic: arithmetic between xarray objects vectorizes based on dimension names, automatically looping (broadcasting) over each distinct dimension. This eliminates the need to insert dummy dimensions of size one to facilitate broadcasting, a common pattern with NumPy.” | concepts[25] |
| concept | Alignment — not defined in this paper's own words; join operation along common coordinates — “Alignment: xarray supports database-like join operations for combining xarray objects along common coordinates.” | concepts[26] |
| concept | common coordinates — not defined in this paper's own words; keys along which xarray objects are combined — “Alignment: xarray supports database-like join operations for combining xarray objects along common coordinates.” | concepts[27] |
| concept | Split-apply-combine — not defined in this paper's own words; N-dimensional grouping strategy — “Split-apply-combine: xarray includes N-dimensional grouped operations implementing the split-apply-combine strategy [24].” | concepts[28] |
| concept | xarray plotting functionality — not defined in this paper's own words; Matplotlib-compatible plotting layer — “Plotting: xarray plotting functionality is a thin wrapper around the popular Matplotlib library. xarray uses the syntax and function names from Matplotlib whenever possible, resulting in a seamless transition between the two.” | concepts[29] |
| concept | thin wrapper — not defined in this paper's own words; architectural class assigned to xarray plotting — “Plotting: xarray plotting functionality is a thin wrapper around the popular Matplotlib library. xarray uses the syntax and function names from Matplotlib whenever possible, resulting in a seamless transition between the two.” | concepts[30] |
| concept | Missing Data — not defined in this paper's own words; missing values handled across xarray operations — “Missing Data: xarray smoothly handles missing data in all operations, including arithmetic, alignment and aggregation.” | concepts[31] |
| concept | serialization and I/O — not defined in this paper's own words; file-format access exposed by xarray backends — “Serialization and I/O: xarray supports direct serialization and I/O to several file formats including pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF by integrating with third-party libraries. Additional serialization formats for 1-dimensional data are available through pandas.” | concepts[32] |
| concept | third-party libraries — not defined in this paper's own words; libraries integrated to implement formats — “Serialization and I/O: xarray supports direct serialization and I/O to several file formats including pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF by integrating with third-party libraries. Additional serialization formats for 1-dimensional data are available through pandas.” | concepts[33] |
| concept | Out-of-core computation — not defined in this paper's own words; processing data that does not fit in memory — “Out-of-core computation: xarray’s data structures can be backed by dask [21] instead of NumPy to support parallel and streaming computation on data that does not fit into memory, up to 100s of GB or TBs in size. Such large datasets (“big data”) are increasingly prevalent in science.” | concepts[34] |
| concept | dask — not defined in this paper's own words; parallel-computing backing for xarray structures — “Out-of-core computation: xarray’s data structures can be backed by dask [21] instead of NumPy to support parallel and streaming computation on data that does not fit into memory, up to 100s of GB or TBs in size. Such large datasets (“big data”) are increasingly prevalent in science.” | concepts[35] |
| concept | parallel and streaming computation — not defined in this paper's own words; execution modes supported by dask backing — “Out-of-core computation: xarray’s data structures can be backed by dask [21] instead of NumPy to support parallel and streaming computation on data that does not fit into memory, up to 100s of GB or TBs in size. Such large datasets (“big data”) are increasingly prevalent in science.” | concepts[36] |
| concept | large test suite — not defined in this paper's own words; unit-test collection for quality control — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” | concepts[37] |
| concept | TravisCI — not defined in this paper's own words; Linux continuous-integration service — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” | concepts[38] |
| concept | Appveyor — not defined in this paper's own words; Windows continuous-integration service — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” | concepts[39] |
| concept | sample data — not defined in this paper's own words; source-distributed inputs for reproducing documentation examples — “xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.” | concepts[40] |
| concept | compiled dependencies — not defined in this paper's own words; dependencies used to accelerate the pure-Python package — “xarray is implemented in pure Python and relies on compiled dependencies for speed.” | concepts[41] |
| concept | core scientific Python libraries — not defined in this paper's own words; domain-agnostic foundations extended by xarray — “xarray was written in a modular, objected-oriented way, to build upon and extend the core scientific Python libraries in a domain-agnostic fashion. The xarray documentation is complete with a wide range of examples and a number of tutorials that use real-world datasets that are available in the xarray repository. We have intentionally avoided including domain-specific functionality in the library, leaving that to third party libraries. It has been widely adopted in the geoscience community [e.g. 6, 10, 9], but has also been used in physics [e.g. 3], time series analytics [8], and finance. The core xarray data structures (the DataArray and the Dataset) are extensible through subclassing or the preferred approach of composition. We also provide an extensible high-level accessor interface to allow users to implement domain specific methods on xarray data objects.” | concepts[42] |
| concept | domain-specific packages — not defined in this paper's own words; specialized packages with which xarray interfaces — “xarray builds on top of, and seamlessly interoperates with, the core scientific Python packages, such as NumPy, SciPy, Matplotlib [13], and pandas. xarray provides a range of backends for serialization and input/output (IO), including the Pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF file formats. Leveraging the dask parallel computing library [21], xarray can optionally perform efficient parallel, out-of-core analysis on datasets that are too large to fit into memory. Finally, xarray interfaces with existing domain-specific packages such as UV-CDAT [25], Iris [17], and Cartopy [18].” | concepts[43] |
| concept | consistency of labels — not defined in this paper's own words; invariant maintained by every xarray operation — “The core motivation for developing xarray was to provide labeled data tools for N-dimensional arrays that render such questions moot. Every operation in xarray both relies on and maintains the consistency of labels.” | concepts[44] |
| concept | labeled, multi-dimensional array — not defined in this paper's own words; array abstraction implemented by DataArray — “The DataArray is xarray’s implementation of a labeled, multi-dimensional array. It has several key properties:” | concepts[45] |
| concept | core metadata-aware operations — not defined in this paper's own words; operations enabled by dimension and coordinate metadata — “xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.” | concepts[46] |
| concept | domain-specific functionality — not defined in this paper's own words; features deliberately delegated to third parties — “xarray was written in a modular, objected-oriented way, to build upon and extend the core scientific Python libraries in a domain-agnostic fashion. The xarray documentation is complete with a wide range of examples and a number of tutorials that use real-world datasets that are available in the xarray repository. We have intentionally avoided including domain-specific functionality in the library, leaving that to third party libraries. It has been widely adopted in the geoscience community [e.g. 6, 10, 9], but has also been used in physics [e.g. 3], time series analytics [8], and finance. The core xarray data structures (the DataArray and the Dataset) are extensible through subclassing or the preferred approach of composition. We also provide an extensible high-level accessor interface to allow users to implement domain specific methods on xarray data objects.” | concepts[47] |
| concept | composition — not defined in this paper's own words; preferred extension approach for xarray structures — “xarray was written in a modular, objected-oriented way, to build upon and extend the core scientific Python libraries in a domain-agnostic fashion. The xarray documentation is complete with a wide range of examples and a number of tutorials that use real-world datasets that are available in the xarray repository. We have intentionally avoided including domain-specific functionality in the library, leaving that to third party libraries. It has been widely adopted in the geoscience community [e.g. 6, 10, 9], but has also been used in physics [e.g. 3], time series analytics [8], and finance. The core xarray data structures (the DataArray and the Dataset) are extensible through subclassing or the preferred approach of composition. We also provide an extensible high-level accessor interface to allow users to implement domain specific methods on xarray data objects.” | concepts[48] |
| concept | high-level accessor interface — not defined in this paper's own words; extension interface for domain-specific methods — “xarray was written in a modular, objected-oriented way, to build upon and extend the core scientific Python libraries in a domain-agnostic fashion. The xarray documentation is complete with a wide range of examples and a number of tutorials that use real-world datasets that are available in the xarray repository. We have intentionally avoided including domain-specific functionality in the library, leaving that to third party libraries. It has been widely adopted in the geoscience community [e.g. 6, 10, 9], but has also been used in physics [e.g. 3], time series analytics [8], and finance. The core xarray data structures (the DataArray and the Dataset) are extensible through subclassing or the preferred approach of composition. We also provide an extensible high-level accessor interface to allow users to implement domain specific methods on xarray data objects.” | concepts[49] |
| concept | team of volunteers — not defined in this paper's own words; development and support workforce — “xarray is developed and supported by a team of volunteers. The primary avenue for user support is StackOverflow [4], with the “xarray-python” tag. Additionally, we use GitHub for a bug tracker (https://github.com/pydata/xarray/issues) and maintain the “xarray” mailing list on Google Groups (https://groups.google.com/forum/#!forum/xarray).” | concepts[50] |
Full text — tagged
The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.
xarrayterm: N-D labeled Arrays and Datasetsterm in Python Abstract xarrayterm is an open source project and Python packageterm that provides a toolkit and data structures for N-dimensional labeled arraysterm. Our approach combines an application programing interface (API) inspired by pandasterm with the Common Data Modelterm for self-described scientific dataterm. Key features of the xarrayterm package include label-based indexing and arithmetic, interoperability with the core scientific Python packagesterm (e.g., pandasterm, NumPy, Matplotlib), out-of-core computationterm on datasetsterm that don’t fit into memory, a wide range of serialization and input/output (I/O) options, and advanced multi-dimensional data manipulation tools such as group-by and resampling. xarrayterm, as a data model and analytics toolkit, has been widely adopted in the geoscience community but is also used more broadly for multi-dimensional data analysis in physics, machine learning and finance.
Python pandasterm netCDF multidimensional data data handling data analysis Metrics (1) Overview Introduction Python has emerged as a leading programing language for both the physical sciences and data sciences. At the core of modern scientific computing and analysis in Python are the NumPy [14] and SciPy [22] packages, which provide a robust N-dimensional array object and the fundamental operations required for science and engineering applications. Much of the success of Python in data science and business analytics is due to pandasterm [16], which introduced intuitive and fast tabular data analysis tools to Python, inspired by R’s data.frame [19]. The pandasterm DataFrameterm and Series objects provide unparalleled analysis tools for data alignmentterm, resampling, grouping, pivoting, and aggregation in Python.
xarrayterm implements data structures and an analytics toolkit for multi-dimensional labeled arrays strongly inspired by pandasterm. While pandasterm includes a data structure called the Panelterm for three dimensional data, its fixed rank design make it unsuitable for applications that require arbitrary rank arrays. Additionally, many of the features that make the pandasterm DataFrameterm and Series objects so useful, are not fully available on the Panelterm. Our approach with xarrayterm adopts Unidata’s self-describing Common Data Modelterm on which the network Common Data Formterm (netCDF) is built [20, 7]. NetCDF provides a well-defined data model for labeled N-dimensional array-oriented scientific dataterm analysis.
xarrayterm builds on top of, and seamlessly interoperates with, the core scientific Python packagesterm, such as NumPy, SciPy, Matplotlib [13], and pandasterm. xarrayterm provides a range of backends for serialization and input/output (IO), including the Pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF file formats. Leveraging the daskterm parallel computing library [21], xarrayterm can optionally perform efficient parallel, out-of-core analysis on datasetsterm that are too large to fit into memory. Finally, xarrayterm interfaces with existing domain-specific packagesterm such as UV-CDAT [25], Iris [17], and Cartopy [18].
Purpose: Your data has labels; you should use them Scientific dataterm is inherently labeled. For example, time series data includes timestampsterm·var that label individual periods or points in time, spatial data has coordinatesterm (e.g. longitude, latitude, elevation), and model or laboratory experiments are often identified by unique identifiers. Figure 1 provides an example of a labeled datasetterm. In this case the data is a map of global air temperature from a numeric weather model. The labels on this particular datasetterm are time (e.g. “2016-05-01”), longitude (x-axis), and latitude (y-axis).
An example of a multidimensional labeled array. This figure (map) is showing the global surface air temperature for May 1, 2016 from ERA-Interim Reanalyis [11]. The map is labeled with the array's coordinatesterm: longitude and latitude.
Unlabeled, N-dimensional arraysterm of numbers (e.g., NumPy’s ndarray) are the most widely used data structure in scientific computing. However, they lack a meaningful representation of the metadataterm associated with their data. Implementing such functionality is left to individual users and domain-specific packagesterm. As a result, programmers frequently encounter pitfalls in the form of questions like “is the time axis of my array in the first or third index position?” or “does my array of timestampsterm·var still align with my data after resampling?”.
The core motivation for developing xarrayterm was to provide labeled data tools for N-dimensional arrays that render such questions moot. Every operation in xarrayterm both relies on and maintains the consistency of labelsterm.
NetCDF The network Common Data Formterm is a collection of self-describing, machine-independent binary data formats and software toolsterm. These data formats and tools facilitate the creation, access, and sharing of scientific dataterm stored in N-dimensional arrays, along with metadataterm describing the contents of each array [20]. NetCDF has become very popular in the geoscience community, and there are existing libraries for reading and writing netCDF in many programming languages, including C, Fortran, Python, Java, Matlab, and Julia.
The principal data structure in the netCDF data model is the datasetterm. Each netCDF datasetterm contains dimensions, variables, and attributesterm, each of which are identified by a hierarchy of unique names. The datasetterm and variable objects may contain attributes that describe the contents, units, history, or other metadataterm of the object. Standardized conventions, such as the Climate and Forecast (CF) Conventions [12], allow for the associations of coordinateterm variables with dimensionsterm.
Implementation and architecture NetCDF forms the basis of the xarrayterm data model and provides a natural and portable serialization format. Building on netCDF, xarrayterm features two main data structures: the DataArrayterm and the Datasetterm. The API for these data structures is summarized in the following sections and in Figure 2.
An overview of xarrayterm’s main data structures. Types are annotated using Python 3 style type hints [23]. “Mapping of” denotes an ordered mapping with values of the given type.
DataArrayterm The DataArrayterm is xarrayterm’s implementation of a labeled, multi-dimensional arrayterm. It has several key properties: data: N-dimensional array (NumPy or daskterm) holding the array's values, coords: dict-like containerterm of arrays (coordinatesterm) that label each point (e.g., 1-dimensional arrays of numbers, datetime objects or strings), dims: dimension namesterm for each axis [e.g., (‘time’, ‘latitude’, ‘longitude’)], attrsterm·var: OrderedDict holding arbitrary metadataterm (e.g. units or descriptions), and name: an arbitrary name for the array.
xarrayterm uses dims and coords to enable its core metadata-aware operationsterm. Dimensionsterm provide names that xarrayterm uses instead of the axis argument found in many NumPy functions. Coordinatesterm are ancillary variablesterm used to enable fast label based indexing and alignmentterm, building on the functionality of the pandasterm Index. DataArrayterm objects also can have a name and can hold arbitrary metadataterm in the form of their attrsterm·var property, which can be used to further describe data (e.g. by providing units). Names and attributesterm are strictly for users and user-written code; in general xarrayterm makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarrayterm does interpret and persist coordinatesterm in operations that transform xarrayterm objects.
Datasetterm The Datasetterm is xarrayterm’s multi-dimensional equivalent of a DataFrameterm. It is a dict-like containerterm of labeled arrays (DataArraysterm) with aligned dimensionsterm. It is designed as an in-memory representation of a netCDF datasetterm. In addition to the dict-like interface of the datasetterm itself, which can be used to access any DataArrayterm in a Datasetterm, datasetsterm have four key properties: data_vars: OrderedDict of DataArrayterm objects corresponding to data variablesterm·var, coords: OrderedDict of DataArrayterm objects intended to label points used in data_vars (e.g., 1-dimensional arrays of numbers, datetime objects or strings), dims: dictionary mapping from dimension namesterm to the fixed length of each dimensionterm (e.g., {‘x’: 6, ‘y’: 6, ‘time’: 8}), and attrsterm·var: OrderedDict to hold arbitrary metadataterm pertaining to the datasetterm.
DataArrayterm objects inside a Datasetterm may have any number of dimensionsterm but are presumed to share a common coordinateterm system. Coordinatesterm can also have any number of dimensionsterm but denote constant/independent quantities, unlike the varying/dependent quantitiesterm that belong in data. Figure 3 illustrates these concepts for an example Datasetterm containing meteorological data.
An example of how a datasetterm (netCDF or xarrayterm) for a weather forecast might be structured. This datasetterm has three dimensionsterm, time, y, and xterm·var, each of which is also a one-dimensional coordinateterm. Temperature and precipitationterm·var are three-dimensional data variablesterm·var. Also included in the datasetterm are two-dimensional coordinatesterm latitude and longitudeterm·var, having dimensionsterm y and x, and reference time, a zero-dimensional (scalar) coordinateterm.
Core xarrayterm Features xarrayterm includes a powerful and growing feature set. The following list highlights some of the key features available in xarrayterm. The xarrayterm documentation [2] includes a complete description of available features and their usage.
Label-based indexing: Similarly to pandasterm objects, xarrayterm objects support both integer- and label-based lookups along each dimensionterm. However, xarrayterm objects also have named dimensionsterm, so you can optionally use dimension namesterm instead of relying on the positional ordering of dimensionsterm.
Arithmetic: arithmetic between xarrayterm objects vectorizes based on dimension namesterm, automatically looping (broadcasting) over each distinct dimensionterm. This eliminates the need to insert dummy dimensions of size oneterm to facilitate broadcasting, a common pattern with NumPy.
Aggregation: calculation of statistics (e.g. sum) along a dimensionterm of an xarrayterm object can be done by dimension nameterm instead of an integer axis number.
Alignmentterm: xarrayterm supports database-like join operations for combining xarrayterm objects along common coordinatesterm.
Split-apply-combineterm: xarrayterm includes N-dimensional grouped operations implementing the split-apply-combineterm strategy [24].
Resampling and rolling window operations: Utilizing the efficient resampling methods from pandasterm and rolling window operations from Bottleneck [15], xarrayterm offers a robust set of resampling and rolling window operations along a single dimensionterm.
Plotting: xarray plotting functionalityterm is a thin wrapperterm around the popular Matplotlib library. xarrayterm uses the syntax and function names from Matplotlib whenever possible, resulting in a seamless transition between the two.
Missing Dataterm: xarrayterm smoothly handles missing dataterm in all operations, including arithmetic, alignmentterm and aggregation.
Interactivity with pandasterm: xarrayterm objects seamlessly to convert to and from pandasterm objects to interact with the rest of the PyData ecosystem.
Serialization and I/Oterm: xarrayterm supports direct serialization and I/Oterm to several file formats including pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF by integrating with third-party librariesterm. Additional serialization formats for 1-dimensional data are available through pandasterm.
Out-of-core computationterm: xarrayterm’s data structures can be backed by daskterm [21] instead of NumPy to support parallel and streaming computationterm on data that does not fit into memory, up to 100s of GB or TBs in size. Such large datasetsterm (“big data”) are increasingly prevalent in science.
Quality control xarrayterm is provided with a large test suiteterm comprised of over 1,500 unit tests. These tests cover the core xarrayterm functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCIterm (Linux) [5] and Appveyorterm (Windows) [1] continuous integration systems. A selection of sample dataterm is also distributed with the source code, allowing users to reproduce any examples in the xarrayterm documentation.
(2) Availability Operating system Linux, Windows and Mac OS X.
Programming language Python, versions 2.7, 3.4 and later.
Additional system requirements None.
Dependencies xarrayterm is implemented in pure Python and relies on compiled dependenciesterm for speed.
NumPy: 1.7 or later pandasterm: 0.15.0 or later netcdf4-python: (optional) used for reading and writing netCDF files SciPy: (optional) used as a fallback for reading/writing netCDF3 Pydap: (optional) used as a fallback for accessing OPeNDAP h5netcdf: (optional) an alternative library for reading and writing netCDF4 files that does not use the netCDF-C libraries PyNIO: (optional) for reading GRIB1/2 and other geoscience specific file formats Bottleneck: (optional) speeds up NaN-skipping and rolling window aggregations by a large factor cyordereddict: (optional) speeds up most internal operations with xarrayterm data structures Daskterm: (optional) required for out-of-core parallel computation Matplotlib: (optional) required for plotting Cartopy: (optional) required for plotting maps seaborn: (optional) additional plotting functionality List of contributors Alex Kleeman, The Climate Corporation Alistair Miles, Wellcome Trust Centre for Human Genetics, University of Oxford Andreas Hilboll, Institute of Environmental Physics, University of Bremen Anna Kuznetsova, The Climate Corporation Antony Lee, Department of Physics, University of California, Berkeley Bas Hoonhout, Deltares, The Netherlands Benjamin Root, Atmospheric and Environmental Research, Inc.
Benoit Bovy, Department of Astrophysics, Geophysics and Oceanography, University of Lige Chun-Wei Yuan, Institute for Health Metrics and Evaluation, University of Washington Christoph Deil, Max Planck Institute for Nuclear Physics Clark Fitzgerald, Department of Statistics, University of California, Davis Dean Pospisil, Institute for Learning and Brain Sciences, University of Washington Edward Richards, Scripps Institution of Oceanography, University of California, San Diego Erik Welch, Continuum Analytics Eugene Brevdo, Google Research Fabien Maussion, Institute of Atmospheric and Cryospheric Sciences, University of Innsbruck Filipe Fernandes, Universidade de So Paulo Guido Imperiale, Legal & General Hauser Mathias, Institute for Atmospheric and Climate Science, ETH Zurich Igor Babuschkin, School of Physics and Astronomy, University of Manchester Jeffrey Gerard, The Climate Corporation Jeremy McGibbon, Department of Atmospheric Sciences, University of Washington Jonathan Chambers, UCL Energy Institute, London UK Julia Signell, Department of Civil and Environmental Engineering, Princeton University Joseph HammanpersName, National Center for Atmospheric Research Johnnie Gray, University College London Jonathan Helmus, Environmental Science Division, Argonne National Laboratory Kelsey Jordahl, Planet Labs Maciek Swat, University of Pennsylvania Marco Zhlke, Brockmann Consult GmbH Markel Garca-Dez, Catalan Institute of Climate Sciences Markus Gonser Maximilian Roos, Sixty Capital Mike Graham, Edgestream Partners Nikolay Koldunov, Climate Service Center Germany Peter Cable, Raytheon Phillip J. Wolfram, Fluid and Solid Mechanics (T-3), Theoretical Division, Los Alamos National Laboratory Rafael Guedes, MetOcean Solutions Ltd Robin Wilson, Geography & Environment, University of Southampton, UK Ryan Abernathey, Department of Earth and Environmental Sciences, Columbia University Scott Sinclair, Satellite Applications and Hydrology Group, Dept. Civil Engineering, University of KwaZulu-Natal Sébastien Celles, Thermal Science and Energy Department, Poitiers Institute of Technology (IUT) Spencer A. Hill, Program in Atmospheric and Oceanic Sciences, Princeton University Stefan Pfenninger, Department of Environmental Systems Science, ETH Zrich Stephan HoyerpersName, Google Research Steven E. Pav, Gilgamath Consulting Takeshi Kanmae, National Institute of Polar Research, Japan Thomas Kluyver, Computational Modeling Group, University of Southampton Todd Small, The Climate Corporation Valliappa Lakshmanan, Google Yves Delley, Department of Physics, ETH Zurich Software location Archive Name: Zenodo Persistent identifier: https://doi.org/10.5281/zenodo.264282 License: Apache, v2.0 Publisher: Zenodo Version published: 0.9.1 Date published: January 30, 2017 Code repository Name: GitHub Identifier: http://github.com/pydata/xarrayterm License: Apache, v2.0 Date published: January 30, 2017 Language English.
(3) Reuse potential xarrayterm was written in a modular, objected-oriented way, to build upon and extend the core scientific Python librariesterm in a domain-agnostic fashion. The xarrayterm documentation is complete with a wide range of examples and a number of tutorials that use real-world datasetsterm that are available in the xarrayterm repository. We have intentionally avoided including domain-specific functionalityterm in the library, leaving that to third party librariesterm. It has been widely adopted in the geoscience community [e.g. 6, 10, 9], but has also been used in physics [e.g. 3], time series analytics [8], and finance. The core xarrayterm data structures (the DataArrayterm and the Datasetterm) are extensible through subclassing or the preferred approach of compositionterm. We also provide an extensible high-level accessor interfaceterm to allow users to implement domain specific methods on xarrayterm data objects.
xarrayterm is developed and supported by a team of volunteersterm. The primary avenue for user support is StackOverflow [4], with the “xarray-python” tag. Additionally, we use GitHub for a bug tracker (https://github.com/pydata/xarrayterm/issues) and maintain the “xarrayterm” mailing list on Google Groups (https://groups.google.com/forum/#!forum/xarrayterm).
Acknowledgements Initial development of xarrayterm was supported by The Climate Corporation. We thank Matthew Rocklin and Jim Crist for their assistance integrating xarrayterm with Daskterm, and Todd Small, Francisco Alvarez and Fabien Maussion for their feedback on early drafts of this manuscript.
Competing Interests The authors have no competing interests to declare.