xarray: N-D labeled Arrays and Datasets in Python

xarray: N-D labeled Arrays and Datasets in Python Stephan Hoyer Joe Hamman

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

xarray: N-D labeled Arrays and Datasets in Python 2017 Journal of Open Research Software 10.5334/jors.148 hoyer2017-xarray-labeled-arrays-datasets-python hoyer2017-xarray-labeled-arrays-datasets-python.txt, the project's staged copy 71fc769aac2ee47505983c27d71b7c6affc6dd1e cc-by earth-science

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

145 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them xarray's data model, architecture, capabilities, quality controls, availability, and reuse potential how to make labeled N-dimensional scientific arrays support metadata-aware analysis, interoperability, serialization, and out-of-core computation the xarray Python package, combining a pandas-inspired API with the self-describing Common Data Model for arbitrary-rank labeled arrays method metadata-aware manipulation of labeled multidimensional scientific data system-or-tool-buildingsystem-or-tool-building Every operation consumes and preserves labels so data and metadata stay aligned.label consistency Dimension names replace positional axes for indexing and aggregation.named dimensions Ancillary coordinate variables build on pandas Index for fast lookup and alignment.coordinate indexing Arithmetic loops over distinct named dimensions without dummy size-one axes.dimension-name broadcasting Database-like joins combine xarray objects along coordinates shared between them.coordinate alignment Multiple serialization formats are exposed by integrating their libraries.third-party I/O integration Dask-backed structures execute parallel and streaming computations outside memory.dask backing The unit suite is run automatically on Linux and Windows services.continuous integration Composition, subclassing, and accessor interfaces add domain methods without putting them in the core.composition and accessors Pandas Panel has a fixed three-dimensional rank and lacks features available to DataFrame and Series. Unlabeled arrays do not meaningfully represent associated metadata, leaving alignment to users or domain packages. Names and attributes are not interpreted by xarray and are propagated only in unambiguous cases. DataArrays in a Dataset are presumed to share a common coordinate system. OPeNDAP and GRIB serialization backends are read-only. The pure-Python implementation relies on compiled dependencies for speed. Domain-specific functionality is intentionally left to third-party libraries. Development and support depend on a volunteer team. over 1,500 unit testseliminates: regressions in core and optional-dependency functionality · outcome: held TravisCI and Appveyoreliminates: platform-specific failures missed by a single operating system · outcome: held sample data distributed with the source codeeliminates: examples that users cannot reproduce · outcome: held xarrayxarray Python packagePython package N-dimensional labeled arraysN-dimensional labeled arrays Common Data ModelCommon Data Model pandaspandas PanelPanel Scientific dataScientific data Unlabeled, N-dimensional arraysUnlabeled, N-dimensional arrays metadatametadata network Common Data Formnetwork Common Data Form collection of self-describing, machine-independent binary data formats and software toolscollection of self-describing, machine-independent binary data formats and software tools dimensions, variables, and attributesdimensions, variables, and attributes netCDF datasetnetCDF dataset DataArrayDataArray DatasetDataset DataFrameDataFrame dict-like containerdict-like container DimensionsDimensions ancillary variablesancillary variables Names and attributesNames and attributes coordinatescoordinates varying/dependent quantitiesvarying/dependent quantities named dimensionsnamed dimensions positional ordering of dimensionspositional ordering of dimensions dimension namesdimension names dummy dimensions of size onedummy dimensions of size one AlignmentAlignment common coordinatescommon coordinates Split-apply-combineSplit-apply-combine xarray plotting functionalityxarray plotting functionality thin wrapperthin wrapper Missing DataMissing Data serialization and I/Oserialization and I/O third-party librariesthird-party libraries Out-of-core computationOut-of-core computation daskdask parallel and streaming computationparallel and streaming computation large test suitelarge test suite TravisCITravisCI AppveyorAppveyor sample datasample data compiled dependenciescompiled dependencies core scientific Python librariescore scientific Python libraries domain-specific packagesdomain-specific packages consistency of labelsconsistency of labels labeled, multi-dimensional arraylabeled, multi-dimensional array core metadata-aware operationscore metadata-aware operations domain-specific functionalitydomain-specific functionality compositioncomposition high-level accessor interfacehigh-level accessor interface team of volunteersteam of volunteers xarray → core scientific Python packagesxarray builds on and interoperates with core scientific Python packages. xarray → domain-specific packagesxarray interfaces with domain-specific packages. xarray → consistency of labelsEvery xarray operation relies on and maintains label consistency. DataArray → labeled, multi-dimensional arrayDataArray implements a labeled multidimensional array. Dimensions → core metadata-aware operationsDimensions enable xarray's metadata-aware operations. dimension names → dummy dimensions of size oneDimension-name broadcasting eliminates the need for dummy size-one dimensions. dask → parallel and streaming computationDask backing supports parallel and streaming computation beyond memory. serialization and I/O → third-party librariesxarray serialization and I/O integrate third-party libraries. large test suite → TravisCIThe unit-test suite is executed automatically on TravisCI. xarray → compiled dependenciesThe pure-Python xarray implementation relies on compiled dependencies for speed. high-level accessor interface → domain-specific functionalityThe accessor interface lets users implement domain-specific methods on xarray objects. xarray as a pandas-inspired, netCDF-grounded labeled-array toolkit scientific labels as timestamps, coordinates, or experiment identifiers netCDF as both a self-describing data model and portable serialization basis DataArray as one labeled multidimensional array Dataset as aligned DataArrays sharing a coordinate system dimensions as named axes and coordinates as ancillary index variables alignment as database-like joining on common coordinates out-of-core as dask-backed parallel or streaming work beyond memory quality control as cross-platform automated unit testing plus reproducible sample data reuse as a domain-agnostic core extended through third-party composition and accessors builds-on → pandasnote: The labeled-array structures and analytics are strongly inspired by pandas. builds-on → netCDFnote: NetCDF explicitly forms the basis of the xarray model. extends → core scientific Python librariesnote: The modular design builds upon and extends core libraries without embedding domains. complements → domain-specific packagesnote: xarray interfaces with domain-specific packages while retaining a domain-agnostic core. xarray —is-a→ Python packagethe relation itself is separately evidenced · strength of the relation: 1 · note: Flat categorical classification. · the relation stated: xarray is a Python package.xarrayPython package xarray —contrasted-with→ Panelthe relation itself is separately evidenced · strength of the relation: 0.95 · note: Direct while-contrast and stated suitability difference. · the relation stated: xarray's arbitrary-rank labeled arrays contrast with pandas Panel's fixed three-dimensional rank.xarrayPanel network Common Data Form —is-a→ collection of self-describing, machine-independent binary data formats and software toolsthe relation itself is separately evidenced · strength of the relation: 1 · note: Explicit categorical definition. · the relation stated: The network Common Data Form is a collection of self-describing binary formats and software tools.network Common Data Formcollection of self-describing, machine-independent binary data formats and software tools dimensions, variables, and attributes —part-of→ netCDF datasetthe relation itself is separately evidenced · strength of the relation: 1 · note: Categorical containment stated by 'contains'. · the relation stated: Dimensions, variables, and attributes are contained in each netCDF dataset.dimensions, variables, and attributesnetCDF dataset Dataset —same-as→ DataFramethe relation itself is separately evidenced · strength of the relation: 0.9 · note: Scoped multidimensional equivalence, so below full identity strength. · the relation stated: Dataset is xarray's multidimensional equivalent of a DataFrame. Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the Dataset is the multi-dimensional EQUIVALENT of a DataFrame; same-as overstates.DatasetDataFrame Dataset —is-a→ dict-like containerthe relation itself is separately evidenced · strength of the relation: 1 · note: Explicit categorical classification. · the relation stated: Dataset is a dict-like container of aligned DataArrays.Datasetdict-like container Names and attributes —contrasted-with→ coordinatesthe relation itself is separately evidenced · strength of the relation: 0.95 · about this quote: DOUBTFUL, found by an audit that read the edge against its span: weak contrast. The edge is kept live; a reader should weigh it with this in mind. · note: Explicit in-contrast comparison. · the relation stated: Names and attributes are not interpreted generally, in contrast to coordinates persisted through transformations.Names and attributescoordinates coordinates —distinct-from→ varying/dependent quantitiesthe relation itself is separately evidenced · strength of the relation: 0.95 · note: Explicit unlike distinction. · the relation stated: Coordinates denote constant or independent quantities, unlike varying or dependent data quantities.coordinatesvarying/dependent quantities named dimensions —contrasted-with→ positional ordering of dimensionsthe relation itself is separately evidenced · strength of the relation: 0.95 · note: Direct instead-of contrast. · the relation stated: Named dimensions are used instead of relying on positional dimension ordering.named dimensionspositional ordering of dimensions xarray plotting functionality —is-a→ thin wrapperthe relation itself is separately evidenced · strength of the relation: 1 · note: Explicit categorical classification. · the relation stated: xarray plotting functionality is a thin wrapper around Matplotlib.xarray plotting functionalitythin wrapper xarray combines a pandas-inspired API with a self-describing scientific-data model. Pandas Panel's fixed rank is unsuitable for arbitrary-rank array applications. Every xarray operation relies on and maintains label consistency. NetCDF supplies xarray's data-model basis and portable serialization format. DataArray represents a labeled multidimensional array. Dataset is a dictionary-like container of aligned DataArrays and an in-memory netCDF representation. Named dimensions replace positional axis arguments for xarray indexing. Dimension-name broadcasting removes the need to insert dummy size-one dimensions. Dask backing supports parallel and streaming computation on data too large for memory. The test suite contains more than 1,500 unit tests run on Linux and Windows continuous integration. The core data structures are extensible through composition, subclassing, and high-level accessors. an open-source Python project providing labeled N-dimensional array structures and analyticsxarray self-describing, machine-independent formats and tools for array scientific datanetwork Common Data Form the principal netCDF structure containing named dimensions, variables, and attributesdataset xarray's labeled multidimensional array implementationDataArray a dictionary-like container of aligned DataArrays and in-memory netCDF representationDataset names used in place of positional axis argumentsDimensions ancillary variables used for label-based indexing and alignmentCoordinates database-like joins along common coordinatesAlignment N-dimensional grouped operations following the split-apply-combine strategySplit-apply-combine parallel or streaming work on data that does not fit into memoryOut-of-core computation 100s of GB or TBs — scale of out-of-memory datasets supported through dask over 1,500 unit tests — size of the xarray test suite two main data structures — DataArray and Dataset in the xarray model any number of dimensions — rank permitted for DataArrays and coordinates timestampstimestamps coordinatescoordinates dimension namesdimension names attrsattrs data variablesdata variables time, y, and xtime, y, and x Temperature and precipitationTemperature and precipitation latitude and longitudelatitude and longitude

xarray: N-D labeled Arrays and Datasets in Python

Abstract

xarray is an open source project and Python package that provides a toolkit and data structures for N-dimensional labeled arrays. Our approach combines an application programing interface (API) inspired by pandas with the Common Data Model for self-described scientific data. Key features of the xarray package include label-based indexing and arithmetic, interoperability with the core scientific Python packages (e.g., pandas, NumPy, Matplotlib), out-of-core computation on datasets that don’t fit into memory, a wide range of serialization and input/output (I/O) options, and advanced multi-dimensional data manipulation tools such as group-by and resampling. xarray, as a data model and analytics toolkit, has been widely adopted in the geoscience community but is also used more broadly for multi-dimensional data analysis in physics, machine learning and finance.

Python

pandas

netCDF

multidimensional

data

data handling

data analysis

Metrics

(1) Overview

Introduction

Python has emerged as a leading programing language for both the physical sciences and data sciences. At the core of modern scientific computing and analysis in Python are the NumPy [14] and SciPy [22] packages, which provide a robust N-dimensional array object and the fundamental operations required for science and engineering applications. Much of the success of Python in data science and business analytics is due to pandas [16], which introduced intuitive and fast tabular data analysis tools to Python, inspired by R’s data.frame [19]. The pandas DataFrame and Series objects provide unparalleled analysis tools for data alignment, resampling, grouping, pivoting, and aggregation in Python.

xarray implements data structures and an analytics toolkit for multi-dimensional labeled arrays strongly inspired by pandas. While pandas includes a data structure called the Panel for three dimensional data, its fixed rank design make it unsuitable for applications that require arbitrary rank arrays. Additionally, many of the features that make the pandas DataFrame and Series objects so useful, are not fully available on the Panel. Our approach with xarray adopts Unidata’s self-describing Common Data Model on which the network Common Data Form (netCDF) is built [20, 7]. NetCDF provides a well-defined data model for labeled N-dimensional array-oriented scientific data analysis.

xarray builds on top of, and seamlessly interoperates with, the core scientific Python packages, such as NumPy, SciPy, Matplotlib [13], and pandas. xarray provides a range of backends for serialization and input/output (IO), including the Pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF file formats. Leveraging the dask parallel computing library [21], xarray can optionally perform efficient parallel, out-of-core analysis on datasets that are too large to fit into memory. Finally, xarray interfaces with existing domain-specific packages such as UV-CDAT [25], Iris [17], and Cartopy [18].

Purpose: Your data has labels; you should use them

Scientific data is inherently labeled. For example, time series data includes timestamps that label individual periods or points in time, spatial data has coordinates (e.g. longitude, latitude, elevation), and model or laboratory experiments are often identified by unique identifiers. Figure 1 provides an example of a labeled dataset. In this case the data is a map of global air temperature from a numeric weather model. The labels on this particular dataset are time (e.g. “2016-05-01”), longitude (x-axis), and latitude (y-axis).

An example of a multidimensional labeled array. This figure (map) is showing the global surface air temperature for May 1, 2016 from ERA-Interim Reanalyis [11]. The map is labeled with the array's coordinates: longitude and latitude.

Unlabeled, N-dimensional arrays of numbers (e.g., NumPy’s ndarray) are the most widely used data structure in scientific computing. However, they lack a meaningful representation of the metadata associated with their data. Implementing such functionality is left to individual users and domain-specific packages. As a result, programmers frequently encounter pitfalls in the form of questions like “is the time axis of my array in the first or third index position?” or “does my array of timestamps still align with my data after resampling?”.

The core motivation for developing xarray was to provide labeled data tools for N-dimensional arrays that render such questions moot. Every operation in xarray both relies on and maintains the consistency of labels.

NetCDF

The network Common Data Form is a collection of self-describing, machine-independent binary data formats and software tools. These data formats and tools facilitate the creation, access, and sharing of scientific data stored in N-dimensional arrays, along with metadata describing the contents of each array [20]. NetCDF has become very popular in the geoscience community, and there are existing libraries for reading and writing netCDF in many programming languages, including C, Fortran, Python, Java, Matlab, and Julia.

The principal data structure in the netCDF data model is the dataset. Each netCDF dataset contains dimensions, variables, and attributes, each of which are identified by a hierarchy of unique names. The dataset and variable objects may contain attributes that describe the contents, units, history, or other metadata of the object. Standardized conventions, such as the Climate and Forecast (CF) Conventions [12], allow for the associations of coordinate variables with dimensions.

Implementation and architecture

NetCDF forms the basis of the xarray data model and provides a natural and portable serialization format. Building on netCDF, xarray features two main data structures: the DataArray and the Dataset. The API for these data structures is summarized in the following sections and in Figure 2.

An overview of xarray’s main data structures. Types are annotated using Python 3 style type hints [23]. “Mapping of” denotes an ordered mapping with values of the given type.

DataArray

The DataArray is xarray’s implementation of a labeled, multi-dimensional array. It has several key properties:

data: N-dimensional array (NumPy or dask) holding the array's values,

coords: dict-like container of arrays (coordinates) that label each point (e.g., 1-dimensional arrays of numbers, datetime objects or strings),

dims: dimension names for each axis [e.g., (‘time’, ‘latitude’, ‘longitude’)],

attrs: OrderedDict holding arbitrary metadata (e.g. units or descriptions), and

name: an arbitrary name for the array.

xarray uses dims and coords to enable its core metadata-aware operations. Dimensions provide names that xarray uses instead of the axis argument found in many NumPy functions. Coordinates are ancillary variables used to enable fast label based indexing and alignment, building on the functionality of the pandas Index. DataArray objects also can have a name and can hold arbitrary metadata in the form of their attrs property, which can be used to further describe data (e.g. by providing units). Names and attributes are strictly for users and user-written code; in general xarray makes no attempt to interpret them, and propagates them only in unambiguous cases. In contrast, xarray does interpret and persist coordinates in operations that transform xarray objects.

Dataset

The Dataset is xarray’s multi-dimensional equivalent of a DataFrame. It is a dict-like container of labeled arrays (DataArrays) with aligned dimensions. It is designed as an in-memory representation of a netCDF dataset. In addition to the dict-like interface of the dataset itself, which can be used to access any DataArray in a Dataset, datasets have four key properties:

data_vars: OrderedDict of DataArray objects corresponding to data variables,

coords: OrderedDict of DataArray objects intended to label points used in data_vars (e.g., 1-dimensional arrays of numbers, datetime objects or strings),

dims: dictionary mapping from dimension names to the fixed length of each dimension (e.g., {‘x’: 6, ‘y’: 6, ‘time’: 8}), and

attrs: OrderedDict to hold arbitrary metadata pertaining to the dataset.

DataArray objects inside a Dataset may have any number of dimensions but are presumed to share a common coordinate system. Coordinates can also have any number of dimensions but denote constant/independent quantities, unlike the varying/dependent quantities that belong in data. Figure 3 illustrates these concepts for an example Dataset containing meteorological data.

An example of how a dataset (netCDF or xarray) for a weather forecast might be structured. This dataset has three dimensions, time, y, and x, each of which is also a one-dimensional coordinate. Temperature and precipitation are three-dimensional data variables. Also included in the dataset are two-dimensional coordinates latitude and longitude, having dimensions y and x, and reference time, a zero-dimensional (scalar) coordinate.

Core xarray Features

xarray includes a powerful and growing feature set. The following list highlights some of the key features available in xarray. The xarray documentation [2] includes a complete description of available features and their usage.

Label-based indexing: Similarly to pandas objects, xarray objects support both integer- and label-based lookups along each dimension. However, xarray objects also have named dimensions, so you can optionally use dimension names instead of relying on the positional ordering of dimensions.

Arithmetic: arithmetic between xarray objects vectorizes based on dimension names, automatically looping (broadcasting) over each distinct dimension. This eliminates the need to insert dummy dimensions of size one to facilitate broadcasting, a common pattern with NumPy.

Aggregation: calculation of statistics (e.g. sum) along a dimension of an xarray object can be done by dimension name instead of an integer axis number.

Alignment: xarray supports database-like join operations for combining xarray objects along common coordinates.

Split-apply-combine: xarray includes N-dimensional grouped operations implementing the split-apply-combine strategy [24].

Resampling and rolling window operations: Utilizing the efficient resampling methods from pandas and rolling window operations from Bottleneck [15], xarray offers a robust set of resampling and rolling window operations along a single dimension.

Plotting: xarray plotting functionality is a thin wrapper around the popular Matplotlib library. xarray uses the syntax and function names from Matplotlib whenever possible, resulting in a seamless transition between the two.

Missing Data: xarray smoothly handles missing data in all operations, including arithmetic, alignment and aggregation.

Interactivity with pandas: xarray objects seamlessly to convert to and from pandas objects to interact with the rest of the PyData ecosystem.

Serialization and I/O: xarray supports direct serialization and I/O to several file formats including pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF by integrating with third-party libraries. Additional serialization formats for 1-dimensional data are available through pandas.

Out-of-core computation: xarray’s data structures can be backed by dask [21] instead of NumPy to support parallel and streaming computation on data that does not fit into memory, up to 100s of GB or TBs in size. Such large datasets (“big data”) are increasingly prevalent in science.

Quality control

xarray is provided with a large test suite comprised of over 1,500 unit tests. These tests cover the core xarray functionality as well as features facilitated by optional dependencies. The unit tests are executed automatically on the TravisCI (Linux) [5] and Appveyor (Windows) [1] continuous integration systems. A selection of sample data is also distributed with the source code, allowing users to reproduce any examples in the xarray documentation.

(2) Availability

Operating system

Linux, Windows and Mac OS X.

Programming language

Python, versions 2.7, 3.4 and later.

Additional system requirements

None.

Dependencies

xarray is implemented in pure Python and relies on compiled dependencies for speed.

NumPy: 1.7 or later

pandas: 0.15.0 or later

netcdf4-python: (optional) used for reading and writing netCDF files

SciPy: (optional) used as a fallback for reading/writing netCDF3

Pydap: (optional) used as a fallback for accessing OPeNDAP

h5netcdf: (optional) an alternative library for reading and writing netCDF4 files that does not use the netCDF-C libraries

PyNIO: (optional) for reading GRIB1/2 and other geoscience specific file formats

Bottleneck: (optional) speeds up NaN-skipping and rolling window aggregations by a large factor

cyordereddict: (optional) speeds up most internal operations with xarray data structures

Dask: (optional) required for out-of-core parallel computation

Matplotlib: (optional) required for plotting

Cartopy: (optional) required for plotting maps

seaborn: (optional) additional plotting functionality

List of contributors

Alex Kleeman, The Climate Corporation

Alistair Miles, Wellcome Trust Centre for Human Genetics, University of Oxford

Andreas Hilboll, Institute of Environmental Physics, University of Bremen

Anna Kuznetsova, The Climate Corporation

Antony Lee, Department of Physics, University of California, Berkeley

Bas Hoonhout, Deltares, The Netherlands

Benjamin Root, Atmospheric and Environmental Research, Inc.

Benoit Bovy, Department of Astrophysics, Geophysics and Oceanography, University of Lige

Chun-Wei Yuan, Institute for Health Metrics and Evaluation, University of Washington

Christoph Deil, Max Planck Institute for Nuclear Physics

Clark Fitzgerald, Department of Statistics, University of California, Davis

Dean Pospisil, Institute for Learning and Brain Sciences, University of Washington

Edward Richards, Scripps Institution of Oceanography, University of California, San Diego

Erik Welch, Continuum Analytics

Eugene Brevdo, Google Research

Fabien Maussion, Institute of Atmospheric and Cryospheric Sciences, University of Innsbruck

Filipe Fernandes, Universidade de So Paulo

Guido Imperiale, Legal & General

Hauser Mathias, Institute for Atmospheric and Climate Science, ETH Zurich

Igor Babuschkin, School of Physics and Astronomy, University of Manchester

Jeffrey Gerard, The Climate Corporation

Jeremy McGibbon, Department of Atmospheric Sciences, University of Washington

Jonathan Chambers, UCL Energy Institute, London UK

Julia Signell, Department of Civil and Environmental Engineering, Princeton University

Joseph Hamman, National Center for Atmospheric Research

Johnnie Gray, University College London

Jonathan Helmus, Environmental Science Division, Argonne National Laboratory

Kelsey Jordahl, Planet Labs

Maciek Swat, University of Pennsylvania

Marco Zhlke, Brockmann Consult GmbH

Markel Garca-Dez, Catalan Institute of Climate Sciences

Markus Gonser

Maximilian Roos, Sixty Capital

Mike Graham, Edgestream Partners

Nikolay Koldunov, Climate Service Center Germany

Peter Cable, Raytheon

Phillip J. Wolfram, Fluid and Solid Mechanics (T-3), Theoretical Division, Los Alamos National Laboratory

Rafael Guedes, MetOcean Solutions Ltd

Robin Wilson, Geography & Environment, University of Southampton, UK

Ryan Abernathey, Department of Earth and Environmental Sciences, Columbia University

Scott Sinclair, Satellite Applications and Hydrology Group, Dept. Civil Engineering, University of KwaZulu-Natal

Sébastien Celles, Thermal Science and Energy Department, Poitiers Institute of Technology (IUT)

Spencer A. Hill, Program in Atmospheric and Oceanic Sciences, Princeton University

Stefan Pfenninger, Department of Environmental Systems Science, ETH Zrich

Stephan Hoyer, Google Research

Steven E. Pav, Gilgamath Consulting

Takeshi Kanmae, National Institute of Polar Research, Japan

Thomas Kluyver, Computational Modeling Group, University of Southampton

Todd Small, The Climate Corporation

Valliappa Lakshmanan, Google

Yves Delley, Department of Physics, ETH Zurich

Software location