F. Alexander WolfpersName, Philipp AngererpersName, Fabian J. TheispersName (2018date). “SCANPY: large-scale single-cell gene expression data analysis.” Genome biologytitle·venue.
Object of study — single-cell gene expression data at a scale existing frameworks do not reach
Research question — how to give the established single-cell analyses a form that scales past a million cells
Contribution — SCANPYterm·dom, and ANNDATAterm·dom - a generic annotated data matrixterm·dom that every tool reads and writes without changing its structure
phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | analysis frameworks failing at data sizes their science has already reached — “However, these frameworks do not scale to the increasingly available large data sets with up to and more than one million cells. Here, we present a framework that overcomes this limitation and provides similar analysis possibilities.” | |
| claim-type | method — “However, these frameworks do not scale to the increasingly available large data sets with up to and more than one million cells. Here, we present a framework that overcomes this limitation and provides similar analysis possibilities.” | |
| method — evidence | — “Typically, SCANPY ’s tools reuse a once-computed, single graph representation of data and hence, avoid the use of different, potentially inconsistent, and computationally expensive representations of data.” | |
| object of study — evidence | — “However, these frameworks do not scale to the increasingly available large data sets with up to and more than one million cells. Here, we present a framework that overcomes this limitation and provides similar analysis possibilities.” | |
| research question — evidence | — “However, these frameworks do not scale to the increasingly available large data sets with up to and more than one million cells. Here, we present a framework that overcomes this limitation and provides similar analysis possibilities.” | |
| contribution — evidence | — “With SCANPY , we introduce the class ANNDATA —with a corresponding package ANNDATA —which stores a data matrix with the most general annotations possible: annotations of observations (samples, cells) and variables (features, genes), and unstructured annotations.” | |
| mechanism | ONE REPRESENTATION, COMPUTED ONCE, REUSED: the tools share a single graph of the data instead of each building its own, which the paper justifies by what it avoids - DIFFERENT, POTENTIALLY INCONSISTENT representations of the same data — “Typically, SCANPY ’s tools reuse a once-computed, single graph representation of data and hence, avoid the use of different, potentially inconsistent, and computationally expensive representations of data.” | mechanisms[0] |
| mechanism | EXTENSIBILITY THROUGH A FIXED STRUCTURE: a tool extracts from the matrix and writes its result back into the same object, and the object's structure is unaffected - so adding a method does not oblige anything else to change — “As SCANPY is built around that class, it is easy to add new functionality to the toolkit. All statistics and machine-learning tools extract information from a data matrix, which can be added to an ANNDATA object while leaving the structure of ANNDATA unaffected.” | mechanisms[1] |
| mechanism | SCALE BY CHANGING RESIDENCY, NOT ALGORITHM: HDF5 backing lets the object be operated on without being loaded, so the limit moves from memory size to disk — “This allows operating on an ANNDATA object without fully loading it into memory—the functionality is offered via ANNDATA ’s backed mode as opposed to its memory mode.” | mechanisms[2] |
| mechanism | SPEED FROM BLOCKING, NOT FROM APPROXIMATION: neighbourhood distances are computed by aggregating rows into submatrices and multiplying, rather than by sampling — “The computation of neighborhood relations is much faster than in the popular reference package [ 27 ]. This is achieved by aggregating rows (observations) in a data matrix to submatrices and computing distances for each submatrix using fast parallelized matrix multiplication.” | mechanisms[3] |
| sense | ANNDATA - THE ANNOTATION IS THE POINT, NOT THE MATRIX: a data matrix plus annotations of observations, of variables, and unstructured ones, held together so a tool can add a result without changing the structure everything else reads — “With SCANPY , we introduce the class ANNDATA —with a corresponding package ANNDATA —which stores a data matrix with the most general annotations possible: annotations of observations (samples, cells) and variables (features, genes), and unstructured annotations.” | senses[0] |
| sense | backed mode - the same object read WITHOUT being loaded into memory, as opposed to memory mode. One class, two residencies, and the analysis code does not change — “This allows operating on an ANNDATA object without fully loading it into memory—the functionality is offered via ANNDATA ’s backed mode as opposed to its memory mode.” | senses[1] |
| sense | scalable toolkit - the software class to which the abstract assigns Scanpy — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | senses[2] |
| sense | preprocessing - one analysis capability included in Scanpy — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | senses[3] |
| sense | visualization - one analysis capability included in Scanpy — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | senses[4] |
| sense | clustering - one analysis capability included in Scanpy — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | senses[5] |
| sense | pseudotime and trajectory inference - temporal and lineage reconstruction capabilities included in Scanpy — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | senses[6] |
| sense | differential expression testing - one analysis capability included in Scanpy — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | senses[7] |
| sense | simulation of gene regulatory networks - the network-simulation capability included in Scanpy — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | senses[8] |
| sense | generic class - the software class assigned to AnnData for handling annotated matrices — “Along with Scanpy , we present AnnData , a generic class for handling annotated data matrices” | senses[9] |
| sense | annotated data matrices - the matrix form handled by the generic AnnData class — “Along with Scanpy , we present AnnData , a generic class for handling annotated data matrices” | senses[10] |
| sense | HDF5-based backing - on-disk backing that permits operating without fully loading an AnnData object — “ANNDATA is similar to R’s EXPRESSIONSET [ 26 ], but supports sparse data and allows HDF5-based backing of ANNDATA objects on disk, a format independent of platform, framework, and language. This allows operating on an ANNDATA object without fully loading it into memory—the functionality is offered via ANNDATA ’s backed mode as opposed to its memory mode” | senses[11] |
| sense | in-place - Scanpy's default object transformation mode for memory-efficient pipelines — “To simplify memory-efficient pipelines, SCANPY ’s functions operate in-place by default but allow the optional non-destructive transformation of objects. Pipelines written this way can then also be run in backed mode to exploit online-learning formulations of algorithms. Almost all of SCANPY ’s tools are parallelized” | senses[12] |
| sense | non-destructive transformation - the optional alternative to in-place transformation — “To simplify memory-efficient pipelines, SCANPY ’s functions operate in-place by default but allow the optional non-destructive transformation of objects. Pipelines written this way can then also be run in backed mode to exploit online-learning formulations of algorithms. Almost all of SCANPY ’s tools are parallelized” | senses[13] |
| sense | online-learning formulations - algorithm formulations that pipelines can exploit in backed mode — “To simplify memory-efficient pipelines, SCANPY ’s functions operate in-place by default but allow the optional non-destructive transformation of objects. Pipelines written this way can then also be run in backed mode to exploit online-learning formulations of algorithms. Almost all of SCANPY ’s tools are parallelized” | senses[14] |
| sense | class for representing a graph of neighborhood relations - Scanpy's representation of neighborhood relations among data points — “SCANPY introduces a class for representing a graph of neighborhood relations among data points. The computation of neighborhood relations is much faster than in the popular reference package [ 27 ]. This is achieved by aggregating rows (observations) in a data matrix to submatrices and computing distances for each submatrix using fast parallelized matrix multiplication. Moreover, the class provides several functions to compute random-walk-based metrics that are not available in other graph software” | senses[15] |
| sense | random-walk-based metrics - metrics exposed by Scanpy's neighborhood-graph class but absent from other graph software — “SCANPY introduces a class for representing a graph of neighborhood relations among data points. The computation of neighborhood relations is much faster than in the popular reference package [ 27 ]. This is achieved by aggregating rows (observations) in a data matrix to submatrices and computing distances for each submatrix using fast parallelized matrix multiplication. Moreover, the class provides several functions to compute random-walk-based metrics that are not available in other graph software” | senses[16] |
| sense | static - AnnData's earlier HDF5 backing state — “In this context, we acknowledge the discussions with S. Linnarson, which motivated us to extend ANNDATA ’s previously static to a dynamic HDF5 backing” | senses[17] |
| sense | dynamic HDF5 backing - the HDF5 backing state to which AnnData was extended — “In this context, we acknowledge the discussions with S. Linnarson, which motivated us to extend ANNDATA ’s previously static to a dynamic HDF5 backing” | senses[18] |
| sense | FLOTILLA - the other Python framework that solves more than one analysis task — “Hence, only the latter provides a data analysis framework that solves more than one specific task. In contrast to SCANPY , however, FLOTILLA is neither targeted at single-cell nor at large-scale data and does not provide any graph-based methods, which are the core of SCANPY . Also, FLOTILLA is built around a complicated class STUDY , which contains data, tools, and plotting functions. SCANPY , by contrast, is built around a simple HDF5-backed class ANNDATA , which makes SCANPY both scalable and extendable (law of Demeter)” | senses[19] |
| sense | data analysis framework - the multi-task software class the paper assigns to FLOTILLA — “Hence, only the latter provides a data analysis framework that solves more than one specific task. In contrast to SCANPY , however, FLOTILLA is neither targeted at single-cell nor at large-scale data and does not provide any graph-based methods, which are the core of SCANPY . Also, FLOTILLA is built around a complicated class STUDY , which contains data, tools, and plotting functions. SCANPY , by contrast, is built around a simple HDF5-backed class ANNDATA , which makes SCANPY both scalable and extendable (law of Demeter)” | senses[20] |
| sense | STUDY - FLOTILLA's class combining data, tools, and plotting functions — “Hence, only the latter provides a data analysis framework that solves more than one specific task. In contrast to SCANPY , however, FLOTILLA is neither targeted at single-cell nor at large-scale data and does not provide any graph-based methods, which are the core of SCANPY . Also, FLOTILLA is built around a complicated class STUDY , which contains data, tools, and plotting functions. SCANPY , by contrast, is built around a simple HDF5-backed class ANNDATA , which makes SCANPY both scalable and extendable (law of Demeter)” | senses[21] |
| sense | simple HDF5-backed class ANNDATA - the contrasting class around which Scanpy is built — “Hence, only the latter provides a data analysis framework that solves more than one specific task. In contrast to SCANPY , however, FLOTILLA is neither targeted at single-cell nor at large-scale data and does not provide any graph-based methods, which are the core of SCANPY . Also, FLOTILLA is built around a complicated class STUDY , which contains data, tools, and plotting functions. SCANPY , by contrast, is built around a simple HDF5-backed class ANNDATA , which makes SCANPY both scalable and extendable (law of Demeter)” | senses[22] |
| sense | large-scale data - a target that the paper says FLOTILLA does not address — “Hence, only the latter provides a data analysis framework that solves more than one specific task. In contrast to SCANPY , however, FLOTILLA is neither targeted at single-cell nor at large-scale data and does not provide any graph-based methods, which are the core of SCANPY . Also, FLOTILLA is built around a complicated class STUDY , which contains data, tools, and plotting functions. SCANPY , by contrast, is built around a simple HDF5-backed class ANNDATA , which makes SCANPY both scalable and extendable (law of Demeter)” | senses[23] |
| sense | graph-based methods - Scanpy's core method family, absent from FLOTILLA — “Hence, only the latter provides a data analysis framework that solves more than one specific task. In contrast to SCANPY , however, FLOTILLA is neither targeted at single-cell nor at large-scale data and does not provide any graph-based methods, which are the core of SCANPY . Also, FLOTILLA is built around a complicated class STUDY , which contains data, tools, and plotting functions. SCANPY , by contrast, is built around a simple HDF5-backed class ANNDATA , which makes SCANPY both scalable and extendable (law of Demeter)” | senses[24] |
| control | benchmarking SCANPY against established single-cell analysis packages on a clustering tutorial of 2,700 peripheral blood mononuclear cells — held — “In a detailed clustering tutorial of 2700 peripheral blood mononuclear cells (PBMCs), adapted from one of SEURAT ’s tutorials ( http://satijalab.org/seurat/pbmc3k_tutorial.html ) [ 2 ], all steps starting from raw count data to the identification of cell types are carried out, providing speedups between 5 and 90 times in each step ( https://github.com/theislab/scanpy_usage/tree/master/170505_seurat ).” — rules out that SCANPY's clustering results differ from what the established tools produce on the same data, so that its speed advantage comes at a cost in output — “In a detailed clustering tutorial of 2700 peripheral blood mononuclear cells (PBMCs), adapted from one of SEURAT ’s tutorials ( http://satijalab.org/seurat/pbmc3k_tutorial.html ) [ 2 ], all steps starting from raw count data to the identification of cell types are carried out, providing speedups between 5 and 90 times in each step ( https://github.com/theislab/scanpy_usage/tree/master/170505_seurat ).” | controls[0] |
| audit | ANNDATA —contrasted-with→ EXPRESSIONSET — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span calls ANNDATA SIMILAR to EXPRESSIONSET, with differences; contrasted-with is loose. — “ANNDATA is similar to R’s EXPRESSIONSET [ 26 ], but supports sparse data and allows HDF5-based backing of ANNDATA objects on disk, a format independent of platform, framework, and language.” | concept_edges[0] |
| differs in | ANNDATA —contrasted-with→ EXPRESSIONSET — sparse data support and HDF5-based backing on disk, which is what lets an object be operated on without being loaded — “ANNDATA is similar to R’s EXPRESSIONSET [ 26 ], but supports sparse data and allows HDF5-based backing of ANNDATA objects on disk, a format independent of platform, framework, and language.” — “but supports sparse data and allows HDF5-based backing of ANNDATA objects on disk” | concept_edges[0] |
| audit | ANNDATA —part-of→ SCANPY — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says SCANPY is built around the ANNDATA class; part-of is loose. — “As SCANPY is built around that class, it is easy to add new functionality to the toolkit. All statistics and machine-learning tools extract information from a data matrix, which can be added to an ANNDATA object while leaving the structure of ANNDATA unaffected.” | concept_edges[1] |
| edge | backed mode —distinct-from→ memory mode — “This allows operating on an ANNDATA object without fully loading it into memory—the functionality is offered via ANNDATA ’s backed mode as opposed to its memory mode.” | concept_edges[2] |
| note | SCANPY —is-a→ scalable toolkit — The abstract gives an unhedged class definition. — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | concept_edges[3] |
| note | preprocessing —part-of→ SCANPY — The abstract unhedgedly includes preprocessing among Scanpy's methods. — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | concept_edges[4] |
| note | visualization —part-of→ SCANPY — The abstract unhedgedly includes visualization among Scanpy's methods. — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | concept_edges[5] |
| note | clustering —part-of→ SCANPY — The abstract unhedgedly includes clustering among Scanpy's methods. — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | concept_edges[6] |
| note | pseudotime and trajectory inference —part-of→ SCANPY — The abstract unhedgedly includes pseudotime and trajectory inference among Scanpy's methods. — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | concept_edges[7] |
| note | differential expression testing —part-of→ SCANPY — The abstract unhedgedly includes differential-expression testing among Scanpy's methods. — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | concept_edges[8] |
| note | simulation of gene regulatory networks —part-of→ SCANPY — The abstract unhedgedly includes regulatory-network simulation among Scanpy's methods. — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | concept_edges[9] |
| differs in | FLOTILLA —distinct-from→ SCANPY — single-cell and large-scale targeting, graph-based methods, and the complexity of the central data class — “Hence, only the latter provides a data analysis framework that solves more than one specific task. In contrast to SCANPY , however, FLOTILLA is neither targeted at single-cell nor at large-scale data and does not provide any graph-based methods, which are the core of SCANPY . Also, FLOTILLA is built around a complicated class STUDY , which contains data, tools, and plotting functions. SCANPY , by contrast, is built around a simple HDF5-backed class ANNDATA , which makes SCANPY both scalable and extendable (law of Demeter)” | concept_edges[10] |
| note | FLOTILLA —distinct-from→ SCANPY — The paper explicitly says 'in contrast to SCANPY' and names the dimensions of difference. — “Hence, only the latter provides a data analysis framework that solves more than one specific task. In contrast to SCANPY , however, FLOTILLA is neither targeted at single-cell nor at large-scale data and does not provide any graph-based methods, which are the core of SCANPY . Also, FLOTILLA is built around a complicated class STUDY , which contains data, tools, and plotting functions. SCANPY , by contrast, is built around a simple HDF5-backed class ANNDATA , which makes SCANPY both scalable and extendable (law of Demeter)” | concept_edges[10] |
| differs in | STUDY —contrasted-with→ simple HDF5-backed class ANNDATA — a complicated class containing data, tools, and plotting functions versus a simple HDF5-backed class — “Hence, only the latter provides a data analysis framework that solves more than one specific task. In contrast to SCANPY , however, FLOTILLA is neither targeted at single-cell nor at large-scale data and does not provide any graph-based methods, which are the core of SCANPY . Also, FLOTILLA is built around a complicated class STUDY , which contains data, tools, and plotting functions. SCANPY , by contrast, is built around a simple HDF5-backed class ANNDATA , which makes SCANPY both scalable and extendable (law of Demeter)” | concept_edges[11] |
| note | STUDY —contrasted-with→ simple HDF5-backed class ANNDATA — The paper's explicit 'by contrast' comparison is unhedged. — “Hence, only the latter provides a data analysis framework that solves more than one specific task. In contrast to SCANPY , however, FLOTILLA is neither targeted at single-cell nor at large-scale data and does not provide any graph-based methods, which are the core of SCANPY . Also, FLOTILLA is built around a complicated class STUDY , which contains data, tools, and plotting functions. SCANPY , by contrast, is built around a simple HDF5-backed class ANNDATA , which makes SCANPY both scalable and extendable (law of Demeter)” | concept_edges[11] |
| outcome | control — held — “In a detailed clustering tutorial of 2700 peripheral blood mononuclear cells (PBMCs), adapted from one of SEURAT ’s tutorials ( http://satijalab.org/seurat/pbmc3k_tutorial.html ) [ 2 ], all steps starting from raw count data to the identification of cell types are carried out, providing speedups between 5 and 90 times in each step ( https://github.com/theislab/scanpy_usage/tree/master/170505_seurat ).” | controls[0] |
| no type for this | static → dynamic HDF5 backing — The paper says AnnData's HDF5 backing was extended from static to dynamic. A state transition is not one of the eight edge types. — “In this context, we acknowledge the discussions with S. Linnarson, which motivated us to extend ANNDATA ’s previously static to a dynamic HDF5 backing” | unhoused_relations[0] |
| limitation | The comparison offered is against R-based frameworks and CELL RANGER; the claim of no comparable toolkit is scoped to the PYTHON ecosystem, in the paper's own words — “SCANPY integrates well into the existing Python ecosystem, in which no comparable toolkit yet exists.” | limitations[0] |
| limitation | The scaling evidence is a demonstration on named datasets - 68,579 cells for the speedup and 1.3 million for feasibility - not a complexity result — “Moreover, we demonstrate the feasibility of analyzing 1.3 million cells without subsampling in a few hours of computing time on eight cores of a small computing server” | limitations[1] |
| concept | SCANPY — “However, these frameworks do not scale to the increasingly available large data sets with up to and more than one million cells. Here, we present a framework that overcomes this limitation and provides similar analysis possibilities. Moreover, in contrast to the existing R-based frameworks, SCANPY ’s Python-based implementation is easy to interface with advanced machine-learning packages, such as TENSORFLOW [ 9 ].” | concepts[0] |
| concept | ANNDATA — “With SCANPY , we introduce the class ANNDATA —with a corresponding package ANNDATA —which stores a data matrix with the most general annotations possible: annotations of observations (samples, cells) and variables (features, genes), and unstructured annotations.” | concepts[1] |
| concept | EXPRESSIONSET — “ANNDATA is similar to R’s EXPRESSIONSET [ 26 ], but supports sparse data and allows HDF5-based backing of ANNDATA objects on disk, a format independent of platform, framework, and language.” | concepts[2] |
| concept | backed mode — “This allows operating on an ANNDATA object without fully loading it into memory—the functionality is offered via ANNDATA ’s backed mode as opposed to its memory mode.” | concepts[3] |
| concept | memory mode — “This allows operating on an ANNDATA object without fully loading it into memory—the functionality is offered via ANNDATA ’s backed mode as opposed to its memory mode.” | concepts[4] |
| concept | annotations — “With SCANPY , we introduce the class ANNDATA —with a corresponding package ANNDATA —which stores a data matrix with the most general annotations possible: annotations of observations (samples, cells) and variables (features, genes), and unstructured annotations.” | concepts[5] |
| concept | data matrix — “With SCANPY , we introduce the class ANNDATA —with a corresponding package ANNDATA —which stores a data matrix with the most general annotations possible: annotations of observations (samples, cells) and variables (features, genes), and unstructured annotations.” | concepts[6] |
| concept | neighborhood relations — “The computation of neighborhood relations is much faster than in the popular reference package [ 27 ]. This is achieved by aggregating rows (observations) in a data matrix to submatrices and computing distances for each submatrix using fast parallelized matrix multiplication.” | concepts[7] |
| concept | single graph representation — “Typically, SCANPY ’s tools reuse a once-computed, single graph representation of data and hence, avoid the use of different, potentially inconsistent, and computationally expensive representations of data.” | concepts[8] |
| concept | sparse data — “ANNDATA is similar to R’s EXPRESSIONSET [ 26 ], but supports sparse data and allows HDF5-based backing of ANNDATA objects on disk, a format independent of platform, framework, and language.” | concepts[9] |
| concept | CELL RANGER — “Benchmarking against the more run-time optimized CELL RANGER R kit [ 6 ], we demonstrate a speedup of 5 to 16 times for a data set of 68,579 PBMCs” | concepts[10] |
| concept | SEURAT — “Simple integrated analysis work flows for single-cell transcriptomic data [ 1 ] have been enabled by frameworks such as SEURAT [ 2 ], MONOCLE [ 3 ], SCDE/PAGODA [ 4 ], MAST [ 5 ], CELL RANGER [ 6 ], SCATER [ 7 ], and SCRAN [ 8 ]. However, these frameworks do not scale to the increasingly available large data sets with up to and more than one million cells. Here, we present a framework that overcomes this limitation and provides similar analysis possibilities.” | concepts[11] |
| concept | Python ecosystem — “SCANPY integrates well into the existing Python ecosystem, in which no comparable toolkit yet exists.” | concepts[12] |
| concept | scalable toolkit — not defined in this paper's own words; the software class to which the abstract assigns Scanpy — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | concepts[13] |
| concept | preprocessing — not defined in this paper's own words; one analysis capability included in Scanpy — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | concepts[14] |
| concept | visualization — not defined in this paper's own words; one analysis capability included in Scanpy — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | concepts[15] |
| concept | clustering — not defined in this paper's own words; one analysis capability included in Scanpy — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | concepts[16] |
| concept | pseudotime and trajectory inference — not defined in this paper's own words; temporal and lineage reconstruction capabilities included in Scanpy — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | concepts[17] |
| concept | differential expression testing — not defined in this paper's own words; one analysis capability included in Scanpy — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | concepts[18] |
| concept | simulation of gene regulatory networks — not defined in this paper's own words; the network-simulation capability included in Scanpy — “Scanpy is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing, visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene regulatory networks” | concepts[19] |
| concept | generic class — not defined in this paper's own words; the software class assigned to AnnData for handling annotated matrices — “Along with Scanpy , we present AnnData , a generic class for handling annotated data matrices” | concepts[20] |
| concept | annotated data matrices — not defined in this paper's own words; the matrix form handled by the generic AnnData class — “Along with Scanpy , we present AnnData , a generic class for handling annotated data matrices” | concepts[21] |
| concept | HDF5-based backing — not defined in this paper's own words; on-disk backing that permits operating without fully loading an AnnData object — “ANNDATA is similar to R’s EXPRESSIONSET [ 26 ], but supports sparse data and allows HDF5-based backing of ANNDATA objects on disk, a format independent of platform, framework, and language. This allows operating on an ANNDATA object without fully loading it into memory—the functionality is offered via ANNDATA ’s backed mode as opposed to its memory mode” | concepts[22] |
| concept | in-place — not defined in this paper's own words; Scanpy's default object transformation mode for memory-efficient pipelines — “To simplify memory-efficient pipelines, SCANPY ’s functions operate in-place by default but allow the optional non-destructive transformation of objects. Pipelines written this way can then also be run in backed mode to exploit online-learning formulations of algorithms. Almost all of SCANPY ’s tools are parallelized” | concepts[23] |
| concept | non-destructive transformation — not defined in this paper's own words; the optional alternative to in-place transformation — “To simplify memory-efficient pipelines, SCANPY ’s functions operate in-place by default but allow the optional non-destructive transformation of objects. Pipelines written this way can then also be run in backed mode to exploit online-learning formulations of algorithms. Almost all of SCANPY ’s tools are parallelized” | concepts[24] |
| concept | online-learning formulations — not defined in this paper's own words; algorithm formulations that pipelines can exploit in backed mode — “To simplify memory-efficient pipelines, SCANPY ’s functions operate in-place by default but allow the optional non-destructive transformation of objects. Pipelines written this way can then also be run in backed mode to exploit online-learning formulations of algorithms. Almost all of SCANPY ’s tools are parallelized” | concepts[25] |
| concept | class for representing a graph of neighborhood relations — not defined in this paper's own words; Scanpy's representation of neighborhood relations among data points — “SCANPY introduces a class for representing a graph of neighborhood relations among data points. The computation of neighborhood relations is much faster than in the popular reference package [ 27 ]. This is achieved by aggregating rows (observations) in a data matrix to submatrices and computing distances for each submatrix using fast parallelized matrix multiplication. Moreover, the class provides several functions to compute random-walk-based metrics that are not available in other graph software” | concepts[26] |
| concept | random-walk-based metrics — not defined in this paper's own words; metrics exposed by Scanpy's neighborhood-graph class but absent from other graph software — “SCANPY introduces a class for representing a graph of neighborhood relations among data points. The computation of neighborhood relations is much faster than in the popular reference package [ 27 ]. This is achieved by aggregating rows (observations) in a data matrix to submatrices and computing distances for each submatrix using fast parallelized matrix multiplication. Moreover, the class provides several functions to compute random-walk-based metrics that are not available in other graph software” | concepts[27] |
| concept | static — not defined in this paper's own words; AnnData's earlier HDF5 backing state — “In this context, we acknowledge the discussions with S. Linnarson, which motivated us to extend ANNDATA ’s previously static to a dynamic HDF5 backing” | concepts[28] |
| concept | dynamic HDF5 backing — not defined in this paper's own words; the HDF5 backing state to which AnnData was extended — “In this context, we acknowledge the discussions with S. Linnarson, which motivated us to extend ANNDATA ’s previously static to a dynamic HDF5 backing” | concepts[29] |
| concept | FLOTILLA — not defined in this paper's own words; the other Python framework that solves more than one analysis task — “Hence, only the latter provides a data analysis framework that solves more than one specific task. In contrast to SCANPY , however, FLOTILLA is neither targeted at single-cell nor at large-scale data and does not provide any graph-based methods, which are the core of SCANPY . Also, FLOTILLA is built around a complicated class STUDY , which contains data, tools, and plotting functions. SCANPY , by contrast, is built around a simple HDF5-backed class ANNDATA , which makes SCANPY both scalable and extendable (law of Demeter)” | concepts[30] |
| concept | data analysis framework — not defined in this paper's own words; the multi-task software class the paper assigns to FLOTILLA — “Hence, only the latter provides a data analysis framework that solves more than one specific task. In contrast to SCANPY , however, FLOTILLA is neither targeted at single-cell nor at large-scale data and does not provide any graph-based methods, which are the core of SCANPY . Also, FLOTILLA is built around a complicated class STUDY , which contains data, tools, and plotting functions. SCANPY , by contrast, is built around a simple HDF5-backed class ANNDATA , which makes SCANPY both scalable and extendable (law of Demeter)” | concepts[31] |
| concept | STUDY — not defined in this paper's own words; FLOTILLA's class combining data, tools, and plotting functions — “Hence, only the latter provides a data analysis framework that solves more than one specific task. In contrast to SCANPY , however, FLOTILLA is neither targeted at single-cell nor at large-scale data and does not provide any graph-based methods, which are the core of SCANPY . Also, FLOTILLA is built around a complicated class STUDY , which contains data, tools, and plotting functions. SCANPY , by contrast, is built around a simple HDF5-backed class ANNDATA , which makes SCANPY both scalable and extendable (law of Demeter)” | concepts[32] |
| concept | simple HDF5-backed class ANNDATA — not defined in this paper's own words; the contrasting class around which Scanpy is built — “Hence, only the latter provides a data analysis framework that solves more than one specific task. In contrast to SCANPY , however, FLOTILLA is neither targeted at single-cell nor at large-scale data and does not provide any graph-based methods, which are the core of SCANPY . Also, FLOTILLA is built around a complicated class STUDY , which contains data, tools, and plotting functions. SCANPY , by contrast, is built around a simple HDF5-backed class ANNDATA , which makes SCANPY both scalable and extendable (law of Demeter)” | concepts[33] |
| concept | large-scale data — not defined in this paper's own words; a target that the paper says FLOTILLA does not address — “Hence, only the latter provides a data analysis framework that solves more than one specific task. In contrast to SCANPY , however, FLOTILLA is neither targeted at single-cell nor at large-scale data and does not provide any graph-based methods, which are the core of SCANPY . Also, FLOTILLA is built around a complicated class STUDY , which contains data, tools, and plotting functions. SCANPY , by contrast, is built around a simple HDF5-backed class ANNDATA , which makes SCANPY both scalable and extendable (law of Demeter)” | concepts[34] |
| concept | graph-based methods — not defined in this paper's own words; Scanpy's core method family, absent from FLOTILLA — “Hence, only the latter provides a data analysis framework that solves more than one specific task. In contrast to SCANPY , however, FLOTILLA is neither targeted at single-cell nor at large-scale data and does not provide any graph-based methods, which are the core of SCANPY . Also, FLOTILLA is built around a complicated class STUDY , which contains data, tools, and plotting functions. SCANPY , by contrast, is built around a simple HDF5-backed class ANNDATA , which makes SCANPY both scalable and extendable (law of Demeter)” | concepts[35] |
Full text — tagged
The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.
SCANPYterm·dom : large-scale single-cell gene expression data analysis Software Open access Published: 06 February 2018 Volume 19 , article number 15 ( 2018 ) Cite this article You have full access to this open access article Download PDF Save article View saved research Genome Biology Aims and scope Submit manuscript SCANPYterm·dom : large-scale single-cell gene expression data analysis Download PDF Abstract Scanpyterm·dom is a scalable toolkitterm for analyzing single-cell gene expression data. It includes methods for preprocessingterm, visualizationterm, clusteringterm, pseudotime and trajectory inferenceterm, differential expression testingterm, and simulation of gene regulatory networksterm. Its Python-based implementation efficiently deals with data sets of more than one million cells ( https://github.com/theislab/Scanpyterm·dom ). Along with Scanpyterm·dom , we present AnnDataterm·dom , a generic classterm·dom for handling annotated data matricesterm·dom ( https://github.com/theislab/anndataterm·dom ).
Ask a research question Get related insights from Springer Nature content.
How can lineage reconstruction be robustly performed on million-cell data?
How to scale single-cell analysis tools for large, high-dimensional datasets?
Or ask your own question Similar content being viewed by others scAlign: a tool for alignment, integration, and rare cell identification from scRNA-seq data Article Open access 14 August 2019 SingleScan: a comprehensive resource for single-cell sequencing data processing and mining Article Open access 07 December 2023 Single-Cell RNA Sequencing Data Analysis Chapter © 2025 Explore related subjects Discover the latest articles, books and news in related subjects, suggested using machine learning.
Gene Expression Analysis Gene expression profiling Genome-wide analysis of gene expression Microarray analysis Transcriptomics High-throughput Cell Screening Gene Expression Analysis in Cancer Genomics Background Simple integrated analysis work flows for single-cell transcriptomic data [ 1 ] have been enabled by frameworks such as SEURATterm·dom [ 2 ], MONOCLE [ 3 ], SCDE/PAGODA [ 4 ], MAST [ 5 ], CELL RANGERterm·dom [ 6 ], SCATER [ 7 ], and SCRAN [ 8 ]. However, these frameworks do not scale to the increasingly available large data sets with up to and more than one million cells. Here, we present a framework that overcomes this limitation and provides similar analysis possibilities. Moreover, in contrast to the existing R-based frameworks, SCANPYterm·dom ’s Python-based implementation is easy to interface with advanced machine-learning packages, such as TENSORFLOW [ 9 ].
Results SCANPYterm·dom integrates canonical analysis methods in a scalable way SCANPYterm·dom integrates the analysis possibilities of established R-based frameworks and provides them in a scalable and modular form. Specifically, SCANPYterm·dom provides preprocessingterm comparable to SEURATterm·dom [ 10 ] and CELL RANGERterm·dom [ 6 ], visualizationterm through TSNE [ 11 , 12 ], graph-drawing [ 13 – 15 ] and diffusion maps [ 11 , 16 , 17 ], clusteringterm similar to PHENOGRAPH [ 18 – 20 ], identification of marker genes for clusters via differential expression tests and pseudotemporal ordering via diffusion pseudotime [ 21 ], which compares favorably [ 22 ] with MONOCLE 2 [ 22 ], and WISHBONE [ 23 ] (Fig. 1 a).
Fig. 1 Full size image a SCANPYterm·dom ’s analysis features. We use the example of 68,579 peripheral blood mononuclear cells of [ 6 ]. We regress out confounding variables, normalize, and identify highly variable genes. TSNE and graph-drawing (Fruchterman–Reingold) visualizationsterm show cell-type annotationsterm·dom obtained by comparisons with bulk expression. Cells are clustered using the Louvain algorithm. Ranking differentially expressed genes in clusters identifies the MS4A1 marker gene for B cells in cluster 7, which agrees with the bulk labels. We use pseudotemporal ordering from a root cell in the CD34+ cluster and detect a branching trajectory, visualized with TSNE and diffusion maps. b Speedup over CELL RANGERterm·dom R kit. We consider representative steps of the analysis [ 6 ]. c Visualizing and clusteringterm 1.3 million cells. The data, brain cells from E18 mice, are publicly available from 10x Genomics. PCA = principal component analysis, DC = diffusion component SCANPYterm·dom is benchmarked in comparisons with established packages In a detailed clusteringterm tutorial of 2700 peripheral blood mononuclear cells (PBMCs), adapted from one of SEURATterm·dom ’s tutorials ( http://satijalab.org/seuratterm·dom/pbmc3k_tutorial.html ) [ 2 ], all steps starting from raw count data to the identification of cell types are carried out, providing speedups between 5 and 90 times in each step ( https://github.com/theislab/scanpyterm·dom_usage/tree/master/170505_seuratterm·dom ). Benchmarking against the more run-time optimized CELL RANGERterm·dom R kit [ 6 ], we demonstrate a speedup of 5 to 16 times for a data set of 68,579 PBMCs (Fig. 1 a,b, https://github.com/theislab/scanpyterm·dom_usage/tree/master/170503_zheng17 ) [ 6 ]. Moreover, we demonstrate the feasibility of analyzing 1.3 million cells without subsampling in a few hours of computing time on eight cores of a small computing server (Fig. 1 c, https://github.com/theislab/scanpyterm·dom_usage/tree/master/170522_visualizing_one_million_cells ). Thus, SCANPYterm·dom provides tools with speedups that enable an analysis of data sets with more than one million cells and an interactive analysis with run times of the order of seconds for about 100,000 cells.
In addition to the mentioned standard clustering-based analyses approaches, we demonstrate the reconstruction of branching developmental processes via diffusion pseudotime [ 21 ] as in the original paper ( https://github.com/theislab/scanpyterm·dom_usage/tree/master/170502_haghverdi16 ), the simulation of single cells using literature-curated gene regulatory networks based on the ideas of [ 24 ] ( https://github.com/theislab/scanpyterm·dom_usage/tree/master/170430_krumsiek11 ), and the analysis of deep-learning results for single-cell imaging data [ 25 ] ( https://github.com/theislab/scanpyterm·dom_usage/tree/master/170529_images ).
SCANPYterm·dom introduces efficient modular implementation choices With SCANPYterm·dom , we introduce the class ANNDATAterm·dom —with a corresponding package ANNDATAterm·dom —which stores a data matrixterm·dom with the most general annotationsterm·dom possible: annotationsterm·dom of observations (samples, cells) and variables (features, genes), and unstructured annotationsterm·dom. As SCANPYterm·dom is built around that class, it is easy to add new functionality to the toolkit. All statistics and machine-learning tools extract information from a data matrixterm·dom, which can be added to an ANNDATAterm·dom object while leaving the structure of ANNDATAterm·dom unaffected. ANNDATAterm·dom is similar to R’s EXPRESSIONSETterm·dom [ 26 ], but supports sparse dataterm·dom and allows HDF5-based backingterm of ANNDATAterm·dom objects on disk, a format independent of platform, framework, and language. This allows operating on an ANNDATAterm·dom object without fully loading it into memory—the functionality is offered via ANNDATAterm·dom ’s backed modeterm·dom as opposed to its memory modeterm·dom. To simplify memory-efficient pipelines, SCANPYterm·dom ’s functions operate in-placeterm by default but allow the optional non-destructive transformationterm of objects. Pipelines written this way can then also be run in backed modeterm·dom to exploit online-learning formulationsterm of algorithms. Almost all of SCANPYterm·dom ’s tools are parallelized.
SCANPYterm·dom introduces a class for representing a graph of neighborhood relationsterm among data points. The computation of neighborhood relationsterm·dom is much faster than in the popular reference package [ 27 ]. This is achieved by aggregating rows (observations) in a data matrixterm·dom to submatrices and computing distances for each submatrix using fast parallelized matrix multiplication. Moreover, the class provides several functions to compute random-walk-based metricsterm that are not available in other graph software [ 14 , 28 , 29 ]. Typically, SCANPYterm·dom ’s tools reuse a once-computed, single graph representationterm·dom of data and hence, avoid the use of different, potentially inconsistent, and computationally expensive representations of data.
Conclusions SCANPYterm·dom ’s scalability directly addresses the strongly increasing need for aggregating larger and larger data sets [ 30 ] across different experimental setups, for example within challenges such as the Human Cell Atlas [ 31 ]. Moreover, being implemented in a highly modular fashion, SCANPYterm·dom can be easily developed further and maintained by a community. The transfer of the results obtained with different tools used within the community is simple, as SCANPYterm·dom ’s data storage formats and objects are language independent and cross-platform. SCANPYterm·dom integrates well into the existing Python ecosystemterm·dom, in which no comparable toolkit yet exists.
During the revision of this article, the loom file format ( https://github.com/linnarsson-lab/loompy ) was proposed for HDF5-based storage of annotated data. Within a joint effort of facilitating data exchange across different labs, ANNDATAterm·dom now supports importing and exporting to loom ( https://github.com/linnarsson-lab/loompy ). In this context, we acknowledge the discussions with S. Linnarson, which motivated us to extend ANNDATAterm·dom ’s previously staticterm·dom to a dynamic HDF5 backingterm·dom. Just before submission of this manuscript, a C++ library that provides simple interfacing of HDF5-backed matrices in R was made available as a preprint [ 32 ].
Methods SCANPYterm·dom ’s technological foundations SCANPYterm·dom ’s core relies on NUMPY [ 33 ], SCIPY [ 34 ], MATPLOTLIB [ 35 ], PANDAS [ 36 ], and H5PY [ 37 ]. Parts of the toolkit rely on SCIKIT-LEARN [ 27 ], STATSMODELS [ 38 ], SEABORN [ 39 ], NETWORKX [ 28 ], IGRAPH [ 14 ], the TSNE package of [ 40 ], and the Louvain clusteringterm package of [ 41 ]. The ANNDATAterm·dom class—available within the package ANNDATAterm·dom —relies only on NUMPY , SCIPY , PANDAS , and H5PY .
SCANPYterm·dom ’s Python-based implementation allows easy interfacing to advanced machine-learning packages such as TENSORFLOW [ 9 ] for deep learning [ 42 ], LIMIX for linear mixed models [ 43 ], and GPY/GPFLOW for Gaussian processes [ 44 , 45 ]. However, we note that the Python ecosystemterm·dom comes with less possibilities for classical statistical analyses compared to R.
Comparison with existing Python packages for single-cell analysis Aside from the highly popular SCLVM ( https://github.com/PMBio/scLVM ) [ 46 , 47 ], which uses Gaussian process latent variable models for inferring hidden sources of variation, there are, among others, the visualizationterm frameworks FASTPROJECT ( https://github.com/YosefLab/FastProject ) [ 48 ], ACCENSE ( http://www.cellaccense.com/ ) [ 49 ], and SPRING ( https://github.com/AllonKleinLab/SPRING ) [ 15 ]—the latter uses the JavaScript package ( http://d3js.org D3.js for the actual visualizationterm and Python only for preprocessingterm—the trajectory inference tool SCIMITAR ( https://github.com/dimenwarper/scimitar ), the clusteringterm tool PHENOGRAPH ( https://github.com/jacoblevine/PhenoGraph ) [ 19 ], the single-cell experiment design tool MIMOSCA ( https://github.com/asncd/MIMOSCA )[ 50 ], UMIS ( https://github.com/vals/umis ) for handling raw read data [ 51 ], the tree-inference tool ECLAIR ( https://github.com/GGiecold/ECLAIR ) [ 52 ], and the framework FLOTILLAterm ( https://github.com/yeolab/flotillaterm ), which comes with modules for simple visualizationterm, simple clusteringterm, and differential expression testingterm. Hence, only the latter provides a data analysis frameworkterm that solves more than one specific task. In contrast to SCANPYterm·dom , however, FLOTILLAterm is neither targeted at single-cell nor at large-scale dataterm and does not provide any graph-based methodsterm, which are the core of SCANPYterm·dom . Also, FLOTILLAterm is built around a complicated class STUDYterm·dom , which contains data, tools, and plotting functions. SCANPYterm·dom , by contrast, is built around a simple HDF5-backed class ANNDATAterm·dom , which makes SCANPYterm·dom both scalable and extendable (law of Demeter).
Availability and requirements SCANPYterm·dom ’s and ANNDATAterm·dom ’s open-source code are maintained on GITHUB ( https://github.com/theislab/scanpyterm·dom , https://github.com/theislab/anndataterm·dom ) and published under the BSD3 license.
SCANPYterm·dom and ANNDATAterm·dom are released via the Python packaging index: https://pypi.python.org/pypi/scanpyterm·dom and https://pypi.python.org/pypi/anndataterm·dom .
Demonstrations and benchmarks discussed in the main text are all stored at https://github.com/theislab/scanpyterm·dom_usage and summarized here: Analyzing 68,579 PBMCs (Fig. 1 ) in a comparison with the Cell Rangerterm·dom R kit [ 6 ]: https://github.com/theislab/scanpyterm·dom_usage/tree/master/170503_zheng17 .
Clusteringterm and identifying cell types, adapted from and benchmarked with http://satijalab.org/seuratterm·dom/pbmc3k_tutorial.html and one of Seuratterm·dom ’s tutorials [ 2 ]: https://github.com/theislab/scanpyterm·dom_usage/tree/master/170505_seuratterm·dom .
Visualizing and clusteringterm 1.3 million cells (Fig. 1 c): https://github.com/theislab/scanpyterm·dom_usage/tree/master/170522_visualizing_one_million_cells .
Reconstructing branching processes via diffusion pseudotime [ 21 ]: https://github.com/theislab/scanpyterm·dom_usage/tree/master/170502_haghverdi16 .
Simulating single cells using gene regulatory networks [ 24 ]: https://github.com/theislab/scanpyterm·dom_usage/tree/master/170430_krumsiek11 .
Analyzing deep-learning results for single-cell images [ 25 ]: https://github.com/theislab/scanpyterm·dom_usage/tree/master/170529_images .
The data sets used in demonstrations and benchmarks are three data sets from 10x Genomics.
Programming language: Python Operating system: Linux, Mac OS and Windows