standOff. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote shown on the same row, so the borrowing can be checked against the thing borrowed from without leaving this page. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.| second reading | alternatives | an independent re-reading of a facet, recorded whether or not it agrees |
| relation to other work | claim_relations | how this paper stands to another: agrees, contradicts, extends, qualifies |
| claim type | claim_type | the shape of the claim: existence, mechanism, method, critique, prediction, null |
| concept edge | concept_edges | an asserted relation between two concepts, with the sentence asserting it |
| concept | concepts | an idea the paper works with, quoted where the paper uses it |
| content tag | content_tags | a finding, definition, quantity, statistic or variable named in the paper |
| contribution | contribution | what the paper claims to add |
| limitation | limitations | what the paper says its evidence cannot support |
| mechanism | mechanisms | how the reported effect is said to come about |
| method | method | how the paper produced its evidence |
| object of study | object_of_study | what the paper is about — the thing it investigates |
| phenomenon | phenomenon | the effect or regularity under study |
| research question | research_question | the question the paper sets itself |
| sense | senses | a distinct use of a term, quoted where the paper fixes it |
| transfer | transfers | a method or concept carried in from another field |
| unrecordable relation | unhoused_relations | a relation the paper asserts that this vocabulary cannot express, with the reason |
The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.
Abstract
research_questionconcepts↗concept_edgescontent_tags↗Genomics is a Big Data science and is going to get much bigger, very soon, but limitationsit is not known whether the needs of genomics will exceed other Big Data domains. sensesalternativesclaim_relationscontent_tags↗Projecting to the year 2025, we compared genomics with three other major generators of Big Data: astronomy, YouTube, and Twitter. contributioncontent_tagsOur estimates show that genomics is a “four-headed beast”—content_tagsit is either on par with or the most demanding of the domains analyzed here in terms of data acquisition, storage, distribution, and analysis. We discuss aspects of new technologies that will need to be developed to rise up and meet the computational challenges that genomics poses for the near future. Now is the time for concerted, community-wide planning for the “genomical” challenges of the next decade.
This perspective considers the growth of genomics over the next ten years and assesses the computational needs that we will face relative to other "Big Data" activities such as astronomy, YouTube, and Twitter.
We compared genomics with three other major generators of Big Data: astronomy, YouTube, and Twitter. Astronomy has faced the challenges of Big Data for over 20 years and continues with ever-more ambitious studies of the universe. YouTube burst on the scene in 2005 and has sparked extraordinary worldwide interest in creating and sharing huge numbers of videos. Twitter, created in 2006, has become the poster child of the burgeoning movement in computational social science [ 6 ], with unprecedented opportunities for new insights by mining the enormous and ever-growing amount of textual data [ 7 ]. mechanisms↗concepts↗sensesParticle physics also produces massive quantities of raw data, although the footprint is surprisingly limited since the vast majority of data are discarded soon after acquisition using the processing power that is coupled to the sensors [ 8 ]. limitationslimitationsConsequently, we do not include the domain in full detail here, although that model of rapid filtering and analysis will surely play an increasingly important role in genomics as the field matures.
object_of_studymethod↗mechanisms↗concepts↗concepts↗concepts↗concepts↗sensesconcept_edgesconcept_edgesconcept_edgesconcept_edgesconcept_edgescontent_tags↗To compare these four disparate domains, we considered the four components that comprise the “life cycle” of a dataset: acquisition, storage, distribution, and analysis ( Table 1 ).
Table 1. Four domains of Big Data in 2025.
In each of the four domains, the projected annual storage and computing needs are presented across the data lifecycle.
Data Phase
Astronomy
YouTube
Genomics
Acquisition
25 zetta-bytes/year 0.5–15 billion tweets/year 500–900 million hours/year 1 zetta-bases/year
Storage
1 EB/year 1–17 PB/year 1–2 EB/year 2–40 EB/year
Analysis
In situ data reduction Topic and sentiment mining Limited requirements Heterogeneous data and analysis
Real-time processing Metadata analysis
Variant calling, ~2 trillion central processing unit (CPU) hours
Massive volumes
All-pairs genome alignments, ~10,000 trillion CPU hours
Distribution
Dedicated lines from antennae to server (600 TB/s) Small units of distribution Major component of modern user’s bandwidth (10 MB/s) Many small (10 MB/s) and fewer massive (10 TB/s) data movement
Open in a new tab
content_tagsThe four Big Data domains differ sharply in how data are acquired. concepts↗Most astronomy data are acquired from a few highly centralized facilities [ 9 ]. mechanisms↗By contrast, YouTube and Twitter acquire data in a highly distributed manner, but under a few standardized protocols. Astronomy, YouTube, and Twitter are expected to show continued dramatic growth in the volume of data to be acquired. For example, content_tagsthe Australian Square Kilometre Array Pathfinder (ASKAP) project content_tagscurrently acquires 7.5 terabytes/second of sample image data, a rate projected to increase 100-fold to 750 terabytes/second (~25 zettabytes per year) by 2025 [ 9 , 10 ]. content_tagscontent_tags↗YouTube currently has 300 hours of video being uploaded every minute, and this could grow to 1,000–1,700 hours per minute (1–2 exabytes of video data per year) by 2025 if we extrapolate from current trends ( S1 Note ). content_tagsToday, content_tagsTwitter generates 500 million tweets/day, each about 3 kilobytes including metadata ( S2 Note ). While this figure is beginning to plateau, a projected logarithmic growth rate would suggest a 2.4-fold growth by 2025, to 1.2 billion tweets per day, 1.36 petabytes/year. content_tagscontent_tags↗In short, content_tagsdata acquisition in these domains is expected to grow by up to two orders of magnitude in the next decade.
concepts↗For genomics, data acquisition is highly distributed and involves heterogeneous formats. phenomenonalternativescontent_tagscontent_tagscontent_tags↗The rate of growth over the last decade has also been truly astonishing, with the total amount of sequence data produced content_tagsdoubling approximately every seven months ( Fig 1 ). content_tagscontent_tagsThe OmicsMaps catalog of all known sequencing instruments in the world [ 11 ] reports that currently there are more than 2,500 high-throughput instruments, manufactured by several different companies, located in nearly 1,000 sequencing centers in 55 countries in universities, hospitals, and other research laboratories. These centers range in size from small laboratories with a few instruments generating a few terabases per year to large dedicated facilities producing several petabases a year. limitationscontent_tags↗(content_tags↗An approximate conversion factor to use in interpreting these numbers is 4 bases = 1 byte, though we will revisit this below.)
Fig 1. Growth of DNA sequencing.
Open in a new tab
The plot shows the growth of DNA sequencing both in the total number of human genomes sequenced (left axis) as well as the worldwide annual sequencing capacity (right axis: Tera-basepairs (Tbp), Peta-basepairs (Pbp), Exa-basepairs (Ebp), Zetta-basepairs (Zbps)). The values through 2015 are based on the historical publication record, with selected milestones in sequencing (first Sanger through first PacBio human genome published) as well as three exemplar projects using large-scale sequencing: the 1000 Genomes Project, aggregating hundreds of human genomes by 2012 [ 3 ]; content_tags↗The Cancer Genome Atlas (TCGA), aggregating over several thousand tumor/normal genome pairs [ 4 ]; and content_tags↗the Exome Aggregation Consortium (ExAC), aggregating over 60,000 human exomes [ 5 ]. Many of the genomes sequenced to date have been whole exome rather than whole genome, but we expect the ratio to be increasingly favored towards whole genome in the future. The values beyond 2015 represent our projection under three possible growth curves as described in the main text.
The raw sequencing reads used in most published studies are archived at either the content_tags↗Sequence Read Archive (SRA) maintained by the United States National Institutes of Health National Center for Biotechnology Information (NIH/NCBI) or one of the international counterparts. content_tagsThe SRA currently contains more than 3.6 petabases of raw sequence data ( S1 Fig ), which reflects the ~32,000 microbial genomes, ~5,000 plant and animal genomes, and ~250,000 individual human genomes that have been sequenced or are in progress thus far [ 12 ]. limitationssensescontent_tagsHowever, the 3.6 petabases represent a small fraction of the total produced; most of it is not yet in these archives. concepts↗Based on manufacturer specifications of the instruments, we estimate the current content_tags↗worldwide sequencing capacity to exceed 35 petabases per year, including the sixteen Illumina X-Ten systems that have been sold so far [ 13 ], each with a capacity of ~2 petabases per year [ 14 ].
limitationslimitationsOver the next ten years, we expect sequencing capacities will continue to grow very rapidly, although the project growth becomes more unpredictable the further out we consider. alternativescontent_tagsIf the growth continues at the current rate by doubling every seven months, then we should reach more than one exabase of sequence per year in the next five years and approach one zettabase of sequence per year by 2025 ( Fig 1 , Table 1 ). mechanisms↗concepts↗claim_relationscontent_tagsInterestingly, even at the more conservative estimates of doubling every 12 months (Illumina’s current own estimate [ 12 ]) or every 18 months (equivalent to Moore’s law), we should reach exabase-scale genomics well within the next decade. We anticipate this sequencing will encompass genome sequences for most of the approximately 1.2 million described species of plants and animals [ 15 ]. With these genomes, plus those of thousands of individuals of “high value” species for energy, environmental, and agricultural reasons, we estimate that there will be at least 2.5 million plant and animal genome sequences by 2025. For example, the genomics powerhouse BGI, in conjunction with the International Rice Research Institute and the Chinese Academy of Agricultural Sciences, has already sequenced 3,000 varieties of rice [ 16 ] and announced a massive project of their own to sequence one million plant and animal genomes [ 17 ]. The Smithsonian Institute also has similar plans to “capture and catalog all the DNA from the world’s flora and fauna.” There also will be genomes for several millions of microbes, with explosive growth projected for both medical and environmental microbe metagenomic sequencing [ 18 , 19 ].
These estimates, however, are dwarfed by the very reasonable possibility that a significant fraction of the world’s human population will have their genomes sequenced. The leading driver of this trend is the promise of genomic medicine to revolutionize the diagnosis and treatment of disease, with some countries contemplating sequencing large portions of their populations: both England [ 20 ] and Saudi Arabia [ 21 ] have announced plans to sequence 100,000 of their citizens, one-third of Iceland’s 320,000 citizens have donated blood for genetic testing [ 22 ], and researchers in both the US [ 23 ] and China [ 17 ] both aim to sequence 1 million genomes in the next few years. With the world’s population projected to top 8 billion by 2025, it is possible that as many as 25% of the population in developed nations and half of that in less-developed nations will have their genomes sequenced (comparable to the current worldwide distribution of Internet users [ 24 ]).
claim_typecontent_tagsWe therefore estimate between 100 million and as many as 2 billion human genomes could be sequenced by 2025, representing four to five orders of magnitude growth in ten years and far exceeding the growth for the three other Big Data domains. Indeed, this number could grow even larger, especially since new single-cell genome sequencing technologies are starting to reveal previously unimagined levels of variation, especially in cancers, necessitating sequencing the genomes of thousands of separate cells in a single tumor [ 10 ].
Moreover, the technology used to sequence DNA is deployed creatively for other applications (e.g., transcriptome, epigenome, proteome, metabolome, and microbiome sequencing) necessitating generating new sequencing data multiple times per person to monitor molecular activity [ 25 ]. These applications require precise quantitative counts of sequencing reads to capture diversity of expression or diversity of abundances, thus requiring millions of reads to accurately estimate underlying distributions as they change over time. For medicine, just having the genome will not be sufficient: for each individual, it will need to be coupled with other relevant ‘omics data sets, some collected periodically and from different tissues, to compare healthy and diseased states [ 26 ]. Computational challenges will increase because of dramatic increases in the total volume of genomic data per person, as will the complexities of integrating these diverse data sources to improve health and cure diseases. Genomics thus appears to pose the greatest challenges for data acquisition of the four Big Data domains.
Data Storage
Data storage requirements for all four domains are projected to be enormous. Today, the largest astronomy data center devotes ~100 petabytes to storage, and the completion of the Square Kilometre Array (SKA) project is expected to lead to a storage demand of 1 exabyte per year. YouTube currently requires from 100 petabytes to 1 exabyte for storage and may be projected to require between 1 and 2 exabytes additional storage per year by 2025. Twitter’s storage needs today are estimated at 0.5 petabytes per year, which may increase to 1.5 petabytes in the next ten years. limitations(Our estimates here ignore the “replication factor” that multiplies storage needs by ~4, for redundancy.) For genomics, we have determined more than 100 petabytes of storage are currently used by only 20 of the largest institutions ( S1 Table ).
limitationsconcepts↗Projections of storage requirements for sequence data depend on the accuracy and application of the sequencing. For every 3 billion bases of human genome sequence, 30-fold more data (~100 gigabases) must be collected because of errors in sequencing, base calling, and genome alignment. This means that as much as 2–40 exabytes of storage capacity will be needed by 2025 just for the human genomes ( S3 Note ). concepts↗concepts↗sensesconcept_edgesThese needs can be diminished with effective data compression [ 27 ], but decompression times and fidelity are a major concern in compressive genomics [ 28 ].
concepts↗concepts↗unhoused_relationssensesAre the emerging “third-generation” single-molecule sequencing technologies with much longer reads, such as those from Pacific Biosciences and Oxford Nanopore, a computational panacea? Though error rates currently are higher and throughput lower than short-read technologies, as they mature, these technologies are starting to be used to sequence and assemble nearly entire chromosomes [ 29 ]. This will minimize the need to oversample as much, and eventually, the raw sequence data may not need to be stored at all. However, eliminating the need to store raw sequence data and only retaining complete genomes will have relatively little impact overall—perhaps one or two orders of magnitude less data storage. concepts↗concepts↗unhoused_relationssensesMore significant reductions in storage demand will come when improvements in sequencing accuracy and database comprehensiveness reach the point at which genome sequences themselves do not need to be stored, just the list of variants relative to a reference collection (“delta encoding”) [ 30 ]. This works well for cataloging the simplest variants in a human genome, but it may not be as useful for complex samples, such as cancer genomes, that have many novel rearrangements and mutations. While certainly helpful, we thus do not expect long-read sequencing technology or delta encoding to solve the storage challenges for genome sequencing in 2025.
concepts↗concepts↗concepts↗concepts↗sensesIn contrast, we do see great opportunities for data reduction and real-time analysis of other ‘omics analysis. For example, once sequencing becomes fast enough and the methods mature enough to correctly infer transcript expression levels in real time, we anticipate that raw RNA-seq reads will no longer be stored, except for specific research purposes. Already several such “streaming” algorithms have been published for this purpose, performing as well as or superior to their nonstreaming counterparts [ 31 ]. transfersFor RNA-seq and other ‘omics applications, genomics will benefit greatly from the lessons learned in particle physics, in which in most cases raw data are discarded almost as fast as they are generated in favor of higher level and greatly compressed summaries.
Altogether, we anticipate the development of huge genomics archives used for storing millions of genomes along with the associated ‘omics measurements over time. Ideally, these archives will also collect or be linked to the patient phenotypic data, especially disease outcomes and treatments provided to support retrospective analysis as new relationships are discovered. concepts↗sensesTo make it practical to search and query through such vast collections, the data will be stored in hierarchical systems that make data and their statistical summaries available at different levels of compression and latency, as used in astronomy [ 32 ] and text analysis [ 33 ]. Thus, although total genomic data could far exceed the demands for the others, with the right new innovations the net requirements could be similar to the domains of astronomy and YouTube.
Data Distribution
Astronomy, YouTube, Twitter, and genomics also differ greatly in data distribution patterns. The major bandwidth requirement of the SKA project is to get data from its 3,000 antennae to a central server, requiring as much as 600 terabytes/second [ 34 ]. The bandwidth usage of YouTube is relatively small for a single download and well supported by the average consumer’s 10 Mbps connection, but aggregate needs worldwide are enormous, with estimates up to 240 petabytes/day ( S4 Note ) [ 35 ]. The distribution patterns of genomics data are much more heterogeneous, involving elements of both situations [ 36 ].
Genomic data are distributed in units spanning a wide range of sizes, from comparisons of a few bases or gene sequences to large multiterabyte bulk downloads from central repositories. concepts↗concepts↗concepts↗concepts↗sensesconcept_edgesconcept_edgesFor large-scale analysis, cloud computing is particularly suited to decreasing the bandwidth for distribution of genomic data [ 37 ] so that applications can run on remote machines that already have data [ 38 ]. Only small segments of code are uploaded and highly processed outputs are downloaded, thus significantly reducing the computing resources necessary for distribution.
But in addition to tailoring genomics applications for the cloud, concepts↗concepts↗concepts↗concepts↗unhoused_relationssensesnew methods of data reliability and security are required to ensure privacy, much more so than for the other three domains. A serious breach of medically sensitive genomic data would have permanent consequences and could seriously hinder the development of genomic medicine. Homomorphic encryption systems, in which encrypted data can be analyzed and manipulated for certain controlled queries without disclosing the raw data, are currently too computationally expensive for widespread use, but these and related cryptographic techniques are promising areas of research [ 39 ].
Data Analysis
Astronomy, YouTube, Twitter, and genomics differ most in computational requirements for data analysis. Astronomy data require extensive specialized analysis, but the bulk of this requirement is for in situ processing and reduction of data by computers located near the telescopes [ 40 ]. This initial analysis is daunting because of its real-time nature and huge data volumes but can often be effectively performed in parallel on thousands of cores. YouTube videos are primarily meant to be viewed, along with some automated analysis for advertisements or copyright infringements. Twitter data are the subject of intense research in the social sciences [ 41 ], especially for topic and sentiment mining, which is performed chiefly on textual “tweets” in the context of associated metadata (e.g., user demographics and temporal information).
Analysis of genomic data involves a more diverse range of approaches because of the variety of steps involved in reading a genome sequence and deriving useful information from it. concepts↗sensesFor population and medical genomics, identifying the genomic variants in each individual genome is currently one of the most computationally complex phases. Variant calling on 2 billion genomes per year, with 100,000 CPUs in parallel, would require methods that process 2 genomes per CPU-hour, three-to-four orders of magnitude faster than current capabilities [ 42 ]. concepts↗sensesWhole genome alignment is another important form of genomic data analysis, used for a variety of goals, from phylogeny reconstruction to genome annotation via comparative methodologies. Just a single whole genome alignment between human and mouse consumes ~100 CPU hours [ 43 ]. Aligning all pairs of the ~2.5 million species expected to be available by 2025 amounts to 50–100 trillion such whole genome alignments, which would need to be six orders of magnitude faster than possible today.
concepts↗concepts↗concepts↗sensesconcept_edgesImprovements to CPU capabilities, as anticipated by Moore’s Law, should help close the gap, but trends in computing power are often geared towards floating point operations and do not necessarily provide improvements in genome analysis, in which string operations and memory management often pose the most significant challenges. Moreover, the bigger bottleneck of Big Data analysis in the future may not be in CPU capabilities but in the input/output (I/O) hardware that shuttles data between storage and processors [ 44 ], a problem requiring research into new parallel I/O hardware and algorithms that can effectively utilize them.
The Long Road Ahead
Genomics clearly poses some of the most severe computational challenges facing us in the next decade. Genomics is a “four-headed beast”; considering the computational demands across the lifecycle of a dataset—acquisition, storage, distribution, and analysis—genomics is either on par with or the most demanding of the Big Data domains. concepts↗concepts↗New integrative approaches need to be developed that take into account the challenges in all four aspects: it is unlikely that a single advance or technology will solve the genomics data problem. Several key technologies that are most critically needed to support future solutions are discussed in Box 1 .
In human health, the major needs are driven by the realization that for precision medicine and similar efforts to be most effective, genomes and related ‘omics data need to be shared and compared in huge numbers. concepts↗sensesIf we do not commit as a scientific community to sharing now, we run the risk of establishing thousands of isolated, private data collections, each too underpowered to allow subtle signals to be extracted. More than anything else, concepts↗unhoused_relationsconnecting these resources requires trust among institutions, scientists, and the public to ensure the collections will be used for medical purposes and not to discriminate or penalize individuals because of their genetic makeup.
Finally, the exascale data and computing centers that are emerging today to meet Big Data challenges in several domains (YouTube [ 50 ], Google [ 51 ], Facebook [ 51 ], and the National Security Agency [ 52 ]), are the result of far-sighted planning and commitment by the respective organizations. Now is the time for concerted, community-wide planning for the “genomical” challenges of the next decade.
Box 1. Key Technological Needs for Big Data Genomics
(1) Acquisition
The most important need to sustain the explosive growth in genomic data acquisition is continued advances in sequencing technologies to reduce costs, improve throughput, and achieve very high accuracy. The current costs of ~US$1,000 per human genome begin to make it practical to sequence human genomes in large numbers, especially for critical medical treatments, but to scale to populations of hundreds of millions to billions of genomes, costs must be reduced by at least another one to two orders of magnitude or more. For many medical applications, the time for sequencing must also be reduced so that it can be completed in near real time, especially to rapidly diagnosis acute infections and conditions. Finally, to make a genome sequence most useful, it must be paired with automated methods to collect metadata and phenotype data, all according to appropriate standards so that data collected in one environment can be compared to those collected in another.
(2) Storage
The community needs to start designing and constructing data centers with fast, tiered storage systems to query and aggregate over large collections of genomes and ‘omics data. There are new technologies on the horizon that will help support these needs, including 3-D memory, integrated computing technologies that overcome the I/O bottleneck, and networks that are two-to-five orders of magnitude faster because of optical switching [ 45 , 46 ]. Similarly, efficient compression and indexing systems are critical to make the best use out of each available byte while making the data highly accessible. We also expect algorithmic developments that can represent large collections of personal genomes as a compact graph, making it more efficient and robust to compare one genome to many others. Beyond these approaches, we see the rise of streaming approaches to make on-the-fly comparisons that will allow us to rapidly discard data, especially for sequencing applications that use the sequence data as a means to infer abundances or other molecular activity.
(3) Distribution
The most practical, and perhaps only, solution for distributing genome sequences at a population scale is to use cloud-computing systems that minimize data movement and maximize code federation [ 47 ]. New developments from companies such as Google, Amazon, and Facebook that include applications built to fit the frameworks of distributed computing efficient data centers and distributed storage and cloud computing paradigms will be part of the solution. Already, large cloud-based genomic resources are being developed using these technologies, especially to support the needs of the largest sequencing centers or to support the needs of large communities (BGI-cloud, TCGA, the International Cancer Genome Consortium [ICGC], etc.). To make these online systems most useful, concepts↗unhoused_relationsthe community needs to develop application programming interfaces (APIs) for discovering and querying large datasets on remote systems. The Global Alliance for Genomics and Health [ 48 ] and others are beginning to develop such standards for human genomic data, and we expect other communities to follow. Finally, authentication, encryption, and other security safeguards must be developed to ensure that genomic data remain private.
(4) Analysis
concepts↗Our ultimate goal is to be able to interpret genomic sequences and answer how DNA mutations, expression changes, or other molecular measurements relate to disease, development, behavior, or evolution. Accomplishing this goal will clearly require integration of biological domain expertise, transferslarge-scale machine learning systems, and a computing infrastructure that can support flexible and dynamic queries to search for patterns over very large collections in very high dimensions. A number of “data science technologies,” including R, Mahout, and other machine learning systems powered by Hadoop and other highly scalable systems, are a start, but the current offerings are still difficult and expensive to use. The community would also benefit from libraries of highly optimized algorithms within a simple interface that can be combined and reused in many contexts as the problems emerge. Data science companies as well as open-source initiatives are already starting to develop such components, such as Amazon’s recent “Amazon Machine Learning” prediction system. But because genomics poses unique challenges in terms of data acquisition, distribution, storage, and especially analysis, waiting for innovations from outside our field is unlikely to be sufficient. We must face these challenges ourselves, concepts↗starting with integrating data science into graduate, undergraduate, and high-school curricula to train the next generations of quantitative biologists, bioinformaticians, and computer scientists and engineers [ 49 ].
Supporting Information
S1 Fig. Growth of GenBank. The y -axis shows the total sequence in bp. (Blue = GenBank, red = whole genome shotgun [WGS] sequences.) Each line is double of the previous. The x -axis indicates time. Each line is 6 months after the previous. Source: http://www.ncbi.nlm.nih.gov/genbank/statistics .
(TIF)
Click here for additional data file. (1.9MB, tif)
S1 Note. YouTube data estimates. (DOCX)
Click here for additional data file. (65.6KB, docx)
S2 Note. Twitter data estimates. (DOCX)
Click here for additional data file. (84.9KB, docx)
S3 Note. Human genomic data storage estimates for 2025. (DOCX)
Click here for additional data file. (39.9KB, docx)
S4 Note. YouTube distribution statistics (current). (DOCX)
Click here for additional data file. (37.8KB, docx)
S1 Table. Capacities of 20 major genomics institutions. The number of sequencers as listed from OmicsMaps.com and their storage capacities from the listed citation. These 20 institutions alone collectively have more than 100 PB of storage available.
(DOCX)
Click here for additional data file. (171.6KB, docx)
Acknowledgments
We thank V. Jongeneel, M. Baker, and the anonymous reviewers for comments that improved the manuscript and the Illinois CompGen Initiative for providing the collaborative structure that fostered the idea for this analysis. We would also like to thank all of the participants of the 2014 Keystone Symposium on “Big Data in Biology” (organized by Lincoln D. Stein, Doreen Ware, and MCS) for their inspiration for this analysis.
Abbreviations
API application programming interface
ASKAP Australian Square Kilometre Array Pathfinder
CPU central processing unit
ExAC Exome Aggregation Consortium
ICGC International Cancer Genome Consortium
I/O input/output
NIH/NCBI National Institutes of Health National Center for Biotechnology Information
SKA Square Kilometre Array
SRA Sequence Read Archive
TCGA The Cancer Genome Atlas
WGS whole genome shotgun
Funding Statement
This research was supported, in part, by grant 1U54GM114838 awarded by NIGMS to SS through funds provided by the trans-NIH Big Data to Knowledge (BD2K) initiative and by National Institutes of Health award (R01-HG006677) to MCS. ZDS and RKI were supported by NSF grant MRI13-37732 (S.S. Lumetta, PI). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; 7 of 10 relations here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.
This record holds no controls. A layer is empty when the tagger recorded nothing in it — which is not the same as the paper having nothing to record.
| facet | tag | field | conf. | whose words |
|---|---|---|---|---|
| object_of_study | The computational demands of genomics in 2025 - acquisition, storage, distribution and analysis - set against astronomy, YouTube and Twitter. | object_of_study | the paper's words | |
| research_question | Genomics is growing fast, but is it about to become the most computationally demanding Big Data science, or merely one among several? | research_question | the paper's words | |
| contribution | A like-for-like projection across four domains showing genomics is on par with or ahead of all of them on every phase of the data lifecycle. | contribution | the paper's words | |
| claim_type | prediction | claim_type | the paper's words | |
| phenomenon | growth of genomic data volume | phenomenon | the paper's words | |
| method | computational-model also in: computational-model | method | the paper's words | |
| mechanisms | mechanisms | mechanisms[0] | 1.0 | the tagger's words |
| mechanisms | mechanisms | mechanisms[1] | 1.0 | the tagger's words |
| mechanisms | mechanisms | mechanisms[2] | 1.0 | the tagger's words |
| mechanisms | mechanisms | mechanisms[3] | 1.0 | the tagger's words |
| limitations | it is not known whether genomics needs will exceed other Big Data domains what this limits: the paper's central comparative question, which it leaves open: IT IS NOT KNOWN WHETHER THE NEEDS OF GENOMICS WILL EXCEED OTHER BIG DATA DOMAINS | limitations[0] | the tagger's words | |
| limitations | Projections become less reliable the further ahead they reach. what this limits: the far end of the projections: THE PROJECT GROWTH BECOMES MORE UNPREDICTABLE THE FURTHER OUT WE CONSIDER | limitations[1] | the tagger's words | |
| limitations | Storage estimates omit the replication factor for redundancy. what this limits: every storage figure quoted: OUR ESTIMATES HERE IGNORE THE REPLICATION FACTOR THAT MULTIPLIES STORAGE NEEDS BY ~4, FOR REDUNDANCY | limitations[2] | the tagger's words | |
| limitations | Particle physics, a fifth massive data producer, was deliberately left out of the comparison. what this limits: the scope of the four-domain comparison: WE DO NOT INCLUDE THE DOMAIN IN FULL DETAIL HERE - particle physics, a fifth massive producer, is left out | limitations[3] | the tagger's words | |
| limitations | Storage projections are contingent on sequencing accuracy and application. what this limits: the storage projections specifically: PROJECTIONS OF STORAGE REQUIREMENTS FOR SEQUENCE DATA DEPEND ON THE ACCURACY AND APPLICATION OF THE SEQUENCING | limitations[4] | the tagger's words | |
| limitations | Particle physics was excluded from the full comparison, so the framework is applied to four domains rather than to the five the authors identify. duplicate of: limitations[3] · what this limits: the same scope as limitations[3] - four domains compared, not five | limitations[5] | 1.0 | the tagger's words |
| limitations | The authors state the projections become less reliable the further out they run. duplicate of: limitations[1] · what this limits: the same scope as limitations[1] - reliability falls with horizon | limitations[6] | 1.0 | the tagger's words |
| limitations | The archived data is a small and unquantified fraction of what is produced, so the baseline for the whole projection is an estimate from instrument specifications rather than a count. what this limits: the baseline the whole projection is built from: THE 3.6 PETABASES REPRESENT A SMALL FRACTION OF THE TOTAL PRODUCED; MOST OF IT IS NOT YET IN THESE ARCHIVES | limitations[7] | 1.0 | the tagger's words |
| limitations | Bases are converted to bytes at a stated approximate factor, which propagates into every storage figure. what this limits: every bases-to-bytes conversion in the paper: AN APPROXIMATE CONVERSION FACTOR TO USE IN INTERPRETING THESE NUMBERS IS 4 BASES = 1 BYTE - stated as approximate, and it propagates | limitations[8] | 0.9 | the tagger's words |
| transfers | large-scale machine learning and big-data computing infrastructure applied to genomics | transfers[0] | the tagger's words | |
| transfers | the data-retention practice of particle physics - discard the raw data after processing and keep only the reduced product - held up as the lesson genomics should learn as sequencing volumes outgrow storage | transfers[1] | 0.85 | the tagger's words |
| concepts | genomics also in: genomics | concepts[0] | the paper's words | |
| concepts | big data also in: big data | concepts[1] | the paper's words | |
| concepts | life cycle also in: life cycle | concepts[4] | 1.0 | the paper's words |
| concepts | acquisition also in: acquisition | concepts[5] | 0.85 | the paper's words |
| concepts | astronomy also in: astronomy | concepts[6] | 1.0 | the paper's words |
| concepts | particle physics also in: particle physics | concepts[7] | 1.0 | the paper's words |
| concepts | sequencing capacity also in: sequencing capacity | concepts[8] | 1.0 | the paper's words |
| concepts | Moore’s law also in: Moore’s law | concepts[9] | 1.0 | the paper's words |
| concepts | storage also in: storage | concepts[10] | 0.95 | the paper's words |
| concepts | distribution also in: distribution | concepts[11] | 0.95 | the paper's words |
| concepts | analysis also in: analysis | concepts[12] | 0.95 | the paper's words |
| concepts | effective data compression also in: effective data compression | concepts[13] | 0.95 | the paper's words |
| concepts | storage requirements also in: storage requirements | concepts[14] | 0.95 | the paper's words |
| concepts | compressive genomics also in: compressive genomics | concepts[15] | 0.95 | the paper's words |
| concepts | single-molecule sequencing technologies | concepts[16] | 0.95 | the paper's words |
| concepts | raw sequence data also in: raw sequence data | concepts[17] | 0.95 | the paper's words |
| concepts | delta encoding also in: delta encoding | concepts[18] | 0.95 | the paper's words |
| concepts | list of variants relative to a reference collection | concepts[19] | 0.95 | the paper's words |
| concepts | data reduction also in: data reduction | concepts[20] | 0.95 | the paper's words |
| concepts | real-time analysis also in: real-time analysis | concepts[21] | 0.95 | the paper's words |
| concepts | “streaming” algorithms also in: “streaming” algorithms | concepts[22] | 0.95 | the paper's words |
| concepts | nonstreaming counterparts also in: nonstreaming counterparts | concepts[23] | 0.95 | the paper's words |
| concepts | hierarchical systems also in: hierarchical systems | concepts[24] | 0.95 | the paper's words |
| concepts | cloud computing also in: cloud computing | concepts[25] | 0.95 | the paper's words |
| concepts | bandwidth for distribution of genomic data | concepts[26] | 0.95 | the paper's words |
| concepts | remote machines that already have data | concepts[27] | 0.95 | the paper's words |
| concepts | computing resources necessary for distribution | concepts[28] | 0.95 | the paper's words |
| concepts | data reliability also in: data reliability | concepts[29] | 0.95 | the paper's words |
| concepts | security also in: security | concepts[30] | 0.95 | the paper's words |
| concepts | privacy also in: privacy | concepts[31] | 0.95 | the paper's words |
| concepts | Homomorphic encryption systems also in: Homomorphic encryption systems | concepts[32] | 0.95 | the paper's words |
| concepts | Variant calling also in: Variant calling | concepts[33] | 0.95 | the paper's words |
| concepts | Whole genome alignment also in: Whole genome alignment | concepts[34] | 0.95 | the paper's words |
| concepts | CPU capabilities also in: CPU capabilities | concepts[35] | 0.95 | the paper's words |
| concepts | input/output (I/O) hardware also in: input/output (I/O) hardware | concepts[36] | 0.95 | the paper's words |
| concepts | parallel I/O hardware and algorithms also in: parallel I/O hardware and algorithms | concepts[37] | 0.95 | the paper's words |
| concepts | integrative approaches also in: integrative approaches | concepts[38] | 0.95 | the paper's words |
| concepts | single advance or technology also in: single advance or technology | concepts[39] | 0.95 | the paper's words |
| concepts | isolated, private data collections also in: isolated, private data collections | concepts[40] | 0.95 | the paper's words |
| concepts | trust among institutions, scientists, and the public | concepts[41] | 0.95 | the paper's words |
| concepts | application programming interfaces (APIs) | concepts[42] | 0.95 | the paper's words |
| concepts | large-scale machine learning systems also in: large-scale machine learning systems | concepts[43] | 0.95 | the paper's words |
| concepts | data science into graduate, undergraduate, and high-school curricula | concepts[44] | 0.95 | the paper's words |
| unhoused_relations | delta encoding → list of variants relative to a reference collection Delta encoding represents genomes as reference-relative variant lists; represents-as is absent from the edge registry. | unhoused_relations[0] | 0.95 | the tagger's words |
| unhoused_relations | Homomorphic encryption systems → privacy Homomorphic encryption enables controlled queries without raw-data disclosure; enables is absent from the edge registry. | unhoused_relations[1] | 0.95 | the tagger's words |
| unhoused_relations | trust among institutions, scientists, and the public → connecting these resources Trust is a prerequisite for connecting genomic resources; prerequisite-for is absent from the edge registry. | unhoused_relations[2] | 0.95 | the tagger's words |
| unhoused_relations | application programming interfaces (APIs) → discovering and querying large datasets on remote systems APIs enable discovery and query of remote datasets; enables is absent from the edge registry. | unhoused_relations[3] | 0.95 | the tagger's words |
| unhoused_relations | single-molecule sequencing technologies → raw sequence data Mature long-read methods may eliminate the need to retain raw sequence data; enables-omission is absent from the edge registry. | unhoused_relations[4] | 0.9 | the tagger's words |
| senses | Big Data - as a comparison class of domains, each with its own acquisition architecture, rather than as a property a dataset has | senses[0] | 1.0 | the tagger's words |
| senses | data volume - as ACQUIRED, which for particle physics is enormous | senses[1] | 1.0 | the tagger's words |
| senses | data volume - as STORED, which for genomics is a small fraction of what is produced and for particle physics is smaller still | senses[2] | 1.0 | the tagger's words |
| senses | Storage / distribution / analysis - three lifecycle components alongside acquisition | senses[3] | 0.95 | the tagger's words |
| senses | Compressive genomics - storage reduction constrained by decompression time and fidelity | senses[4] | 0.95 | the tagger's words |
| senses | Long-read sequencing - mature technology that may remove the need to retain raw reads | senses[5] | 0.9 | the tagger's words |
| senses | Delta encoding - retention of reference-relative variant lists instead of complete genomes | senses[6] | 0.95 | the tagger's words |
| senses | Streaming omics analysis - real-time inference performing at least as well as nonstreaming methods | senses[7] | 0.95 | the tagger's words |
| senses | Hierarchical genomic archive - data and summaries exposed at multiple compression and latency levels | senses[8] | 0.95 | the tagger's words |
| senses | Cloud genomic analysis - code moves to remote machines already holding the data | senses[9] | 0.95 | the tagger's words |
| senses | Homomorphic encryption - controlled analysis of encrypted data without revealing raw genomes | senses[10] | 0.95 | the tagger's words |
| senses | Variant calling - computationally intensive identification of variants in each individual genome | senses[11] | 0.95 | the tagger's words |
| senses | Whole-genome alignment - comparative method for phylogeny reconstruction and genome annotation | senses[12] | 0.95 | the tagger's words |
| senses | I/O bottleneck - data shuttling between storage and processors rather than CPU arithmetic | senses[13] | 0.9 | the tagger's words |
| senses | Genomic data sharing - protection against isolated collections too underpowered for subtle signals | senses[14] | 0.95 | the tagger's words |
| alternatives | claim_type: prediction a second reading reaching the same value. Recorded: prediction — note: a second reading reaching the same value from a different sentence. Not applied - a person decides. | alternatives.claim_type | the tagger's words | |
| alternatives | method: comparative a second reading DIFFERING from the recorded value. Recorded: computational-model — note: a second reading, verbatim-verified, disagreeing with the value on the record. Not applied - a person decides. | alternatives.method | the tagger's words | |
| alternatives | phenomenon: exponential growth of genomic sequence data relative to other Big Data domains a second reading DIFFERING from the recorded value. Recorded: growth of genomic data volume — note: a second reading, verbatim-verified, disagreeing with the value on the record. Not applied - a person decides. | alternatives.phenomenon | the tagger's words | |
| claim_relations | builds-on → the OmicsMaps catalogue of sequencing instruments and Illumina's own growth estimate | claim_relations[0] | 1.0 | the tagger's words |
| claim_relations | extends → the Big Data literature on astronomy, computational social science and video platforms | claim_relations[1] | 1.0 | the tagger's words |
| concept_edges | acquisition —part-of→ life cycle strength of the relation: 1 also in: acquisition · life cycle | concept_edges[0] | the tagger's words | |
| concept_edges | genomics —is-a→ Big Data strength of the relation: 1 | concept_edges[1] | the tagger's words | |
| concept_edges | Big Data —operationalised-as→ life cycle strength of the relation: 1
Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the life cycle is the frame for comparing the domains; it is not an operationalisation of Big Data. also in: Big Data · life cycle | concept_edges[4] | the tagger's words | |
| concept_edges | storage —part-of→ life cycle the relation itself is separately evidenced · strength of the relation: 1 · note: Storage is explicitly one of four components comprising the dataset lifecycle. also in: storage · life cycle | concept_edges[7] | 0.95 | the tagger's words |
| concept_edges | distribution —part-of→ life cycle the relation itself is separately evidenced · strength of the relation: 1 · note: Distribution is explicitly one of four components comprising the dataset lifecycle. also in: distribution · life cycle | concept_edges[8] | 0.95 | the tagger's words |
| concept_edges | analysis —part-of→ life cycle the relation itself is separately evidenced · strength of the relation: 1 · note: Analysis is explicitly one of four components comprising the dataset lifecycle. also in: analysis · life cycle | concept_edges[9] | 0.95 | the tagger's words |
| concept_edges | effective data compression —causes→ storage requirements direction on the measure: reduces · the relation itself is separately evidenced · strength of the relation: 1 · note: The paper states storage needs can be diminished with effective compression.
Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says these needs CAN be diminished with effective compression; hedged. also in: effective data compression · storage requirements | concept_edges[10] | 0.95 | the tagger's words |
| concept_edges | cloud computing —causes→ bandwidth for distribution of genomic data direction on the measure: reduces · the relation itself is separately evidenced · strength of the relation: 1 · note: Cloud computing is stated to decrease distribution bandwidth. | concept_edges[11] | 0.95 | the tagger's words |
| concept_edges | remote machines that already have data —causes→ computing resources necessary for distribution direction on the measure: reduces · the relation itself is separately evidenced · strength of the relation: 1 · note: Running applications where data reside is stated to reduce distribution resources. | concept_edges[12] | 0.95 | the tagger's words |
| concept_edges | CPU capabilities —contrasted-with→ input/output (I/O) hardware the relation itself is separately evidenced · strength of the relation: 0.8 · note: The paper hedges that the larger future bottleneck may be I/O rather than CPU capability.
Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says the bigger bottleneck MAY not be in CPU but in I/O hardware; hedged. also in: CPU capabilities · input/output (I/O) hardware | concept_edges[13] | 0.9 | the tagger's words |
| content_tags | 7 — doubling time of sequence data produced | content_tags.quantities[0] | the tagger's words | |
| content_tags | 2,500 — high-throughput sequencers worldwide | content_tags.quantities[1] | the tagger's words | |
| content_tags | 1,000 — global spread of sequencing centers | content_tags.quantities[2] | the tagger's words | |
| content_tags | 3.6 — raw sequence data held in the SRA archive | content_tags.quantities[3] | the tagger's words | |
| content_tags | 7.5 — ASKAP telescope image-data acquisition rate | content_tags.quantities[4] | the tagger's words | |
| content_tags | 500 — current Twitter data generation | content_tags.quantities[5] | the tagger's words | |
| content_tags | doubling every seven months — observed growth rate of sequence data production, the basis of the central projection | content_tags.quantities[6] | 1.0 | the tagger's words |
| content_tags | one zettabase per year by 2025 — projected genomic acquisition under the seven-month doubling | content_tags.quantities[7] | 1.0 | the tagger's words |
| content_tags | ~25 zettabytes per year by 2025 — projected astronomy acquisition, from ASKAP scaling 7.5 to 750 terabytes/second | content_tags.quantities[8] | 1.0 | the tagger's words |
| content_tags | 1–2 exabytes of video per year by 2025 — projected YouTube acquisition, from 300 hours uploaded per minute today | content_tags.quantities[9] | 1.0 | the tagger's words |
| content_tags | 500 million tweets/day at ~3 kilobytes each — Twitter's acquisition baseline, the smallest of the four by orders of magnitude | content_tags.quantities[10] | 1.0 | the tagger's words |
| content_tags | 3.6 petabases archived; ~35 petabases/year capacity — what is in the SRA against what the world's instruments can produce - the gap the paper flags | content_tags.quantities[11] | 1.0 | the tagger's words |
| content_tags | genomics is on par with or the most demanding of the Big Data domains | content_tags.findings[0] | the tagger's words | |
| content_tags | data acquisition is expected to grow by up to two orders of magnitude about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. | content_tags.findings[1] | the tagger's words | |
| content_tags | human genomes may grow four to five orders of magnitude by 2025 | content_tags.findings[2] | the tagger's words | |
| content_tags | EVIDENCE CORRECTED (RE-GROUNDED - THE POINTER WAS WRONG, NOT THE CLAIM) · even conservative doubling (12-18 months) still yields exabase-scale genomics EVIDENCE CORRECTED: RE-GROUNDED - THE POINTER WAS WRONG, NOT THE CLAIM. The old evidence was verbatim, anchored and correctly cut, and it contained NONE of this claim's numbers (12, 18). The paper states the claim in one sentence elsewhere and the tagger did not use it. the project, 2026-08-23: 'WE HAVE SPENT THE DAY VERIFYING THAT QUOTES ARE REAL AND NONE OF IT ASKS WHETHER THEY ARE RELEVANT.' Confirmed by reading the claim, the old span and the candidate together; the candidate carries every number AND the claim's own content words. | content_tags.findings[3] | the tagger's words | |
| content_tags | Genomics is a “four-headed beast”: on par with or the most demanding of the domains analysed in data acquisition, storage, distribution and analysis. | content_tags.findings[4] | 1.0 | the tagger's words |
| content_tags | Sequence data has been doubling roughly every seven months. | content_tags.findings[5] | 1.0 | the tagger's words |
| content_tags | The four Big Data domains differ sharply in how data are acquired. | content_tags.findings[6] | 1.0 | the tagger's words |
| content_tags | Across all domains, acquisition volume is expected to rise by up to two orders of magnitude in a decade. | content_tags.findings[7] | 1.0 | the tagger's words |
| content_tags | worldwide sequencing capacity also in: worldwide sequencing capacity | content_tags.variables[0] | the tagger's words | |
| content_tags | conversion factor also in: conversion factor | content_tags.variables[2] | the tagger's words | |
| content_tags | domain also in: domain | content_tags.variables[4] | 1.0 | the tagger's words |
| content_tags | Sequence Read Archive (SRA) also in: Sequence Read Archive (SRA) | content_tags.definitions[0] | the tagger's words | |
| content_tags | The Cancer Genome Atlas (TCGA) also in: The Cancer Genome Atlas (TCGA) | content_tags.definitions[2] | the tagger's words | |
| content_tags | Exome Aggregation Consortium (ExAC) also in: Exome Aggregation Consortium (ExAC) | content_tags.definitions[3] | the tagger's words | |
| content_tags | life cycle also in: life cycle | content_tags.definitions[4] | 1.0 | the tagger's words |
| content_tags | genomics also in: genomics | content_tags.definitions[5] | 1.0 | the tagger's words |
| content_tags | conversion factor also in: conversion factor | content_tags.definitions[6] | 1.0 | the tagger's words |
| content_tags | exponential-growth also in: exponential-growth | content_tags.statistics[0] | 1.0 | the tagger's words |
| content_tags | extrapolation also in: extrapolation | content_tags.statistics[1] | 1.0 | the tagger's words |
| content_tags | order-of-magnitude also in: order-of-magnitude | content_tags.statistics[2] | 1.0 | the tagger's words |
Tags that could not be placed in the text:
content_tags.definitions[1] — WGS, whole genome…What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.