standOff. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote shown on the same row, so the borrowing can be checked against the thing borrowed from without leaving this page. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.| relation to other work | claim_relations | how this paper stands to another: agrees, contradicts, extends, qualifies |
| claim type | claim_type | the shape of the claim: existence, mechanism, method, critique, prediction, null |
| concept edge | concept_edges | an asserted relation between two concepts, with the sentence asserting it |
| concept | concepts | an idea the paper works with, quoted where the paper uses it |
| content tag | content_tags | a finding, definition, quantity, statistic or variable named in the paper |
| contribution | contribution | what the paper claims to add |
| limitation | limitations | what the paper says its evidence cannot support |
| mechanism | mechanisms | how the reported effect is said to come about |
| method | method | how the paper produced its evidence |
| object of study | object_of_study | what the paper is about — the thing it investigates |
| phenomenon | phenomenon | the effect or regularity under study |
| research question | research_question | the question the paper sets itself |
| sense | senses | a distinct use of a term, quoted where the paper fixes it |
| unrecordable relation | unhoused_relations | a relation the paper asserts that this vocabulary cannot express, with the reason |
The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.
Ten Simple Rules for Reproducible Computational Research
research_questionlimitationsconcepts↗concepts↗unhoused_relationscontent_tagscontent_tags↗Replication is the cornerstone of a cumulative science [1] . However, new tools and technologies, massive amounts of data, interdisciplinary approaches, and the complexity of the questions being asked are complicating replication efforts, as are increased pressures on scientists to advance their research [2] . As full replication of studies on independently collected data is often not feasible, there has recently been a call for reproducible research as an attainable minimum standard for assessing the value of scientific claims [3] . This requires that papers in experimental science describe the results and provide a sufficiently clear protocol to allow successful repetition and extension of analyses based on original data [4] .
concepts↗claim_relationsThe importance of replication and reproducibility has recently been exemplified through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5] , studies showing difficulties with replicating published experimental results [6] , an increase in retracted papers [7] , and through a high number of failing clinical trials [8] , [9] . This has led to discussions on how individual researchers, institutions, funding bodies, and journals can establish routines that increase transparency and reproducibility. In order to foster such aspects, it has been suggested that the scientific community needs to develop a “culture of reproducibility” for computational science, and to require it for published claims [3] .
unhoused_relationscontent_tagsWe want to emphasize that reproducibility is not only a moral responsibility with respect to the scientific field, but that a lack of reproducibility can also be a burden for you as an individual researcher. As an example, a good practice of reproducibility is necessary in order to allow previously developed methodology to be effectively applied on new data, or to allow reuse of code and results for new projects. In other words, good habits of reproducibility may actually turn out to be a time-saver in the longer run.
object_of_studymethod↗limitationsunhoused_relationsclaim_relationscontent_tagsWe further note that reproducibility is just as much about the habits that ensure reproducible research as the technologies that can make these processes efficient and realistic. Each of the following ten rules captures a specific aspect of reproducibility, and discusses what is needed in terms of information handling and tracking of procedures. If you are taking a bare-bones approach to bioinformatics analysis, i.e., running various custom scripts from the command line, you will probably need to handle each rule explicitly. If you are instead performing your analyses through an integrated framework (such as GenePattern [10] , Galaxy [11] , LONI pipeline [12] , or Taverna [13] ), the system may already provide full or partial support for most of the rules. What is needed on your part is then merely the knowledge of how to exploit these existing possibilities.
limitationsunhoused_relationsIn a pragmatic setting, with publication pressure and deadlines, one may face the need to make a trade-off between the ideals of reproducibility and the need to get the research out while it is still relevant. This trade-off becomes more important when considering that a large part of the analyses being tried out never end up yielding any results. However, frequently one will, with the wisdom of hindsight, contemplate the missed opportunity to ensure reproducibility, as it may already be too late to take the necessary notes from memory (or at least much more difficult than to do it while underway). We believe that the rewards of reproducibility will compensate for the risk of having spent valuable time developing an annotated catalog of analyses that turned out as blind alleys.
phenomenonlimitationsconcepts↗sensescontent_tagsAs a minimal requirement, you should at least be able to reproduce the results yourself. This would satisfy the most basic requirements of sound research, allowing any substantial future questioning of the research to be met with a precise explanation. Although it may sound like a very weak requirement, even this level of reproducibility will often require a certain level of care in order to be met. There will for a given analysis be an exponential number of possible combinations of software versions, parameter values, pre-processing steps, and so on, meaning that a failure to take notes may make exact reproduction essentially impossible.
sensessensesWith this basic level of reproducibility in place, there is much more that can be wished for. An obvious extension is to go from a level where you can reproduce results in case of a critical situation to a level where you can practically and routinely reuse your previous work and increase your productivity. A second extension is to ensure that peers have a practical possibility of reproducing your results, which can lead to increased trust in, interest for, and citations of your work [6] , [14] .
contributionclaim_typeconcepts↗We here present ten simple rules for reproducibility of computational research. These rules can be at your disposal for whenever you want to make your research more accessible—be it for peers or for your future self.
Rule 1: For Every Result, Keep Track of How It Was Produced
mechanisms↗concepts↗concepts↗concepts↗concepts↗concepts↗unhoused_relationssensesconcept_edgesconcept_edgescontent_tags↗content_tags↗Whenever a result may be of potential interest, keep track of how it was produced. When doing this, one will frequently find that getting from raw data to the final result involves many interrelated steps (single commands, scripts, programs). We refer to such a sequence of steps, whether it is automated or performed manually, as an analysis workflow. While the essential part of an analysis is often represented by only one of the steps, the full sequence of pre- and post-processing steps are often critical in order to reach the achieved result. For every involved step, you should ensure that every detail that may influence the execution of the step is recorded. If the step is performed by a computer program, the critical details include the name and version of the program, as well as the exact parameters and inputs that were used.
mechanisms↗concepts↗concepts↗concepts↗unhoused_relationsunhoused_relationssensessensescontent_tagscontent_tags↗Although manually noting the precise sequence of steps taken allows for an analysis to be reproduced, the documentation can easily get out of sync with how the analysis was really performed in its final version. By instead specifying the full analysis workflow in a form that allows for direct execution, one can ensure that the specification matches the analysis that was (subsequently) performed, and that the analysis can be reproduced by yourself or others in an automated way. Such executable descriptions [10] might come in the form of simple shell scripts or makefiles [15] , [16] at the command line, or in the form of stored workflows in a workflow management system [10] , [11] , [13] , [17] , [18] .
As a minimum, you should at least record sufficient details on programs, parameters, and manual procedures to allow yourself, in a year or so, to approximately reproduce the results.
Rule 2: Avoid Manual Data Manipulation Steps
mechanisms↗limitationsconcepts↗concepts↗claim_relationsconcept_edgescontent_tagsWhenever possible, rely on the execution of programs instead of manual procedures to modify data. Such manual procedures are not only inefficient and error-prone, they are also difficult to reproduce. If working at the UNIX command line, manual modification of files can usually be replaced by the use of standard UNIX commands or small custom scripts. If working with integrated frameworks, there will typically be a quite rich collection of components for data manipulation. As an example, manual tweaking of data files to attain format compatibility should be replaced by format converters that can be reenacted and included into executable workflows. Other manual operations like the use of copy and paste between documents should also be avoided. If manual operations cannot be avoided, you should as a minimum note down which data files were modified or moved, and for what purpose.
Rule 3: Archive the Exact Versions of All External Programs Used
mechanisms↗concepts↗concepts↗content_tagsIn order to exactly reproduce a given result, it may be necessary to use programs in the exact versions used originally. Also, as both input and output formats may change between versions, a newer version of a program may not even run without modifying its inputs. Even having noted which version was used of a given program, it is not always trivial to get hold of a program in anything but the current version. Archiving the exact versions of programs actually used may thus save a lot of hassle at later stages. In some cases, all that is needed is to store a single executable or source code file. In other cases, a given program may again have specific requirements to other installed programs/packages, or dependencies to specific operating system components. To ensure future availability, the only viable solution may then be to store a full virtual machine image of the operating system and program. As a minimum, you should note the exact names and versions of the main programs you use.
Rule 4: Version Control All Custom Scripts
concepts↗content_tags↗Even the slightest change to a computer program can have large intended or unintended consequences. When a continually developed piece of code (typically a small script) has been used to generate a certain result, only that exact state of the script may be able to produce that exact output, even given the same input data and parameters. As also discussed for rules 3 and 6, exact reproduction of results may in certain situations be essential. If computer code is not systematically archived along its evolution, backtracking to a code state that gave a certain result may be a hopeless task. This can cast doubt on previous results, as it may be impossible to know if they were partly the result of a bug or otherwise unfortunate behavior.
mechanisms↗concepts↗content_tags↗The standard solution to track evolution of code is to use a version control system [15] , such as Subversion, Git, or Mercurial. These systems are relatively easy to set up and use, and may be used to systematically store the state of the code throughout development at any desired time granularity.
As a minimum, you should archive copies of your scripts from time to time, so that you keep a rough record of the various states the code has taken during development.
Rule 5: Record All Intermediate Results, When Possible in Standardized Formats
mechanisms↗limitationsconcepts↗concepts↗content_tagscontent_tags↗In principle, as long as the full process used to produce a given result is tracked, all intermediate data can also be regenerated. In practice, having easily accessible intermediate results may be of great value. Quickly browsing through intermediate results can reveal discrepancies toward what is assumed, and can in this way uncover bugs or faulty interpretations that are not apparent in the final results. Secondly, it more directly reveals consequences of alternative programs and parameter choices at individual steps. Thirdly, when the full process is not readily executable, it allows parts of the process to be rerun. Fourthly, when reproducing results, it allows any experienced inconsistencies to be tracked to the steps where the problems arise. Fifth, it allows critical examination of the full process behind a result, without the need to have all executables operational. When possible, store such intermediate results in standardized formats. As a minimum, archive any intermediate result files that are produced when running an analysis (as long as the required storage space is not prohibitive).
Rule 6: For Analyses That Include Randomness, Note Underlying Random Seeds
mechanisms↗concepts↗unhoused_relationscontent_tags↗Many analyses and predictions include some element of randomness, meaning the same program will typically give slightly different results every time it is executed (even when receiving identical inputs and parameters). However, given the same initial seed, all random numbers used in an analysis will be equal, thus giving identical results every time it is run. There is a large difference between observing that a result has been reproduced exactly or only approximately. While achieving equal results is a strong indication that a procedure has been reproduced exactly, it is often hard to conclude anything when achieving only approximately equal results. For analyses that involve random numbers, this means that the random seed should be recorded. This allows results to be reproduced exactly by providing the same seed to the random number generator in future runs. As a minimum, you should note which analysis steps involve randomness, so that a certain level of discrepancy can be anticipated when reproducing the results.
Rule 7: Always Store Raw Data behind Plots
mechanisms↗concepts↗concepts↗unhoused_relationscontent_tags↗From the time a figure is first generated to it being part of a published article, it is often modified several times. In some cases, such modifications are merely visual adjustments to improve readability, or to ensure visual consistency between figures. If raw data behind figures are stored in a systematic manner, so as to allow raw data for a given figure to be easily retrieved, one can simply modify the plotting procedure, instead of having to redo the whole analysis. An additional advantage of this is that if one really wants to read fine values in a figure, one can consult the raw numbers. In cases where plotting involves more than a direct visualization of underlying numbers, it can be useful to store both the underlying data and the processed values that are directly visualized. An example of this is the plotting of histograms, where both the values before binning (original data) and the counts per bin (heights of visualized bars) could be stored. When plotting is performed using a command-based system like R, it is convenient to also store the code used to make the plot. One can then apply slight modifications to these commands, instead of having to specify the plot from scratch. As a minimum, one should note which data formed the basis of a given plot and how this data could be reconstructed.
Rule 8: Generate Hierarchical Analysis Output, Allowing Layers of Increasing Detail to Be Inspected
mechanisms↗concepts↗concepts↗sensessensescontent_tagscontent_tags↗The final results that make it to an article, be it plots or tables, often represent highly summarized data. For instance, each value along a curve may in turn represent averages from an underlying distribution. In order to validate and fully understand the main result, it is often useful to inspect the detailed values underlying the summaries. A common but impractical way of doing this is to incorporate various debug outputs in the source code of scripts and programs. When the storage context allows, it is better to simply incorporate permanent output of all underlying data when a main result is generated, using a systematic naming convention to allow the full data underlying a given summarized value to be easily found. We find hypertext (i.e., html file output) to be particularly useful for this purpose. This allows summarized results to be generated along with links that can be very conveniently followed (by simply clicking) to the full data underlying each summarized value. When working with summarized results, you should as a minimum at least once generate, inspect, and validate the detailed values underlying the summaries.
Rule 9: Connect Textual Statements to Underlying Results
sensescontent_tagsThroughout a typical research project, a range of different analyses are tried and interpretation of the results made. Although the results of analyses and their corresponding textual interpretations are clearly interconnected at the conceptual level, they tend to live quite separate lives in their representations: results usually live on a data area on a server or personal computer, while interpretations live in text documents in the form of personal notes or emails to collaborators. Such textual interpretations are not generally mere shadows of the results—they often involve viewing the results in light of other theories and results. As such, they carry extra information, while at the same time having their necessary support in a given result.
concepts↗concepts↗unhoused_relationsIf you want to reevaluate your previous interpretations, or allow peers to make their own assessment of claims you make in a scientific paper, you will have to connect a given textual statement (interpretation, claim, conclusion) to the precise results underlying the statement. Making this connection when it is needed may be difficult and error-prone, as it may be hard to locate the exact result underlying and supporting the statement from a large pool of different analyses with various versions.
mechanisms↗To allow efficient retrieval of details behind textual statements, we suggest that statements are connected to underlying results already from the time the statements are initially formulated (for instance in notes or emails). Such a connection can for instance be a simple file path to detailed results, or the ID of a result in an analysis framework, included within the text itself. For an even tighter integration, there are tools available to help integrate reproducible analyses directly into textual documents, such as Sweave [19] , the GenePattern Word add-in [4] , and Galaxy Pages [20] . These solutions can also subsequently be used in connection with publications, as discussed in the next rule.
As a minimum, you should provide enough details along with your textual interpretations so as to allow the exact underlying results, or at least some related results, to be tracked down in the future.
Rule 10: Provide Public Access to Scripts, Runs, and Results
mechanisms↗concepts↗concepts↗unhoused_relationscontent_tagsLast, but not least, all input data, scripts, versions, parameters, and intermediate results should be made publicly and easily accessible. Various solutions have now become available to make data sharing more convenient, standardized, and accessible in particular domains, such as for gene expression data [21] – [23] . Most journals allow articles to be supplemented with online material, and some journals have initiated further efforts for making data and code more integrated with publications [3] , [24] . As a minimum, you should submit the main data and source code as supplementary material, and be prepared to respond to any requests for further data or methodology details by peers.
concepts↗concepts↗unhoused_relationscontent_tagsMaking reproducibility of your work by peers a realistic possibility sends a strong signal of quality, trustworthiness, and transparency. This could increase the quality and speed of the reviewing process on your work, the chances of your work getting published, and the chances of your work being taken further and cited by other researchers after publication [25] .
Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; 3 of 3 relations here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.
This record holds no controls. A layer is empty when the tagger recorded nothing in it — which is not the same as the paper having nothing to record.
| facet | tag | field | conf. | whose words |
|---|---|---|---|---|
| object_of_study | practical information-handling and provenance practices for reproducible computational research | object_of_study | the paper's words | |
| research_question | what minimum habits and technical practices let researchers and peers reproduce computational analyses and trace claims to results | research_question | the paper's words | |
| contribution | presents ten implementable rules covering workflow provenance, automation, environment capture, versioning, intermediates, randomness, plots, layered outputs, claim links, and public access | contribution | the paper's words | |
| claim_type | method | claim_type | the paper's words | |
| phenomenon | failure or success in exactly reproducing computational research results | phenomenon | the paper's words | |
| method | practice-report also in: practice-report | method | the paper's words | |
| mechanisms | Recording the full sequence, software, versions, parameters, and inputs preserves how a result was produced. also in: provenance capture | mechanisms[0] | 0.95 | the tagger's words |
| mechanisms | Directly executable specifications keep documentation aligned with the analysis actually run. also in: workflow execution | mechanisms[1] | 0.95 | the tagger's words |
| mechanisms | Commands, scripts, or framework components replace manual edits and make transformations reenactable. also in: automation of data manipulation | mechanisms[2] | 0.95 | the tagger's words |
| mechanisms | Preserving exact program versions or a full virtual machine protects against software and dependency change. also in: environment archiving | mechanisms[3] | 0.95 | the tagger's words |
| mechanisms | Systematic code-state history permits recovery of the exact script that produced a result. also in: version control | mechanisms[4] | 0.95 | the tagger's words |
| mechanisms | Stored intermediate outputs expose discrepancies and localize the step at which reproduction diverges. also in: intermediate checkpointing | mechanisms[5] | 0.95 | the tagger's words |
| mechanisms | Reusing the same initial random seed yields the same random numbers and exact stochastic results. also in: seed capture | mechanisms[6] | 0.95 | the tagger's words |
| mechanisms | Stored source and processed values let a figure be modified or interrogated without rerunning the entire analysis. also in: plot-data retention | mechanisms[7] | 0.95 | the tagger's words |
| mechanisms | Links connect summary values to permanent detailed output for inspection and validation. also in: hierarchical hypertext output | mechanisms[8] | 0.9 | the tagger's words |
| mechanisms | A path or result identifier embedded with a statement preserves access to its exact support. also in: claim-result linking | mechanisms[9] | 0.95 | the tagger's words |
| mechanisms | Sharing data, code, versions, parameters, and intermediates makes peer reproduction practically possible. also in: public research packaging | mechanisms[10] | 0.95 | the tagger's words |
| limitations | Full replication on independently collected data is often infeasible, motivating the weaker reproducibility standard. | limitations[0] | 0.95 | the tagger's words |
| limitations | Publication pressure creates a tradeoff between comprehensive provenance work and releasing results while relevant. | limitations[1] | 0.95 | the tagger's words |
| limitations | Even self-reproduction requires care because software versions, parameters, and preprocessing create exponentially many configurations. | limitations[2] | 0.95 | the tagger's words |
| limitations | A bare command-line workflow may require every rule to be handled explicitly, while integrated frameworks provide only partial support. | limitations[3] | 0.9 | the tagger's words |
| limitations | Storage of every intermediate result may be limited by available space. | limitations[4] | 0.95 | the tagger's words |
| limitations | If manual operations cannot be avoided, the approach falls back to documenting modified or moved files and purposes. | limitations[5] | 0.95 | the tagger's words |
| concepts | replication also in: replication | concepts[0] | 0.95 | the paper's words |
| concepts | reproducible research also in: reproducible research | concepts[1] | 0.95 | the paper's words |
| concepts | culture of reproducibility also in: culture of reproducibility | concepts[2] | 0.95 | the paper's words |
| concepts | computational research also in: computational research | concepts[3] | 0.95 | the paper's words |
| concepts | analysis workflow also in: analysis workflow | concepts[4] | 0.95 | the paper's words |
| concepts | sequence of steps also in: sequence of steps | concepts[5] | 0.95 | the paper's words |
| concepts | critical details also in: critical details | concepts[6] | 0.95 | the paper's words |
| concepts | name and version of the program also in: name and version of the program | concepts[7] | 0.95 | the paper's words |
| concepts | exact parameters and inputs also in: exact parameters and inputs | concepts[8] | 0.95 | the paper's words |
| concepts | executable descriptions also in: executable descriptions | concepts[9] | 0.95 | the paper's words |
| concepts | shell scripts also in: shell scripts | concepts[10] | 0.95 | the paper's words |
| concepts | makefiles also in: makefiles | concepts[11] | 0.95 | the paper's words |
| concepts | manual procedures also in: manual procedures | concepts[12] | 0.95 | the paper's words |
| concepts | execution of programs also in: execution of programs | concepts[13] | 0.95 | the paper's words |
| concepts | exact versions of programs also in: exact versions of programs | concepts[14] | 0.95 | the paper's words |
| concepts | virtual machine image also in: virtual machine image | concepts[15] | 0.9 | the paper's words |
| concepts | version control system also in: version control system | concepts[16] | 0.95 | the paper's words |
| concepts | code state also in: code state | concepts[17] | 0.95 | the paper's words |
| concepts | intermediate results also in: intermediate results | concepts[18] | 0.95 | the paper's words |
| concepts | standardized formats also in: standardized formats | concepts[19] | 0.95 | the paper's words |
| concepts | random seed also in: random seed | concepts[20] | 0.95 | the paper's words |
| concepts | raw data also in: raw data | concepts[21] | 0.95 | the paper's words |
| concepts | processed values also in: processed values | concepts[22] | 0.95 | the paper's words |
| concepts | hierarchical analysis output about this quote: THE TERM IS NOT IN THE QUOTE - the term is the rule the quote sits under ('Rule 8: Generate Hierarchical Analysis Output, Allowing Layers of Increasing Detail to Be Inspected'). also in: hierarchical analysis output | concepts[23] | 0.95 | the paper's words |
| concepts | summarized results also in: summarized results | concepts[24] | 0.95 | the paper's words |
| concepts | textual statements also in: textual statements | concepts[25] | 0.95 | the paper's words |
| concepts | underlying results also in: underlying results | concepts[26] | 0.95 | the paper's words |
| concepts | public access also in: public access | concepts[27] | 0.95 | the paper's words |
| concepts | input data also in: input data | concepts[28] | 0.95 | the paper's words |
| concepts | transparency also in: transparency | concepts[29] | 0.9 | the paper's words |
| concepts | trustworthiness also in: trustworthiness | concepts[30] | 0.9 | the paper's words |
| concepts | reproduce the results yourself also in: reproduce the results yourself | concepts[31] | 0.9 | the paper's words |
| unhoused_relations | replication → cumulative science Replication is described as the cornerstone of cumulative science. — the relation stated: Replication is the cornerstone of cumulative science. | unhoused_relations[0] | 0.95 | the tagger's words |
| unhoused_relations | sequence of steps → analysis workflow The authors refer to an automated or manual sequence of steps as an analysis workflow. — the relation stated: A sequence of interrelated analysis steps is called an analysis workflow. | unhoused_relations[1] | 0.95 | the tagger's words |
| unhoused_relations | lack of reproducibility → individual researcher burden Poor reproducibility creates extra work for the original researcher. — the relation stated: Lack of reproducibility can burden the individual researcher. | unhoused_relations[2] | 0.95 | the tagger's words |
| unhoused_relations | random seed → exact stochastic result The same seed permits exact reproduction of analyses involving randomness. — the relation stated: Recording and reusing a seed permits identical random sequences. | unhoused_relations[3] | 0.95 | the tagger's words |
| unhoused_relations | raw data → plot Raw and processed values are retained behind a plot for retrieval and reconstruction. — the relation stated: Stored raw data supports modification and fine-value inspection of figures. | unhoused_relations[4] | 0.95 | the tagger's words |
| unhoused_relations | textual statements → underlying results Interpretive statements should be connected to the precise results supporting them. — the relation stated: Claims and interpretations require a retrievable connection to their supporting results. | unhoused_relations[5] | 0.95 | the tagger's words |
| unhoused_relations | peer reproducibility → trustworthiness Making peer reproduction realistic sends a signal of trustworthiness. — the relation stated: Peer reproducibility signals quality, trustworthiness, and transparency. | unhoused_relations[6] | 0.9 | the tagger's words |
| unhoused_relations | reproducibility practices → publication timing Researchers face a tradeoff between ideal provenance and publishing while work remains relevant. — the relation stated: Reproducibility ideals can compete with deadlines and relevance. | unhoused_relations[7] | 0.95 | the tagger's words |
| unhoused_relations | integrated framework → ten reproducibility rules Integrated analysis systems may already support some or most of the rules. — the relation stated: Integrated frameworks may provide partial or full procedural support. | unhoused_relations[8] | 0.9 | the tagger's words |
| unhoused_relations | public access → peer reproducibility Sharing scripts, data, configurations, and results makes peer reproduction practical. — the relation stated: Publicly accessible research materials permit peers to attempt reproduction. | unhoused_relations[9] | 0.95 | the tagger's words |
| unhoused_relations | shell scripts → executable descriptions Shell scripts are one possible form of directly executable workflow description. — the relation stated: Executable descriptions might take the form of simple shell scripts. | unhoused_relations[10] | 0.95 | the tagger's words |
| unhoused_relations | makefiles → executable descriptions Makefiles are one possible form of directly executable workflow description. — the relation stated: Executable descriptions might take the form of makefiles. | unhoused_relations[11] | 0.95 | the tagger's words |
| senses | reproducibility - minimum standard: the original researcher can repeat an analysis on original data | senses[0] | 0.95 | the tagger's words |
| senses | reproducibility - peer standard: others have a practical possibility of reproducing results | senses[1] | 0.95 | the tagger's words |
| senses | reproducibility - reuse standard: prior work can be routinely reused for productivity | senses[2] | 0.9 | the tagger's words |
| senses | workflow - documented: a manually recorded sequence that may drift from performed analysis | senses[3] | 0.95 | the tagger's words |
| senses | workflow - executable: a directly runnable specification synchronized with the analysis | senses[4] | 0.95 | the tagger's words |
| senses | result provenance - computational: programs, versions, parameters, inputs, and code states | senses[5] | 0.95 | the tagger's words |
| senses | result provenance - interpretive: links between textual statements and supporting outputs | senses[6] | 0.95 | the tagger's words |
| senses | analysis output - summary layer: plots and tables used in an article | senses[7] | 0.95 | the tagger's words |
| senses | analysis output - detail layer: values underlying summarized results | senses[8] | 0.95 | the tagger's words |
| claim_relations | supports → culture of reproducibility in computational science note: The rules operationalize the community-level call for transparent, reproducible published claims. | claim_relations[0] | 0.95 | the tagger's words |
| claim_relations | critiques → manual data manipulation in computational workflows note: Manual procedures are presented as inefficient, error-prone, and hard to reproduce. | claim_relations[1] | 0.95 | the tagger's words |
| claim_relations | builds-on → workflow management and integrated analysis frameworks note: Existing frameworks are described as already supporting many of the rules. | claim_relations[2] | 0.9 | the tagger's words |
| concept_edges | name and version of the program —part-of→ critical details the relation itself is separately evidenced · strength of the relation: 1 · note: Categorical constituent relation. · the relation stated: Critical execution details explicitly include the program name and version. also in: name and version of the program · critical details | concept_edges[0] | 0.95 | the tagger's words |
| concept_edges | exact parameters and inputs —part-of→ critical details the relation itself is separately evidenced · strength of the relation: 1 · note: Categorical constituent relation. · the relation stated: Critical execution details explicitly include exact parameters and inputs. also in: exact parameters and inputs · critical details | concept_edges[1] | 0.95 | the tagger's words |
| concept_edges | manual procedures —contrasted-with→ execution of programs the relation itself is separately evidenced · strength of the relation: 0.95 · note: Flat procedural contrast. · the relation stated: Program execution is explicitly recommended instead of manual data manipulation procedures. also in: manual procedures · execution of programs | concept_edges[2] | 0.95 | the tagger's words |
| content_tags | Reproducibility is proposed as an attainable minimum standard when independent full replication is infeasible. | content_tags.findings[0] | 0.95 | the tagger's words |
| content_tags | Reproducibility depends on researcher habits as well as enabling technologies. | content_tags.findings[1] | 0.95 | the tagger's words |
| content_tags | The authors argue that reproducibility can save time and enable reuse of methods, code, and results. | content_tags.findings[2] | 0.9 | the tagger's words |
| content_tags | At minimum, a researcher should be able to reproduce their own result and explain it precisely. | content_tags.findings[3] | 0.95 | the tagger's words |
| content_tags | Executable workflow descriptions remain synchronized with performed analyses better than manually maintained documentation. | content_tags.findings[4] | 0.95 | the tagger's words |
| content_tags | Manual data manipulation is characterized as inefficient, error-prone, and difficult to reproduce. | content_tags.findings[5] | 0.95 | the tagger's words |
| content_tags | Exact program versions, code states, parameters, inputs, and random seeds can be necessary for exact reproduction. | content_tags.findings[6] | 0.95 | the tagger's words |
| content_tags | Intermediate results support debugging, parameter comparison, partial reruns, inconsistency localization, and process inspection. | content_tags.findings[7] | 0.95 | the tagger's words |
| content_tags | Hierarchical output lets readers move from summarized results to the detailed values beneath them. | content_tags.findings[8] | 0.95 | the tagger's words |
| content_tags | Textual scientific claims carry information beyond numerical results but require traceable links to supporting outputs. | content_tags.findings[9] | 0.95 | the tagger's words |
| content_tags | The recommended public research package includes input data, scripts, versions, parameters, and intermediate results. | content_tags.findings[10] | 0.95 | the tagger's words |
| content_tags | Peer reproducibility is argued to signal quality, trustworthiness, and transparency. | content_tags.findings[11] | 0.9 | the tagger's words |
| content_tags | a minimum standard in which results and protocols permit repetition and extension of analyses on the original data also in: reproducible research | content_tags.definitions[0] | 0.95 | the tagger's words |
| content_tags | an automated or manual sequence of interrelated commands, scripts, or programs from raw data to a final result also in: analysis workflow | content_tags.definitions[1] | 0.95 | the tagger's words |
| content_tags | a directly runnable workflow specification such as a shell script, makefile, or stored workflow also in: executable description | content_tags.definitions[2] | 0.95 | the tagger's words |
| content_tags | a system such as Subversion, Git, or Mercurial that stores code states throughout development also in: version control system | content_tags.definitions[3] | 0.95 | the tagger's words |
| content_tags | layered output connecting summarized results with inspectable underlying detail also in: hierarchical analysis output | content_tags.definitions[4] | 0.9 | the tagger's words |
| content_tags | parameters and inputs also in: parameters and inputs | content_tags.variables[1] | 0.95 | the tagger's words |
| content_tags | code state also in: code state | content_tags.variables[2] | 0.95 | the tagger's words |
| content_tags | random seed also in: random seed | content_tags.variables[3] | 0.95 | the tagger's words |
| content_tags | raw data also in: raw data | content_tags.variables[4] | 0.95 | the tagger's words |
| content_tags | intermediate result also in: intermediate result | content_tags.variables[5] | 0.95 | the tagger's words |
Tags that could not be placed in the text:
What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.