standOff. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote shown on the same row, so the borrowing can be checked against the thing borrowed from without leaving this page. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.| relation to other work | claim_relations | how this paper stands to another: agrees, contradicts, extends, qualifies |
| claim type | claim_type | the shape of the claim: existence, mechanism, method, critique, prediction, null |
| concept edge | concept_edges | an asserted relation between two concepts, with the sentence asserting it |
| concept | concepts | an idea the paper works with, quoted where the paper uses it |
| contribution | contribution | what the paper claims to add |
| limitation | limitations | what the paper says its evidence cannot support |
| mechanism | mechanisms | how the reported effect is said to come about |
| method | method | how the paper produced its evidence |
| object of study | object_of_study | what the paper is about — the thing it investigates |
| phenomenon | phenomenon | the effect or regularity under study |
| research question | research_question | the question the paper sets itself |
| sense | senses | a distinct use of a term, quoted where the paper fixes it |
| unrecordable relation | unhoused_relations | a relation the paper asserts that this vocabulary cannot express, with the reason |
Best Practices for Scientific Computing
Abstract
object_of_studyclaim_typeconcepts↗concepts↗concepts↗claim_relationsWe describe a set of best practices for scientific software development, based on research and experience, that will improve scientists' productivity and the reliability of their software.
Introduction
Scientists spend an increasing amount of time building and using software. However, most scientists are never taught how to do this efficiently. As a result, many are unaware of tools and practices that would allow them to write more reliable and maintainable code with less effort. We describe a set of best practices for scientific software development that have solid foundations in research and experience, and that improve scientists' productivity and the reliability of their software.
concepts↗Software is as important to modern scientific research as telescopes and test tubes. From groups that work exclusively on computational problems, to traditional laboratory and field scientists, more and more of the daily operation of science revolves around developing new algorithms, managing and analyzing the large amounts of data that are generated in single research projects, combining disparate datasets to assess synthetic problems, and other computational tasks.
limitationsconcepts↗concepts↗unhoused_relationsScientists typically develop their own software for these purposes because doing so requires substantial domain-specific knowledge. As a result, recent studies have found that scientists typically spend 30% or more of their time developing software [1] , [2] . However, 90% or more of them are primarily self-taught [1] , [2] , and therefore lack exposure to basic software development practices such as writing maintainable code, using version control and issue trackers, code reviews, unit testing, and task automation.
phenomenonconcepts↗concepts↗concepts↗concepts↗concepts↗unhoused_relationsunhoused_relationssensesclaim_relationsWe believe that software is just another kind of experimental apparatus [3] and should be built, checked, and used as carefully as any physical apparatus. However, while most scientists are careful to validate their laboratory and field equipment, most do not know how reliable their software is [4] , [5] . This can lead to serious errors impacting the central conclusions of published research [6] : recent high-profile retractions, technical comments, and corrections because of errors in computational methods include papers in Science [7] , [8] , PNAS [9] , the Journal of Molecular Biology [10] , Ecology Letters [11] , [12] , the Journal of Mammalogy [13] , Journal of the American College of Cardiology [14] , Hypertension [15] , and The American Economic Review [16] .
mechanisms↗concepts↗concepts↗concept_edgesIn addition, because software is often used for more than a single project, and is often reused by other scientists, computing errors can have disproportionate impacts on the scientific process. This type of cascading impact caused several prominent retractions when an error from another group's code was not discovered until after publication [6] . As with bench experiments, not everything must be done to the most exacting standards; however, scientists need to be aware of best practices both to improve their own approaches and for reviewing computational work by others.
research_questionmethod↗limitationsconcepts↗concepts↗unhoused_relationsThis paper describes a set of practices that are easy to adopt and have proven effective in many research settings. Our recommendations are based on several decades of collective experience both building scientific software and teaching computing to scientists [17] , [18] , reports from many other groups [19] –, guidelines for commercial and open source software development [26] ,, and on empirical studies of scientific computing [28] – [31] and software development in general (summarized in [32] ). None of these practices will guarantee efficient, error-free software development, but used in concert they will reduce the number of errors in scientific software, make it easier to reuse, and save the authors of the software time and effort that can used for focusing on the underlying scientific questions.
limitationsconcepts↗concepts↗unhoused_relationsOur practices are summarized in Box 1 ; labels in the main text such as “(1a)” refer to items in that summary. For reasons of space, we do not discuss the equally important (but independent) issues of reproducible research, publication and citation of code and data, and open science. We do believe, however, that all of these will be much easier to implement if scientists have the skills we describe.
Box 1. Summary of Best Practices
Write programs for people, not computers. A program should not require its readers to hold more than a handful of facts in memory at once. Make names consistent, distinctive, and meaningful. Make code style and formatting consistent.
Let the computer do the work. Make the computer repeat tasks. Save recent commands in a file for re-use. Use a build tool to automate workflows.
Make incremental changes. Work in small steps with frequent feedback and course correction. Use a version control system. Put everything that has been created manually in version control.
Don't repeat yourself (or others). Every piece of data must have a single authoritative representation in the system. Modularize code rather than copying and pasting. Re-use code instead of rewriting it.
Plan for mistakes. Add assertions to programs to check their operation. Use an off-the-shelf unit testing library. Turn bugs into test cases. Use a symbolic debugger.
Optimize software only after it works correctly. Use a profiler to identify bottlenecks. Write code in the highest-level language possible.
Document design and purpose, not mechanics. Document interfaces and reasons, not implementations. Refactor code in preference to explaining how it works. Embed the documentation for a piece of software in that software.
Collaborate. Use pre-merge code reviews. Use pair programming when bringing someone new up to speed and when tackling particularly tricky problems. Use an issue tracking tool.
Write Programs for People, Not Computers
concepts↗claim_relationsScientists writing software need to write code that both executes correctly and can be easily read and understood by other programmers (especially the author's future self). If software cannot be easily read and understood, it is much more difficult to know that it is actually doing what it is intended to do. To be productive, software developers must therefore take several aspects of human cognition into account: in particular, that human working memory is limited, human pattern matching abilities are finely tuned, and human attention span is short [33] – [37] .
mechanisms↗concepts↗concepts↗unhoused_relationsFirst, a program should not require its readers to hold more than a handful of facts in memory at once (1a) . Human working memory can hold only a handful of items at a time, where each item is either a single fact or a “chunk” aggregating several facts [33] , [34] , so programs should limit the total number of items to be remembered to accomplish a task. The primary way to accomplish this is to break programs up into easily understood functions, each of which conducts a single, easily understood, task. This serves to make each piece of the program easier to understand in the same way that breaking up a scientific paper using sections and paragraphs makes it easier to read.
concepts↗Second, scientists should make names consistent, distinctive, and meaningful (1b) . For example, using non-descriptive names, like a and foo, or names that are very similar, like results and results2, is likely to cause confusion.
concepts↗concepts↗unhoused_relationsThird, scientists should make code style and formatting consistent (1c) . If different parts of a scientific paper used different formatting and capitalization, it would make that paper more difficult to read. Likewise, if different parts of a program are indented differently, or if programmers mix CamelCaseNaming and pothole_case_naming, code takes longer to read and readers make more mistakes [35] , [36] .
Let the Computer Do the Work
concepts↗unhoused_relationsScience often involves repetition of computational tasks such as processing large numbers of data files in the same way or regenerating figures each time new data are added to an existing analysis. Computers were invented to do these kinds of repetitive tasks but, even today, many scientists type the same commands in over and over again or click the same buttons repeatedly [17] . In addition to wasting time, sooner or later even the most careful researcher will lose focus while doing this and make mistakes.
concepts↗Scientists should therefore make the computer repeat tasks (2a) and save recent commands in a file for re-use (2b) . For example, most command-line tools have a “history” option that lets users display and re-execute recent commands, with minor edits to filenames or parameters. This is often cited as one reason command-line interfaces remain popular [38] , [39] : “do this again” saves time and reduces errors.
concepts↗concepts↗concepts↗unhoused_relationssensesA file containing commands for an interactive system is often called a script , though there is real no difference between this and a program. When these scripts are repeatedly used in the same way, or in combination, a workflow management tool can be used. The paradigmatic example is compiling and linking programs in languages such as Fortran, C++, Java, and C# [40] . The most widely used tool for this task is probably Make ( http://www.gnu.org/software/make ), although many alternatives are now available [41] . All of these allow people to express dependencies between files, i.e., to say that if A or B has changed, then C needs to be updated using a specific set of commands. These tools have been successfully adopted for scientific workflows as well [42] .
mechanisms↗concepts↗unhoused_relationsclaim_relationsTo avoid errors and inefficiencies from repeating commands manually, we recommend that scientists use a build tool to automate workflows (2c) , e.g., specify the ways in which intermediate data files and final results depend on each other, and on the programs that create them, so that a single command will regenerate anything that needs to be regenerated.
mechanisms↗concepts↗concepts↗concepts↗concepts↗unhoused_relationssensesIn order to maximize reproducibility, everything needed to re-create the output should be recorded automatically in a format that other programs can read. (Borrowing a term from archaeology and forensics, this is often called the provenance of data.) There have been some initiatives to automate the collection of this information, and standardize its format [43] , but it is already possible to record the following without additional tools:
unique identifiers and version numbers for raw data records (which scientists may need to create themselves);
unique identifiers and version numbers for programs and libraries;
the values of parameters used to generate any given output; and
the names and version numbers of programs (however small) used to generate those outputs.
Make Incremental Changes
limitationsconcepts↗concepts↗concept_edgesUnlike traditional commercial software developers, but very much like developers in open source projects or startups, scientific programmers usually don't get their requirements from customers, and their requirements are rarely frozen [31] , [44] . In fact, scientists often can't know what their programs should do next until the current version has produced some results. This challenges design approaches that rely on specifying requirements in advance.
mechanisms↗concepts↗concepts↗concepts↗unhoused_relationsclaim_relationsProgrammers are most productive when they work in small steps with frequent feedback and course correction (3a) rather than trying to plan months or years of work in advance. While the details vary from team to team, these developers typically work in steps that are sized to be about an hour long, and these steps are often grouped in iterations that last roughly one week. This accommodates the cognitive constraints discussed in the first section, and acknowledges the reality that real-world requirements are constantly changing. The goal is to produce working (if incomplete) code after each iteration. While these practices have been around for decades, they gained prominence starting in the late 1990s under the banner of agile development [45] , [46] .
concepts↗unhoused_relationsTwo of the biggest challenges scientists and other programmers face when working with code and data are keeping track of changes (and being able to revert them if things go wrong), and collaborating on a program or dataset [23] . Typical solutions are to email software to colleagues or to copy successive versions of it to a shared folder, e.g., Dropbox ( http://www.dropbox.com ). However, both approaches are fragile and can lead to confusion and lost work when important changes are overwritten or out-of-date files are used. It's also difficult to find out which changes are in which versions or to say exactly how particular results were computed at a later date.
concepts↗concepts↗concepts↗concepts↗concepts↗sensesThe standard solution in both industry and open source is to use a version control system (3b) (VCS) [27] , [47] . A VCS stores snapshots of a project's files in a repository (or a set of repositories). Programmers can modify their working copy of the project at will, then commit changes to the repository when they are satisfied with the results to share them with colleagues.
mechanisms↗concepts↗concepts↗concepts↗unhoused_relationsCrucially, if several people have edited files simultaneously, the VCS highlights the differences and requires them to resolve any conflicts before accepting the changes. The VCS also stores the entire history of those files, allowing arbitrary versions to be retrieved and compared, together with metadata such as comments on what was changed and the author of the changes. All of this information can be extracted to provide provenance for both code and data.
Many good VCSes are open source and freely available, including Git ( http://git-scm.com ), Subversion ( http://subversion.apache.org ), and Mercurial ( http://mercurial.selenic.com ). Many free hosting services are available as well, with GitHub ( https://github.com ), BitBucket ( https://bitbucket.org ), SourceForge ( http://sourceforge.net ), and Google Code ( http://code.google.com ) being the most popular. As with coding style, the best one to use is almost always whatever your colleagues are already using [27] .
limitationsconcepts↗unhoused_relationsReproducibility is maximized when scientists put everything that has been created manually in version control (3c) , including programs, original field observations, and the source files for papers. Automated output and intermediate files can be regenerated as needed. Binary files (e.g., images and audio clips) may be stored in version control, but it is often more sensible to use an archiving system for them, and store the metadata describing their contents in version control instead [48] .
Don't Repeat Yourself (or Others)
concepts↗concepts↗unhoused_relationsAnything that is repeated in two or more places is more difficult to maintain. Every time a change or correction is made, multiple locations must be updated, which increases the chance of errors and inconsistencies. To avoid this, programmers follow the DRY Principle [49] , for “don't repeat yourself,” which applies to both data and code.
concepts↗sensesFor data, this maxim holds that every piece of data must have a single authoritative representation in the system (4a) . Physical constants ought to be defined exactly once to ensure that the entire program is using the same value; raw data files should have a single canonical version, every geographic location from which data has been collected should be given an ID that can be used to look up its latitude and longitude, and so on.
mechanisms↗concepts↗concepts↗unhoused_relationsclaim_relationsThe DRY Principle applies to code at two scales. At small scales, modularize code rather than copying and pasting (4b) . Avoiding “code clones” has been shown to reduce error rates [50] : when a change is made or a bug is fixed, that change or fix takes effect everywhere, and people's mental model of the program (i.e., their belief that “this one's been fixed”) remains accurate. As a side effect, modularizing code allows people to remember its functionality as a single mental chunk, which in turn makes code easier to understand. Modularized code can also be more easily repurposed for other projects.
concepts↗concepts↗unhoused_relationsAt larger scales, it is vital that scientific programmers re-use code instead of rewriting it (4c) . Tens of millions of lines of high-quality open source software are freely available on the web, and at least as much is available commercially. It is typically better to find an established library or package that solves a problem than to attempt to write one's own routines for well established problems (e.g., numerical integration, matrix inversions, etc.).
Plan for Mistakes
limitationsconcepts↗unhoused_relationsMistakes are inevitable, so verifying and maintaining the validity of code over time is immensely challenging [51] . While no single practice has been shown to catch or prevent all mistakes, several are very effective when used in combination [47] , [52] , [53] .
concepts↗concepts↗concepts↗unhoused_relationssensesThe first line of defense is defensive programming . Experienced programmers add assertions to programs to check their operation (5a) because experience has taught them that everyone (including their future self) makes mistakes. An assertion is simply a statement that something holds true at a particular point in a program; as the example below shows, assertions can be used to ensure that inputs are valid, outputs are consistent, and so on.
mechanisms↗concepts↗concepts↗unhoused_relationsAssertions can make up a sizeable fraction of the code in well-written applications, just as tools for calibrating scientific instruments can make up a sizeable fraction of the equipment in a lab. These assertions serve two purposes. First, they ensure that if something does go wrong, the program will halt immediately, which simplifies debugging. Second, assertions are executable documentation , i.e., they explain the program as well as checking its behavior. This makes them more useful in many cases than comments since the reader can be sure that they are accurate and up to date.
mechanisms↗concepts↗concepts↗concepts↗concepts↗concepts↗sensesclaim_relationsThe second layer of defense is automated testing . Automated tests can check to make sure that a single unit of code is returning correct results ( unit tests ), that pieces of code work correctly when combined ( integration tests ), and that the behavior of a program doesn't change when the details are modified ( regression tests ). These tests are conducted by the computer, so that they are easy to rerun every time the program is modified. Creating and managing tests is easier if programmers use an off-the-shelf unit testing library (5b) to initialize inputs, run tests, and report their results in a uniform way. These libraries are available for all major programming languages including those commonly used in scientific computing [54] – [56] .
mechanisms↗limitationsconcepts↗concepts↗concepts↗concepts↗unhoused_relationsTests check to see whether the code matches the researcher's expectations of its behavior, which depends on the researcher's understanding of the problem at hand [57] – [59] . For example, in scientific computing, tests are often conducted by comparing output to simplified cases, experimental data, or the results of earlier programs that are trusted. Another approach for generating tests is to turn bugs into test cases (5c) by writing tests that trigger a bug that has been found in the code and (once fixed) will prevent the bug from reappearing unnoticed. In combination these kinds of testing can improve our confidence that scientific code is operating properly and that the results it produces are valid. An additional benefit of testing is that it encourages programmers to design and build code that is testable (i.e., self-contained functions and classes that can run more or less independently of one another). Code that is designed this way is also easier to understand and more reusable.
limitationsconcepts↗concepts↗unhoused_relationssensesNo matter how good one's computational practice is, reasonably complex code will always initially contain bugs. Fixing bugs that have been identified is often easier if you use a symbolic debugger (5d) to track them down. A better name for this kind of tool would be “interactive program inspector” since a debugger allows users to pause a program at any line (or when some condition is true), inspect the values of variables, and walk up and down active function calls to figure out why things are behaving the way they are. Debuggers are usually more productive than adding and removing print statements or scrolling through hundreds of lines of log output [60] , because they allow the user to see exactly how the code is executing rather than just snapshots of state of the program at a few moments in time. In other words, the debugger allows the scientist to witness what is going wrong directly, rather than having to anticipate the error or infer the problem using indirect evidence.
Optimize Software Only after It Works Correctly
mechanisms↗concepts↗concepts↗unhoused_relationsclaim_relationsToday's computers and software are so complex that even experts find it hard to predict which parts of any particular program will be performance bottlenecks [61] . The most productive way to make code fast is therefore to make it work correctly, determine whether it's actually worth speeding it up, and—in those cases where it is—to use a profiler to identify bottlenecks (6a) .
concepts↗concepts↗unhoused_relationsThis strategy also has interesting implications for choice of programming language. Research has confirmed that most programmers write roughly the same number of lines of code per unit time regardless of the language they use [62] . Since faster, lower level, languages require more lines of code to accomplish the same task, scientists are most productive when they write code in the highest-level language possible (6b) , and shift to low-level languages like C and Fortran only when they are sure the performance boost is needed. (Using higher-level languages also helps program comprehensibility, since such languages have, in a sense, “pre-chunked” the facts that programmers need to have in short-term memory.)
mechanisms↗concepts↗concepts↗unhoused_relationsTaking this approach allows more code to be written (and tested) in the same amount of time. Even when it is known before coding begins that a low-level language will ultimately be necessary, rapid prototyping in a high-level language helps programmers make and evaluate design decisions quickly. Programmers can also use a high-level prototype as a test oracle for a high-performance low-level reimplementation, i.e., compare the output of the optimized (and usually more complex) program against the output from its unoptimized (but usually simpler) predecessor in order to check its correctness.
Document Design and Purpose, Not Mechanics
concepts↗concepts↗concepts↗concepts↗unhoused_relationsunhoused_relationsclaim_relationsIn the same way that a well documented experimental protocol makes research methods easier to reproduce, good documentation helps people understand code. This makes the code more reusable and lowers maintenance costs [47] . As a result, code that is well documented makes it easier to transition when the graduate students and postdocs who have been writing code in a lab transition to the next career phase. Reference documentation and descriptions of design decisions are key for improving the understandability of code. However, inline documentation that recapitulates code is not useful. Therefore we recommend that scientific programmers document interfaces and reasons, not implementations (7a) . For example, a clear description like this at the beginning of a function that describes what it does and its inputs and outputs is useful:
In contrast, the comment in the code fragment below does nothing to aid comprehension:
limitationsconcepts↗concepts↗unhoused_relationsIf a substantial description of the implementation of a piece of software is needed, it is better to refactor code in preference to explaining how it works (7b) , i.e., rather than write a paragraph to explain a complex piece of code, reorganize the code itself so that it doesn't need such an explanation. This may not always be possible—some pieces of code are intrinsically difficult—but the onus should always be on the author to convince his or her peers of that.
concepts↗unhoused_relationsThe best way to create and maintain reference documentation is to embed the documentation for a piece of software in that software (7c) . Doing this increases the probability that when programmers change the code, they will update the documentation at the same time.
mechanisms↗concepts↗concepts↗Embedded documentation usually takes the form of specially-formatted and placed comments. Typically, a documentation generator such as Javadoc, Doxygen, or Sphinx extracts these comments and generates well-formatted web pages and other human-friendly documents ( http://en.wikipedia.org/wiki/Comparison_of_documentation_generators ). Alternatively, code can be embedded in a larger document that includes information about what the code is doing (i.e., literate programming). Common approaches to this include this use of knitr [63] and IPython Notebooks [64] .
Collaborate
mechanisms↗concepts↗concepts↗unhoused_relationsclaim_relationsIn the same way that having manuscripts reviewed by other scientists can reduce errors and make research easier to understand, reviews of source code can eliminate bugs and improve readability. A large body of research has shown that code reviews are the most cost-effective way of finding bugs in code [65] , [66] . They are also a good way to spread knowledge and good practices around a team. In projects with shifting membership, such as most academic labs, code reviews help ensure that critical knowledge isn't lost when a student or postdoc leaves the lab.
concepts↗concepts↗unhoused_relationsCode can be reviewed either before or after it has been committed to a shared version control repository. Experience shows that if reviews don't have to be done in order to get code into the repository, they will soon not be done at all [27] . We therefore recommend that projects use pre-merge code reviews (8a) .
limitationsconcepts↗concepts↗concepts↗concepts↗concepts↗unhoused_relationsunhoused_relationssensesconcept_edgesAn extreme form of code review is pair programming , in which two developers sit together while writing code. One (the driver) actually writes the code; the other (the navigator) provides real-time feedback and is free to track larger issues of design and consistency. Several studies have found that pair programming improves productivity [67] , but many programmers find it intrusive. We therefore recommend that teams use pair programming when bringing someone new up to speed and when tackling particularly tricky problems (8b) .
concepts↗concepts↗unhoused_relationsOnce a team grows beyond a certain size, it becomes difficult to keep track of what needs to be reviewed, or of who's doing what. Teams can avoid a lot of duplicated effort and dropped balls if they use an issue tracking tool (8c) to maintain a list of tasks to be performed and bugs to be fixed [68] . This helps avoid duplicated work and makes it easier for tasks to be transferred to different people. Free repository hosting services like GitHub include issue tracking tools, and many good standalone tools exist as well, such as Trac ( http://trac.edgewall.org ).
Conclusion
contributionconcepts↗unhoused_relationssensesWe have outlined a series of recommended best practices for scientific computing based on extensive research, as well as our collective experience. These practices can be applied to individual work as readily as group work; separately and together, they improve the productivity of scientific programming and the reliability of the resulting code, and therefore the speed with which we produce results and our confidence in them. They are also, we believe, prerequisites for reproducible computational research: if software is not version controlled, readable, and tested, the chances of its authors (much less anyone else) being able to re-create results are remote.
limitationsconcepts↗concepts↗unhoused_relationsOur 24 recommendations are a beginning, not an end. Individuals and groups who have incorporated them into their work will find links to more advanced practices at Software Carpentry ( http://software-carpentry.org ).
limitationsconcepts↗concepts↗unhoused_relationsResearch suggests that the time cost of implementing these kinds of tools and approaches in scientific computing is almost immediately offset by the gains in productivity of the programmers involved [17] . Even so, the recommendations described above may seem intimidating to implement. Fortunately, the different practices reinforce and support one another, so the effort required is less than the sum of adding each component separately. Nevertheless, we do not recommend that research groups attempt to implement all of these recommendations at once, but instead suggest that these tools be introduced incrementally over a period of time.
concepts↗concepts↗unhoused_relationsHow to implement the recommended practices can be learned from many excellent tutorials available online or through workshops and classes organized by groups like Software Carpentry. This type of training has proven effective at driving adoption of these tools in scientific settings [17] , [69] .
For computing to achieve the level of rigor that is expected throughout other parts of science, it is necessary for scientists to begin to adopt the tools and approaches that are known to improve both the quality of software and the efficiency with which it is produced. To facilitate this adoption, universities and funding agencies need to support the training of scientists in the use of these tools and the investment of time and money in building better scientific software. Investment in these approaches by both individuals and institutions will improve our confidence in the results of computational science and will allow us to make more rapid progress on important scientific questions than would otherwise be possible.
Acknowledgments
We are grateful to Joel Adamson, Aron Ahmadia, Roscoe Bartlett, Erik Bray, Stephen Crouch, Michael Jackson, Justin Kitzes, Adam Obeng, Karthik Ram, Yoav Ram, and Tracy Teal for feedback on this paper.
Funding Statement
Neil Chue Hong was supported by the UK Engineering and Physical Sciences Research Council (EPSRC) Grant EP/H043160/1 for the UK Software Sustainability Institute. Ian M. Mitchell was supported by NSERC Discovery Grant #298211. Mark Plumbley was supported by EPSRC through a Leadership Fellowship (EP/G007144/1) and a grant (EP/H043101/1) for SoundSoftware.ac.uk. Ethan White was supported by a CAREER grant from the US National Science Foundation (DEB 0953694). Greg Wilson was supported by a grant from the Sloan Foundation. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
References
Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; 3 of 3 relations here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.
This record holds no controls. A layer is empty when the tagger recorded nothing in it — which is not the same as the paper having nothing to record.
| facet | tag | field | conf. | whose words |
|---|---|---|---|---|
| object_of_study | practical software-development techniques for improving the productivity, reliability, maintainability, and reproducibility of scientific computing | object_of_study | the paper's words | |
| research_question | which established development practices scientists can adopt to reduce software errors, support reuse, and preserve reproducible computational results | research_question | the paper's words | |
| contribution | twenty-four recommendations organized around readability, automation, incremental change, reuse, testing, optimization, documentation, and collaboration | contribution | the paper's words | |
| claim_type | method | claim_type | the paper's words | |
| phenomenon | scientific conclusions made unreliable or irreproducible by fragile, opaque, untested, or poorly managed research software | phenomenon | the paper's words | |
| method | practice-report also in: practice-report | method | the paper's words | |
| mechanisms | mechanisms | mechanisms[0] | 0.99 | the tagger's words |
| mechanisms | mechanisms | mechanisms[1] | 0.99 | the tagger's words |
| mechanisms | mechanisms | mechanisms[2] | 0.99 | the tagger's words |
| mechanisms | mechanisms | mechanisms[3] | 0.99 | the tagger's words |
| mechanisms | mechanisms | mechanisms[4] | 0.99 | the tagger's words |
| mechanisms | mechanisms | mechanisms[5] | 0.99 | the tagger's words |
| mechanisms | mechanisms | mechanisms[6] | 0.99 | the tagger's words |
| mechanisms | mechanisms | mechanisms[7] | 0.99 | the tagger's words |
| mechanisms | mechanisms | mechanisms[8] | 0.99 | the tagger's words |
| mechanisms | mechanisms | mechanisms[9] | 0.99 | the tagger's words |
| mechanisms | mechanisms | mechanisms[10] | 0.99 | the tagger's words |
| mechanisms | mechanisms | mechanisms[11] | 0.99 | the tagger's words |
| mechanisms | mechanisms | mechanisms[12] | 0.99 | the tagger's words |
| mechanisms | mechanisms | mechanisms[13] | 0.99 | the tagger's words |
| limitations | most scientists are self-taught programmers without exposure to basic development practices | limitations[0] | 0.99 | the tagger's words |
| limitations | the recommendations cannot guarantee efficient or error-free software development | limitations[1] | 0.99 | the tagger's words |
| limitations | the paper does not cover reproducible-research policy, code and data citation, publication, or open science in detail | limitations[2] | 0.99 | the tagger's words |
| limitations | scientific software requirements are rarely frozen and often become knowable only after current results exist | limitations[3] | 0.99 | the tagger's words |
| limitations | binary files may be unsuitable for version control and may require separate archival storage plus metadata | limitations[4] | 0.99 | the tagger's words |
| limitations | no single defensive or testing practice catches or prevents every mistake | limitations[5] | 0.99 | the tagger's words |
| limitations | tests can only compare code against the researcher's expectations and understanding of the problem | limitations[6] | 0.99 | the tagger's words |
| limitations | reasonably complex code will contain initial bugs regardless of overall practice quality | limitations[7] | 0.99 | the tagger's words |
| limitations | some intrinsically difficult code cannot always be made self-explanatory by refactoring | limitations[8] | 0.99 | the tagger's words |
| limitations | pair programming may improve productivity but many programmers find it intrusive | limitations[9] | 0.99 | the tagger's words |
| limitations | the twenty-four recommendations are a starting point rather than a complete endpoint | limitations[10] | 0.99 | the tagger's words |
| limitations | groups should adopt practices incrementally rather than attempt every recommendation at once | limitations[11] | 0.99 | the tagger's words |
| concepts | best practices for scientific software development | concepts[0] | 0.99 | the paper's words |
| concepts | scientists' productivity also in: scientists' productivity | concepts[1] | 0.99 | the paper's words |
| concepts | reliability of their software also in: reliability of their software | concepts[2] | 0.99 | the paper's words |
| concepts | scientific research also in: scientific research | concepts[3] | 0.99 | the paper's words |
| concepts | software also in: software | concepts[4] | 0.99 | the paper's words |
| concepts | experimental apparatus also in: experimental apparatus | concepts[5] | 0.99 | the paper's words |
| concepts | computational methods also in: computational methods | concepts[6] | 0.99 | the paper's words |
| concepts | central conclusions of published research | concepts[7] | 0.99 | the paper's words |
| concepts | computing errors also in: computing errors | concepts[8] | 0.99 | the paper's words |
| concepts | prominent retractions also in: prominent retractions | concepts[9] | 0.99 | the paper's words |
| concepts | best practices also in: best practices | concepts[10] | 0.99 | the paper's words |
| concepts | error-free software development also in: error-free software development | concepts[11] | 0.99 | the paper's words |
| concepts | reproducible research also in: reproducible research | concepts[12] | 0.99 | the paper's words |
| concepts | open science also in: open science | concepts[13] | 0.99 | the paper's words |
| concepts | program also in: program | concepts[14] | 0.99 | the paper's words |
| concepts | human working memory also in: human working memory | concepts[15] | 0.99 | the paper's words |
| concepts | functions also in: functions | concepts[16] | 0.99 | the paper's words |
| concepts | names also in: names | concepts[17] | 0.99 | the paper's words |
| concepts | code style and formatting also in: code style and formatting | concepts[18] | 0.99 | the paper's words |
| concepts | repetition of computational tasks also in: repetition of computational tasks | concepts[19] | 0.99 | the paper's words |
| concepts | recent commands also in: recent commands | concepts[20] | 0.99 | the paper's words |
| concepts | script also in: script | concepts[21] | 0.99 | the paper's words |
| concepts | workflow management tool also in: workflow management tool | concepts[22] | 0.99 | the paper's words |
| concepts | dependencies between files also in: dependencies between files | concepts[23] | 0.99 | the paper's words |
| concepts | build tool also in: build tool | concepts[24] | 0.99 | the paper's words |
| concepts | provenance of data also in: provenance of data | concepts[25] | 0.99 | the paper's words |
| concepts | unique identifiers and version numbers | concepts[26] | 0.99 | the paper's words |
| concepts | values of parameters also in: values of parameters | concepts[27] | 0.99 | the paper's words |
| concepts | scientific programmers also in: scientific programmers | concepts[28] | 0.99 | the paper's words |
| concepts | traditional commercial software developers | concepts[29] | 0.99 | the paper's words |
| concepts | small steps with frequent feedback and course correction | concepts[30] | 0.99 | the paper's words |
| concepts | agile development also in: agile development | concepts[31] | 0.99 | the paper's words |
| concepts | version control system also in: version control system | concepts[32] | 0.99 | the paper's words |
| concepts | VCS also in: VCS | concepts[33] | 0.99 | the paper's words |
| concepts | snapshots of a project's files also in: snapshots of a project's files | concepts[34] | 0.99 | the paper's words |
| concepts | repository also in: repository | concepts[35] | 0.99 | the paper's words |
| concepts | working copy also in: working copy | concepts[36] | 0.99 | the paper's words |
| concepts | conflicts also in: conflicts | concepts[37] | 0.99 | the paper's words |
| concepts | entire history of those files also in: entire history of those files | concepts[38] | 0.99 | the paper's words |
| concepts | everything that has been created manually | concepts[39] | 0.99 | the paper's words |
| concepts | DRY Principle also in: DRY Principle | concepts[40] | 0.99 | the paper's words |
| concepts | single authoritative representation also in: single authoritative representation | concepts[41] | 0.99 | the paper's words |
| concepts | code clones also in: code clones | concepts[42] | 0.99 | the paper's words |
| concepts | modularize code also in: modularize code | concepts[43] | 0.99 | the paper's words |
| concepts | re-use code also in: re-use code | concepts[44] | 0.99 | the paper's words |
| concepts | established library or package also in: established library or package | concepts[45] | 0.99 | the paper's words |
| concepts | defensive programming also in: defensive programming | concepts[46] | 0.99 | the paper's words |
| concepts | assertion also in: assertion | concepts[47] | 0.99 | the paper's words |
| concepts | statement also in: statement | concepts[48] | 0.99 | the paper's words |
| concepts | executable documentation also in: executable documentation | concepts[49] | 0.99 | the paper's words |
| concepts | automated testing also in: automated testing | concepts[50] | 0.99 | the paper's words |
| concepts | unit tests also in: unit tests | concepts[51] | 0.99 | the paper's words |
| concepts | integration tests also in: integration tests | concepts[52] | 0.99 | the paper's words |
| concepts | regression tests also in: regression tests | concepts[53] | 0.99 | the paper's words |
| concepts | unit testing library also in: unit testing library | concepts[54] | 0.99 | the paper's words |
| concepts | bugs also in: bugs | concepts[55] | 0.99 | the paper's words |
| concepts | test cases also in: test cases | concepts[56] | 0.99 | the paper's words |
| concepts | testable also in: testable | concepts[57] | 0.99 | the paper's words |
| concepts | symbolic debugger also in: symbolic debugger | concepts[58] | 0.99 | the paper's words |
| concepts | performance bottlenecks also in: performance bottlenecks | concepts[59] | 0.99 | the paper's words |
| concepts | profiler also in: profiler | concepts[60] | 0.99 | the paper's words |
| concepts | highest-level language possible also in: highest-level language possible | concepts[61] | 0.99 | the paper's words |
| concepts | low-level languages also in: low-level languages | concepts[62] | 0.99 | the paper's words |
| concepts | rapid prototyping also in: rapid prototyping | concepts[63] | 0.99 | the paper's words |
| concepts | test oracle also in: test oracle | concepts[64] | 0.99 | the paper's words |
| concepts | good documentation also in: good documentation | concepts[65] | 0.99 | the paper's words |
| concepts | maintenance costs also in: maintenance costs | concepts[66] | 0.99 | the paper's words |
| concepts | interfaces and reasons also in: interfaces and reasons | concepts[67] | 0.99 | the paper's words |
| concepts | implementations also in: implementations | concepts[68] | 0.99 | the paper's words |
| concepts | refactor code also in: refactor code | concepts[69] | 0.99 | the paper's words |
| concepts | reference documentation also in: reference documentation | concepts[70] | 0.99 | the paper's words |
| concepts | documentation generator also in: documentation generator | concepts[71] | 0.99 | the paper's words |
| concepts | literate programming also in: literate programming | concepts[72] | 0.99 | the paper's words |
| concepts | code reviews also in: code reviews | concepts[73] | 0.99 | the paper's words |
| concepts | source code also in: source code | concepts[74] | 0.99 | the paper's words |
| concepts | pre-merge code reviews also in: pre-merge code reviews | concepts[75] | 0.99 | the paper's words |
| concepts | pair programming also in: pair programming | concepts[76] | 0.99 | the paper's words |
| concepts | code review also in: code review | concepts[77] | 0.99 | the paper's words |
| concepts | driver also in: driver | concepts[78] | 0.99 | the paper's words |
| concepts | navigator also in: navigator | concepts[79] | 0.99 | the paper's words |
| concepts | issue tracking tool also in: issue tracking tool | concepts[80] | 0.99 | the paper's words |
| concepts | duplicated effort also in: duplicated effort | concepts[81] | 0.99 | the paper's words |
| concepts | reproducible computational research also in: reproducible computational research | concepts[82] | 0.99 | the paper's words |
| concepts | 24 recommendations also in: 24 recommendations | concepts[83] | 0.99 | the paper's words |
| concepts | time cost also in: time cost | concepts[84] | 0.99 | the paper's words |
| concepts | training also in: training | concepts[85] | 0.99 | the paper's words |
| concepts | primarily self-taught also in: primarily self-taught | concepts[86] | 0.99 | the paper's words |
| concepts | basic software development practices also in: basic software development practices | concepts[87] | 0.99 | the paper's words |
| concepts | how reliable their software is also in: how reliable their software is | concepts[88] | 0.99 | the paper's words |
| concepts | reproducibility also in: reproducibility | concepts[89] | 0.99 | the paper's words |
| concepts | readers make more mistakes also in: readers make more mistakes | concepts[90] | 0.99 | the paper's words |
| concepts | mistakes also in: mistakes | concepts[91] | 0.99 | the paper's words |
| concepts | requirements are constantly changing also in: requirements are constantly changing | concepts[92] | 0.99 | the paper's words |
| concepts | lost work also in: lost work | concepts[93] | 0.99 | the paper's words |
| concepts | provenance for both code and data also in: provenance for both code and data | concepts[94] | 0.99 | the paper's words |
| concepts | errors and inconsistencies also in: errors and inconsistencies | concepts[95] | 0.99 | the paper's words |
| concepts | debugging also in: debugging | concepts[96] | 0.99 | the paper's words |
| concepts | confidence that scientific code is operating properly | concepts[97] | 0.99 | the paper's words |
| concepts | witness what is going wrong directly also in: witness what is going wrong directly | concepts[98] | 0.99 | the paper's words |
| concepts | explaining how it works also in: explaining how it works | concepts[99] | 0.99 | the paper's words |
| concepts | shared version control repository also in: shared version control repository | concepts[100] | 0.99 | the paper's words |
| concepts | productivity also in: productivity | concepts[101] | 0.99 | the paper's words |
| concepts | advanced practices also in: advanced practices | concepts[102] | 0.99 | the paper's words |
| concepts | productivity of the programmers involved | concepts[103] | 0.99 | the paper's words |
| concepts | adoption of these tools also in: adoption of these tools | concepts[104] | 0.99 | the paper's words |
| unhoused_relations | software → experimental apparatus Software is another kind of experimental apparatus. | unhoused_relations[0] | 0.99 | the tagger's words |
| unhoused_relations | script → program There is no real difference between a script and a program. | unhoused_relations[1] | 0.99 | the tagger's words |
| unhoused_relations | assertion → statement An assertion is simply a statement that something holds true at a particular point in a program. | unhoused_relations[2] | 0.99 | the tagger's words |
| unhoused_relations | pair programming → code review Pair programming is an extreme form of code review. | unhoused_relations[3] | 0.99 | the tagger's words |
| unhoused_relations | primarily self-taught → basic software development practices Most scientific programmers are self-taught and lack exposure to basic development practices. | unhoused_relations[4] | 0.99 | the tagger's words |
| unhoused_relations | how reliable their software is → central conclusions of published research Unknown software reliability can lead to errors that affect central research conclusions. | unhoused_relations[5] | 0.99 | the tagger's words |
| unhoused_relations | best practices → error-free software development No combination of the recommended practices guarantees efficient error-free development. | unhoused_relations[6] | 0.99 | the tagger's words |
| unhoused_relations | best practices → reproducible research The described skills make reproducible research and open science easier to implement. | unhoused_relations[7] | 0.99 | the tagger's words |
| unhoused_relations | human working memory → functions Limited working memory motivates breaking programs into single-task functions. | unhoused_relations[8] | 0.99 | the tagger's words |
| unhoused_relations | code style and formatting → readers make more mistakes Inconsistent formatting makes code slower to read and causes more reader mistakes. | unhoused_relations[9] | 0.99 | the tagger's words |
| unhoused_relations | repetition of computational tasks → mistakes Manual repetition wastes time and eventually produces mistakes. | unhoused_relations[10] | 0.99 | the tagger's words |
| unhoused_relations | build tool → dependencies between files A build tool uses declared file dependencies to automate scientific workflows. | unhoused_relations[11] | 0.99 | the tagger's words |
| unhoused_relations | provenance of data → reproducibility Machine-readable provenance records everything needed to recreate output. | unhoused_relations[12] | 0.99 | the tagger's words |
| unhoused_relations | small steps with frequent feedback and course correction → requirements are constantly changing Small iterative steps accommodate changing scientific requirements. | unhoused_relations[13] | 0.99 | the tagger's words |
| unhoused_relations | version control system → lost work Version control replaces fragile copying practices that can cause confusion and lost work. | unhoused_relations[14] | 0.99 | the tagger's words |
| unhoused_relations | version control system → provenance for both code and data Stored histories and metadata provide provenance for code and data. | unhoused_relations[15] | 0.99 | the tagger's words |
| unhoused_relations | everything that has been created manually → version control system Reproducibility is maximized when all manually created materials are version controlled. | unhoused_relations[16] | 0.99 | the tagger's words |
| unhoused_relations | DRY Principle → errors and inconsistencies Avoiding repeated representations reduces the chance of update errors and inconsistencies. | unhoused_relations[17] | 0.99 | the tagger's words |
| unhoused_relations | modularize code → code clones Modularization avoids code clones and lets fixes take effect everywhere. | unhoused_relations[18] | 0.99 | the tagger's words |
| unhoused_relations | re-use code → established library or package Scientific programmers should use established packages for solved problems rather than rewrite them. | unhoused_relations[19] | 0.99 | the tagger's words |
| unhoused_relations | defensive programming → mistakes Assertions and tests form complementary defenses because no single practice catches every mistake. | unhoused_relations[20] | 0.99 | the tagger's words |
| unhoused_relations | assertion → debugging An assertion halts a failing program immediately and simplifies debugging. | unhoused_relations[21] | 0.99 | the tagger's words |
| unhoused_relations | automated testing → confidence that scientific code is operating properly Combined tests improve confidence that code and results are valid. | unhoused_relations[22] | 0.99 | the tagger's words |
| unhoused_relations | symbolic debugger → witness what is going wrong directly A debugger exposes execution directly rather than requiring inference from logs. | unhoused_relations[23] | 0.99 | the tagger's words |
| unhoused_relations | profiler → performance bottlenecks A profiler is used to identify bottlenecks only after correctness and need are established. | unhoused_relations[24] | 0.99 | the tagger's words |
| unhoused_relations | highest-level language possible → scientists' productivity Higher-level languages increase productivity because equivalent tasks require fewer lines of code. | unhoused_relations[25] | 0.99 | the tagger's words |
| unhoused_relations | rapid prototyping → test oracle A high-level prototype can serve as a test oracle for a low-level reimplementation. | unhoused_relations[26] | 0.99 | the tagger's words |
| unhoused_relations | good documentation → maintenance costs Good documentation improves reuse and lowers software maintenance costs. | unhoused_relations[27] | 0.99 | the tagger's words |
| unhoused_relations | interfaces and reasons → implementations Documentation should cover interfaces and reasons rather than implementation mechanics. | unhoused_relations[28] | 0.99 | the tagger's words |
| unhoused_relations | refactor code → explaining how it works Complex code should be refactored instead of explained in a long implementation description. | unhoused_relations[29] | 0.99 | the tagger's words |
| unhoused_relations | reference documentation → software Embedding reference documentation in software increases the chance it changes with the code. | unhoused_relations[30] | 0.99 | the tagger's words |
| unhoused_relations | code reviews → bugs Code reviews eliminate bugs, improve readability, and spread knowledge and practice. | unhoused_relations[31] | 0.99 | the tagger's words |
| unhoused_relations | pre-merge code reviews → shared version control repository Requiring review before merge prevents review from disappearing in practice. | unhoused_relations[32] | 0.99 | the tagger's words |
| unhoused_relations | pair programming → productivity Studies report productivity improvement from pair programming, but many programmers find it intrusive. | unhoused_relations[33] | 0.99 | the tagger's words |
| unhoused_relations | issue tracking tool → duplicated effort Issue tracking helps teams avoid duplicated effort and dropped tasks. | unhoused_relations[34] | 0.99 | the tagger's words |
| unhoused_relations | best practices → reproducible computational research Readable, tested, version-controlled software is presented as a prerequisite for reproducible computational research. | unhoused_relations[35] | 0.99 | the tagger's words |
| unhoused_relations | 24 recommendations → advanced practices The twenty-four recommendations are a beginning that leads toward more advanced practice. | unhoused_relations[36] | 0.99 | the tagger's words |
| unhoused_relations | time cost → productivity of the programmers involved The time cost of adopting the practices is almost immediately offset by productivity gains. | unhoused_relations[37] | 0.99 | the tagger's words |
| unhoused_relations | training → adoption of these tools Tutorials, workshops, and classes have proven effective at driving adoption. | unhoused_relations[38] | 0.99 | the tagger's words |
| senses | software - a kind of experimental apparatus that requires validation and careful use | senses[0] | 0.99 | the tagger's words |
| senses | script - a command file with no substantive difference from a program | senses[1] | 0.99 | the tagger's words |
| senses | provenance of data - machine-readable identifiers, versions, parameters, and programs needed to recreate output | senses[2] | 0.99 | the tagger's words |
| senses | VCS - a version control system storing snapshots and project history in repositories | senses[3] | 0.99 | the tagger's words |
| senses | DRY Principle - one authoritative representation for each datum and reusable rather than copied code | senses[4] | 0.99 | the tagger's words |
| senses | assertion - an executable statement that a condition holds at a point in a program | senses[5] | 0.99 | the tagger's words |
| senses | automated testing - unit, integration, and regression checks rerun by a computer | senses[6] | 0.99 | the tagger's words |
| senses | symbolic debugger - an interactive program inspector exposing live execution state | senses[7] | 0.99 | the tagger's words |
| senses | pair programming - an extreme code-review form joining a code-writing driver and feedback-oriented navigator | senses[8] | 0.99 | the tagger's words |
| senses | reproducible computational research - results supported by software that is version controlled, readable, and tested | senses[9] | 0.99 | the tagger's words |
| claim_relations | proposes → a research- and experience-based set of scientific software-development practices note: The paper's stated contribution. | claim_relations[0] | 0.99 | the tagger's words |
| claim_relations | argues → scientific software should be validated as carefully as physical experimental apparatus note: The central scientific-accountability analogy. | claim_relations[1] | 0.99 | the tagger's words |
| claim_relations | recommends → programs designed for limited human working memory and reliable comprehension note: The human-readable-program practice. | claim_relations[2] | 0.99 | the tagger's words |
| claim_relations | recommends → automated workflows with machine-readable provenance note: The automation and provenance practice. | claim_relations[3] | 0.99 | the tagger's words |
| claim_relations | supports → small incremental changes and version control under changing research requirements note: The incremental-development practice. | claim_relations[4] | 0.99 | the tagger's words |
| claim_relations | recommends → single-source data, modular code, and reuse of established libraries note: The DRY and reuse practice. | claim_relations[5] | 0.99 | the tagger's words |
| claim_relations | supports → layered defensive programming, automated tests, regression cases, and interactive debugging note: The plan-for-mistakes practice. | claim_relations[6] | 0.99 | the tagger's words |
| claim_relations | recommends → correctness before profiler-guided optimization and low-level reimplementation note: The optimization practice. | claim_relations[7] | 0.99 | the tagger's words |
| claim_relations | recommends → documentation of interfaces, reasons, design, and purpose rather than mechanics note: The documentation practice. | claim_relations[8] | 0.99 | the tagger's words |
| claim_relations | supports → pre-merge review, selective pair programming, and issue tracking for collaboration note: The collaboration practice. | claim_relations[9] | 0.99 | the tagger's words |
| concept_edges | computing errors —causes→ prominent retractions the relation itself is separately evidenced · strength of the relation: 0.8 · note: Direct positive causal direction. · the relation stated: A cascading error from reused code caused several prominent retractions. also in: computing errors · prominent retractions | concept_edges[0] | 0.99 | the tagger's words |
| concept_edges | scientific programmers —distinct-from→ traditional commercial software developers the relation itself is separately evidenced · strength of the relation: 1 · note: Explicit unlike contrast. · the relation stated: Scientific programmers differ from traditional commercial developers because their requirements are rarely fixed in advance. | concept_edges[1] | 0.99 | the tagger's words |
| concept_edges | driver —distinct-from→ navigator the relation itself is separately evidenced · strength of the relation: 1 · note: Explicit role contrast within one method. · the relation stated: The driver writes code whereas the navigator provides real-time feedback and tracks larger design issues. | concept_edges[2] | 0.99 | the tagger's words |
Tags that could not be placed in the text:
What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.