Marcus R. MunafòpersName, Brian A. NosekpersName, Dorothy V. M. BishoppersName, Katherine S. ButtonpersName, Christopher D. ChamberspersName, Nathalie Percie du SertpersName, Uri SimonsohnpersName, Eric-Jan WagenmakerspersName, Jennifer J. WarepersName, John P. A. IoannidispersName (2017date). “A manifesto for reproducible science.” Nature Human Behaviourtitle·j.
Object of study — systemic threats to reliable and reproducible scienceterm and practical reforms across methodsterm, reporting, openness, evaluationterm, and incentivesterm
Research question — which interventions could improve scientific reliability, transparencyterm, reproducibilityterm, and efficiency, and how should their effectiveness be tested
Contribution — an evidence-informed manifesto organizing practical reforms into methodsterm, reporting and disseminationterm, reproducibilityterm, evaluationterm, and incentivesterm
Cross-disciplinary analysis
content-tags & facets:
phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | systemic erosion of scientific credibility through biased reasoning, weak design, selective reporting, opacity, narrow evaluation, and dysfunctional rewards — “Threats to reproducible science. An idealized version of the hypothetico-deductive model of the scientific method is shown. Various potential threats to this model exist (indicated in red), including lack of replication 5 , hypothesizing after the results are known (HARKing) 7 , poor study design, low statistical power 2 , analytical flexibility 51 , P -hacking 4 , publication bias 3 and lack of data sharing 6 . Together these will serve to undermine the robustness of published research, and may also impact on the ability of science to self-correct.” | |
| claim-type | critique — “The challenges to reproducible science are systemic and cultural, but that does not mean they cannot be met. The measures we have described constitute practical and achievable steps toward improving rigor and reproducibility. All of them have shown some effectiveness, and are well suited to wider adoption, evaluation and improvement. Equally, these proposals are not an exhaustive list; there are many other nascent and maturing ideas for making research practices more efficient and reliable 73 . Offering a solution to a problem does not guarantee its effectiveness, and making changes to cultural norms and incentives can spur additional behavioural changes that are difficult to anticipate. Some solutions may be ineffective or even harmful to the efficiency and reliability of science, even if conceptually they appear sensible.” | |
| method — evidence | — “Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.” | |
| object of study — evidence | — “Here we propose a series of measures that we believe will improve research efficiency and robustness of scientific findings by directly targeting specific threats to reproducible science. We argue for the adoption, evaluation and ongoing improvement of these measures to optimize the pace and efficiency of knowledge accumulation. The measures are organized into the following categories 16 : methods, reporting and dissemination, reproducibility, evaluation and incentives. They are not intended to be exhaustive, but provide a broad, practical and evidence-based set of actions that can be implemented by researchers, institutions, journals and funders. The measures and their current implementation are summarized in Table 1 .” | |
| research question — evidence | — “Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.” | |
| contribution — evidence | — “Here we propose a series of measures that we believe will improve research efficiency and robustness of scientific findings by directly targeting specific threats to reproducible science. We argue for the adoption, evaluation and ongoing improvement of these measures to optimize the pace and efficiency of knowledge accumulation. The measures are organized into the following categories 16 : methods, reporting and dissemination, reproducibility, evaluation and incentives. They are not intended to be exhaustive, but provide a broad, practical and evidence-based set of actions that can be implemented by researchers, institutions, journals and funders. The measures and their current implementation are summarized in Table 1 .” | |
| mechanism | cognitive biases combine to make random structure look meaningful and expected after the fact — “A hallmark of scientific creativity is the ability to see novel and unexpected patterns in data. John Snow's identification of links between cholera and water supply 17 , Paul Broca's work on language lateralization 18 and Jocelyn Bell Burnell's discovery of pulsars 19 are examples of breakthroughs achieved by interpreting observations in a new way. However, a major challenge for scientists is to be open to new and important insights while simultaneously avoiding being misled by our tendency to see structure in randomness. The combination of apophenia (the tendency to see patterns in random data), confirmation bias (the tendency to focus on evidence that is in line with our expectations or favoured explanation) and hindsight bias (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusions 20 . Thomas Levenson documents the example of astronomers who became convinced they had seen the fictitious planet Vulcan because their contemporary theories predicted its existence 21 . Experimenter effects are an example of this kind of bias 22 .” | mechanisms[0] |
| mechanism | high-dimensional data multiplied by flexible pipelines creates many opportunities for false-positive findings — “Over-interpretation of noise is facilitated by the extent to which data analysis is rapid, flexible and automated 23 . In a high-dimensional dataset, there may be hundreds or thousands of reasonable alternative approaches to analysing the same data 24 , 25 . For example, in a systematic review of functional magnetic resonance imaging (fMRI) studies, Carp showed that there were almost as many unique analytical pipelines as there were studies 26 . If several thousand potential analytical pipelines can be applied to high-dimensional data, the generation of false-positive findings is highly likely. For example, applying almost 7,000 analytical pipelines to a single fMRI dataset resulted in over 90% of brain voxels showing significant activation in at least one analysis 27 .” | mechanisms[1] |
| mechanism | blinding masks hypotheses, conditions, labels, or results so decisions cannot be steered toward desired outcomes — “Protecting against cognitive biases. There is a substantial literature on the difficulty of avoiding cognitive biases. An effective solution to mitigate self-deception and unwanted biases is blinding. In some research contexts, participants and data collectors can be blinded to the experimental condition that participants are assigned to, and to the research hypotheses, while the data analyst can be blinded to key parts of the data. For example, during data preparation and cleaning, the identity of experimental conditions or the variable labels can be masked so that the output is not interpretable in terms of the research hypothesis. In some physical sciences this approach has been extended to include deliberate perturbations in or masking of data to allow data preparation (for example, identification of outliers) to proceed without the analyst being able to see the corresponding results 30 . Pre-registration of the study design, primary outcome(s) and analysis plan (see ‘Promoting study pre-registration’ section’, below) is a highly effective form of blinding because the data do not exist and the outcomes are not yet known.” | mechanisms[2] |
| mechanism | continuing modular training updates statistical and methodological practice across career stages — “Without formal requirements for continuing education, the most effective solutions may be to develop educational resources that are accessible, easy-to-digest and immediately and effectively applicable to research (for example, brief, web-based modules for specific topics, and combinations of modules that are customized for particular research applications). A modular approach simplifies the process of iterative updating of those materials. Demonstration software and hands-on examples may also make the lessons and implications particularly tangible to researchers at any career stage: the Experimental Design Assistant ( https://eda.nc3rs.org.uk ) supports research design for whole animal experiments, while P -hacker ( http://shinyapps.org/apps/p-hacker/ ) shows just how easy it is to generate apparently statistically significant findings by exploiting analytic flexibility.” | mechanisms[3] |
| mechanism | independent methodologists reduce personal investment in design, analysis, and interpretation decisions — “Implementing independent methodological support. The need for independent methodological support is well-established in some areas — many clinical trials, for example, have multidisciplinary trial steering committees to provide advice and oversee the design and conduct of the trial. The need for these committees grew out of the well-understood financial conflicts of interest that exist in many clinical trials. The sponsor of a trial may be the company manufacturing the product, and any intentional or unintentional influence can distort the study design, analysis and interpretation of results for the ultimate financial benefit of the manufacturer at the cost of the accuracy of the science and the health benefit to the consumers 37 , 38 . Non-financial conflicts of interest also exist, such as the beliefs and preconceptions of individual scientists and the stakes that researchers have in obtaining publishable results in order to progress their career 39 , 40 . Including independent researchers (particularly methodologists with no personal investment in a research topic) in the design, monitoring, analysis or interpretation of research outcomes may mitigate some of those influences, and can be done either at the level of the individual research project or through a process facilitated by a funding agency (see Box 1 ).” | mechanisms[4] |
| mechanism | distributed team science pools sites and resources to increase power and test generalizability — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | mechanisms[5] |
| mechanism | advance registration makes studies discoverable and keeps confirmatory decisions independent of observed data — “The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P -hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory research that is important for testing hypotheses, and data-contingent exploratory research that is important for generating hypotheses.” | mechanisms[6] |
| mechanism | Registered Reports review protocols before outcomes and condition publication on adherence and evidence-bound conclusions — “The Registered Reports (RR) initiative seeks to eliminate various forms of bias in hypothesis-driven research 52 , 53 , and in particular, the evaluation of a study based on the results. Unlike conventional journal articles, RRs split the peer review process into two stages, before and after results are known. At the first stage, reviewers and editors assess a detailed protocol that includes the study rationale, procedure and a detailed analysis plan. Following favourable reviews (and probably revision to meet strict methodological standards), the journal offers in-principle acceptance: publication of study outcomes is guaranteed provided the authors adhere to the approved protocol, the study meets pre-specified quality checks, and conclusions are appropriately evidence-bound. Once the study is completed, the authors resubmit a complete manuscript that includes the results and discussion. The article is published at the end of this two-stage process. By accepting articles before results are known, RRs prevent publication bias. By reviewing the hypotheses and analysis plans in advance, RRs should also help neutralize P- hacking and HARKing (hypothesizing after the results are known) by authors, and CARKing (critiquing after the results are known) by reviewers with their own investments in the research outcomes, although empirical evidence will be required to confirm that this is the case.” | mechanisms[7] |
| mechanism | transparent access lets communities review, critique, extend, and reproduce supporting evidence — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | mechanisms[8] |
| mechanism | public data repositories support reanalysis, error detection, accountability, longevity, efficiency, and quality — “Sharing data in public repositories offers field-wide advantages in terms of accountability, data longevity, efficiency and quality (for example, reanalyses may detect crucial mistakes or even data fabrication) 84 . Unfortunately, many scientific disciplines, including most of those devoted to the study of human behaviour, do not have a culture that values open data 6 . In the past, data sharing has rarely been enforced or facilitated. Recent initiatives, however, aim to change the normative culture. Hopefully, these initiatives will change the culture on data sharing. Once accepted as the norm, we doubt that data sharing will ever go out of fashion.” | mechanisms[9] |
| mechanism | preprints and public post-publication commentary accelerate and diversify evaluation — “Both pre- and post-publication peer review mechanisms dramatically accelerate and expand the evaluation process 68 . By sharing preprints, researchers can obtain rapid feedback on their work from a diverse community, rather than waiting several months for a few reviews in the conventional, closed peer review process. Using post-publication services, reviewers can make positive and critical commentary on articles instantly, rather than relying on the laborious, uncertain and lengthy process of authoring a commentary and submitting it to the publishing journal for possible publication, eventually.” | mechanisms[10] |
| mechanism | changing appointment and promotion rewards changes rational research strategies and researcher behaviour — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | mechanisms[11] |
| mechanism | iterative evaluation winnows proposed reforms toward a broadly effective set — “These cautions are not a rationale for inaction. Reproducible research practices are at the heart of sound research and integral to the scientific method. How best to achieve rigorous and efficient knowledge accumulation is a scientific question; the most effective solutions will be identified by a combination of brilliant hypothesizing and blind luck, by iterative examination of the effectiveness of each change, and by a winnowing of many possibilities to the broadly enacted few. True understanding of how best to structure and incentivize science will emerge slowly and will never be finished. That is how science works. The key to fostering a robust metascience that evaluates and improves practices is that the stakeholders of science must not embrace the status quo, but instead pursue self-examination continuously for improvement and self-correction of the scientific process itself.” | mechanisms[12] |
| argues | adoption and iterative evaluation of measures across five elements of the scientific process — “Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.” | claim_relations[0] |
| critiques | cognitive and analytic processes that turn random patterns into false conclusions — “A hallmark of scientific creativity is the ability to see novel and unexpected patterns in data. John Snow's identification of links between cholera and water supply 17 , Paul Broca's work on language lateralization 18 and Jocelyn Bell Burnell's discovery of pulsars 19 are examples of breakthroughs achieved by interpreting observations in a new way. However, a major challenge for scientists is to be open to new and important insights while simultaneously avoiding being misled by our tendency to see structure in randomness. The combination of apophenia (the tendency to see patterns in random data), confirmation bias (the tendency to focus on evidence that is in line with our expectations or favoured explanation) and hindsight bias (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusions 20 . Thomas Levenson documents the example of astronomers who became convinced they had seen the fictitious planet Vulcan because their contemporary theories predicted its existence 21 . Experimenter effects are an example of this kind of bias 22 .” | claim_relations[1] |
| recommends | blinding analysts, participants, collectors, conditions, labels, hypotheses, or results where feasible — “Protecting against cognitive biases. There is a substantial literature on the difficulty of avoiding cognitive biases. An effective solution to mitigate self-deception and unwanted biases is blinding. In some research contexts, participants and data collectors can be blinded to the experimental condition that participants are assigned to, and to the research hypotheses, while the data analyst can be blinded to key parts of the data. For example, during data preparation and cleaning, the identity of experimental conditions or the variable labels can be masked so that the output is not interpretable in terms of the research hypothesis. In some physical sciences this approach has been extended to include deliberate perturbations in or masking of data to allow data preparation (for example, identification of outliers) to proceed without the analyst being able to see the corresponding results 30 . Pre-registration of the study design, primary outcome(s) and analysis plan (see ‘Promoting study pre-registration’ section’, below) is a highly effective form of blinding because the data do not exist and the outcomes are not yet known.” | claim_relations[2] |
| supports | team science and distributed collaboration for power and generalizability — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | claim_relations[3] |
| recommends | strong pre-registration of design, outcomes, and analysis before data or outcomes are known — “The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P -hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory research that is important for testing hypotheses, and data-contingent exploratory research that is important for generating hypotheses.” | claim_relations[4] |
| qualifies | reporting guidelines as necessary but insufficient without adoption, enforcement, and evaluation — “The positive outcome of reporting guidelines is that they make it possible to detect and study these behaviours and their impact. Otherwise, these behaviours are simply unknowable in any systematic way. The negative outcome is the empirical evidence that reporting guidelines may be necessary, but will not alone be sufficient, to address reporting biases. The impact of guidelines and how best to optimize their use and impact will be best assessed by randomized trials (see Box 4 ).” | claim_relations[5] |
| supports | transparent and accessible research content, data, methods, workflows, and interpretation — “Open science refers to the process of making the content and process of producing evidence and claims transparent and accessible to others. Transparency is a scientific ideal, and adding ‘open’ should therefore be redundant. In reality, science often lacks openness: many published articles are not available to people without a personal or institutional subscription, and most data, materials and code supporting research outcomes are not made accessible, for example, in a public repository (refs 63 , 64 ; Box 5 ).” | claim_relations[6] |
| supports | diversified pre-publication and post-publication evaluation — “Both pre- and post-publication peer review mechanisms dramatically accelerate and expand the evaluation process 68 . By sharing preprints, researchers can obtain rapid feedback on their work from a diverse community, rather than waiting several months for a few reviews in the conventional, closed peer review process. Using post-publication services, reviewers can make positive and critical commentary on articles instantly, rather than relying on the laborious, uncertain and lengthy process of authoring a commentary and submitting it to the publishing journal for possible publication, eventually.” | claim_relations[7] |
| critiques | reward systems favoring positive, novel, and clean results over rigor and reproducibility — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | claim_relations[8] |
| qualifies | all proposed reforms as hypotheses requiring continuing evaluation for ineffectiveness or harm — “The challenges to reproducible science are systemic and cultural, but that does not mean they cannot be met. The measures we have described constitute practical and achievable steps toward improving rigor and reproducibility. All of them have shown some effectiveness, and are well suited to wider adoption, evaluation and improvement. Equally, these proposals are not an exhaustive list; there are many other nascent and maturing ideas for making research practices more efficient and reliable 73 . Offering a solution to a problem does not guarantee its effectiveness, and making changes to cultural norms and incentives can spur additional behavioural changes that are difficult to anticipate. Some solutions may be ineffective or even harmful to the efficiency and reliability of science, even if conceptually they appear sensible.” | claim_relations[9] |
| sense | metascience - the scientific study of science itself — “What proportion of published research is likely to be false? Low sample size, small effect sizes, data dredging (also known as P -hacking), conflicts of interest, large numbers of scientists working competitively in silos without combining their efforts, and so on, may conspire to dramatically increase the probability that a published finding is incorrect 1 . The field of metascience — the scientific study of science itself — is flourishing and has generated substantial empirical evidence for the existence and prevalence of threats to efficiency in knowledge accumulation (refs 2 , 3 , 4 , 5 , 6 , 7 ; Fig. 1 ).” | senses[0] |
| sense | apophenia - seeing patterns in random data — “A hallmark of scientific creativity is the ability to see novel and unexpected patterns in data. John Snow's identification of links between cholera and water supply 17 , Paul Broca's work on language lateralization 18 and Jocelyn Bell Burnell's discovery of pulsars 19 are examples of breakthroughs achieved by interpreting observations in a new way. However, a major challenge for scientists is to be open to new and important insights while simultaneously avoiding being misled by our tendency to see structure in randomness. The combination of apophenia (the tendency to see patterns in random data), confirmation bias (the tendency to focus on evidence that is in line with our expectations or favoured explanation) and hindsight bias (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusions 20 . Thomas Levenson documents the example of astronomers who became convinced they had seen the fictitious planet Vulcan because their contemporary theories predicted its existence 21 . Experimenter effects are an example of this kind of bias 22 .” | senses[1] |
| sense | confirmation bias - favoring evidence aligned with expectations or a preferred explanation — “A hallmark of scientific creativity is the ability to see novel and unexpected patterns in data. John Snow's identification of links between cholera and water supply 17 , Paul Broca's work on language lateralization 18 and Jocelyn Bell Burnell's discovery of pulsars 19 are examples of breakthroughs achieved by interpreting observations in a new way. However, a major challenge for scientists is to be open to new and important insights while simultaneously avoiding being misled by our tendency to see structure in randomness. The combination of apophenia (the tendency to see patterns in random data), confirmation bias (the tendency to focus on evidence that is in line with our expectations or favoured explanation) and hindsight bias (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusions 20 . Thomas Levenson documents the example of astronomers who became convinced they had seen the fictitious planet Vulcan because their contemporary theories predicted its existence 21 . Experimenter effects are an example of this kind of bias 22 .” | senses[2] |
| sense | hindsight bias - seeing an event as predictable only after it occurred — “A hallmark of scientific creativity is the ability to see novel and unexpected patterns in data. John Snow's identification of links between cholera and water supply 17 , Paul Broca's work on language lateralization 18 and Jocelyn Bell Burnell's discovery of pulsars 19 are examples of breakthroughs achieved by interpreting observations in a new way. However, a major challenge for scientists is to be open to new and important insights while simultaneously avoiding being misled by our tendency to see structure in randomness. The combination of apophenia (the tendency to see patterns in random data), confirmation bias (the tendency to focus on evidence that is in line with our expectations or favoured explanation) and hindsight bias (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusions 20 . Thomas Levenson documents the example of astronomers who became convinced they had seen the fictitious planet Vulcan because their contemporary theories predicted its existence 21 . Experimenter effects are an example of this kind of bias 22 .” | senses[3] |
| sense | P-hacking - data-contingent selection among analyses, models, exclusions, covariates, or outcomes — “Promoting study pre-registration. Pre-registration of study protocols for randomized controlled trials in clinical medicine has become standard practice 46 . In its simplest form it may simply comprise the registration of the basic study design, but it can also include a detailed pre-specification of the study procedures, outcomes and statistical analysis plan. It was introduced to address two problems: publication bias and analytical flexibility (in particular outcome switching in the case of clinical medicine). Publication bias 47 , also known as the file drawer problem 48 , refers to the fact that many more studies are conducted than published. Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results 47 , 49 , 50 . The consequence is that the published literature indicates stronger evidence for findings than exists in reality. Outcome switching refers to the possibility of changing the outcomes of interest in the study depending on the observed results. A researcher may include ten variables that could be considered outcomes of the research, and — once the results are known — intentionally or unintentionally select the subset of outcomes that show statistically significant results as the outcomes of interest. The consequence is an increase in the likelihood that reported results are spurious by leveraging chance, while negative evidence gets ignored. This is one of several related research practices that can inflate spurious findings when analysis decisions are made with knowledge of the observed data, such as selection of models, exclusion rules and covariates. Such data-contingent analysis decisions constitute what has become known as P -hacking 51 , and pre-registration can protect against all of these.” | senses[4] |
| sense | pre-registration - advance specification that separates data-independent confirmation from data-contingent exploration — “The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P -hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory research that is important for testing hypotheses, and data-contingent exploratory research that is important for generating hypotheses.” | senses[5] |
| sense | Registered Reports - two-stage review with conditional acceptance before results are known — “The Registered Reports (RR) initiative seeks to eliminate various forms of bias in hypothesis-driven research 52 , 53 , and in particular, the evaluation of a study based on the results. Unlike conventional journal articles, RRs split the peer review process into two stages, before and after results are known. At the first stage, reviewers and editors assess a detailed protocol that includes the study rationale, procedure and a detailed analysis plan. Following favourable reviews (and probably revision to meet strict methodological standards), the journal offers in-principle acceptance: publication of study outcomes is guaranteed provided the authors adhere to the approved protocol, the study meets pre-specified quality checks, and conclusions are appropriately evidence-bound. Once the study is completed, the authors resubmit a complete manuscript that includes the results and discussion. The article is published at the end of this two-stage process. By accepting articles before results are known, RRs prevent publication bias. By reviewing the hypotheses and analysis plans in advance, RRs should also help neutralize P- hacking and HARKing (hypothesizing after the results are known) by authors, and CARKing (critiquing after the results are known) by reviewers with their own investments in the research outcomes, although empirical evidence will be required to confirm that this is the case.” | senses[6] |
| sense | transparency - inspectable evidence and process rather than credibility based only on authority — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | senses[7] |
| sense | open science - transparent and accessible production of evidence and claims — “Open science refers to the process of making the content and process of producing evidence and claims transparent and accessible to others. Transparency is a scientific ideal, and adding ‘open’ should therefore be redundant. In reality, science often lacks openness: many published articles are not available to people without a personal or institutional subscription, and most data, materials and code supporting research outcomes are not made accessible, for example, in a public repository (refs 63 , 64 ; Box 5 ).” | senses[8] |
| sense | Full disclosure - reporting the full design and collected data instead of a curated subset — “Full disclosure refers to the process of describing in full the study design and data collected that underlie the results reported, rather than a curated version of the design, and/or a subset of the data collected. The need for disclosure is clear: in order to adequately evaluate results we need to know how they were obtained. For example, the informational value of a dependent variable exhibiting an effect of interest is different if only one variable was collected or if fifteen were. The probability of a single variable achieving P < 0.05 just by chance is 5%, but the probability of one of fifteen variables achieving P < 0.05 is 54% 1 . It is obvious that cherry-picking one from fifteen variables invalidates the results unless it is clear that this has happened. If readers know, then they can adjust their interpretation accordingly. From this simple fact it follows that if authors do not tell us whether they collected one or fifteen variables readers cannot evaluate their research 51 .” | senses[9] |
| note | adoption and iterative evaluation of measures across five elements of the scientific process — The manifesto's central argument. — “Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.” | claim_relations[0] |
| weight | adoption and iterative evaluation of measures across five elements of the scientific process — 0.99 — “Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.” | claim_relations[0] |
| note | cognitive and analytic processes that turn random patterns into false conclusions — The problem diagnosis is explicit. — “A hallmark of scientific creativity is the ability to see novel and unexpected patterns in data. John Snow's identification of links between cholera and water supply 17 , Paul Broca's work on language lateralization 18 and Jocelyn Bell Burnell's discovery of pulsars 19 are examples of breakthroughs achieved by interpreting observations in a new way. However, a major challenge for scientists is to be open to new and important insights while simultaneously avoiding being misled by our tendency to see structure in randomness. The combination of apophenia (the tendency to see patterns in random data), confirmation bias (the tendency to focus on evidence that is in line with our expectations or favoured explanation) and hindsight bias (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusions 20 . Thomas Levenson documents the example of astronomers who became convinced they had seen the fictitious planet Vulcan because their contemporary theories predicted its existence 21 . Experimenter effects are an example of this kind of bias 22 .” | claim_relations[1] |
| weight | cognitive and analytic processes that turn random patterns into false conclusions — 0.99 — “A hallmark of scientific creativity is the ability to see novel and unexpected patterns in data. John Snow's identification of links between cholera and water supply 17 , Paul Broca's work on language lateralization 18 and Jocelyn Bell Burnell's discovery of pulsars 19 are examples of breakthroughs achieved by interpreting observations in a new way. However, a major challenge for scientists is to be open to new and important insights while simultaneously avoiding being misled by our tendency to see structure in randomness. The combination of apophenia (the tendency to see patterns in random data), confirmation bias (the tendency to focus on evidence that is in line with our expectations or favoured explanation) and hindsight bias (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusions 20 . Thomas Levenson documents the example of astronomers who became convinced they had seen the fictitious planet Vulcan because their contemporary theories predicted its existence 21 . Experimenter effects are an example of this kind of bias 22 .” | claim_relations[1] |
| note | blinding analysts, participants, collectors, conditions, labels, hypotheses, or results where feasible — A concrete method intervention. — “Protecting against cognitive biases. There is a substantial literature on the difficulty of avoiding cognitive biases. An effective solution to mitigate self-deception and unwanted biases is blinding. In some research contexts, participants and data collectors can be blinded to the experimental condition that participants are assigned to, and to the research hypotheses, while the data analyst can be blinded to key parts of the data. For example, during data preparation and cleaning, the identity of experimental conditions or the variable labels can be masked so that the output is not interpretable in terms of the research hypothesis. In some physical sciences this approach has been extended to include deliberate perturbations in or masking of data to allow data preparation (for example, identification of outliers) to proceed without the analyst being able to see the corresponding results 30 . Pre-registration of the study design, primary outcome(s) and analysis plan (see ‘Promoting study pre-registration’ section’, below) is a highly effective form of blinding because the data do not exist and the outcomes are not yet known.” | claim_relations[2] |
| weight | blinding analysts, participants, collectors, conditions, labels, hypotheses, or results where feasible — 0.99 — “Protecting against cognitive biases. There is a substantial literature on the difficulty of avoiding cognitive biases. An effective solution to mitigate self-deception and unwanted biases is blinding. In some research contexts, participants and data collectors can be blinded to the experimental condition that participants are assigned to, and to the research hypotheses, while the data analyst can be blinded to key parts of the data. For example, during data preparation and cleaning, the identity of experimental conditions or the variable labels can be masked so that the output is not interpretable in terms of the research hypothesis. In some physical sciences this approach has been extended to include deliberate perturbations in or masking of data to allow data preparation (for example, identification of outliers) to proceed without the analyst being able to see the corresponding results 30 . Pre-registration of the study design, primary outcome(s) and analysis plan (see ‘Promoting study pre-registration’ section’, below) is a highly effective form of blinding because the data do not exist and the outcomes are not yet known.” | claim_relations[2] |
| note | team science and distributed collaboration for power and generalizability — The paper identifies team science as a solution to resource limits. — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | claim_relations[3] |
| weight | team science and distributed collaboration for power and generalizability — 0.95 — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | claim_relations[3] |
| note | strong pre-registration of design, outcomes, and analysis before data or outcomes are known — A direct reporting and design intervention. — “The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P -hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory research that is important for testing hypotheses, and data-contingent exploratory research that is important for generating hypotheses.” | claim_relations[4] |
| weight | strong pre-registration of design, outcomes, and analysis before data or outcomes are known — 0.99 — “The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P -hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory research that is important for testing hypotheses, and data-contingent exploratory research that is important for generating hypotheses.” | claim_relations[4] |
| note | reporting guidelines as necessary but insufficient without adoption, enforcement, and evaluation — The passage explicitly preserves value while denying sufficiency. — “The positive outcome of reporting guidelines is that they make it possible to detect and study these behaviours and their impact. Otherwise, these behaviours are simply unknowable in any systematic way. The negative outcome is the empirical evidence that reporting guidelines may be necessary, but will not alone be sufficient, to address reporting biases. The impact of guidelines and how best to optimize their use and impact will be best assessed by randomized trials (see Box 4 ).” | claim_relations[5] |
| weight | reporting guidelines as necessary but insufficient without adoption, enforcement, and evaluation — 0.99 — “The positive outcome of reporting guidelines is that they make it possible to detect and study these behaviours and their impact. Otherwise, these behaviours are simply unknowable in any systematic way. The negative outcome is the empirical evidence that reporting guidelines may be necessary, but will not alone be sufficient, to address reporting biases. The impact of guidelines and how best to optimize their use and impact will be best assessed by randomized trials (see Box 4 ).” | claim_relations[5] |
| note | transparent and accessible research content, data, methods, workflows, and interpretation — The open-science recommendation is explicit. — “Open science refers to the process of making the content and process of producing evidence and claims transparent and accessible to others. Transparency is a scientific ideal, and adding ‘open’ should therefore be redundant. In reality, science often lacks openness: many published articles are not available to people without a personal or institutional subscription, and most data, materials and code supporting research outcomes are not made accessible, for example, in a public repository (refs 63 , 64 ; Box 5 ).” | claim_relations[6] |
| weight | transparent and accessible research content, data, methods, workflows, and interpretation — 0.99 — “Open science refers to the process of making the content and process of producing evidence and claims transparent and accessible to others. Transparency is a scientific ideal, and adding ‘open’ should therefore be redundant. In reality, science often lacks openness: many published articles are not available to people without a personal or institutional subscription, and most data, materials and code supporting research outcomes are not made accessible, for example, in a public repository (refs 63 , 64 ; Box 5 ).” | claim_relations[6] |
| note | diversified pre-publication and post-publication evaluation — The paper states pace and diversity benefits. — “Both pre- and post-publication peer review mechanisms dramatically accelerate and expand the evaluation process 68 . By sharing preprints, researchers can obtain rapid feedback on their work from a diverse community, rather than waiting several months for a few reviews in the conventional, closed peer review process. Using post-publication services, reviewers can make positive and critical commentary on articles instantly, rather than relying on the laborious, uncertain and lengthy process of authoring a commentary and submitting it to the publishing journal for possible publication, eventually.” | claim_relations[7] |
| weight | diversified pre-publication and post-publication evaluation — 0.95 — “Both pre- and post-publication peer review mechanisms dramatically accelerate and expand the evaluation process 68 . By sharing preprints, researchers can obtain rapid feedback on their work from a diverse community, rather than waiting several months for a few reviews in the conventional, closed peer review process. Using post-publication services, reviewers can make positive and critical commentary on articles instantly, rather than relying on the laborious, uncertain and lengthy process of authoring a commentary and submitting it to the publishing journal for possible publication, eventually.” | claim_relations[7] |
| note | reward systems favoring positive, novel, and clean results over rigor and reproducibility — The incentive diagnosis is explicit. — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | claim_relations[8] |
| weight | reward systems favoring positive, novel, and clean results over rigor and reproducibility — 0.99 — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | claim_relations[8] |
| note | all proposed reforms as hypotheses requiring continuing evaluation for ineffectiveness or harm — The conclusion explicitly warns against assuming effectiveness. — “The challenges to reproducible science are systemic and cultural, but that does not mean they cannot be met. The measures we have described constitute practical and achievable steps toward improving rigor and reproducibility. All of them have shown some effectiveness, and are well suited to wider adoption, evaluation and improvement. Equally, these proposals are not an exhaustive list; there are many other nascent and maturing ideas for making research practices more efficient and reliable 73 . Offering a solution to a problem does not guarantee its effectiveness, and making changes to cultural norms and incentives can spur additional behavioural changes that are difficult to anticipate. Some solutions may be ineffective or even harmful to the efficiency and reliability of science, even if conceptually they appear sensible.” | claim_relations[9] |
| weight | all proposed reforms as hypotheses requiring continuing evaluation for ineffectiveness or harm — 0.99 — “The challenges to reproducible science are systemic and cultural, but that does not mean they cannot be met. The measures we have described constitute practical and achievable steps toward improving rigor and reproducibility. All of them have shown some effectiveness, and are well suited to wider adoption, evaluation and improvement. Equally, these proposals are not an exhaustive list; there are many other nascent and maturing ideas for making research practices more efficient and reliable 73 . Offering a solution to a problem does not guarantee its effectiveness, and making changes to cultural norms and incentives can spur additional behavioural changes that are difficult to anticipate. Some solutions may be ineffective or even harmful to the efficiency and reliability of science, even if conceptually they appear sensible.” | claim_relations[9] |
| note | data dredging —same-as→ P -hacking — Explicit also-known-as equivalence. — “What proportion of published research is likely to be false? Low sample size, small effect sizes, data dredging (also known as P -hacking), conflicts of interest, large numbers of scientists working competitively in silos without combining their efforts, and so on, may conspire to dramatically increase the probability that a published finding is incorrect 1 . The field of metascience — the scientific study of science itself — is flourishing and has generated substantial empirical evidence for the existence and prevalence of threats to efficiency in knowledge accumulation (refs 2 , 3 , 4 , 5 , 6 , 7 ; Fig. 1 ).” | concept_edges[0] |
| relation | data dredging —same-as→ P -hacking — Data dredging is also known as P-hacking. — “What proportion of published research is likely to be false? Low sample size, small effect sizes, data dredging (also known as P -hacking), conflicts of interest, large numbers of scientists working competitively in silos without combining their efforts, and so on, may conspire to dramatically increase the probability that a published finding is incorrect 1 . The field of metascience — the scientific study of science itself — is flourishing and has generated substantial empirical evidence for the existence and prevalence of threats to efficiency in knowledge accumulation (refs 2 , 3 , 4 , 5 , 6 , 7 ; Fig. 1 ).” | concept_edges[0] |
| note | publication bias —same-as→ file drawer problem — Explicit also-known-as equivalence. — “Promoting study pre-registration. Pre-registration of study protocols for randomized controlled trials in clinical medicine has become standard practice 46 . In its simplest form it may simply comprise the registration of the basic study design, but it can also include a detailed pre-specification of the study procedures, outcomes and statistical analysis plan. It was introduced to address two problems: publication bias and analytical flexibility (in particular outcome switching in the case of clinical medicine). Publication bias 47 , also known as the file drawer problem 48 , refers to the fact that many more studies are conducted than published. Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results 47 , 49 , 50 . The consequence is that the published literature indicates stronger evidence for findings than exists in reality. Outcome switching refers to the possibility of changing the outcomes of interest in the study depending on the observed results. A researcher may include ten variables that could be considered outcomes of the research, and — once the results are known — intentionally or unintentionally select the subset of outcomes that show statistically significant results as the outcomes of interest. The consequence is an increase in the likelihood that reported results are spurious by leveraging chance, while negative evidence gets ignored. This is one of several related research practices that can inflate spurious findings when analysis decisions are made with knowledge of the observed data, such as selection of models, exclusion rules and covariates. Such data-contingent analysis decisions constitute what has become known as P -hacking 51 , and pre-registration can protect against all of these.” | concept_edges[1] |
| relation | publication bias —same-as→ file drawer problem — Publication bias is also known as the file drawer problem. — “Promoting study pre-registration. Pre-registration of study protocols for randomized controlled trials in clinical medicine has become standard practice 46 . In its simplest form it may simply comprise the registration of the basic study design, but it can also include a detailed pre-specification of the study procedures, outcomes and statistical analysis plan. It was introduced to address two problems: publication bias and analytical flexibility (in particular outcome switching in the case of clinical medicine). Publication bias 47 , also known as the file drawer problem 48 , refers to the fact that many more studies are conducted than published. Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results 47 , 49 , 50 . The consequence is that the published literature indicates stronger evidence for findings than exists in reality. Outcome switching refers to the possibility of changing the outcomes of interest in the study depending on the observed results. A researcher may include ten variables that could be considered outcomes of the research, and — once the results are known — intentionally or unintentionally select the subset of outcomes that show statistically significant results as the outcomes of interest. The consequence is an increase in the likelihood that reported results are spurious by leveraging chance, while negative evidence gets ignored. This is one of several related research practices that can inflate spurious findings when analysis decisions are made with knowledge of the observed data, such as selection of models, exclusion rules and covariates. Such data-contingent analysis decisions constitute what has become known as P -hacking 51 , and pre-registration can protect against all of these.” | concept_edges[1] |
| note | pre-registration —is-a→ blinding — Explicit form-of classification. — “Protecting against cognitive biases. There is a substantial literature on the difficulty of avoiding cognitive biases. An effective solution to mitigate self-deception and unwanted biases is blinding. In some research contexts, participants and data collectors can be blinded to the experimental condition that participants are assigned to, and to the research hypotheses, while the data analyst can be blinded to key parts of the data. For example, during data preparation and cleaning, the identity of experimental conditions or the variable labels can be masked so that the output is not interpretable in terms of the research hypothesis. In some physical sciences this approach has been extended to include deliberate perturbations in or masking of data to allow data preparation (for example, identification of outliers) to proceed without the analyst being able to see the corresponding results 30 . Pre-registration of the study design, primary outcome(s) and analysis plan (see ‘Promoting study pre-registration’ section’, below) is a highly effective form of blinding because the data do not exist and the outcomes are not yet known.” | concept_edges[2] |
| relation | pre-registration —is-a→ blinding — Pre-registration before outcomes exist is a highly effective form of blinding. — “Protecting against cognitive biases. There is a substantial literature on the difficulty of avoiding cognitive biases. An effective solution to mitigate self-deception and unwanted biases is blinding. In some research contexts, participants and data collectors can be blinded to the experimental condition that participants are assigned to, and to the research hypotheses, while the data analyst can be blinded to key parts of the data. For example, during data preparation and cleaning, the identity of experimental conditions or the variable labels can be masked so that the output is not interpretable in terms of the research hypothesis. In some physical sciences this approach has been extended to include deliberate perturbations in or masking of data to allow data preparation (for example, identification of outliers) to proceed without the analyst being able to see the corresponding results 30 . Pre-registration of the study design, primary outcome(s) and analysis plan (see ‘Promoting study pre-registration’ section’, below) is a highly effective form of blinding because the data do not exist and the outcomes are not yet known.” | concept_edges[2] |
| audit | low statistical power —causes→ false-positive findings — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span writes 'both false-positive and false-negative results', not 'false-positive findings'. — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | concept_edges[3] |
| effect direction | low statistical power —causes→ false-positive findings — increases — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” — “Low statistical power increases the likelihood of obtaining both false-positive” | concept_edges[3] |
| note | low statistical power —causes→ false-positive findings — Explicit positive increase in outcome likelihood. — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | concept_edges[3] |
| relation | low statistical power —causes→ false-positive findings — Low statistical power increases the likelihood of false-positive findings. — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | concept_edges[3] |
| weight | low statistical power —causes→ false-positive findings — 0.75 — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | concept_edges[3] |
| effect direction | low statistical power —causes→ false-negative results — increases — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” — “Low statistical power increases the likelihood of obtaining both false-positive and false-negative” | concept_edges[4] |
| note | low statistical power —causes→ false-negative results — Explicit positive increase in outcome likelihood. — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | concept_edges[4] |
| relation | low statistical power —causes→ false-negative results — Low statistical power increases the likelihood of false-negative results. — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | concept_edges[4] |
| weight | low statistical power —causes→ false-negative results — 0.75 — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | concept_edges[4] |
| audit | P -hacking —causes→ false discovery rate — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says self-deception, uncontrolled, CAN dramatically increase the false discovery rate; hedged. — “During data analysis it can be difficult for researchers to recognize P -hacking 28 or data dredging because confirmation and hindsight biases can encourage the acceptance of outcomes that fit expectations or desires as appropriate, and the rejection of outcomes that do not as the result of suboptimal designs or analyses. Hypotheses may emerge that fit the data and are then reported without indication or recognition of their post hoc origin 7 . This, unfortunately, is not scientific discovery, but self-deception 29 . Uncontrolled, it can dramatically increase the false discovery rate. We need measures to counter the natural tendency of enthusiastic scientists who are motivated by discovery to see patterns in noise.” | concept_edges[5] |
| effect direction | P -hacking —causes→ false discovery rate — increases — “During data analysis it can be difficult for researchers to recognize P -hacking 28 or data dredging because confirmation and hindsight biases can encourage the acceptance of outcomes that fit expectations or desires as appropriate, and the rejection of outcomes that do not as the result of suboptimal designs or analyses. Hypotheses may emerge that fit the data and are then reported without indication or recognition of their post hoc origin 7 . This, unfortunately, is not scientific discovery, but self-deception 29 . Uncontrolled, it can dramatically increase the false discovery rate. We need measures to counter the natural tendency of enthusiastic scientists who are motivated by discovery to see patterns in noise.” — “it can dramatically increase the false discovery rate” | concept_edges[5] |
| note | P -hacking —causes→ false discovery rate — The antecedent of 'it' is P-hacking in the same paragraph; the direction is positive and explicit. — “During data analysis it can be difficult for researchers to recognize P -hacking 28 or data dredging because confirmation and hindsight biases can encourage the acceptance of outcomes that fit expectations or desires as appropriate, and the rejection of outcomes that do not as the result of suboptimal designs or analyses. Hypotheses may emerge that fit the data and are then reported without indication or recognition of their post hoc origin 7 . This, unfortunately, is not scientific discovery, but self-deception 29 . Uncontrolled, it can dramatically increase the false discovery rate. We need measures to counter the natural tendency of enthusiastic scientists who are motivated by discovery to see patterns in noise.” | concept_edges[5] |
| relation | P -hacking —causes→ false discovery rate — Uncontrolled P-hacking can dramatically increase the false discovery rate. — “During data analysis it can be difficult for researchers to recognize P -hacking 28 or data dredging because confirmation and hindsight biases can encourage the acceptance of outcomes that fit expectations or desires as appropriate, and the rejection of outcomes that do not as the result of suboptimal designs or analyses. Hypotheses may emerge that fit the data and are then reported without indication or recognition of their post hoc origin 7 . This, unfortunately, is not scientific discovery, but self-deception 29 . Uncontrolled, it can dramatically increase the false discovery rate. We need measures to counter the natural tendency of enthusiastic scientists who are motivated by discovery to see patterns in noise.” | concept_edges[5] |
| weight | P -hacking —causes→ false discovery rate — 0.8 — “During data analysis it can be difficult for researchers to recognize P -hacking 28 or data dredging because confirmation and hindsight biases can encourage the acceptance of outcomes that fit expectations or desires as appropriate, and the rejection of outcomes that do not as the result of suboptimal designs or analyses. Hypotheses may emerge that fit the data and are then reported without indication or recognition of their post hoc origin 7 . This, unfortunately, is not scientific discovery, but self-deception 29 . Uncontrolled, it can dramatically increase the false discovery rate. We need measures to counter the natural tendency of enthusiastic scientists who are motivated by discovery to see patterns in noise.” | concept_edges[5] |
| note | methodology applied —part-of→ supporting evidence — Direct 'includes' containment. — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concept_edges[6] |
| relation | methodology applied —part-of→ supporting evidence — The evidence supporting scientific claims includes the methodology applied. — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concept_edges[6] |
| note | data acquired —part-of→ supporting evidence — Direct 'includes' containment. — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concept_edges[7] |
| relation | data acquired —part-of→ supporting evidence — The evidence supporting scientific claims includes the data acquired. — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concept_edges[7] |
| note | process of methodology implementation —part-of→ supporting evidence — Direct 'includes' containment. — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concept_edges[8] |
| relation | process of methodology implementation —part-of→ supporting evidence — The evidence supporting scientific claims includes the process of methodology implementation. — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concept_edges[8] |
| note | data analysis —part-of→ supporting evidence — Direct 'includes' containment. — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concept_edges[9] |
| relation | data analysis —part-of→ supporting evidence — The evidence supporting scientific claims includes data analysis. — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concept_edges[9] |
| note | outcome interpretation —part-of→ supporting evidence — Direct 'includes' containment. — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concept_edges[10] |
| relation | outcome interpretation —part-of→ supporting evidence — The evidence supporting scientific claims includes outcome interpretation. — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concept_edges[10] |
| effect direction | open data badge —causes→ data sharing — increases — “The Center for Open Science has suggested that journals assign a badge to articles with open data (as well as to other open practices such as pre-registration and open materials). The main purpose of the badges is to signal that the journal values these practices. The journal Psychological Science has adopted these badges, and there is evidence that the open data badge has had a positive effect, increasing data sharing by more than tenfold ( Fig. 2 ).” — “increasing data sharing by more than tenfold” | concept_edges[11] |
| note | open data badge —causes→ data sharing — Observed positive intervention effect. — “The Center for Open Science has suggested that journals assign a badge to articles with open data (as well as to other open practices such as pre-registration and open materials). The main purpose of the badges is to signal that the journal values these practices. The journal Psychological Science has adopted these badges, and there is evidence that the open data badge has had a positive effect, increasing data sharing by more than tenfold ( Fig. 2 ).” | concept_edges[11] |
| relation | open data badge —causes→ data sharing — The open data badge had a positive effect, increasing data sharing by more than tenfold. — “The Center for Open Science has suggested that journals assign a badge to articles with open data (as well as to other open practices such as pre-registration and open materials). The main purpose of the badges is to signal that the journal values these practices. The journal Psychological Science has adopted these badges, and there is evidence that the open data badge has had a positive effect, increasing data sharing by more than tenfold ( Fig. 2 ).” | concept_edges[11] |
| weight | open data badge —causes→ data sharing — 0.9 — “The Center for Open Science has suggested that journals assign a badge to articles with open data (as well as to other open practices such as pre-registration and open materials). The main purpose of the badges is to signal that the journal values these practices. The journal Psychological Science has adopted these badges, and there is evidence that the open data badge has had a positive effect, increasing data sharing by more than tenfold ( Fig. 2 ).” | concept_edges[11] |
| effect direction | incentives —causes→ false positives — increases — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” — “These incentives ultimately increase the likelihood of false positives” | concept_edges[12] |
| note | incentives —causes→ false positives — Explicit positive increase in outcome likelihood. — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | concept_edges[12] |
| relation | incentives —causes→ false positives — Publication incentives increase the likelihood of false positives in the published literature. — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | concept_edges[12] |
| weight | incentives —causes→ false positives — 0.75 — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | concept_edges[12] |
| audit | incentive structures —causes→ researcher behaviour — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says the model SUGGESTS altering incentive structures WOULD encourage a change in behaviour; hedged. — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | concept_edges[13] |
| note | incentive structures —causes→ researcher behaviour — Explicit modal effect of a proposed intervention. — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | concept_edges[13] |
| relation | incentive structures —causes→ researcher behaviour — Altering incentive structures would encourage a change in researcher behaviour. — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | concept_edges[13] |
| weight | incentive structures —causes→ researcher behaviour — 0.6 — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | concept_edges[13] |
| relation | pre- and post-publication peer review mechanisms → evaluation process — Pre- and post-publication peer review mechanisms accelerate and expand the evaluation process. — “Both pre- and post-publication peer review mechanisms dramatically accelerate and expand the evaluation process 68 . By sharing preprints, researchers can obtain rapid feedback on their work from a diverse community, rather than waiting several months for a few reviews in the conventional, closed peer review process. Using post-publication services, reviewers can make positive and critical commentary on articles instantly, rather than relying on the laborious, uncertain and lengthy process of authoring a commentary and submitting it to the publishing journal for possible publication, eventually.” | unhoused_relations[0] |
| weight | pre- and post-publication peer review mechanisms → evaluation process — 0.85 — “Both pre- and post-publication peer review mechanisms dramatically accelerate and expand the evaluation process 68 . By sharing preprints, researchers can obtain rapid feedback on their work from a diverse community, rather than waiting several months for a few reviews in the conventional, closed peer review process. Using post-publication services, reviewers can make positive and critical commentary on articles instantly, rather than relying on the laborious, uncertain and lengthy process of authoring a commentary and submitting it to the publishing journal for possible publication, eventually.” | unhoused_relations[0] |
| relation | metascience → scientific study of science itself — Metascience is the scientific study of science itself. — “What proportion of published research is likely to be false? Low sample size, small effect sizes, data dredging (also known as P -hacking), conflicts of interest, large numbers of scientists working competitively in silos without combining their efforts, and so on, may conspire to dramatically increase the probability that a published finding is incorrect 1 . The field of metascience — the scientific study of science itself — is flourishing and has generated substantial empirical evidence for the existence and prevalence of threats to efficiency in knowledge accumulation (refs 2 , 3 , 4 , 5 , 6 , 7 ; Fig. 1 ).” | unhoused_relations[1] |
| relation | lack of replication → robustness of published research — Lack of replication is one of several threats that undermine published-research robustness. — “Threats to reproducible science. An idealized version of the hypothetico-deductive model of the scientific method is shown. Various potential threats to this model exist (indicated in red), including lack of replication 5 , hypothesizing after the results are known (HARKing) 7 , poor study design, low statistical power 2 , analytical flexibility 51 , P -hacking 4 , publication bias 3 and lack of data sharing 6 . Together these will serve to undermine the robustness of published research, and may also impact on the ability of science to self-correct.” | unhoused_relations[2] |
| weight | lack of replication → robustness of published research — 0.8 — “Threats to reproducible science. An idealized version of the hypothetico-deductive model of the scientific method is shown. Various potential threats to this model exist (indicated in red), including lack of replication 5 , hypothesizing after the results are known (HARKing) 7 , poor study design, low statistical power 2 , analytical flexibility 51 , P -hacking 4 , publication bias 3 and lack of data sharing 6 . Together these will serve to undermine the robustness of published research, and may also impact on the ability of science to self-correct.” | unhoused_relations[2] |
| relation | apophenia → false conclusions — Apophenia combines with confirmation and hindsight bias to lead researchers toward false conclusions. — “A hallmark of scientific creativity is the ability to see novel and unexpected patterns in data. John Snow's identification of links between cholera and water supply 17 , Paul Broca's work on language lateralization 18 and Jocelyn Bell Burnell's discovery of pulsars 19 are examples of breakthroughs achieved by interpreting observations in a new way. However, a major challenge for scientists is to be open to new and important insights while simultaneously avoiding being misled by our tendency to see structure in randomness. The combination of apophenia (the tendency to see patterns in random data), confirmation bias (the tendency to focus on evidence that is in line with our expectations or favoured explanation) and hindsight bias (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusions 20 . Thomas Levenson documents the example of astronomers who became convinced they had seen the fictitious planet Vulcan because their contemporary theories predicted its existence 21 . Experimenter effects are an example of this kind of bias 22 .” | unhoused_relations[3] |
| weight | apophenia → false conclusions — 0.8 — “A hallmark of scientific creativity is the ability to see novel and unexpected patterns in data. John Snow's identification of links between cholera and water supply 17 , Paul Broca's work on language lateralization 18 and Jocelyn Bell Burnell's discovery of pulsars 19 are examples of breakthroughs achieved by interpreting observations in a new way. However, a major challenge for scientists is to be open to new and important insights while simultaneously avoiding being misled by our tendency to see structure in randomness. The combination of apophenia (the tendency to see patterns in random data), confirmation bias (the tendency to focus on evidence that is in line with our expectations or favoured explanation) and hindsight bias (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusions 20 . Thomas Levenson documents the example of astronomers who became convinced they had seen the fictitious planet Vulcan because their contemporary theories predicted its existence 21 . Experimenter effects are an example of this kind of bias 22 .” | unhoused_relations[3] |
| relation | analytical flexibility → false-positive findings — Many reasonable pipelines on high-dimensional data make false-positive findings highly likely. — “Over-interpretation of noise is facilitated by the extent to which data analysis is rapid, flexible and automated 23 . In a high-dimensional dataset, there may be hundreds or thousands of reasonable alternative approaches to analysing the same data 24 , 25 . For example, in a systematic review of functional magnetic resonance imaging (fMRI) studies, Carp showed that there were almost as many unique analytical pipelines as there were studies 26 . If several thousand potential analytical pipelines can be applied to high-dimensional data, the generation of false-positive findings is highly likely. For example, applying almost 7,000 analytical pipelines to a single fMRI dataset resulted in over 90% of brain voxels showing significant activation in at least one analysis 27 .” | unhoused_relations[4] |
| weight | analytical flexibility → false-positive findings — 0.85 — “Over-interpretation of noise is facilitated by the extent to which data analysis is rapid, flexible and automated 23 . In a high-dimensional dataset, there may be hundreds or thousands of reasonable alternative approaches to analysing the same data 24 , 25 . For example, in a systematic review of functional magnetic resonance imaging (fMRI) studies, Carp showed that there were almost as many unique analytical pipelines as there were studies 26 . If several thousand potential analytical pipelines can be applied to high-dimensional data, the generation of false-positive findings is highly likely. For example, applying almost 7,000 analytical pipelines to a single fMRI dataset resulted in over 90% of brain voxels showing significant activation in at least one analysis 27 .” | unhoused_relations[4] |
| relation | blinding → cognitive biases — Blinding mitigates self-deception and unwanted cognitive biases. — “Protecting against cognitive biases. There is a substantial literature on the difficulty of avoiding cognitive biases. An effective solution to mitigate self-deception and unwanted biases is blinding. In some research contexts, participants and data collectors can be blinded to the experimental condition that participants are assigned to, and to the research hypotheses, while the data analyst can be blinded to key parts of the data. For example, during data preparation and cleaning, the identity of experimental conditions or the variable labels can be masked so that the output is not interpretable in terms of the research hypothesis. In some physical sciences this approach has been extended to include deliberate perturbations in or masking of data to allow data preparation (for example, identification of outliers) to proceed without the analyst being able to see the corresponding results 30 . Pre-registration of the study design, primary outcome(s) and analysis plan (see ‘Promoting study pre-registration’ section’, below) is a highly effective form of blinding because the data do not exist and the outcomes are not yet known.” | unhoused_relations[5] |
| weight | blinding → cognitive biases — 0.9 — “Protecting against cognitive biases. There is a substantial literature on the difficulty of avoiding cognitive biases. An effective solution to mitigate self-deception and unwanted biases is blinding. In some research contexts, participants and data collectors can be blinded to the experimental condition that participants are assigned to, and to the research hypotheses, while the data analyst can be blinded to key parts of the data. For example, during data preparation and cleaning, the identity of experimental conditions or the variable labels can be masked so that the output is not interpretable in terms of the research hypothesis. In some physical sciences this approach has been extended to include deliberate perturbations in or masking of data to allow data preparation (for example, identification of outliers) to proceed without the analyst being able to see the corresponding results 30 . Pre-registration of the study design, primary outcome(s) and analysis plan (see ‘Promoting study pre-registration’ section’, below) is a highly effective form of blinding because the data do not exist and the outcomes are not yet known.” | unhoused_relations[5] |
| relation | methodological training → Common misperceptions — Improved methodological training can address common statistical misperceptions. — “Improving methodological training. Research design and statistical analysis are mutually dependent. Common misperceptions, such as the interpretation of P values 31 , limitations of null-hypothesis significance testing 32 , the meaning and importance of statistical power 2 , the accuracy of reported effect sizes 33 , and the likelihood that a sample size that generated a statistically significant finding will also be adequate to replicate a true finding 34 , could all be addressed through improved statistical training. Similarly, basic design principles are important, such as blinding to reduce experimenter bias, randomization or counterbalancing to control for confounding, and the use of within-subjects designs, where possible, to maximize power. However, integrative training in research practices that can protect oneself against cognitive biases and the effects of distorted incentives is arguably more important. Moreover, statistical and methodological best practices are under constant revision and improvement, so that senior as well as junior researchers need continuing methodological education, not least because much training of early-career researchers is informal and flows from their supervisor or mentor. A failure to adopt advances in methodology — such as the very slow progress in increasing statistical power 35 , 36 — may be partly a function of failing to inculcate a continuing professional education and development ethic.” | unhoused_relations[6] |
| weight | methodological training → Common misperceptions — 0.8 — “Improving methodological training. Research design and statistical analysis are mutually dependent. Common misperceptions, such as the interpretation of P values 31 , limitations of null-hypothesis significance testing 32 , the meaning and importance of statistical power 2 , the accuracy of reported effect sizes 33 , and the likelihood that a sample size that generated a statistically significant finding will also be adequate to replicate a true finding 34 , could all be addressed through improved statistical training. Similarly, basic design principles are important, such as blinding to reduce experimenter bias, randomization or counterbalancing to control for confounding, and the use of within-subjects designs, where possible, to maximize power. However, integrative training in research practices that can protect oneself against cognitive biases and the effects of distorted incentives is arguably more important. Moreover, statistical and methodological best practices are under constant revision and improvement, so that senior as well as junior researchers need continuing methodological education, not least because much training of early-career researchers is informal and flows from their supervisor or mentor. A failure to adopt advances in methodology — such as the very slow progress in increasing statistical power 35 , 36 — may be partly a function of failing to inculcate a continuing professional education and development ethic.” | unhoused_relations[6] |
| relation | independent methodological support → conflicts of interest — Independent methodological support may mitigate financial and non-financial conflicts of interest. — “Implementing independent methodological support. The need for independent methodological support is well-established in some areas — many clinical trials, for example, have multidisciplinary trial steering committees to provide advice and oversee the design and conduct of the trial. The need for these committees grew out of the well-understood financial conflicts of interest that exist in many clinical trials. The sponsor of a trial may be the company manufacturing the product, and any intentional or unintentional influence can distort the study design, analysis and interpretation of results for the ultimate financial benefit of the manufacturer at the cost of the accuracy of the science and the health benefit to the consumers 37 , 38 . Non-financial conflicts of interest also exist, such as the beliefs and preconceptions of individual scientists and the stakes that researchers have in obtaining publishable results in order to progress their career 39 , 40 . Including independent researchers (particularly methodologists with no personal investment in a research topic) in the design, monitoring, analysis or interpretation of research outcomes may mitigate some of those influences, and can be done either at the level of the individual research project or through a process facilitated by a funding agency (see Box 1 ).” | unhoused_relations[7] |
| weight | independent methodological support → conflicts of interest — 0.65 — “Implementing independent methodological support. The need for independent methodological support is well-established in some areas — many clinical trials, for example, have multidisciplinary trial steering committees to provide advice and oversee the design and conduct of the trial. The need for these committees grew out of the well-understood financial conflicts of interest that exist in many clinical trials. The sponsor of a trial may be the company manufacturing the product, and any intentional or unintentional influence can distort the study design, analysis and interpretation of results for the ultimate financial benefit of the manufacturer at the cost of the accuracy of the science and the health benefit to the consumers 37 , 38 . Non-financial conflicts of interest also exist, such as the beliefs and preconceptions of individual scientists and the stakes that researchers have in obtaining publishable results in order to progress their career 39 , 40 . Including independent researchers (particularly methodologists with no personal investment in a research topic) in the design, monitoring, analysis or interpretation of research outcomes may mitigate some of those influences, and can be done either at the level of the individual research project or through a process facilitated by a funding agency (see Box 1 ).” | unhoused_relations[7] |
| relation | team science → lack of resources to improve power — Team science is presented as a solution to individual investigators' limited resources. — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | unhoused_relations[8] |
| weight | team science → lack of resources to improve power — 0.9 — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | unhoused_relations[8] |
| relation | distributed collaboration → high-powered designs — Distributed collaboration across sites facilitates high-powered designs. — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | unhoused_relations[9] |
| weight | distributed collaboration → high-powered designs — 0.85 — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | unhoused_relations[9] |
| relation | pre-registration → publication bias — Pre-registration addresses publication bias by making research discoverable regardless of publication outcome. — “The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P -hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory research that is important for testing hypotheses, and data-contingent exploratory research that is important for generating hypotheses.” | unhoused_relations[10] |
| weight | pre-registration → publication bias — 0.9 — “The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P -hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory research that is important for testing hypotheses, and data-contingent exploratory research that is important for generating hypotheses.” | unhoused_relations[10] |
| relation | pre-registration → outcome switching — Pre-registration addresses outcome switching by fixing decisions before data are observed. — “The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P -hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory research that is important for testing hypotheses, and data-contingent exploratory research that is important for generating hypotheses.” | unhoused_relations[11] |
| weight | pre-registration → outcome switching — 0.9 — “The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P -hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory research that is important for testing hypotheses, and data-contingent exploratory research that is important for generating hypotheses.” | unhoused_relations[11] |
| relation | pre-registration → P -hacking — Pre-registration protects against data-contingent P-hacking. — “Promoting study pre-registration. Pre-registration of study protocols for randomized controlled trials in clinical medicine has become standard practice 46 . In its simplest form it may simply comprise the registration of the basic study design, but it can also include a detailed pre-specification of the study procedures, outcomes and statistical analysis plan. It was introduced to address two problems: publication bias and analytical flexibility (in particular outcome switching in the case of clinical medicine). Publication bias 47 , also known as the file drawer problem 48 , refers to the fact that many more studies are conducted than published. Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results 47 , 49 , 50 . The consequence is that the published literature indicates stronger evidence for findings than exists in reality. Outcome switching refers to the possibility of changing the outcomes of interest in the study depending on the observed results. A researcher may include ten variables that could be considered outcomes of the research, and — once the results are known — intentionally or unintentionally select the subset of outcomes that show statistically significant results as the outcomes of interest. The consequence is an increase in the likelihood that reported results are spurious by leveraging chance, while negative evidence gets ignored. This is one of several related research practices that can inflate spurious findings when analysis decisions are made with knowledge of the observed data, such as selection of models, exclusion rules and covariates. Such data-contingent analysis decisions constitute what has become known as P -hacking 51 , and pre-registration can protect against all of these.” | unhoused_relations[12] |
| weight | pre-registration → P -hacking — 0.9 — “Promoting study pre-registration. Pre-registration of study protocols for randomized controlled trials in clinical medicine has become standard practice 46 . In its simplest form it may simply comprise the registration of the basic study design, but it can also include a detailed pre-specification of the study procedures, outcomes and statistical analysis plan. It was introduced to address two problems: publication bias and analytical flexibility (in particular outcome switching in the case of clinical medicine). Publication bias 47 , also known as the file drawer problem 48 , refers to the fact that many more studies are conducted than published. Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results 47 , 49 , 50 . The consequence is that the published literature indicates stronger evidence for findings than exists in reality. Outcome switching refers to the possibility of changing the outcomes of interest in the study depending on the observed results. A researcher may include ten variables that could be considered outcomes of the research, and — once the results are known — intentionally or unintentionally select the subset of outcomes that show statistically significant results as the outcomes of interest. The consequence is an increase in the likelihood that reported results are spurious by leveraging chance, while negative evidence gets ignored. This is one of several related research practices that can inflate spurious findings when analysis decisions are made with knowledge of the observed data, such as selection of models, exclusion rules and covariates. Such data-contingent analysis decisions constitute what has become known as P -hacking 51 , and pre-registration can protect against all of these.” | unhoused_relations[12] |
| relation | pre-registration → confirmatory research — Pre-registration makes confirmatory research distinguishable from exploratory research. — “The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P -hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory research that is important for testing hypotheses, and data-contingent exploratory research that is important for generating hypotheses.” | unhoused_relations[13] |
| weight | pre-registration → confirmatory research — 0.95 — “The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P -hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory research that is important for testing hypotheses, and data-contingent exploratory research that is important for generating hypotheses.” | unhoused_relations[13] |
| relation | reporting guidelines → research quality — Improved reporting may be insufficient on its own to maximize research quality. — “Improving the quality of reporting. Pre-registration will improve discoverability of research, but discoverability does not guarantee usability. Poor usability reflects difficulty in evaluating what was done, in reusing the methodology to assess reproducibility, and in incorporating the evidence into systematic reviews and meta-analyses. Improving the quality and transparency in the reporting of research is necessary to address this. The Transparency and Openness Promotion (TOP) guidelines offer standards as a basis for journals and funders to incentivize or require greater transparency in planning and reporting of research 54 . TOP provides principles for how transparency and usability can be increased, while other guidelines provide concrete steps for how to maximize the quality of reporting in particular areas. For example, the Consolidated Standards of Reporting Trials (CONSORT) statement provides guidance for clear, complete and accurate reporting of randomized controlled trials 55 – 57 . Over 300 reporting guidelines now exist for observational studies, prognostic studies, predictive models, diagnostic tests, systematic reviews and meta-analyses in humans, a large variety of studies using different laboratory methods, and animal studies. The Equator Network ( http://www.equator-network.org/ ) aggregates these guidelines to improve discoverability 58 . There are also guidelines for improving the reporting of research planning; for example, the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement for reporting of systematic reviews and meta-analyses 59 , and PRISMA-P for protocols of systematic reviews 60 . The Preregistration Challenge workflow and the pre-registration recipe for social-behavioural research 61 also illustrate guidelines for reporting research plans.” | unhoused_relations[14] |
| weight | reporting guidelines → research quality — 0.8 — “Improving the quality of reporting. Pre-registration will improve discoverability of research, but discoverability does not guarantee usability. Poor usability reflects difficulty in evaluating what was done, in reusing the methodology to assess reproducibility, and in incorporating the evidence into systematic reviews and meta-analyses. Improving the quality and transparency in the reporting of research is necessary to address this. The Transparency and Openness Promotion (TOP) guidelines offer standards as a basis for journals and funders to incentivize or require greater transparency in planning and reporting of research 54 . TOP provides principles for how transparency and usability can be increased, while other guidelines provide concrete steps for how to maximize the quality of reporting in particular areas. For example, the Consolidated Standards of Reporting Trials (CONSORT) statement provides guidance for clear, complete and accurate reporting of randomized controlled trials 55 – 57 . Over 300 reporting guidelines now exist for observational studies, prognostic studies, predictive models, diagnostic tests, systematic reviews and meta-analyses in humans, a large variety of studies using different laboratory methods, and animal studies. The Equator Network ( http://www.equator-network.org/ ) aggregates these guidelines to improve discoverability 58 . There are also guidelines for improving the reporting of research planning; for example, the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement for reporting of systematic reviews and meta-analyses 59 , and PRISMA-P for protocols of systematic reviews 60 . The Preregistration Challenge workflow and the pre-registration recipe for social-behavioural research 61 also illustrate guidelines for reporting research plans.” | unhoused_relations[14] |
| relation | Registered Reports → publication bias — Registered Reports prevent publication bias by accepting articles before results are known. — “The Registered Reports (RR) initiative seeks to eliminate various forms of bias in hypothesis-driven research 52 , 53 , and in particular, the evaluation of a study based on the results. Unlike conventional journal articles, RRs split the peer review process into two stages, before and after results are known. At the first stage, reviewers and editors assess a detailed protocol that includes the study rationale, procedure and a detailed analysis plan. Following favourable reviews (and probably revision to meet strict methodological standards), the journal offers in-principle acceptance: publication of study outcomes is guaranteed provided the authors adhere to the approved protocol, the study meets pre-specified quality checks, and conclusions are appropriately evidence-bound. Once the study is completed, the authors resubmit a complete manuscript that includes the results and discussion. The article is published at the end of this two-stage process. By accepting articles before results are known, RRs prevent publication bias. By reviewing the hypotheses and analysis plans in advance, RRs should also help neutralize P- hacking and HARKing (hypothesizing after the results are known) by authors, and CARKing (critiquing after the results are known) by reviewers with their own investments in the research outcomes, although empirical evidence will be required to confirm that this is the case.” | unhoused_relations[15] |
| weight | Registered Reports → publication bias — 0.95 — “The Registered Reports (RR) initiative seeks to eliminate various forms of bias in hypothesis-driven research 52 , 53 , and in particular, the evaluation of a study based on the results. Unlike conventional journal articles, RRs split the peer review process into two stages, before and after results are known. At the first stage, reviewers and editors assess a detailed protocol that includes the study rationale, procedure and a detailed analysis plan. Following favourable reviews (and probably revision to meet strict methodological standards), the journal offers in-principle acceptance: publication of study outcomes is guaranteed provided the authors adhere to the approved protocol, the study meets pre-specified quality checks, and conclusions are appropriately evidence-bound. Once the study is completed, the authors resubmit a complete manuscript that includes the results and discussion. The article is published at the end of this two-stage process. By accepting articles before results are known, RRs prevent publication bias. By reviewing the hypotheses and analysis plans in advance, RRs should also help neutralize P- hacking and HARKing (hypothesizing after the results are known) by authors, and CARKing (critiquing after the results are known) by reviewers with their own investments in the research outcomes, although empirical evidence will be required to confirm that this is the case.” | unhoused_relations[15] |
| relation | Registered Reports → P -hacking — Advance review in Registered Reports should help neutralize P-hacking. — “The Registered Reports (RR) initiative seeks to eliminate various forms of bias in hypothesis-driven research 52 , 53 , and in particular, the evaluation of a study based on the results. Unlike conventional journal articles, RRs split the peer review process into two stages, before and after results are known. At the first stage, reviewers and editors assess a detailed protocol that includes the study rationale, procedure and a detailed analysis plan. Following favourable reviews (and probably revision to meet strict methodological standards), the journal offers in-principle acceptance: publication of study outcomes is guaranteed provided the authors adhere to the approved protocol, the study meets pre-specified quality checks, and conclusions are appropriately evidence-bound. Once the study is completed, the authors resubmit a complete manuscript that includes the results and discussion. The article is published at the end of this two-stage process. By accepting articles before results are known, RRs prevent publication bias. By reviewing the hypotheses and analysis plans in advance, RRs should also help neutralize P- hacking and HARKing (hypothesizing after the results are known) by authors, and CARKing (critiquing after the results are known) by reviewers with their own investments in the research outcomes, although empirical evidence will be required to confirm that this is the case.” | unhoused_relations[16] |
| weight | Registered Reports → P -hacking — 0.8 — “The Registered Reports (RR) initiative seeks to eliminate various forms of bias in hypothesis-driven research 52 , 53 , and in particular, the evaluation of a study based on the results. Unlike conventional journal articles, RRs split the peer review process into two stages, before and after results are known. At the first stage, reviewers and editors assess a detailed protocol that includes the study rationale, procedure and a detailed analysis plan. Following favourable reviews (and probably revision to meet strict methodological standards), the journal offers in-principle acceptance: publication of study outcomes is guaranteed provided the authors adhere to the approved protocol, the study meets pre-specified quality checks, and conclusions are appropriately evidence-bound. Once the study is completed, the authors resubmit a complete manuscript that includes the results and discussion. The article is published at the end of this two-stage process. By accepting articles before results are known, RRs prevent publication bias. By reviewing the hypotheses and analysis plans in advance, RRs should also help neutralize P- hacking and HARKing (hypothesizing after the results are known) by authors, and CARKing (critiquing after the results are known) by reviewers with their own investments in the research outcomes, although empirical evidence will be required to confirm that this is the case.” | unhoused_relations[16] |
| relation | Registered Reports → exploratory research — Registered Reports permit unregistered exploration when it is transparently labeled post hoc. — “Perhaps the most commonly voiced objection to RRs is that the format somehow limits exploration or creativity by requiring authors to adhere to a pre-specified methodology. However, RRs place no restrictions on creative analysis practices or serendipity. Authors are free to report the outcomes of any unregistered exploratory analyses, provided such tests are clearly labelled as post hoc . Thus, the sole requirement is that exploratory outcomes are identified transparently as exploratory (for a list of frequently asked questions see https://cos.io/rr/#faq ). Of course, RRs are not intended for research that is solely exploratory.” | unhoused_relations[17] |
| weight | Registered Reports → exploratory research — 0.95 — “Perhaps the most commonly voiced objection to RRs is that the format somehow limits exploration or creativity by requiring authors to adhere to a pre-specified methodology. However, RRs place no restrictions on creative analysis practices or serendipity. Authors are free to report the outcomes of any unregistered exploratory analyses, provided such tests are clearly labelled as post hoc . Thus, the sole requirement is that exploratory outcomes are identified transparently as exploratory (for a list of frequently asked questions see https://cos.io/rr/#faq ). Of course, RRs are not intended for research that is solely exploratory.” | unhoused_relations[17] |
| relation | open science → transparency — Open science makes the production of evidence and claims transparent and accessible. — “Open science refers to the process of making the content and process of producing evidence and claims transparent and accessible to others. Transparency is a scientific ideal, and adding ‘open’ should therefore be redundant. In reality, science often lacks openness: many published articles are not available to people without a personal or institutional subscription, and most data, materials and code supporting research outcomes are not made accessible, for example, in a public repository (refs 63 , 64 ; Box 5 ).” | unhoused_relations[18] |
| relation | transparency → trust — Transparency is superior to credibility based only on trust in authority. — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | unhoused_relations[19] |
| relation | public repositories → accountability — Sharing data in public repositories offers accountability, longevity, efficiency, and quality advantages. — “Sharing data in public repositories offers field-wide advantages in terms of accountability, data longevity, efficiency and quality (for example, reanalyses may detect crucial mistakes or even data fabrication) 84 . Unfortunately, many scientific disciplines, including most of those devoted to the study of human behaviour, do not have a culture that values open data 6 . In the past, data sharing has rarely been enforced or facilitated. Recent initiatives, however, aim to change the normative culture. Hopefully, these initiatives will change the culture on data sharing. Once accepted as the norm, we doubt that data sharing will ever go out of fashion.” | unhoused_relations[20] |
| weight | public repositories → accountability — 0.9 — “Sharing data in public repositories offers field-wide advantages in terms of accountability, data longevity, efficiency and quality (for example, reanalyses may detect crucial mistakes or even data fabrication) 84 . Unfortunately, many scientific disciplines, including most of those devoted to the study of human behaviour, do not have a culture that values open data 6 . In the past, data sharing has rarely been enforced or facilitated. Recent initiatives, however, aim to change the normative culture. Hopefully, these initiatives will change the culture on data sharing. Once accepted as the norm, we doubt that data sharing will ever go out of fashion.” | unhoused_relations[20] |
| relation | Full disclosure → adequately evaluate results — Full disclosure is needed so readers can adequately evaluate how results were obtained. — “Full disclosure refers to the process of describing in full the study design and data collected that underlie the results reported, rather than a curated version of the design, and/or a subset of the data collected. The need for disclosure is clear: in order to adequately evaluate results we need to know how they were obtained. For example, the informational value of a dependent variable exhibiting an effect of interest is different if only one variable was collected or if fifteen were. The probability of a single variable achieving P < 0.05 just by chance is 5%, but the probability of one of fifteen variables achieving P < 0.05 is 54% 1 . It is obvious that cherry-picking one from fifteen variables invalidates the results unless it is clear that this has happened. If readers know, then they can adjust their interpretation accordingly. From this simple fact it follows that if authors do not tell us whether they collected one or fifteen variables readers cannot evaluate their research 51 .” | unhoused_relations[21] |
| weight | Full disclosure → adequately evaluate results — 0.95 — “Full disclosure refers to the process of describing in full the study design and data collected that underlie the results reported, rather than a curated version of the design, and/or a subset of the data collected. The need for disclosure is clear: in order to adequately evaluate results we need to know how they were obtained. For example, the informational value of a dependent variable exhibiting an effect of interest is different if only one variable was collected or if fifteen were. The probability of a single variable achieving P < 0.05 just by chance is 5%, but the probability of one of fifteen variables achieving P < 0.05 is 54% 1 . It is obvious that cherry-picking one from fifteen variables invalidates the results unless it is clear that this has happened. If readers know, then they can adjust their interpretation accordingly. From this simple fact it follows that if authors do not tell us whether they collected one or fifteen variables readers cannot evaluate their research 51 .” | unhoused_relations[21] |
| relation | peer review → dissemination — Scientific publishing historically confounded peer-review evaluation with dissemination. — “Diversifying peer review. For most of the history of scientific publishing, two functions have been confounded — evaluation and dissemination. Journals have provided dissemination via sorting and delivering content to the research community, and gatekeeping via peer review to determine what is worth disseminating. However, with the advent of the internet, individual researchers are no longer dependent on publishers to bind, print and mail their research to subscribers. Dissemination is now easy and can be controlled by researchers themselves. For example, preprint services (arXiv for some physical sciences, bioRxiv and PeerJ for the life sciences, engrXiv for engineering, PsyArXiv for psychology, and SocArXiv and the Social Science Research Network (SSRN) for the social sciences) facilitate easy sharing, sorting and discovery of research prior to publication. This dramatically accelerates the dissemination of information to the research community.” | unhoused_relations[22] |
| weight | peer review → dissemination — 0.95 — “Diversifying peer review. For most of the history of scientific publishing, two functions have been confounded — evaluation and dissemination. Journals have provided dissemination via sorting and delivering content to the research community, and gatekeeping via peer review to determine what is worth disseminating. However, with the advent of the internet, individual researchers are no longer dependent on publishers to bind, print and mail their research to subscribers. Dissemination is now easy and can be controlled by researchers themselves. For example, preprint services (arXiv for some physical sciences, bioRxiv and PeerJ for the life sciences, engrXiv for engineering, PsyArXiv for psychology, and SocArXiv and the Social Science Research Network (SSRN) for the social sciences) facilitate easy sharing, sorting and discovery of research prior to publication. This dramatically accelerates the dissemination of information to the research community.” | unhoused_relations[22] |
| relation | preprint services → dissemination — Preprint services facilitate rapid sharing, sorting, and discovery before publication. — “Diversifying peer review. For most of the history of scientific publishing, two functions have been confounded — evaluation and dissemination. Journals have provided dissemination via sorting and delivering content to the research community, and gatekeeping via peer review to determine what is worth disseminating. However, with the advent of the internet, individual researchers are no longer dependent on publishers to bind, print and mail their research to subscribers. Dissemination is now easy and can be controlled by researchers themselves. For example, preprint services (arXiv for some physical sciences, bioRxiv and PeerJ for the life sciences, engrXiv for engineering, PsyArXiv for psychology, and SocArXiv and the Social Science Research Network (SSRN) for the social sciences) facilitate easy sharing, sorting and discovery of research prior to publication. This dramatically accelerates the dissemination of information to the research community.” | unhoused_relations[23] |
| weight | preprint services → dissemination — 0.9 — “Diversifying peer review. For most of the history of scientific publishing, two functions have been confounded — evaluation and dissemination. Journals have provided dissemination via sorting and delivering content to the research community, and gatekeeping via peer review to determine what is worth disseminating. However, with the advent of the internet, individual researchers are no longer dependent on publishers to bind, print and mail their research to subscribers. Dissemination is now easy and can be controlled by researchers themselves. For example, preprint services (arXiv for some physical sciences, bioRxiv and PeerJ for the life sciences, engrXiv for engineering, PsyArXiv for psychology, and SocArXiv and the Social Science Research Network (SSRN) for the social sciences) facilitate easy sharing, sorting and discovery of research prior to publication. This dramatically accelerates the dissemination of information to the research community.” | unhoused_relations[23] |
| relation | Positive, novel and clean results → publication — Positive, novel, clean results are more likely to be published than negative results and replications. — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | unhoused_relations[24] |
| weight | Positive, novel and clean results → publication — 0.9 — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | unhoused_relations[24] |
| relation | incentive structures → scientific value of research — Changing incentive structures may improve scientific value by changing researcher behaviour. — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | unhoused_relations[25] |
| weight | incentive structures → scientific value of research — 0.6 — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | unhoused_relations[25] |
| relation | rigorous, transparent and reproducible research → reward structures — Coordinated stakeholders can align reward structures with rigorous and reproducible practice. — “Funders, publishers, societies, institutions, editors, reviewers and authors all contribute to the cultural norms that create and sustain dysfunctional incentives. Changing the incentives is therefore a problem that requires a coordinated effort by all stakeholders to alter reward structures. There will always be incentives for innovative outcomes — those who discover new things will be rewarded more than those who do not. However, there can also be incentives for efficiency and effectiveness — those who conduct rigorous, transparent and reproducible research could be rewarded more than those who do not. There are promising examples of effective interventions for nudging incentives. For example, journals are adopting badges to acknowledge open practices ( Fig. 2 ), Registered Reports as a results-blind publishing model (see Box 3 ) and TOP guidelines to promote openness and transparency. Funders are also adopting transparency requirements, and piloting funding mechanisms to promote reproducibility such as the Netherlands Organisation for Scientific Research (NWO) and the US National Science Foundation's Directorate of Social, Behavioral and Economic Sciences, both of which have announced funding opportunities for replication studies. Institutions are wrestling with policy and infrastructure adjustments to promote data sharing, and there are hints of open-science practices becoming part of hiring and performance evaluation (for example, http://www.nicebread.de/open-science-hiring-practices/ ). Collectively, and at scale, such efforts can shift incentives such that what is good for the scientist is also good for science — rigorous, transparent and reproducible research practices producing credible results.” | unhoused_relations[26] |
| weight | rigorous, transparent and reproducible research → reward structures — 0.85 — “Funders, publishers, societies, institutions, editors, reviewers and authors all contribute to the cultural norms that create and sustain dysfunctional incentives. Changing the incentives is therefore a problem that requires a coordinated effort by all stakeholders to alter reward structures. There will always be incentives for innovative outcomes — those who discover new things will be rewarded more than those who do not. However, there can also be incentives for efficiency and effectiveness — those who conduct rigorous, transparent and reproducible research could be rewarded more than those who do not. There are promising examples of effective interventions for nudging incentives. For example, journals are adopting badges to acknowledge open practices ( Fig. 2 ), Registered Reports as a results-blind publishing model (see Box 3 ) and TOP guidelines to promote openness and transparency. Funders are also adopting transparency requirements, and piloting funding mechanisms to promote reproducibility such as the Netherlands Organisation for Scientific Research (NWO) and the US National Science Foundation's Directorate of Social, Behavioral and Economic Sciences, both of which have announced funding opportunities for replication studies. Institutions are wrestling with policy and infrastructure adjustments to promote data sharing, and there are hints of open-science practices becoming part of hiring and performance evaluation (for example, http://www.nicebread.de/open-science-hiring-practices/ ). Collectively, and at scale, such efforts can shift incentives such that what is good for the scientist is also good for science — rigorous, transparent and reproducible research practices producing credible results.” | unhoused_relations[26] |
| relation | Proposed solutions → efficiency and reliability of science — Some proposed solutions may be ineffective or harmful despite conceptual plausibility. — “The challenges to reproducible science are systemic and cultural, but that does not mean they cannot be met. The measures we have described constitute practical and achievable steps toward improving rigor and reproducibility. All of them have shown some effectiveness, and are well suited to wider adoption, evaluation and improvement. Equally, these proposals are not an exhaustive list; there are many other nascent and maturing ideas for making research practices more efficient and reliable 73 . Offering a solution to a problem does not guarantee its effectiveness, and making changes to cultural norms and incentives can spur additional behavioural changes that are difficult to anticipate. Some solutions may be ineffective or even harmful to the efficiency and reliability of science, even if conceptually they appear sensible.” | unhoused_relations[27] |
| weight | Proposed solutions → efficiency and reliability of science — 0.55 — “The challenges to reproducible science are systemic and cultural, but that does not mean they cannot be met. The measures we have described constitute practical and achievable steps toward improving rigor and reproducibility. All of them have shown some effectiveness, and are well suited to wider adoption, evaluation and improvement. Equally, these proposals are not an exhaustive list; there are many other nascent and maturing ideas for making research practices more efficient and reliable 73 . Offering a solution to a problem does not guarantee its effectiveness, and making changes to cultural norms and incentives can spur additional behavioural changes that are difficult to anticipate. Some solutions may be ineffective or even harmful to the efficiency and reliability of science, even if conceptually they appear sensible.” | unhoused_relations[27] |
| relation | replication → false results — Replication alone may not be sufficient to eliminate false results. — “The field of metascience (or metaresearch) is growing rapidly, with over 2,000 relevant publications accruing annually 16 . Much of that literature constitutes the evaluation of existing practices and the identification of alternative approaches. What was previously taken for granted may be questioned, such as widely used statistical methods; for example, the most popular methods and software for spatial extent analysis in fMRI imaging were recently shown to produce unacceptably high false-positive rates 74 . Proposed solutions may also give rise to other challenges; for example, while replication is a hallmark for reinforcing trust in scientific results, there is uncertainty about which studies deserve to be replicated and what would be the most efficient replication strategies. Moreover, a recent simulation suggests that replication alone may not suffice to rid us of false results 71 .” | unhoused_relations[28] |
| weight | replication → false results — 0.7 — “The field of metascience (or metaresearch) is growing rapidly, with over 2,000 relevant publications accruing annually 16 . Much of that literature constitutes the evaluation of existing practices and the identification of alternative approaches. What was previously taken for granted may be questioned, such as widely used statistical methods; for example, the most popular methods and software for spatial extent analysis in fMRI imaging were recently shown to produce unacceptably high false-positive rates 74 . Proposed solutions may also give rise to other challenges; for example, while replication is a hallmark for reinforcing trust in scientific results, there is uncertainty about which studies deserve to be replicated and what would be the most efficient replication strategies. Moreover, a recent simulation suggests that replication alone may not suffice to rid us of false results 71 .” | unhoused_relations[28] |
| relation | self-correction of the scientific process itself → status quo — Stakeholders should pursue continuous self-examination rather than embrace the status quo. — “These cautions are not a rationale for inaction. Reproducible research practices are at the heart of sound research and integral to the scientific method. How best to achieve rigorous and efficient knowledge accumulation is a scientific question; the most effective solutions will be identified by a combination of brilliant hypothesizing and blind luck, by iterative examination of the effectiveness of each change, and by a winnowing of many possibilities to the broadly enacted few. True understanding of how best to structure and incentivize science will emerge slowly and will never be finished. That is how science works. The key to fostering a robust metascience that evaluates and improves practices is that the stakeholders of science must not embrace the status quo, but instead pursue self-examination continuously for improvement and self-correction of the scientific process itself.” | unhoused_relations[29] |
| weight | self-correction of the scientific process itself → status quo — 0.95 — “These cautions are not a rationale for inaction. Reproducible research practices are at the heart of sound research and integral to the scientific method. How best to achieve rigorous and efficient knowledge accumulation is a scientific question; the most effective solutions will be identified by a combination of brilliant hypothesizing and blind luck, by iterative examination of the effectiveness of each change, and by a winnowing of many possibilities to the broadly enacted few. True understanding of how best to structure and incentivize science will emerge slowly and will never be finished. That is how science works. The key to fostering a robust metascience that evaluates and improves practices is that the stakeholders of science must not embrace the status quo, but instead pursue self-examination continuously for improvement and self-correction of the scientific process itself.” | unhoused_relations[29] |
| no type for this | pre- and post-publication peer review mechanisms → evaluation process — Pre- and post-publication peer review mechanisms accelerate and expand the evaluation process. — “Both pre- and post-publication peer review mechanisms dramatically accelerate and expand the evaluation process 68 . By sharing preprints, researchers can obtain rapid feedback on their work from a diverse community, rather than waiting several months for a few reviews in the conventional, closed peer review process. Using post-publication services, reviewers can make positive and critical commentary on articles instantly, rather than relying on the laborious, uncertain and lengthy process of authoring a commentary and submitting it to the publishing journal for possible publication, eventually.” | unhoused_relations[0] |
| no type for this | metascience → scientific study of science itself — Metascience is the scientific study of science itself. — “What proportion of published research is likely to be false? Low sample size, small effect sizes, data dredging (also known as P -hacking), conflicts of interest, large numbers of scientists working competitively in silos without combining their efforts, and so on, may conspire to dramatically increase the probability that a published finding is incorrect 1 . The field of metascience — the scientific study of science itself — is flourishing and has generated substantial empirical evidence for the existence and prevalence of threats to efficiency in knowledge accumulation (refs 2 , 3 , 4 , 5 , 6 , 7 ; Fig. 1 ).” | unhoused_relations[1] |
| no type for this | lack of replication → robustness of published research — Lack of replication is one of several threats that undermine published-research robustness. — “Threats to reproducible science. An idealized version of the hypothetico-deductive model of the scientific method is shown. Various potential threats to this model exist (indicated in red), including lack of replication 5 , hypothesizing after the results are known (HARKing) 7 , poor study design, low statistical power 2 , analytical flexibility 51 , P -hacking 4 , publication bias 3 and lack of data sharing 6 . Together these will serve to undermine the robustness of published research, and may also impact on the ability of science to self-correct.” | unhoused_relations[2] |
| no type for this | apophenia → false conclusions — Apophenia combines with confirmation and hindsight bias to lead researchers toward false conclusions. — “A hallmark of scientific creativity is the ability to see novel and unexpected patterns in data. John Snow's identification of links between cholera and water supply 17 , Paul Broca's work on language lateralization 18 and Jocelyn Bell Burnell's discovery of pulsars 19 are examples of breakthroughs achieved by interpreting observations in a new way. However, a major challenge for scientists is to be open to new and important insights while simultaneously avoiding being misled by our tendency to see structure in randomness. The combination of apophenia (the tendency to see patterns in random data), confirmation bias (the tendency to focus on evidence that is in line with our expectations or favoured explanation) and hindsight bias (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusions 20 . Thomas Levenson documents the example of astronomers who became convinced they had seen the fictitious planet Vulcan because their contemporary theories predicted its existence 21 . Experimenter effects are an example of this kind of bias 22 .” | unhoused_relations[3] |
| no type for this | analytical flexibility → false-positive findings — Many reasonable pipelines on high-dimensional data make false-positive findings highly likely. — “Over-interpretation of noise is facilitated by the extent to which data analysis is rapid, flexible and automated 23 . In a high-dimensional dataset, there may be hundreds or thousands of reasonable alternative approaches to analysing the same data 24 , 25 . For example, in a systematic review of functional magnetic resonance imaging (fMRI) studies, Carp showed that there were almost as many unique analytical pipelines as there were studies 26 . If several thousand potential analytical pipelines can be applied to high-dimensional data, the generation of false-positive findings is highly likely. For example, applying almost 7,000 analytical pipelines to a single fMRI dataset resulted in over 90% of brain voxels showing significant activation in at least one analysis 27 .” | unhoused_relations[4] |
| no type for this | blinding → cognitive biases — Blinding mitigates self-deception and unwanted cognitive biases. — “Protecting against cognitive biases. There is a substantial literature on the difficulty of avoiding cognitive biases. An effective solution to mitigate self-deception and unwanted biases is blinding. In some research contexts, participants and data collectors can be blinded to the experimental condition that participants are assigned to, and to the research hypotheses, while the data analyst can be blinded to key parts of the data. For example, during data preparation and cleaning, the identity of experimental conditions or the variable labels can be masked so that the output is not interpretable in terms of the research hypothesis. In some physical sciences this approach has been extended to include deliberate perturbations in or masking of data to allow data preparation (for example, identification of outliers) to proceed without the analyst being able to see the corresponding results 30 . Pre-registration of the study design, primary outcome(s) and analysis plan (see ‘Promoting study pre-registration’ section’, below) is a highly effective form of blinding because the data do not exist and the outcomes are not yet known.” | unhoused_relations[5] |
| no type for this | methodological training → Common misperceptions — Improved methodological training can address common statistical misperceptions. — “Improving methodological training. Research design and statistical analysis are mutually dependent. Common misperceptions, such as the interpretation of P values 31 , limitations of null-hypothesis significance testing 32 , the meaning and importance of statistical power 2 , the accuracy of reported effect sizes 33 , and the likelihood that a sample size that generated a statistically significant finding will also be adequate to replicate a true finding 34 , could all be addressed through improved statistical training. Similarly, basic design principles are important, such as blinding to reduce experimenter bias, randomization or counterbalancing to control for confounding, and the use of within-subjects designs, where possible, to maximize power. However, integrative training in research practices that can protect oneself against cognitive biases and the effects of distorted incentives is arguably more important. Moreover, statistical and methodological best practices are under constant revision and improvement, so that senior as well as junior researchers need continuing methodological education, not least because much training of early-career researchers is informal and flows from their supervisor or mentor. A failure to adopt advances in methodology — such as the very slow progress in increasing statistical power 35 , 36 — may be partly a function of failing to inculcate a continuing professional education and development ethic.” | unhoused_relations[6] |
| no type for this | independent methodological support → conflicts of interest — Independent methodological support may mitigate financial and non-financial conflicts of interest. — “Implementing independent methodological support. The need for independent methodological support is well-established in some areas — many clinical trials, for example, have multidisciplinary trial steering committees to provide advice and oversee the design and conduct of the trial. The need for these committees grew out of the well-understood financial conflicts of interest that exist in many clinical trials. The sponsor of a trial may be the company manufacturing the product, and any intentional or unintentional influence can distort the study design, analysis and interpretation of results for the ultimate financial benefit of the manufacturer at the cost of the accuracy of the science and the health benefit to the consumers 37 , 38 . Non-financial conflicts of interest also exist, such as the beliefs and preconceptions of individual scientists and the stakes that researchers have in obtaining publishable results in order to progress their career 39 , 40 . Including independent researchers (particularly methodologists with no personal investment in a research topic) in the design, monitoring, analysis or interpretation of research outcomes may mitigate some of those influences, and can be done either at the level of the individual research project or through a process facilitated by a funding agency (see Box 1 ).” | unhoused_relations[7] |
| no type for this | team science → lack of resources to improve power — Team science is presented as a solution to individual investigators' limited resources. — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | unhoused_relations[8] |
| no type for this | distributed collaboration → high-powered designs — Distributed collaboration across sites facilitates high-powered designs. — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | unhoused_relations[9] |
| no type for this | pre-registration → publication bias — Pre-registration addresses publication bias by making research discoverable regardless of publication outcome. — “The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P -hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory research that is important for testing hypotheses, and data-contingent exploratory research that is important for generating hypotheses.” | unhoused_relations[10] |
| no type for this | pre-registration → outcome switching — Pre-registration addresses outcome switching by fixing decisions before data are observed. — “The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P -hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory research that is important for testing hypotheses, and data-contingent exploratory research that is important for generating hypotheses.” | unhoused_relations[11] |
| no type for this | pre-registration → P -hacking — Pre-registration protects against data-contingent P-hacking. — “Promoting study pre-registration. Pre-registration of study protocols for randomized controlled trials in clinical medicine has become standard practice 46 . In its simplest form it may simply comprise the registration of the basic study design, but it can also include a detailed pre-specification of the study procedures, outcomes and statistical analysis plan. It was introduced to address two problems: publication bias and analytical flexibility (in particular outcome switching in the case of clinical medicine). Publication bias 47 , also known as the file drawer problem 48 , refers to the fact that many more studies are conducted than published. Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results 47 , 49 , 50 . The consequence is that the published literature indicates stronger evidence for findings than exists in reality. Outcome switching refers to the possibility of changing the outcomes of interest in the study depending on the observed results. A researcher may include ten variables that could be considered outcomes of the research, and — once the results are known — intentionally or unintentionally select the subset of outcomes that show statistically significant results as the outcomes of interest. The consequence is an increase in the likelihood that reported results are spurious by leveraging chance, while negative evidence gets ignored. This is one of several related research practices that can inflate spurious findings when analysis decisions are made with knowledge of the observed data, such as selection of models, exclusion rules and covariates. Such data-contingent analysis decisions constitute what has become known as P -hacking 51 , and pre-registration can protect against all of these.” | unhoused_relations[12] |
| no type for this | pre-registration → confirmatory research — Pre-registration makes confirmatory research distinguishable from exploratory research. — “The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P -hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory research that is important for testing hypotheses, and data-contingent exploratory research that is important for generating hypotheses.” | unhoused_relations[13] |
| no type for this | reporting guidelines → research quality — Improved reporting may be insufficient on its own to maximize research quality. — “Improving the quality of reporting. Pre-registration will improve discoverability of research, but discoverability does not guarantee usability. Poor usability reflects difficulty in evaluating what was done, in reusing the methodology to assess reproducibility, and in incorporating the evidence into systematic reviews and meta-analyses. Improving the quality and transparency in the reporting of research is necessary to address this. The Transparency and Openness Promotion (TOP) guidelines offer standards as a basis for journals and funders to incentivize or require greater transparency in planning and reporting of research 54 . TOP provides principles for how transparency and usability can be increased, while other guidelines provide concrete steps for how to maximize the quality of reporting in particular areas. For example, the Consolidated Standards of Reporting Trials (CONSORT) statement provides guidance for clear, complete and accurate reporting of randomized controlled trials 55 – 57 . Over 300 reporting guidelines now exist for observational studies, prognostic studies, predictive models, diagnostic tests, systematic reviews and meta-analyses in humans, a large variety of studies using different laboratory methods, and animal studies. The Equator Network ( http://www.equator-network.org/ ) aggregates these guidelines to improve discoverability 58 . There are also guidelines for improving the reporting of research planning; for example, the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement for reporting of systematic reviews and meta-analyses 59 , and PRISMA-P for protocols of systematic reviews 60 . The Preregistration Challenge workflow and the pre-registration recipe for social-behavioural research 61 also illustrate guidelines for reporting research plans.” | unhoused_relations[14] |
| no type for this | Registered Reports → publication bias — Registered Reports prevent publication bias by accepting articles before results are known. — “The Registered Reports (RR) initiative seeks to eliminate various forms of bias in hypothesis-driven research 52 , 53 , and in particular, the evaluation of a study based on the results. Unlike conventional journal articles, RRs split the peer review process into two stages, before and after results are known. At the first stage, reviewers and editors assess a detailed protocol that includes the study rationale, procedure and a detailed analysis plan. Following favourable reviews (and probably revision to meet strict methodological standards), the journal offers in-principle acceptance: publication of study outcomes is guaranteed provided the authors adhere to the approved protocol, the study meets pre-specified quality checks, and conclusions are appropriately evidence-bound. Once the study is completed, the authors resubmit a complete manuscript that includes the results and discussion. The article is published at the end of this two-stage process. By accepting articles before results are known, RRs prevent publication bias. By reviewing the hypotheses and analysis plans in advance, RRs should also help neutralize P- hacking and HARKing (hypothesizing after the results are known) by authors, and CARKing (critiquing after the results are known) by reviewers with their own investments in the research outcomes, although empirical evidence will be required to confirm that this is the case.” | unhoused_relations[15] |
| no type for this | Registered Reports → P -hacking — Advance review in Registered Reports should help neutralize P-hacking. — “The Registered Reports (RR) initiative seeks to eliminate various forms of bias in hypothesis-driven research 52 , 53 , and in particular, the evaluation of a study based on the results. Unlike conventional journal articles, RRs split the peer review process into two stages, before and after results are known. At the first stage, reviewers and editors assess a detailed protocol that includes the study rationale, procedure and a detailed analysis plan. Following favourable reviews (and probably revision to meet strict methodological standards), the journal offers in-principle acceptance: publication of study outcomes is guaranteed provided the authors adhere to the approved protocol, the study meets pre-specified quality checks, and conclusions are appropriately evidence-bound. Once the study is completed, the authors resubmit a complete manuscript that includes the results and discussion. The article is published at the end of this two-stage process. By accepting articles before results are known, RRs prevent publication bias. By reviewing the hypotheses and analysis plans in advance, RRs should also help neutralize P- hacking and HARKing (hypothesizing after the results are known) by authors, and CARKing (critiquing after the results are known) by reviewers with their own investments in the research outcomes, although empirical evidence will be required to confirm that this is the case.” | unhoused_relations[16] |
| no type for this | Registered Reports → exploratory research — Registered Reports permit unregistered exploration when it is transparently labeled post hoc. — “Perhaps the most commonly voiced objection to RRs is that the format somehow limits exploration or creativity by requiring authors to adhere to a pre-specified methodology. However, RRs place no restrictions on creative analysis practices or serendipity. Authors are free to report the outcomes of any unregistered exploratory analyses, provided such tests are clearly labelled as post hoc . Thus, the sole requirement is that exploratory outcomes are identified transparently as exploratory (for a list of frequently asked questions see https://cos.io/rr/#faq ). Of course, RRs are not intended for research that is solely exploratory.” | unhoused_relations[17] |
| no type for this | open science → transparency — Open science makes the production of evidence and claims transparent and accessible. — “Open science refers to the process of making the content and process of producing evidence and claims transparent and accessible to others. Transparency is a scientific ideal, and adding ‘open’ should therefore be redundant. In reality, science often lacks openness: many published articles are not available to people without a personal or institutional subscription, and most data, materials and code supporting research outcomes are not made accessible, for example, in a public repository (refs 63 , 64 ; Box 5 ).” | unhoused_relations[18] |
| no type for this | transparency → trust — Transparency is superior to credibility based only on trust in authority. — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | unhoused_relations[19] |
| no type for this | public repositories → accountability — Sharing data in public repositories offers accountability, longevity, efficiency, and quality advantages. — “Sharing data in public repositories offers field-wide advantages in terms of accountability, data longevity, efficiency and quality (for example, reanalyses may detect crucial mistakes or even data fabrication) 84 . Unfortunately, many scientific disciplines, including most of those devoted to the study of human behaviour, do not have a culture that values open data 6 . In the past, data sharing has rarely been enforced or facilitated. Recent initiatives, however, aim to change the normative culture. Hopefully, these initiatives will change the culture on data sharing. Once accepted as the norm, we doubt that data sharing will ever go out of fashion.” | unhoused_relations[20] |
| no type for this | Full disclosure → adequately evaluate results — Full disclosure is needed so readers can adequately evaluate how results were obtained. — “Full disclosure refers to the process of describing in full the study design and data collected that underlie the results reported, rather than a curated version of the design, and/or a subset of the data collected. The need for disclosure is clear: in order to adequately evaluate results we need to know how they were obtained. For example, the informational value of a dependent variable exhibiting an effect of interest is different if only one variable was collected or if fifteen were. The probability of a single variable achieving P < 0.05 just by chance is 5%, but the probability of one of fifteen variables achieving P < 0.05 is 54% 1 . It is obvious that cherry-picking one from fifteen variables invalidates the results unless it is clear that this has happened. If readers know, then they can adjust their interpretation accordingly. From this simple fact it follows that if authors do not tell us whether they collected one or fifteen variables readers cannot evaluate their research 51 .” | unhoused_relations[21] |
| no type for this | peer review → dissemination — Scientific publishing historically confounded peer-review evaluation with dissemination. — “Diversifying peer review. For most of the history of scientific publishing, two functions have been confounded — evaluation and dissemination. Journals have provided dissemination via sorting and delivering content to the research community, and gatekeeping via peer review to determine what is worth disseminating. However, with the advent of the internet, individual researchers are no longer dependent on publishers to bind, print and mail their research to subscribers. Dissemination is now easy and can be controlled by researchers themselves. For example, preprint services (arXiv for some physical sciences, bioRxiv and PeerJ for the life sciences, engrXiv for engineering, PsyArXiv for psychology, and SocArXiv and the Social Science Research Network (SSRN) for the social sciences) facilitate easy sharing, sorting and discovery of research prior to publication. This dramatically accelerates the dissemination of information to the research community.” | unhoused_relations[22] |
| no type for this | preprint services → dissemination — Preprint services facilitate rapid sharing, sorting, and discovery before publication. — “Diversifying peer review. For most of the history of scientific publishing, two functions have been confounded — evaluation and dissemination. Journals have provided dissemination via sorting and delivering content to the research community, and gatekeeping via peer review to determine what is worth disseminating. However, with the advent of the internet, individual researchers are no longer dependent on publishers to bind, print and mail their research to subscribers. Dissemination is now easy and can be controlled by researchers themselves. For example, preprint services (arXiv for some physical sciences, bioRxiv and PeerJ for the life sciences, engrXiv for engineering, PsyArXiv for psychology, and SocArXiv and the Social Science Research Network (SSRN) for the social sciences) facilitate easy sharing, sorting and discovery of research prior to publication. This dramatically accelerates the dissemination of information to the research community.” | unhoused_relations[23] |
| no type for this | Positive, novel and clean results → publication — Positive, novel, clean results are more likely to be published than negative results and replications. — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | unhoused_relations[24] |
| no type for this | incentive structures → scientific value of research — Changing incentive structures may improve scientific value by changing researcher behaviour. — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | unhoused_relations[25] |
| no type for this | rigorous, transparent and reproducible research → reward structures — Coordinated stakeholders can align reward structures with rigorous and reproducible practice. — “Funders, publishers, societies, institutions, editors, reviewers and authors all contribute to the cultural norms that create and sustain dysfunctional incentives. Changing the incentives is therefore a problem that requires a coordinated effort by all stakeholders to alter reward structures. There will always be incentives for innovative outcomes — those who discover new things will be rewarded more than those who do not. However, there can also be incentives for efficiency and effectiveness — those who conduct rigorous, transparent and reproducible research could be rewarded more than those who do not. There are promising examples of effective interventions for nudging incentives. For example, journals are adopting badges to acknowledge open practices ( Fig. 2 ), Registered Reports as a results-blind publishing model (see Box 3 ) and TOP guidelines to promote openness and transparency. Funders are also adopting transparency requirements, and piloting funding mechanisms to promote reproducibility such as the Netherlands Organisation for Scientific Research (NWO) and the US National Science Foundation's Directorate of Social, Behavioral and Economic Sciences, both of which have announced funding opportunities for replication studies. Institutions are wrestling with policy and infrastructure adjustments to promote data sharing, and there are hints of open-science practices becoming part of hiring and performance evaluation (for example, http://www.nicebread.de/open-science-hiring-practices/ ). Collectively, and at scale, such efforts can shift incentives such that what is good for the scientist is also good for science — rigorous, transparent and reproducible research practices producing credible results.” | unhoused_relations[26] |
| no type for this | Proposed solutions → efficiency and reliability of science — Some proposed solutions may be ineffective or harmful despite conceptual plausibility. — “The challenges to reproducible science are systemic and cultural, but that does not mean they cannot be met. The measures we have described constitute practical and achievable steps toward improving rigor and reproducibility. All of them have shown some effectiveness, and are well suited to wider adoption, evaluation and improvement. Equally, these proposals are not an exhaustive list; there are many other nascent and maturing ideas for making research practices more efficient and reliable 73 . Offering a solution to a problem does not guarantee its effectiveness, and making changes to cultural norms and incentives can spur additional behavioural changes that are difficult to anticipate. Some solutions may be ineffective or even harmful to the efficiency and reliability of science, even if conceptually they appear sensible.” | unhoused_relations[27] |
| no type for this | replication → false results — Replication alone may not be sufficient to eliminate false results. — “The field of metascience (or metaresearch) is growing rapidly, with over 2,000 relevant publications accruing annually 16 . Much of that literature constitutes the evaluation of existing practices and the identification of alternative approaches. What was previously taken for granted may be questioned, such as widely used statistical methods; for example, the most popular methods and software for spatial extent analysis in fMRI imaging were recently shown to produce unacceptably high false-positive rates 74 . Proposed solutions may also give rise to other challenges; for example, while replication is a hallmark for reinforcing trust in scientific results, there is uncertainty about which studies deserve to be replicated and what would be the most efficient replication strategies. Moreover, a recent simulation suggests that replication alone may not suffice to rid us of false results 71 .” | unhoused_relations[28] |
| no type for this | self-correction of the scientific process itself → status quo — Stakeholders should pursue continuous self-examination rather than embrace the status quo. — “These cautions are not a rationale for inaction. Reproducible research practices are at the heart of sound research and integral to the scientific method. How best to achieve rigorous and efficient knowledge accumulation is a scientific question; the most effective solutions will be identified by a combination of brilliant hypothesizing and blind luck, by iterative examination of the effectiveness of each change, and by a winnowing of many possibilities to the broadly enacted few. True understanding of how best to structure and incentivize science will emerge slowly and will never be finished. That is how science works. The key to fostering a robust metascience that evaluates and improves practices is that the stakeholders of science must not embrace the status quo, but instead pursue self-examination continuously for improvement and self-correction of the scientific process itself.” | unhoused_relations[29] |
| limitation | evidence for the proposed reforms is incomplete, so broad adoption requires iterative evaluation and improvement — “Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.” | limitations[0] |
| limitation | whether the current reproducibility problem should be called a crisis remains debatable — “Data from many fields suggests reproducibility is lower than is desirable 8 , 9 , 10 , 11 , 12 , 13 , 14 ; one analysis estimates that 85% of biomedical research efforts are wasted 14 , while 90% of respondents to a recent survey in Nature agreed that there is a ‘reproducibility crisis’ 15 . Whether ‘crisis’ is the appropriate term to describe the current state or trajectory of science is debatable, but accumulated evidence indicates that there is substantial room for improvement with regard to research practices to maximize the efficiency of the research community's use of the public's financial investment in research.” | limitations[1] |
| limitation | the proposed measures are broad and practical but explicitly not exhaustive — “Here we propose a series of measures that we believe will improve research efficiency and robustness of scientific findings by directly targeting specific threats to reproducible science. We argue for the adoption, evaluation and ongoing improvement of these measures to optimize the pace and efficiency of knowledge accumulation. The measures are organized into the following categories 16 : methods, reporting and dissemination, reproducibility, evaluation and incentives. They are not intended to be exhaustive, but provide a broad, practical and evidence-based set of actions that can be implemented by researchers, institutions, journals and funders. The measures and their current implementation are summarized in Table 1 .” | limitations[2] |
| limitation | researchers may fail to recognize their own P-hacking because confirmation and hindsight biases legitimize desired outcomes — “During data analysis it can be difficult for researchers to recognize P -hacking 28 or data dredging because confirmation and hindsight biases can encourage the acceptance of outcomes that fit expectations or desires as appropriate, and the rejection of outcomes that do not as the result of suboptimal designs or analyses. Hypotheses may emerge that fit the data and are then reported without indication or recognition of their post hoc origin 7 . This, unfortunately, is not scientific discovery, but self-deception 29 . Uncontrolled, it can dramatically increase the false discovery rate. We need measures to counter the natural tendency of enthusiastic scientists who are motivated by discovery to see patterns in noise.” | limitations[3] |
| limitation | statistical and methodological best practices continually change, making one-time training inadequate — “Improving methodological training. Research design and statistical analysis are mutually dependent. Common misperceptions, such as the interpretation of P values 31 , limitations of null-hypothesis significance testing 32 , the meaning and importance of statistical power 2 , the accuracy of reported effect sizes 33 , and the likelihood that a sample size that generated a statistically significant finding will also be adequate to replicate a true finding 34 , could all be addressed through improved statistical training. Similarly, basic design principles are important, such as blinding to reduce experimenter bias, randomization or counterbalancing to control for confounding, and the use of within-subjects designs, where possible, to maximize power. However, integrative training in research practices that can protect oneself against cognitive biases and the effects of distorted incentives is arguably more important. Moreover, statistical and methodological best practices are under constant revision and improvement, so that senior as well as junior researchers need continuing methodological education, not least because much training of early-career researchers is informal and flows from their supervisor or mentor. A failure to adopt advances in methodology — such as the very slow progress in increasing statistical power 35 , 36 — may be partly a function of failing to inculcate a continuing professional education and development ethic.” | limitations[4] |
| limitation | reporting guidelines can become bureaucratic exercises and are not sufficient on their own to eliminate reporting bias — “The positive outcome of reporting guidelines is that they make it possible to detect and study these behaviours and their impact. Otherwise, these behaviours are simply unknowable in any systematic way. The negative outcome is the empirical evidence that reporting guidelines may be necessary, but will not alone be sufficient, to address reporting biases. The impact of guidelines and how best to optimize their use and impact will be best assessed by randomized trials (see Box 4 ).” | limitations[5] |
| limitation | the claimed bias-reducing effects of Registered Reports still require empirical confirmation — “The Registered Reports (RR) initiative seeks to eliminate various forms of bias in hypothesis-driven research 52 , 53 , and in particular, the evaluation of a study based on the results. Unlike conventional journal articles, RRs split the peer review process into two stages, before and after results are known. At the first stage, reviewers and editors assess a detailed protocol that includes the study rationale, procedure and a detailed analysis plan. Following favourable reviews (and probably revision to meet strict methodological standards), the journal offers in-principle acceptance: publication of study outcomes is guaranteed provided the authors adhere to the approved protocol, the study meets pre-specified quality checks, and conclusions are appropriately evidence-bound. Once the study is completed, the authors resubmit a complete manuscript that includes the results and discussion. The article is published at the end of this two-stage process. By accepting articles before results are known, RRs prevent publication bias. By reviewing the hypotheses and analysis plans in advance, RRs should also help neutralize P- hacking and HARKing (hypothesizing after the results are known) by authors, and CARKing (critiquing after the results are known) by reviewers with their own investments in the research outcomes, although empirical evidence will be required to confirm that this is the case.” | limitations[6] |
| limitation | open practices face financial interests, weak incentives, and disciplinary cultures that do not value data sharing — “Open science refers to the process of making the content and process of producing evidence and claims transparent and accessible to others. Transparency is a scientific ideal, and adding ‘open’ should therefore be redundant. In reality, science often lacks openness: many published articles are not available to people without a personal or institutional subscription, and most data, materials and code supporting research outcomes are not made accessible, for example, in a public repository (refs 63 , 64 ; Box 5 ).” | limitations[7] |
| limitation | non-anonymous public review introduces reputational risks that require empirical evaluation — “Both pre- and post-publication peer review mechanisms dramatically accelerate and expand the evaluation process 68 . By sharing preprints, researchers can obtain rapid feedback on their work from a diverse community, rather than waiting several months for a few reviews in the conventional, closed peer review process. Using post-publication services, reviewers can make positive and critical commentary on articles instantly, rather than relying on the laborious, uncertain and lengthy process of authoring a commentary and submitting it to the publishing journal for possible publication, eventually.” | limitations[8] |
| limitation | coordinated incentives cannot remove the greater rewards attached to genuinely innovative discoveries — “Funders, publishers, societies, institutions, editors, reviewers and authors all contribute to the cultural norms that create and sustain dysfunctional incentives. Changing the incentives is therefore a problem that requires a coordinated effort by all stakeholders to alter reward structures. There will always be incentives for innovative outcomes — those who discover new things will be rewarded more than those who do not. However, there can also be incentives for efficiency and effectiveness — those who conduct rigorous, transparent and reproducible research could be rewarded more than those who do not. There are promising examples of effective interventions for nudging incentives. For example, journals are adopting badges to acknowledge open practices ( Fig. 2 ), Registered Reports as a results-blind publishing model (see Box 3 ) and TOP guidelines to promote openness and transparency. Funders are also adopting transparency requirements, and piloting funding mechanisms to promote reproducibility such as the Netherlands Organisation for Scientific Research (NWO) and the US National Science Foundation's Directorate of Social, Behavioral and Economic Sciences, both of which have announced funding opportunities for replication studies. Institutions are wrestling with policy and infrastructure adjustments to promote data sharing, and there are hints of open-science practices becoming part of hiring and performance evaluation (for example, http://www.nicebread.de/open-science-hiring-practices/ ). Collectively, and at scale, such efforts can shift incentives such that what is good for the scientist is also good for science — rigorous, transparent and reproducible research practices producing credible results.” | limitations[9] |
| limitation | proposed reforms may prove ineffective or harmful and may induce unanticipated behavioral changes — “The challenges to reproducible science are systemic and cultural, but that does not mean they cannot be met. The measures we have described constitute practical and achievable steps toward improving rigor and reproducibility. All of them have shown some effectiveness, and are well suited to wider adoption, evaluation and improvement. Equally, these proposals are not an exhaustive list; there are many other nascent and maturing ideas for making research practices more efficient and reliable 73 . Offering a solution to a problem does not guarantee its effectiveness, and making changes to cultural norms and incentives can spur additional behavioural changes that are difficult to anticipate. Some solutions may be ineffective or even harmful to the efficiency and reliability of science, even if conceptually they appear sensible.” | limitations[10] |
| limitation | replication alone may not eliminate false results, and efficient choices about what to replicate remain uncertain — “The field of metascience (or metaresearch) is growing rapidly, with over 2,000 relevant publications accruing annually 16 . Much of that literature constitutes the evaluation of existing practices and the identification of alternative approaches. What was previously taken for granted may be questioned, such as widely used statistical methods; for example, the most popular methods and software for spatial extent analysis in fMRI imaging were recently shown to produce unacceptably high false-positive rates 74 . Proposed solutions may also give rise to other challenges; for example, while replication is a hallmark for reinforcing trust in scientific results, there is uncertainty about which studies deserve to be replicated and what would be the most efficient replication strategies. Moreover, a recent simulation suggests that replication alone may not suffice to rid us of false results 71 .” | limitations[11] |
Concepts
terms this paper carries, each with the sentence it was cut from — the layer that joins one paper to another. Terms with no quote yet are on CONCEPTS.html.
| concept | reproducible science — not defined in this paper's own words; science whose findings and supporting processes can be independently checked and reproduced — “Threats to reproducible science. An idealized version of the hypothetico-deductive model of the scientific method is shown. Various potential threats to this model exist (indicated in red), including lack of replication 5 , hypothesizing after the results are known (HARKing) 7 , poor study design, low statistical power 2 , analytical flexibility 51 , P -hacking 4 , publication bias 3 and lack of data sharing 6 . Together these will serve to undermine the robustness of published research, and may also impact on the ability of science to self-correct.” | concepts[0] |
| concept | scientific research — not defined in this paper's own words; the research process whose reliability, efficiency, and credibility the manifesto seeks to improve — “Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.” | concepts[1] |
| concept | published scientific literature — not defined in this paper's own words; the public record whose credibility depends on reliable research practices — “Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.” | concepts[2] |
| concept | scientific process — not defined in this paper's own words; the linked cycle of methods, reporting, reproducibility, evaluation, and incentives — “Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.” | concepts[3] |
| concept | metascience — the scientific study of science itself — “What proportion of published research is likely to be false? Low sample size, small effect sizes, data dredging (also known as P -hacking), conflicts of interest, large numbers of scientists working competitively in silos without combining their efforts, and so on, may conspire to dramatically increase the probability that a published finding is incorrect 1 . The field of metascience — the scientific study of science itself — is flourishing and has generated substantial empirical evidence for the existence and prevalence of threats to efficiency in knowledge accumulation (refs 2 , 3 , 4 , 5 , 6 , 7 ; Fig. 1 ).” | concepts[4] |
| concept | knowledge accumulation — not defined in this paper's own words; the cumulative production of credible scientific understanding — “What proportion of published research is likely to be false? Low sample size, small effect sizes, data dredging (also known as P -hacking), conflicts of interest, large numbers of scientists working competitively in silos without combining their efforts, and so on, may conspire to dramatically increase the probability that a published finding is incorrect 1 . The field of metascience — the scientific study of science itself — is flourishing and has generated substantial empirical evidence for the existence and prevalence of threats to efficiency in knowledge accumulation (refs 2 , 3 , 4 , 5 , 6 , 7 ; Fig. 1 ).” | concepts[5] |
| concept | methods — not defined in this paper's own words; study-design, statistical, support, and collaboration practices — “Here we propose a series of measures that we believe will improve research efficiency and robustness of scientific findings by directly targeting specific threats to reproducible science. We argue for the adoption, evaluation and ongoing improvement of these measures to optimize the pace and efficiency of knowledge accumulation. The measures are organized into the following categories 16 : methods, reporting and dissemination, reproducibility, evaluation and incentives. They are not intended to be exhaustive, but provide a broad, practical and evidence-based set of actions that can be implemented by researchers, institutions, journals and funders. The measures and their current implementation are summarized in Table 1 .” | concepts[6] |
| concept | reporting and dissemination — not defined in this paper's own words; practices for communicating research transparently and completely — “Here we propose a series of measures that we believe will improve research efficiency and robustness of scientific findings by directly targeting specific threats to reproducible science. We argue for the adoption, evaluation and ongoing improvement of these measures to optimize the pace and efficiency of knowledge accumulation. The measures are organized into the following categories 16 : methods, reporting and dissemination, reproducibility, evaluation and incentives. They are not intended to be exhaustive, but provide a broad, practical and evidence-based set of actions that can be implemented by researchers, institutions, journals and funders. The measures and their current implementation are summarized in Table 1 .” | concepts[7] |
| concept | reproducibility — not defined in this paper's own words; the capacity to verify or reproduce research evidence and claims — “Here we propose a series of measures that we believe will improve research efficiency and robustness of scientific findings by directly targeting specific threats to reproducible science. We argue for the adoption, evaluation and ongoing improvement of these measures to optimize the pace and efficiency of knowledge accumulation. The measures are organized into the following categories 16 : methods, reporting and dissemination, reproducibility, evaluation and incentives. They are not intended to be exhaustive, but provide a broad, practical and evidence-based set of actions that can be implemented by researchers, institutions, journals and funders. The measures and their current implementation are summarized in Table 1 .” | concepts[8] |
| concept | evaluation — not defined in this paper's own words; processes that assess research before and after dissemination — “Here we propose a series of measures that we believe will improve research efficiency and robustness of scientific findings by directly targeting specific threats to reproducible science. We argue for the adoption, evaluation and ongoing improvement of these measures to optimize the pace and efficiency of knowledge accumulation. The measures are organized into the following categories 16 : methods, reporting and dissemination, reproducibility, evaluation and incentives. They are not intended to be exhaustive, but provide a broad, practical and evidence-based set of actions that can be implemented by researchers, institutions, journals and funders. The measures and their current implementation are summarized in Table 1 .” | concepts[9] |
| concept | incentives — not defined in this paper's own words; reward structures that shape researcher choices and publication outcomes — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | concepts[10] |
| concept | Low sample size — not defined in this paper's own words; small samples that contribute to incorrect published findings — “What proportion of published research is likely to be false? Low sample size, small effect sizes, data dredging (also known as P -hacking), conflicts of interest, large numbers of scientists working competitively in silos without combining their efforts, and so on, may conspire to dramatically increase the probability that a published finding is incorrect 1 . The field of metascience — the scientific study of science itself — is flourishing and has generated substantial empirical evidence for the existence and prevalence of threats to efficiency in knowledge accumulation (refs 2 , 3 , 4 , 5 , 6 , 7 ; Fig. 1 ).” | concepts[11] |
| concept | small effect sizes — not defined in this paper's own words; weak effects that increase the fragility of research findings — “What proportion of published research is likely to be false? Low sample size, small effect sizes, data dredging (also known as P -hacking), conflicts of interest, large numbers of scientists working competitively in silos without combining their efforts, and so on, may conspire to dramatically increase the probability that a published finding is incorrect 1 . The field of metascience — the scientific study of science itself — is flourishing and has generated substantial empirical evidence for the existence and prevalence of threats to efficiency in knowledge accumulation (refs 2 , 3 , 4 , 5 , 6 , 7 ; Fig. 1 ).” | concepts[12] |
| concept | data dredging — not defined in this paper's own words; data-contingent searching also called P-hacking — “What proportion of published research is likely to be false? Low sample size, small effect sizes, data dredging (also known as P -hacking), conflicts of interest, large numbers of scientists working competitively in silos without combining their efforts, and so on, may conspire to dramatically increase the probability that a published finding is incorrect 1 . The field of metascience — the scientific study of science itself — is flourishing and has generated substantial empirical evidence for the existence and prevalence of threats to efficiency in knowledge accumulation (refs 2 , 3 , 4 , 5 , 6 , 7 ; Fig. 1 ).” | concepts[13] |
| concept | P -hacking — not defined in this paper's own words; analytic flexibility used after seeing data to obtain preferred outcomes — “What proportion of published research is likely to be false? Low sample size, small effect sizes, data dredging (also known as P -hacking), conflicts of interest, large numbers of scientists working competitively in silos without combining their efforts, and so on, may conspire to dramatically increase the probability that a published finding is incorrect 1 . The field of metascience — the scientific study of science itself — is flourishing and has generated substantial empirical evidence for the existence and prevalence of threats to efficiency in knowledge accumulation (refs 2 , 3 , 4 , 5 , 6 , 7 ; Fig. 1 ).” | concepts[14] |
| concept | conflicts of interest — not defined in this paper's own words; interests capable of distorting research decisions — “What proportion of published research is likely to be false? Low sample size, small effect sizes, data dredging (also known as P -hacking), conflicts of interest, large numbers of scientists working competitively in silos without combining their efforts, and so on, may conspire to dramatically increase the probability that a published finding is incorrect 1 . The field of metascience — the scientific study of science itself — is flourishing and has generated substantial empirical evidence for the existence and prevalence of threats to efficiency in knowledge accumulation (refs 2 , 3 , 4 , 5 , 6 , 7 ; Fig. 1 ).” | concepts[15] |
| concept | lack of replication — not defined in this paper's own words; absence of independent repetition among threats to robust science — “Threats to reproducible science. An idealized version of the hypothetico-deductive model of the scientific method is shown. Various potential threats to this model exist (indicated in red), including lack of replication 5 , hypothesizing after the results are known (HARKing) 7 , poor study design, low statistical power 2 , analytical flexibility 51 , P -hacking 4 , publication bias 3 and lack of data sharing 6 . Together these will serve to undermine the robustness of published research, and may also impact on the ability of science to self-correct.” | concepts[16] |
| concept | HARKing — not defined in this paper's own words; hypothesizing after results are known — “Threats to reproducible science. An idealized version of the hypothetico-deductive model of the scientific method is shown. Various potential threats to this model exist (indicated in red), including lack of replication 5 , hypothesizing after the results are known (HARKing) 7 , poor study design, low statistical power 2 , analytical flexibility 51 , P -hacking 4 , publication bias 3 and lack of data sharing 6 . Together these will serve to undermine the robustness of published research, and may also impact on the ability of science to self-correct.” | concepts[17] |
| concept | poor study design — not defined in this paper's own words; design weaknesses that threaten reproducibility — “Threats to reproducible science. An idealized version of the hypothetico-deductive model of the scientific method is shown. Various potential threats to this model exist (indicated in red), including lack of replication 5 , hypothesizing after the results are known (HARKing) 7 , poor study design, low statistical power 2 , analytical flexibility 51 , P -hacking 4 , publication bias 3 and lack of data sharing 6 . Together these will serve to undermine the robustness of published research, and may also impact on the ability of science to self-correct.” | concepts[18] |
| concept | low statistical power — not defined in this paper's own words; design sensitivity below levels needed for reliable positive and negative findings — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | concepts[19] |
| concept | analytical flexibility — not defined in this paper's own words; availability of many defensible analysis choices — “Threats to reproducible science. An idealized version of the hypothetico-deductive model of the scientific method is shown. Various potential threats to this model exist (indicated in red), including lack of replication 5 , hypothesizing after the results are known (HARKing) 7 , poor study design, low statistical power 2 , analytical flexibility 51 , P -hacking 4 , publication bias 3 and lack of data sharing 6 . Together these will serve to undermine the robustness of published research, and may also impact on the ability of science to self-correct.” | concepts[20] |
| concept | publication bias — not defined in this paper's own words; selective publication of positive and novel findings — “Promoting study pre-registration. Pre-registration of study protocols for randomized controlled trials in clinical medicine has become standard practice 46 . In its simplest form it may simply comprise the registration of the basic study design, but it can also include a detailed pre-specification of the study procedures, outcomes and statistical analysis plan. It was introduced to address two problems: publication bias and analytical flexibility (in particular outcome switching in the case of clinical medicine). Publication bias 47 , also known as the file drawer problem 48 , refers to the fact that many more studies are conducted than published. Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results 47 , 49 , 50 . The consequence is that the published literature indicates stronger evidence for findings than exists in reality. Outcome switching refers to the possibility of changing the outcomes of interest in the study depending on the observed results. A researcher may include ten variables that could be considered outcomes of the research, and — once the results are known — intentionally or unintentionally select the subset of outcomes that show statistically significant results as the outcomes of interest. The consequence is an increase in the likelihood that reported results are spurious by leveraging chance, while negative evidence gets ignored. This is one of several related research practices that can inflate spurious findings when analysis decisions are made with knowledge of the observed data, such as selection of models, exclusion rules and covariates. Such data-contingent analysis decisions constitute what has become known as P -hacking 51 , and pre-registration can protect against all of these.” | concepts[21] |
| concept | lack of data sharing — not defined in this paper's own words; inaccessibility of the data needed to inspect and reproduce research — “Threats to reproducible science. An idealized version of the hypothetico-deductive model of the scientific method is shown. Various potential threats to this model exist (indicated in red), including lack of replication 5 , hypothesizing after the results are known (HARKing) 7 , poor study design, low statistical power 2 , analytical flexibility 51 , P -hacking 4 , publication bias 3 and lack of data sharing 6 . Together these will serve to undermine the robustness of published research, and may also impact on the ability of science to self-correct.” | concepts[22] |
| concept | apophenia — the tendency to see patterns in random data — “A hallmark of scientific creativity is the ability to see novel and unexpected patterns in data. John Snow's identification of links between cholera and water supply 17 , Paul Broca's work on language lateralization 18 and Jocelyn Bell Burnell's discovery of pulsars 19 are examples of breakthroughs achieved by interpreting observations in a new way. However, a major challenge for scientists is to be open to new and important insights while simultaneously avoiding being misled by our tendency to see structure in randomness. The combination of apophenia (the tendency to see patterns in random data), confirmation bias (the tendency to focus on evidence that is in line with our expectations or favoured explanation) and hindsight bias (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusions 20 . Thomas Levenson documents the example of astronomers who became convinced they had seen the fictitious planet Vulcan because their contemporary theories predicted its existence 21 . Experimenter effects are an example of this kind of bias 22 .” | concepts[23] |
| concept | confirmation bias — not defined in this paper's own words; attention to evidence consistent with expectations or favored explanations — “A hallmark of scientific creativity is the ability to see novel and unexpected patterns in data. John Snow's identification of links between cholera and water supply 17 , Paul Broca's work on language lateralization 18 and Jocelyn Bell Burnell's discovery of pulsars 19 are examples of breakthroughs achieved by interpreting observations in a new way. However, a major challenge for scientists is to be open to new and important insights while simultaneously avoiding being misled by our tendency to see structure in randomness. The combination of apophenia (the tendency to see patterns in random data), confirmation bias (the tendency to focus on evidence that is in line with our expectations or favoured explanation) and hindsight bias (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusions 20 . Thomas Levenson documents the example of astronomers who became convinced they had seen the fictitious planet Vulcan because their contemporary theories predicted its existence 21 . Experimenter effects are an example of this kind of bias 22 .” | concepts[24] |
| concept | hindsight bias — not defined in this paper's own words; retrospective perception that an event was predictable — “A hallmark of scientific creativity is the ability to see novel and unexpected patterns in data. John Snow's identification of links between cholera and water supply 17 , Paul Broca's work on language lateralization 18 and Jocelyn Bell Burnell's discovery of pulsars 19 are examples of breakthroughs achieved by interpreting observations in a new way. However, a major challenge for scientists is to be open to new and important insights while simultaneously avoiding being misled by our tendency to see structure in randomness. The combination of apophenia (the tendency to see patterns in random data), confirmation bias (the tendency to focus on evidence that is in line with our expectations or favoured explanation) and hindsight bias (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusions 20 . Thomas Levenson documents the example of astronomers who became convinced they had seen the fictitious planet Vulcan because their contemporary theories predicted its existence 21 . Experimenter effects are an example of this kind of bias 22 .” | concepts[25] |
| concept | Experimenter effects — not defined in this paper's own words; biases through which expectations shape research observations or outcomes — “A hallmark of scientific creativity is the ability to see novel and unexpected patterns in data. John Snow's identification of links between cholera and water supply 17 , Paul Broca's work on language lateralization 18 and Jocelyn Bell Burnell's discovery of pulsars 19 are examples of breakthroughs achieved by interpreting observations in a new way. However, a major challenge for scientists is to be open to new and important insights while simultaneously avoiding being misled by our tendency to see structure in randomness. The combination of apophenia (the tendency to see patterns in random data), confirmation bias (the tendency to focus on evidence that is in line with our expectations or favoured explanation) and hindsight bias (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusions 20 . Thomas Levenson documents the example of astronomers who became convinced they had seen the fictitious planet Vulcan because their contemporary theories predicted its existence 21 . Experimenter effects are an example of this kind of bias 22 .” | concepts[26] |
| concept | high-dimensional dataset — not defined in this paper's own words; data supporting hundreds or thousands of plausible analytic paths — “Over-interpretation of noise is facilitated by the extent to which data analysis is rapid, flexible and automated 23 . In a high-dimensional dataset, there may be hundreds or thousands of reasonable alternative approaches to analysing the same data 24 , 25 . For example, in a systematic review of functional magnetic resonance imaging (fMRI) studies, Carp showed that there were almost as many unique analytical pipelines as there were studies 26 . If several thousand potential analytical pipelines can be applied to high-dimensional data, the generation of false-positive findings is highly likely. For example, applying almost 7,000 analytical pipelines to a single fMRI dataset resulted in over 90% of brain voxels showing significant activation in at least one analysis 27 .” | concepts[27] |
| concept | analytical pipelines — not defined in this paper's own words; alternative sequences of data-processing and analysis choices — “Over-interpretation of noise is facilitated by the extent to which data analysis is rapid, flexible and automated 23 . In a high-dimensional dataset, there may be hundreds or thousands of reasonable alternative approaches to analysing the same data 24 , 25 . For example, in a systematic review of functional magnetic resonance imaging (fMRI) studies, Carp showed that there were almost as many unique analytical pipelines as there were studies 26 . If several thousand potential analytical pipelines can be applied to high-dimensional data, the generation of false-positive findings is highly likely. For example, applying almost 7,000 analytical pipelines to a single fMRI dataset resulted in over 90% of brain voxels showing significant activation in at least one analysis 27 .” | concepts[28] |
| concept | false-positive findings — not defined in this paper's own words; reported positive results generated without a reliable underlying effect — “Over-interpretation of noise is facilitated by the extent to which data analysis is rapid, flexible and automated 23 . In a high-dimensional dataset, there may be hundreds or thousands of reasonable alternative approaches to analysing the same data 24 , 25 . For example, in a systematic review of functional magnetic resonance imaging (fMRI) studies, Carp showed that there were almost as many unique analytical pipelines as there were studies 26 . If several thousand potential analytical pipelines can be applied to high-dimensional data, the generation of false-positive findings is highly likely. For example, applying almost 7,000 analytical pipelines to a single fMRI dataset resulted in over 90% of brain voxels showing significant activation in at least one analysis 27 .” | concepts[29] |
| concept | false-negative results — not defined in this paper's own words; negative results missed because a design has inadequate power — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | concepts[30] |
| concept | false discovery rate — not defined in this paper's own words; the proportion of apparent discoveries expected to be false — “During data analysis it can be difficult for researchers to recognize P -hacking 28 or data dredging because confirmation and hindsight biases can encourage the acceptance of outcomes that fit expectations or desires as appropriate, and the rejection of outcomes that do not as the result of suboptimal designs or analyses. Hypotheses may emerge that fit the data and are then reported without indication or recognition of their post hoc origin 7 . This, unfortunately, is not scientific discovery, but self-deception 29 . Uncontrolled, it can dramatically increase the false discovery rate. We need measures to counter the natural tendency of enthusiastic scientists who are motivated by discovery to see patterns in noise.” | concepts[31] |
| concept | cognitive biases — not defined in this paper's own words; predictable reasoning tendencies that foster self-deception — “Protecting against cognitive biases. There is a substantial literature on the difficulty of avoiding cognitive biases. An effective solution to mitigate self-deception and unwanted biases is blinding. In some research contexts, participants and data collectors can be blinded to the experimental condition that participants are assigned to, and to the research hypotheses, while the data analyst can be blinded to key parts of the data. For example, during data preparation and cleaning, the identity of experimental conditions or the variable labels can be masked so that the output is not interpretable in terms of the research hypothesis. In some physical sciences this approach has been extended to include deliberate perturbations in or masking of data to allow data preparation (for example, identification of outliers) to proceed without the analyst being able to see the corresponding results 30 . Pre-registration of the study design, primary outcome(s) and analysis plan (see ‘Promoting study pre-registration’ section’, below) is a highly effective form of blinding because the data do not exist and the outcomes are not yet known.” | concepts[32] |
| concept | blinding — not defined in this paper's own words; masking hypotheses, conditions, labels, or results to limit bias — “Protecting against cognitive biases. There is a substantial literature on the difficulty of avoiding cognitive biases. An effective solution to mitigate self-deception and unwanted biases is blinding. In some research contexts, participants and data collectors can be blinded to the experimental condition that participants are assigned to, and to the research hypotheses, while the data analyst can be blinded to key parts of the data. For example, during data preparation and cleaning, the identity of experimental conditions or the variable labels can be masked so that the output is not interpretable in terms of the research hypothesis. In some physical sciences this approach has been extended to include deliberate perturbations in or masking of data to allow data preparation (for example, identification of outliers) to proceed without the analyst being able to see the corresponding results 30 . Pre-registration of the study design, primary outcome(s) and analysis plan (see ‘Promoting study pre-registration’ section’, below) is a highly effective form of blinding because the data do not exist and the outcomes are not yet known.” | concepts[33] |
| concept | pre-registration — not defined in this paper's own words; advance registration and specification of design, outcomes, and analysis — “The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P -hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory research that is important for testing hypotheses, and data-contingent exploratory research that is important for generating hypotheses.” | concepts[34] |
| concept | research design — not defined in this paper's own words; the planned structure for collecting and analyzing evidence — “Improving methodological training. Research design and statistical analysis are mutually dependent. Common misperceptions, such as the interpretation of P values 31 , limitations of null-hypothesis significance testing 32 , the meaning and importance of statistical power 2 , the accuracy of reported effect sizes 33 , and the likelihood that a sample size that generated a statistically significant finding will also be adequate to replicate a true finding 34 , could all be addressed through improved statistical training. Similarly, basic design principles are important, such as blinding to reduce experimenter bias, randomization or counterbalancing to control for confounding, and the use of within-subjects designs, where possible, to maximize power. However, integrative training in research practices that can protect oneself against cognitive biases and the effects of distorted incentives is arguably more important. Moreover, statistical and methodological best practices are under constant revision and improvement, so that senior as well as junior researchers need continuing methodological education, not least because much training of early-career researchers is informal and flows from their supervisor or mentor. A failure to adopt advances in methodology — such as the very slow progress in increasing statistical power 35 , 36 — may be partly a function of failing to inculcate a continuing professional education and development ethic.” | concepts[35] |
| concept | statistical analysis — not defined in this paper's own words; quantitative analysis whose validity depends on research design — “Improving methodological training. Research design and statistical analysis are mutually dependent. Common misperceptions, such as the interpretation of P values 31 , limitations of null-hypothesis significance testing 32 , the meaning and importance of statistical power 2 , the accuracy of reported effect sizes 33 , and the likelihood that a sample size that generated a statistically significant finding will also be adequate to replicate a true finding 34 , could all be addressed through improved statistical training. Similarly, basic design principles are important, such as blinding to reduce experimenter bias, randomization or counterbalancing to control for confounding, and the use of within-subjects designs, where possible, to maximize power. However, integrative training in research practices that can protect oneself against cognitive biases and the effects of distorted incentives is arguably more important. Moreover, statistical and methodological best practices are under constant revision and improvement, so that senior as well as junior researchers need continuing methodological education, not least because much training of early-career researchers is informal and flows from their supervisor or mentor. A failure to adopt advances in methodology — such as the very slow progress in increasing statistical power 35 , 36 — may be partly a function of failing to inculcate a continuing professional education and development ethic.” | concepts[36] |
| concept | randomization — not defined in this paper's own words; allocation used to control confounding — “Improving methodological training. Research design and statistical analysis are mutually dependent. Common misperceptions, such as the interpretation of P values 31 , limitations of null-hypothesis significance testing 32 , the meaning and importance of statistical power 2 , the accuracy of reported effect sizes 33 , and the likelihood that a sample size that generated a statistically significant finding will also be adequate to replicate a true finding 34 , could all be addressed through improved statistical training. Similarly, basic design principles are important, such as blinding to reduce experimenter bias, randomization or counterbalancing to control for confounding, and the use of within-subjects designs, where possible, to maximize power. However, integrative training in research practices that can protect oneself against cognitive biases and the effects of distorted incentives is arguably more important. Moreover, statistical and methodological best practices are under constant revision and improvement, so that senior as well as junior researchers need continuing methodological education, not least because much training of early-career researchers is informal and flows from their supervisor or mentor. A failure to adopt advances in methodology — such as the very slow progress in increasing statistical power 35 , 36 — may be partly a function of failing to inculcate a continuing professional education and development ethic.” | concepts[37] |
| concept | counterbalancing — not defined in this paper's own words; design balancing used to control confounding — “Improving methodological training. Research design and statistical analysis are mutually dependent. Common misperceptions, such as the interpretation of P values 31 , limitations of null-hypothesis significance testing 32 , the meaning and importance of statistical power 2 , the accuracy of reported effect sizes 33 , and the likelihood that a sample size that generated a statistically significant finding will also be adequate to replicate a true finding 34 , could all be addressed through improved statistical training. Similarly, basic design principles are important, such as blinding to reduce experimenter bias, randomization or counterbalancing to control for confounding, and the use of within-subjects designs, where possible, to maximize power. However, integrative training in research practices that can protect oneself against cognitive biases and the effects of distorted incentives is arguably more important. Moreover, statistical and methodological best practices are under constant revision and improvement, so that senior as well as junior researchers need continuing methodological education, not least because much training of early-career researchers is informal and flows from their supervisor or mentor. A failure to adopt advances in methodology — such as the very slow progress in increasing statistical power 35 , 36 — may be partly a function of failing to inculcate a continuing professional education and development ethic.” | concepts[38] |
| concept | within-subjects designs — not defined in this paper's own words; designs used where possible to maximize statistical power — “Improving methodological training. Research design and statistical analysis are mutually dependent. Common misperceptions, such as the interpretation of P values 31 , limitations of null-hypothesis significance testing 32 , the meaning and importance of statistical power 2 , the accuracy of reported effect sizes 33 , and the likelihood that a sample size that generated a statistically significant finding will also be adequate to replicate a true finding 34 , could all be addressed through improved statistical training. Similarly, basic design principles are important, such as blinding to reduce experimenter bias, randomization or counterbalancing to control for confounding, and the use of within-subjects designs, where possible, to maximize power. However, integrative training in research practices that can protect oneself against cognitive biases and the effects of distorted incentives is arguably more important. Moreover, statistical and methodological best practices are under constant revision and improvement, so that senior as well as junior researchers need continuing methodological education, not least because much training of early-career researchers is informal and flows from their supervisor or mentor. A failure to adopt advances in methodology — such as the very slow progress in increasing statistical power 35 , 36 — may be partly a function of failing to inculcate a continuing professional education and development ethic.” | concepts[39] |
| concept | methodological training — not defined in this paper's own words; education in statistical and research-design practice — “Improving methodological training. Research design and statistical analysis are mutually dependent. Common misperceptions, such as the interpretation of P values 31 , limitations of null-hypothesis significance testing 32 , the meaning and importance of statistical power 2 , the accuracy of reported effect sizes 33 , and the likelihood that a sample size that generated a statistically significant finding will also be adequate to replicate a true finding 34 , could all be addressed through improved statistical training. Similarly, basic design principles are important, such as blinding to reduce experimenter bias, randomization or counterbalancing to control for confounding, and the use of within-subjects designs, where possible, to maximize power. However, integrative training in research practices that can protect oneself against cognitive biases and the effects of distorted incentives is arguably more important. Moreover, statistical and methodological best practices are under constant revision and improvement, so that senior as well as junior researchers need continuing methodological education, not least because much training of early-career researchers is informal and flows from their supervisor or mentor. A failure to adopt advances in methodology — such as the very slow progress in increasing statistical power 35 , 36 — may be partly a function of failing to inculcate a continuing professional education and development ethic.” | concepts[40] |
| concept | continuing methodological education — not defined in this paper's own words; ongoing updating of methodological knowledge across career stages — “Improving methodological training. Research design and statistical analysis are mutually dependent. Common misperceptions, such as the interpretation of P values 31 , limitations of null-hypothesis significance testing 32 , the meaning and importance of statistical power 2 , the accuracy of reported effect sizes 33 , and the likelihood that a sample size that generated a statistically significant finding will also be adequate to replicate a true finding 34 , could all be addressed through improved statistical training. Similarly, basic design principles are important, such as blinding to reduce experimenter bias, randomization or counterbalancing to control for confounding, and the use of within-subjects designs, where possible, to maximize power. However, integrative training in research practices that can protect oneself against cognitive biases and the effects of distorted incentives is arguably more important. Moreover, statistical and methodological best practices are under constant revision and improvement, so that senior as well as junior researchers need continuing methodological education, not least because much training of early-career researchers is informal and flows from their supervisor or mentor. A failure to adopt advances in methodology — such as the very slow progress in increasing statistical power 35 , 36 — may be partly a function of failing to inculcate a continuing professional education and development ethic.” | concepts[41] |
| concept | independent methodological support — not defined in this paper's own words; design and analysis advice from methodologists without investment in outcomes — “Implementing independent methodological support. The need for independent methodological support is well-established in some areas — many clinical trials, for example, have multidisciplinary trial steering committees to provide advice and oversee the design and conduct of the trial. The need for these committees grew out of the well-understood financial conflicts of interest that exist in many clinical trials. The sponsor of a trial may be the company manufacturing the product, and any intentional or unintentional influence can distort the study design, analysis and interpretation of results for the ultimate financial benefit of the manufacturer at the cost of the accuracy of the science and the health benefit to the consumers 37 , 38 . Non-financial conflicts of interest also exist, such as the beliefs and preconceptions of individual scientists and the stakes that researchers have in obtaining publishable results in order to progress their career 39 , 40 . Including independent researchers (particularly methodologists with no personal investment in a research topic) in the design, monitoring, analysis or interpretation of research outcomes may mitigate some of those influences, and can be done either at the level of the individual research project or through a process facilitated by a funding agency (see Box 1 ).” | concepts[42] |
| concept | financial conflicts of interest — not defined in this paper's own words; financial stakes capable of distorting design, analysis, and interpretation — “Implementing independent methodological support. The need for independent methodological support is well-established in some areas — many clinical trials, for example, have multidisciplinary trial steering committees to provide advice and oversee the design and conduct of the trial. The need for these committees grew out of the well-understood financial conflicts of interest that exist in many clinical trials. The sponsor of a trial may be the company manufacturing the product, and any intentional or unintentional influence can distort the study design, analysis and interpretation of results for the ultimate financial benefit of the manufacturer at the cost of the accuracy of the science and the health benefit to the consumers 37 , 38 . Non-financial conflicts of interest also exist, such as the beliefs and preconceptions of individual scientists and the stakes that researchers have in obtaining publishable results in order to progress their career 39 , 40 . Including independent researchers (particularly methodologists with no personal investment in a research topic) in the design, monitoring, analysis or interpretation of research outcomes may mitigate some of those influences, and can be done either at the level of the individual research project or through a process facilitated by a funding agency (see Box 1 ).” | concepts[43] |
| concept | Non-financial conflicts of interest — not defined in this paper's own words; beliefs, preconceptions, and career stakes that can bias research — “Implementing independent methodological support. The need for independent methodological support is well-established in some areas — many clinical trials, for example, have multidisciplinary trial steering committees to provide advice and oversee the design and conduct of the trial. The need for these committees grew out of the well-understood financial conflicts of interest that exist in many clinical trials. The sponsor of a trial may be the company manufacturing the product, and any intentional or unintentional influence can distort the study design, analysis and interpretation of results for the ultimate financial benefit of the manufacturer at the cost of the accuracy of the science and the health benefit to the consumers 37 , 38 . Non-financial conflicts of interest also exist, such as the beliefs and preconceptions of individual scientists and the stakes that researchers have in obtaining publishable results in order to progress their career 39 , 40 . Including independent researchers (particularly methodologists with no personal investment in a research topic) in the design, monitoring, analysis or interpretation of research outcomes may mitigate some of those influences, and can be done either at the level of the individual research project or through a process facilitated by a funding agency (see Box 1 ).” | concepts[44] |
| concept | team science — not defined in this paper's own words; distributed collaboration pooling resources, sites, and perspectives — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | concepts[45] |
| concept | distributed collaboration — not defined in this paper's own words; coordination across study sites to enable stronger designs — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | concepts[46] |
| concept | high-powered designs — not defined in this paper's own words; designs with adequate sensitivity supported by distributed resources — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | concepts[47] |
| concept | Multi-site collaborative projects — not defined in this paper's own words; coordinated research conducted across multiple locations — “Multi-centre and collaborative efforts have a long and successful tradition in fields such as randomized controlled trials in some areas of clinical medicine, and in genetic association analyses, and have improved the robustness of the resulting research literatures. Multi-site collaborative projects have also been advocated for other types of research, such as animal studies 41 – 43 , in an effort to maximize their power, enhance standardization, and optimize transparency and protection from biases. The Many Labs projects illustrate this potential in the social and behavioural sciences, with dozens of laboratories implementing the same research protocol to obtain highly precise estimates of effect sizes, and evaluate variability across samples and settings 44 , 45 . It is also possible, and desirable, to incorporate a team science ethos into student training (see Box 2 ).” | concepts[48] |
| concept | Many Labs — not defined in this paper's own words; multi-laboratory coordinated replication projects — “Multi-centre and collaborative efforts have a long and successful tradition in fields such as randomized controlled trials in some areas of clinical medicine, and in genetic association analyses, and have improved the robustness of the resulting research literatures. Multi-site collaborative projects have also been advocated for other types of research, such as animal studies 41 – 43 , in an effort to maximize their power, enhance standardization, and optimize transparency and protection from biases. The Many Labs projects illustrate this potential in the social and behavioural sciences, with dozens of laboratories implementing the same research protocol to obtain highly precise estimates of effect sizes, and evaluate variability across samples and settings 44 , 45 . It is also possible, and desirable, to incorporate a team science ethos into student training (see Box 2 ).” | concepts[49] |
| concept | study protocols — not defined in this paper's own words; registered plans for study design and conduct — “Promoting study pre-registration. Pre-registration of study protocols for randomized controlled trials in clinical medicine has become standard practice 46 . In its simplest form it may simply comprise the registration of the basic study design, but it can also include a detailed pre-specification of the study procedures, outcomes and statistical analysis plan. It was introduced to address two problems: publication bias and analytical flexibility (in particular outcome switching in the case of clinical medicine). Publication bias 47 , also known as the file drawer problem 48 , refers to the fact that many more studies are conducted than published. Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results 47 , 49 , 50 . The consequence is that the published literature indicates stronger evidence for findings than exists in reality. Outcome switching refers to the possibility of changing the outcomes of interest in the study depending on the observed results. A researcher may include ten variables that could be considered outcomes of the research, and — once the results are known — intentionally or unintentionally select the subset of outcomes that show statistically significant results as the outcomes of interest. The consequence is an increase in the likelihood that reported results are spurious by leveraging chance, while negative evidence gets ignored. This is one of several related research practices that can inflate spurious findings when analysis decisions are made with knowledge of the observed data, such as selection of models, exclusion rules and covariates. Such data-contingent analysis decisions constitute what has become known as P -hacking 51 , and pre-registration can protect against all of these.” | concepts[50] |
| concept | outcome switching — not defined in this paper's own words; changing focal outcomes after observing results — “Promoting study pre-registration. Pre-registration of study protocols for randomized controlled trials in clinical medicine has become standard practice 46 . In its simplest form it may simply comprise the registration of the basic study design, but it can also include a detailed pre-specification of the study procedures, outcomes and statistical analysis plan. It was introduced to address two problems: publication bias and analytical flexibility (in particular outcome switching in the case of clinical medicine). Publication bias 47 , also known as the file drawer problem 48 , refers to the fact that many more studies are conducted than published. Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results 47 , 49 , 50 . The consequence is that the published literature indicates stronger evidence for findings than exists in reality. Outcome switching refers to the possibility of changing the outcomes of interest in the study depending on the observed results. A researcher may include ten variables that could be considered outcomes of the research, and — once the results are known — intentionally or unintentionally select the subset of outcomes that show statistically significant results as the outcomes of interest. The consequence is an increase in the likelihood that reported results are spurious by leveraging chance, while negative evidence gets ignored. This is one of several related research practices that can inflate spurious findings when analysis decisions are made with knowledge of the observed data, such as selection of models, exclusion rules and covariates. Such data-contingent analysis decisions constitute what has become known as P -hacking 51 , and pre-registration can protect against all of these.” | concepts[51] |
| concept | file drawer problem — not defined in this paper's own words; the nonpublication of many conducted studies — “Promoting study pre-registration. Pre-registration of study protocols for randomized controlled trials in clinical medicine has become standard practice 46 . In its simplest form it may simply comprise the registration of the basic study design, but it can also include a detailed pre-specification of the study procedures, outcomes and statistical analysis plan. It was introduced to address two problems: publication bias and analytical flexibility (in particular outcome switching in the case of clinical medicine). Publication bias 47 , also known as the file drawer problem 48 , refers to the fact that many more studies are conducted than published. Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results 47 , 49 , 50 . The consequence is that the published literature indicates stronger evidence for findings than exists in reality. Outcome switching refers to the possibility of changing the outcomes of interest in the study depending on the observed results. A researcher may include ten variables that could be considered outcomes of the research, and — once the results are known — intentionally or unintentionally select the subset of outcomes that show statistically significant results as the outcomes of interest. The consequence is an increase in the likelihood that reported results are spurious by leveraging chance, while negative evidence gets ignored. This is one of several related research practices that can inflate spurious findings when analysis decisions are made with knowledge of the observed data, such as selection of models, exclusion rules and covariates. Such data-contingent analysis decisions constitute what has become known as P -hacking 51 , and pre-registration can protect against all of these.” | concepts[52] |
| concept | data-contingent analysis decisions — not defined in this paper's own words; analysis choices made with knowledge of observed data — “Promoting study pre-registration. Pre-registration of study protocols for randomized controlled trials in clinical medicine has become standard practice 46 . In its simplest form it may simply comprise the registration of the basic study design, but it can also include a detailed pre-specification of the study procedures, outcomes and statistical analysis plan. It was introduced to address two problems: publication bias and analytical flexibility (in particular outcome switching in the case of clinical medicine). Publication bias 47 , also known as the file drawer problem 48 , refers to the fact that many more studies are conducted than published. Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results 47 , 49 , 50 . The consequence is that the published literature indicates stronger evidence for findings than exists in reality. Outcome switching refers to the possibility of changing the outcomes of interest in the study depending on the observed results. A researcher may include ten variables that could be considered outcomes of the research, and — once the results are known — intentionally or unintentionally select the subset of outcomes that show statistically significant results as the outcomes of interest. The consequence is an increase in the likelihood that reported results are spurious by leveraging chance, while negative evidence gets ignored. This is one of several related research practices that can inflate spurious findings when analysis decisions are made with knowledge of the observed data, such as selection of models, exclusion rules and covariates. Such data-contingent analysis decisions constitute what has become known as P -hacking 51 , and pre-registration can protect against all of these.” | concepts[53] |
| concept | confirmatory research — not defined in this paper's own words; data-independent hypothesis testing — “The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P -hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory research that is important for testing hypotheses, and data-contingent exploratory research that is important for generating hypotheses.” | concepts[54] |
| concept | exploratory research — not defined in this paper's own words; data-contingent work used to generate hypotheses — “The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P -hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory research that is important for testing hypotheses, and data-contingent exploratory research that is important for generating hypotheses.” | concepts[55] |
| concept | reporting guidelines — not defined in this paper's own words; standards intended to improve completeness and transparency of research reports — “Improving the quality of reporting. Pre-registration will improve discoverability of research, but discoverability does not guarantee usability. Poor usability reflects difficulty in evaluating what was done, in reusing the methodology to assess reproducibility, and in incorporating the evidence into systematic reviews and meta-analyses. Improving the quality and transparency in the reporting of research is necessary to address this. The Transparency and Openness Promotion (TOP) guidelines offer standards as a basis for journals and funders to incentivize or require greater transparency in planning and reporting of research 54 . TOP provides principles for how transparency and usability can be increased, while other guidelines provide concrete steps for how to maximize the quality of reporting in particular areas. For example, the Consolidated Standards of Reporting Trials (CONSORT) statement provides guidance for clear, complete and accurate reporting of randomized controlled trials 55 – 57 . Over 300 reporting guidelines now exist for observational studies, prognostic studies, predictive models, diagnostic tests, systematic reviews and meta-analyses in humans, a large variety of studies using different laboratory methods, and animal studies. The Equator Network ( http://www.equator-network.org/ ) aggregates these guidelines to improve discoverability 58 . There are also guidelines for improving the reporting of research planning; for example, the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement for reporting of systematic reviews and meta-analyses 59 , and PRISMA-P for protocols of systematic reviews 60 . The Preregistration Challenge workflow and the pre-registration recipe for social-behavioural research 61 also illustrate guidelines for reporting research plans.” | concepts[56] |
| concept | TOP guidelines — not defined in this paper's own words; standards for transparency and openness policies — “Improving the quality of reporting. Pre-registration will improve discoverability of research, but discoverability does not guarantee usability. Poor usability reflects difficulty in evaluating what was done, in reusing the methodology to assess reproducibility, and in incorporating the evidence into systematic reviews and meta-analyses. Improving the quality and transparency in the reporting of research is necessary to address this. The Transparency and Openness Promotion (TOP) guidelines offer standards as a basis for journals and funders to incentivize or require greater transparency in planning and reporting of research 54 . TOP provides principles for how transparency and usability can be increased, while other guidelines provide concrete steps for how to maximize the quality of reporting in particular areas. For example, the Consolidated Standards of Reporting Trials (CONSORT) statement provides guidance for clear, complete and accurate reporting of randomized controlled trials 55 – 57 . Over 300 reporting guidelines now exist for observational studies, prognostic studies, predictive models, diagnostic tests, systematic reviews and meta-analyses in humans, a large variety of studies using different laboratory methods, and animal studies. The Equator Network ( http://www.equator-network.org/ ) aggregates these guidelines to improve discoverability 58 . There are also guidelines for improving the reporting of research planning; for example, the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement for reporting of systematic reviews and meta-analyses 59 , and PRISMA-P for protocols of systematic reviews 60 . The Preregistration Challenge workflow and the pre-registration recipe for social-behavioural research 61 also illustrate guidelines for reporting research plans.” | concepts[57] |
| concept | CONSORT statement — not defined in this paper's own words; guidance for clear, complete, and accurate randomized-trial reporting — “Improving the quality of reporting. Pre-registration will improve discoverability of research, but discoverability does not guarantee usability. Poor usability reflects difficulty in evaluating what was done, in reusing the methodology to assess reproducibility, and in incorporating the evidence into systematic reviews and meta-analyses. Improving the quality and transparency in the reporting of research is necessary to address this. The Transparency and Openness Promotion (TOP) guidelines offer standards as a basis for journals and funders to incentivize or require greater transparency in planning and reporting of research 54 . TOP provides principles for how transparency and usability can be increased, while other guidelines provide concrete steps for how to maximize the quality of reporting in particular areas. For example, the Consolidated Standards of Reporting Trials (CONSORT) statement provides guidance for clear, complete and accurate reporting of randomized controlled trials 55 – 57 . Over 300 reporting guidelines now exist for observational studies, prognostic studies, predictive models, diagnostic tests, systematic reviews and meta-analyses in humans, a large variety of studies using different laboratory methods, and animal studies. The Equator Network ( http://www.equator-network.org/ ) aggregates these guidelines to improve discoverability 58 . There are also guidelines for improving the reporting of research planning; for example, the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement for reporting of systematic reviews and meta-analyses 59 , and PRISMA-P for protocols of systematic reviews 60 . The Preregistration Challenge workflow and the pre-registration recipe for social-behavioural research 61 also illustrate guidelines for reporting research plans.” | concepts[58] |
| concept | Registered Reports — not defined in this paper's own words; a two-stage publishing process that evaluates protocols before results — “The Registered Reports (RR) initiative seeks to eliminate various forms of bias in hypothesis-driven research 52 , 53 , and in particular, the evaluation of a study based on the results. Unlike conventional journal articles, RRs split the peer review process into two stages, before and after results are known. At the first stage, reviewers and editors assess a detailed protocol that includes the study rationale, procedure and a detailed analysis plan. Following favourable reviews (and probably revision to meet strict methodological standards), the journal offers in-principle acceptance: publication of study outcomes is guaranteed provided the authors adhere to the approved protocol, the study meets pre-specified quality checks, and conclusions are appropriately evidence-bound. Once the study is completed, the authors resubmit a complete manuscript that includes the results and discussion. The article is published at the end of this two-stage process. By accepting articles before results are known, RRs prevent publication bias. By reviewing the hypotheses and analysis plans in advance, RRs should also help neutralize P- hacking and HARKing (hypothesizing after the results are known) by authors, and CARKing (critiquing after the results are known) by reviewers with their own investments in the research outcomes, although empirical evidence will be required to confirm that this is the case.” | concepts[59] |
| concept | peer review process — not defined in this paper's own words; editorial assessment divided into stages before and after results — “The Registered Reports (RR) initiative seeks to eliminate various forms of bias in hypothesis-driven research 52 , 53 , and in particular, the evaluation of a study based on the results. Unlike conventional journal articles, RRs split the peer review process into two stages, before and after results are known. At the first stage, reviewers and editors assess a detailed protocol that includes the study rationale, procedure and a detailed analysis plan. Following favourable reviews (and probably revision to meet strict methodological standards), the journal offers in-principle acceptance: publication of study outcomes is guaranteed provided the authors adhere to the approved protocol, the study meets pre-specified quality checks, and conclusions are appropriately evidence-bound. Once the study is completed, the authors resubmit a complete manuscript that includes the results and discussion. The article is published at the end of this two-stage process. By accepting articles before results are known, RRs prevent publication bias. By reviewing the hypotheses and analysis plans in advance, RRs should also help neutralize P- hacking and HARKing (hypothesizing after the results are known) by authors, and CARKing (critiquing after the results are known) by reviewers with their own investments in the research outcomes, although empirical evidence will be required to confirm that this is the case.” | concepts[60] |
| concept | in-principle acceptance — not defined in this paper's own words; a publication commitment conditional on protocol adherence and evidence-bound conclusions — “The Registered Reports (RR) initiative seeks to eliminate various forms of bias in hypothesis-driven research 52 , 53 , and in particular, the evaluation of a study based on the results. Unlike conventional journal articles, RRs split the peer review process into two stages, before and after results are known. At the first stage, reviewers and editors assess a detailed protocol that includes the study rationale, procedure and a detailed analysis plan. Following favourable reviews (and probably revision to meet strict methodological standards), the journal offers in-principle acceptance: publication of study outcomes is guaranteed provided the authors adhere to the approved protocol, the study meets pre-specified quality checks, and conclusions are appropriately evidence-bound. Once the study is completed, the authors resubmit a complete manuscript that includes the results and discussion. The article is published at the end of this two-stage process. By accepting articles before results are known, RRs prevent publication bias. By reviewing the hypotheses and analysis plans in advance, RRs should also help neutralize P- hacking and HARKing (hypothesizing after the results are known) by authors, and CARKing (critiquing after the results are known) by reviewers with their own investments in the research outcomes, although empirical evidence will be required to confirm that this is the case.” | concepts[61] |
| concept | results-blind peer review — not defined in this paper's own words; evaluation of rationale and methods without knowing results — “At the same time, some journals are trialling ‘results-free’ review, where editorial decisions to accept are based solely on review of the rationale and study methods alone (that is, results-blind) 67 .” | concepts[62] |
| concept | transparency — not defined in this paper's own words; accessibility of the evidence and process supporting scientific claims — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concepts[63] |
| concept | trust — not defined in this paper's own words; credibility resting on an originator's confidence or authority rather than inspectable evidence — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concepts[64] |
| concept | open science — making the content and process of producing evidence and claims transparent and accessible — “Open science refers to the process of making the content and process of producing evidence and claims transparent and accessible to others. Transparency is a scientific ideal, and adding ‘open’ should therefore be redundant. In reality, science often lacks openness: many published articles are not available to people without a personal or institutional subscription, and most data, materials and code supporting research outcomes are not made accessible, for example, in a public repository (refs 63 , 64 ; Box 5 ).” | concepts[65] |
| concept | open access — not defined in this paper's own words; public accessibility of research articles — “This has motivated calls for open access, open data and open workflows (including analysis pipelines), but there are substantial barriers to meeting these ideals, including vested financial interests (particularly in scholarly publishing) and few incentives for researchers to pursue open practices.” | concepts[66] |
| concept | open data — not defined in this paper's own words; public accessibility of research data — “Open science refers to the process of making the content and process of producing evidence and claims transparent and accessible to others. Transparency is a scientific ideal, and adding ‘open’ should therefore be redundant. In reality, science often lacks openness: many published articles are not available to people without a personal or institutional subscription, and most data, materials and code supporting research outcomes are not made accessible, for example, in a public repository (refs 63 , 64 ; Box 5 ).” | concepts[67] |
| concept | open workflows — not defined in this paper's own words; accessible research processes including analysis pipelines — “This has motivated calls for open access, open data and open workflows (including analysis pipelines), but there are substantial barriers to meeting these ideals, including vested financial interests (particularly in scholarly publishing) and few incentives for researchers to pursue open practices.” | concepts[68] |
| concept | public repositories — not defined in this paper's own words; shared infrastructure for persistent access to research data — “Sharing data in public repositories offers field-wide advantages in terms of accountability, data longevity, efficiency and quality (for example, reanalyses may detect crucial mistakes or even data fabrication) 84 . Unfortunately, many scientific disciplines, including most of those devoted to the study of human behaviour, do not have a culture that values open data 6 . In the past, data sharing has rarely been enforced or facilitated. Recent initiatives, however, aim to change the normative culture. Hopefully, these initiatives will change the culture on data sharing. Once accepted as the norm, we doubt that data sharing will ever go out of fashion.” | concepts[69] |
| concept | supporting evidence — not defined in this paper's own words; methodology, data, implementation, analysis, and interpretation supporting claims — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concepts[70] |
| concept | methodology applied — not defined in this paper's own words; the methods used to produce a scientific claim — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concepts[71] |
| concept | data acquired — not defined in this paper's own words; the observations supporting a scientific claim — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concepts[72] |
| concept | process of methodology implementation — not defined in this paper's own words; how a stated methodology was carried out — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concepts[73] |
| concept | data analysis — not defined in this paper's own words; the transformation and interpretation of acquired data — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concepts[74] |
| concept | outcome interpretation — not defined in this paper's own words; the inferential reading placed on analyzed results — “Promoting transparency and open science. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust.” | concepts[75] |
| concept | Full disclosure — not defined in this paper's own words; complete description of study design and collected data rather than a curated subset — “Full disclosure refers to the process of describing in full the study design and data collected that underlie the results reported, rather than a curated version of the design, and/or a subset of the data collected. The need for disclosure is clear: in order to adequately evaluate results we need to know how they were obtained. For example, the informational value of a dependent variable exhibiting an effect of interest is different if only one variable was collected or if fifteen were. The probability of a single variable achieving P < 0.05 just by chance is 5%, but the probability of one of fifteen variables achieving P < 0.05 is 54% 1 . It is obvious that cherry-picking one from fifteen variables invalidates the results unless it is clear that this has happened. If readers know, then they can adjust their interpretation accordingly. From this simple fact it follows that if authors do not tell us whether they collected one or fifteen variables readers cannot evaluate their research 51 .” | concepts[76] |
| concept | peer review — not defined in this paper's own words; community evaluation of research claims and methods — “Diversifying peer review. For most of the history of scientific publishing, two functions have been confounded — evaluation and dissemination. Journals have provided dissemination via sorting and delivering content to the research community, and gatekeeping via peer review to determine what is worth disseminating. However, with the advent of the internet, individual researchers are no longer dependent on publishers to bind, print and mail their research to subscribers. Dissemination is now easy and can be controlled by researchers themselves. For example, preprint services (arXiv for some physical sciences, bioRxiv and PeerJ for the life sciences, engrXiv for engineering, PsyArXiv for psychology, and SocArXiv and the Social Science Research Network (SSRN) for the social sciences) facilitate easy sharing, sorting and discovery of research prior to publication. This dramatically accelerates the dissemination of information to the research community.” | concepts[77] |
| concept | dissemination — not defined in this paper's own words; distribution and discovery of scientific work — “Diversifying peer review. For most of the history of scientific publishing, two functions have been confounded — evaluation and dissemination. Journals have provided dissemination via sorting and delivering content to the research community, and gatekeeping via peer review to determine what is worth disseminating. However, with the advent of the internet, individual researchers are no longer dependent on publishers to bind, print and mail their research to subscribers. Dissemination is now easy and can be controlled by researchers themselves. For example, preprint services (arXiv for some physical sciences, bioRxiv and PeerJ for the life sciences, engrXiv for engineering, PsyArXiv for psychology, and SocArXiv and the Social Science Research Network (SSRN) for the social sciences) facilitate easy sharing, sorting and discovery of research prior to publication. This dramatically accelerates the dissemination of information to the research community.” | concepts[78] |
| concept | preprint services — not defined in this paper's own words; infrastructure for sharing research before journal publication — “Diversifying peer review. For most of the history of scientific publishing, two functions have been confounded — evaluation and dissemination. Journals have provided dissemination via sorting and delivering content to the research community, and gatekeeping via peer review to determine what is worth disseminating. However, with the advent of the internet, individual researchers are no longer dependent on publishers to bind, print and mail their research to subscribers. Dissemination is now easy and can be controlled by researchers themselves. For example, preprint services (arXiv for some physical sciences, bioRxiv and PeerJ for the life sciences, engrXiv for engineering, PsyArXiv for psychology, and SocArXiv and the Social Science Research Network (SSRN) for the social sciences) facilitate easy sharing, sorting and discovery of research prior to publication. This dramatically accelerates the dissemination of information to the research community.” | concepts[79] |
| concept | post-publication peer review — not defined in this paper's own words; public evaluation after a work is published — “Other services, such as PubMed Commons and PubPeer, offer public platforms to comment on published works facilitating post-publication peer review.” | concepts[80] |
| concept | pre- and post-publication peer review mechanisms — not defined in this paper's own words; public review processes operating before and after formal publication — “Both pre- and post-publication peer review mechanisms dramatically accelerate and expand the evaluation process 68 . By sharing preprints, researchers can obtain rapid feedback on their work from a diverse community, rather than waiting several months for a few reviews in the conventional, closed peer review process. Using post-publication services, reviewers can make positive and critical commentary on articles instantly, rather than relying on the laborious, uncertain and lengthy process of authoring a commentary and submitting it to the publishing journal for possible publication, eventually.” | concepts[81] |
| concept | evaluation process — not defined in this paper's own words; the collective assessment of research quality — “Both pre- and post-publication peer review mechanisms dramatically accelerate and expand the evaluation process 68 . By sharing preprints, researchers can obtain rapid feedback on their work from a diverse community, rather than waiting several months for a few reviews in the conventional, closed peer review process. Using post-publication services, reviewers can make positive and critical commentary on articles instantly, rather than relying on the laborious, uncertain and lengthy process of authoring a commentary and submitting it to the publishing journal for possible publication, eventually.” | concepts[82] |
| concept | open data badge — not defined in this paper's own words; a journal signal recognizing public data — “The Center for Open Science has suggested that journals assign a badge to articles with open data (as well as to other open practices such as pre-registration and open materials). The main purpose of the badges is to signal that the journal values these practices. The journal Psychological Science has adopted these badges, and there is evidence that the open data badge has had a positive effect, increasing data sharing by more than tenfold ( Fig. 2 ).” | concepts[83] |
| concept | data sharing — not defined in this paper's own words; making research data available to others — “The Center for Open Science has suggested that journals assign a badge to articles with open data (as well as to other open practices such as pre-registration and open materials). The main purpose of the badges is to signal that the journal values these practices. The journal Psychological Science has adopted these badges, and there is evidence that the open data badge has had a positive effect, increasing data sharing by more than tenfold ( Fig. 2 ).” | concepts[84] |
| concept | Positive, novel and clean results — not defined in this paper's own words; outcomes favored by conventional publication incentives — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | concepts[85] |
| concept | negative results — not defined in this paper's own words; results disfavored relative to positive novel findings — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | concepts[86] |
| concept | replications — not defined in this paper's own words; repeated tests of prior findings — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | concepts[87] |
| concept | false positives — not defined in this paper's own words; published positive findings encouraged by dysfunctional incentives — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | concepts[88] |
| concept | incentive structures — not defined in this paper's own words; appointment and promotion reward rules shaping research strategy — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | concepts[89] |
| concept | researcher behaviour — not defined in this paper's own words; research choices altered by reward structures — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | concepts[90] |
| concept | rigorous, transparent and reproducible research — not defined in this paper's own words; the practices that coordinated incentives should reward — “Funders, publishers, societies, institutions, editors, reviewers and authors all contribute to the cultural norms that create and sustain dysfunctional incentives. Changing the incentives is therefore a problem that requires a coordinated effort by all stakeholders to alter reward structures. There will always be incentives for innovative outcomes — those who discover new things will be rewarded more than those who do not. However, there can also be incentives for efficiency and effectiveness — those who conduct rigorous, transparent and reproducible research could be rewarded more than those who do not. There are promising examples of effective interventions for nudging incentives. For example, journals are adopting badges to acknowledge open practices ( Fig. 2 ), Registered Reports as a results-blind publishing model (see Box 3 ) and TOP guidelines to promote openness and transparency. Funders are also adopting transparency requirements, and piloting funding mechanisms to promote reproducibility such as the Netherlands Organisation for Scientific Research (NWO) and the US National Science Foundation's Directorate of Social, Behavioral and Economic Sciences, both of which have announced funding opportunities for replication studies. Institutions are wrestling with policy and infrastructure adjustments to promote data sharing, and there are hints of open-science practices becoming part of hiring and performance evaluation (for example, http://www.nicebread.de/open-science-hiring-practices/ ). Collectively, and at scale, such efforts can shift incentives such that what is good for the scientist is also good for science — rigorous, transparent and reproducible research practices producing credible results.” | concepts[91] |
| concept | self-correction of the scientific process itself — not defined in this paper's own words; continuous examination and improvement of research practice — “These cautions are not a rationale for inaction. Reproducible research practices are at the heart of sound research and integral to the scientific method. How best to achieve rigorous and efficient knowledge accumulation is a scientific question; the most effective solutions will be identified by a combination of brilliant hypothesizing and blind luck, by iterative examination of the effectiveness of each change, and by a winnowing of many possibilities to the broadly enacted few. True understanding of how best to structure and incentivize science will emerge slowly and will never be finished. That is how science works. The key to fostering a robust metascience that evaluates and improves practices is that the stakeholders of science must not embrace the status quo, but instead pursue self-examination continuously for improvement and self-correction of the scientific process itself.” | concepts[92] |
| concept | scientific study of science itself — not defined in this paper's own words; the explicit definitional gloss for metascience — “What proportion of published research is likely to be false? Low sample size, small effect sizes, data dredging (also known as P -hacking), conflicts of interest, large numbers of scientists working competitively in silos without combining their efforts, and so on, may conspire to dramatically increase the probability that a published finding is incorrect 1 . The field of metascience — the scientific study of science itself — is flourishing and has generated substantial empirical evidence for the existence and prevalence of threats to efficiency in knowledge accumulation (refs 2 , 3 , 4 , 5 , 6 , 7 ; Fig. 1 ).” | concepts[93] |
| concept | robustness of published research — not defined in this paper's own words; resilience of published findings to the identified threats — “Threats to reproducible science. An idealized version of the hypothetico-deductive model of the scientific method is shown. Various potential threats to this model exist (indicated in red), including lack of replication 5 , hypothesizing after the results are known (HARKing) 7 , poor study design, low statistical power 2 , analytical flexibility 51 , P -hacking 4 , publication bias 3 and lack of data sharing 6 . Together these will serve to undermine the robustness of published research, and may also impact on the ability of science to self-correct.” | concepts[94] |
| concept | false conclusions — not defined in this paper's own words; incorrect conclusions encouraged by interacting cognitive biases — “A hallmark of scientific creativity is the ability to see novel and unexpected patterns in data. John Snow's identification of links between cholera and water supply 17 , Paul Broca's work on language lateralization 18 and Jocelyn Bell Burnell's discovery of pulsars 19 are examples of breakthroughs achieved by interpreting observations in a new way. However, a major challenge for scientists is to be open to new and important insights while simultaneously avoiding being misled by our tendency to see structure in randomness. The combination of apophenia (the tendency to see patterns in random data), confirmation bias (the tendency to focus on evidence that is in line with our expectations or favoured explanation) and hindsight bias (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusions 20 . Thomas Levenson documents the example of astronomers who became convinced they had seen the fictitious planet Vulcan because their contemporary theories predicted its existence 21 . Experimenter effects are an example of this kind of bias 22 .” | concepts[95] |
| concept | Common misperceptions — not defined in this paper's own words; recurring misunderstandings of statistical analysis and design — “Improving methodological training. Research design and statistical analysis are mutually dependent. Common misperceptions, such as the interpretation of P values 31 , limitations of null-hypothesis significance testing 32 , the meaning and importance of statistical power 2 , the accuracy of reported effect sizes 33 , and the likelihood that a sample size that generated a statistically significant finding will also be adequate to replicate a true finding 34 , could all be addressed through improved statistical training. Similarly, basic design principles are important, such as blinding to reduce experimenter bias, randomization or counterbalancing to control for confounding, and the use of within-subjects designs, where possible, to maximize power. However, integrative training in research practices that can protect oneself against cognitive biases and the effects of distorted incentives is arguably more important. Moreover, statistical and methodological best practices are under constant revision and improvement, so that senior as well as junior researchers need continuing methodological education, not least because much training of early-career researchers is informal and flows from their supervisor or mentor. A failure to adopt advances in methodology — such as the very slow progress in increasing statistical power 35 , 36 — may be partly a function of failing to inculcate a continuing professional education and development ethic.” | concepts[96] |
| concept | lack of resources to improve power — not defined in this paper's own words; the resource constraint that team science is proposed to address — “Encouraging collaboration and team science. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical power increases the likelihood of obtaining both false-positive and false-negative results 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentives, poor understanding of the consequences of low power, and lack of resources to improve power. Team science is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaboration across many study sites facilitates high-powered designs and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.” | concepts[97] |
| concept | research quality — not defined in this paper's own words; the quality that reporting alone may not maximize — “Improving the quality of reporting. Pre-registration will improve discoverability of research, but discoverability does not guarantee usability. Poor usability reflects difficulty in evaluating what was done, in reusing the methodology to assess reproducibility, and in incorporating the evidence into systematic reviews and meta-analyses. Improving the quality and transparency in the reporting of research is necessary to address this. The Transparency and Openness Promotion (TOP) guidelines offer standards as a basis for journals and funders to incentivize or require greater transparency in planning and reporting of research 54 . TOP provides principles for how transparency and usability can be increased, while other guidelines provide concrete steps for how to maximize the quality of reporting in particular areas. For example, the Consolidated Standards of Reporting Trials (CONSORT) statement provides guidance for clear, complete and accurate reporting of randomized controlled trials 55 – 57 . Over 300 reporting guidelines now exist for observational studies, prognostic studies, predictive models, diagnostic tests, systematic reviews and meta-analyses in humans, a large variety of studies using different laboratory methods, and animal studies. The Equator Network ( http://www.equator-network.org/ ) aggregates these guidelines to improve discoverability 58 . There are also guidelines for improving the reporting of research planning; for example, the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement for reporting of systematic reviews and meta-analyses 59 , and PRISMA-P for protocols of systematic reviews 60 . The Preregistration Challenge workflow and the pre-registration recipe for social-behavioural research 61 also illustrate guidelines for reporting research plans.” | concepts[98] |
| concept | adequately evaluate results — not defined in this paper's own words; the reader's ability to assess findings from full information — “Full disclosure refers to the process of describing in full the study design and data collected that underlie the results reported, rather than a curated version of the design, and/or a subset of the data collected. The need for disclosure is clear: in order to adequately evaluate results we need to know how they were obtained. For example, the informational value of a dependent variable exhibiting an effect of interest is different if only one variable was collected or if fifteen were. The probability of a single variable achieving P < 0.05 just by chance is 5%, but the probability of one of fifteen variables achieving P < 0.05 is 54% 1 . It is obvious that cherry-picking one from fifteen variables invalidates the results unless it is clear that this has happened. If readers know, then they can adjust their interpretation accordingly. From this simple fact it follows that if authors do not tell us whether they collected one or fifteen variables readers cannot evaluate their research 51 .” | concepts[99] |
| concept | accountability — not defined in this paper's own words; one field-wide advantage of sharing data in repositories — “Sharing data in public repositories offers field-wide advantages in terms of accountability, data longevity, efficiency and quality (for example, reanalyses may detect crucial mistakes or even data fabrication) 84 . Unfortunately, many scientific disciplines, including most of those devoted to the study of human behaviour, do not have a culture that values open data 6 . In the past, data sharing has rarely been enforced or facilitated. Recent initiatives, however, aim to change the normative culture. Hopefully, these initiatives will change the culture on data sharing. Once accepted as the norm, we doubt that data sharing will ever go out of fashion.” | concepts[100] |
| concept | publication — not defined in this paper's own words; the dissemination outcome shaped by result-dependent incentives — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | concepts[101] |
| concept | scientific value of research — not defined in this paper's own words; the social value that better incentives are intended to improve — “Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentives ultimately increase the likelihood of false positives in the published literature 71 . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results. For example, with simulations, Munafò and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research.” | concepts[102] |
| concept | reward structures — not defined in this paper's own words; institutional incentives sustained by scientific stakeholders — “Funders, publishers, societies, institutions, editors, reviewers and authors all contribute to the cultural norms that create and sustain dysfunctional incentives. Changing the incentives is therefore a problem that requires a coordinated effort by all stakeholders to alter reward structures. There will always be incentives for innovative outcomes — those who discover new things will be rewarded more than those who do not. However, there can also be incentives for efficiency and effectiveness — those who conduct rigorous, transparent and reproducible research could be rewarded more than those who do not. There are promising examples of effective interventions for nudging incentives. For example, journals are adopting badges to acknowledge open practices ( Fig. 2 ), Registered Reports as a results-blind publishing model (see Box 3 ) and TOP guidelines to promote openness and transparency. Funders are also adopting transparency requirements, and piloting funding mechanisms to promote reproducibility such as the Netherlands Organisation for Scientific Research (NWO) and the US National Science Foundation's Directorate of Social, Behavioral and Economic Sciences, both of which have announced funding opportunities for replication studies. Institutions are wrestling with policy and infrastructure adjustments to promote data sharing, and there are hints of open-science practices becoming part of hiring and performance evaluation (for example, http://www.nicebread.de/open-science-hiring-practices/ ). Collectively, and at scale, such efforts can shift incentives such that what is good for the scientist is also good for science — rigorous, transparent and reproducible research practices producing credible results.” | concepts[103] |
| concept | efficiency and reliability of science — not defined in this paper's own words; joint criteria against which proposed reforms must be assessed — “The challenges to reproducible science are systemic and cultural, but that does not mean they cannot be met. The measures we have described constitute practical and achievable steps toward improving rigor and reproducibility. All of them have shown some effectiveness, and are well suited to wider adoption, evaluation and improvement. Equally, these proposals are not an exhaustive list; there are many other nascent and maturing ideas for making research practices more efficient and reliable 73 . Offering a solution to a problem does not guarantee its effectiveness, and making changes to cultural norms and incentives can spur additional behavioural changes that are difficult to anticipate. Some solutions may be ineffective or even harmful to the efficiency and reliability of science, even if conceptually they appear sensible.” | concepts[104] |
| concept | Proposed solutions — not defined in this paper's own words; candidate reforms that may create new problems or fail — “The field of metascience (or metaresearch) is growing rapidly, with over 2,000 relevant publications accruing annually 16 . Much of that literature constitutes the evaluation of existing practices and the identification of alternative approaches. What was previously taken for granted may be questioned, such as widely used statistical methods; for example, the most popular methods and software for spatial extent analysis in fMRI imaging were recently shown to produce unacceptably high false-positive rates 74 . Proposed solutions may also give rise to other challenges; for example, while replication is a hallmark for reinforcing trust in scientific results, there is uncertainty about which studies deserve to be replicated and what would be the most efficient replication strategies. Moreover, a recent simulation suggests that replication alone may not suffice to rid us of false results 71 .” | concepts[105] |
| concept | replication — not defined in this paper's own words; repeating studies to reinforce trust in results — “The field of metascience (or metaresearch) is growing rapidly, with over 2,000 relevant publications accruing annually 16 . Much of that literature constitutes the evaluation of existing practices and the identification of alternative approaches. What was previously taken for granted may be questioned, such as widely used statistical methods; for example, the most popular methods and software for spatial extent analysis in fMRI imaging were recently shown to produce unacceptably high false-positive rates 74 . Proposed solutions may also give rise to other challenges; for example, while replication is a hallmark for reinforcing trust in scientific results, there is uncertainty about which studies deserve to be replicated and what would be the most efficient replication strategies. Moreover, a recent simulation suggests that replication alone may not suffice to rid us of false results 71 .” | concepts[106] |
| concept | false results — not defined in this paper's own words; incorrect findings that replication alone may not eliminate — “The field of metascience (or metaresearch) is growing rapidly, with over 2,000 relevant publications accruing annually 16 . Much of that literature constitutes the evaluation of existing practices and the identification of alternative approaches. What was previously taken for granted may be questioned, such as widely used statistical methods; for example, the most popular methods and software for spatial extent analysis in fMRI imaging were recently shown to produce unacceptably high false-positive rates 74 . Proposed solutions may also give rise to other challenges; for example, while replication is a hallmark for reinforcing trust in scientific results, there is uncertainty about which studies deserve to be replicated and what would be the most efficient replication strategies. Moreover, a recent simulation suggests that replication alone may not suffice to rid us of false results 71 .” | concepts[107] |
| concept | status quo — not defined in this paper's own words; existing scientific practice that stakeholders should not accept uncritically — “These cautions are not a rationale for inaction. Reproducible research practices are at the heart of sound research and integral to the scientific method. How best to achieve rigorous and efficient knowledge accumulation is a scientific question; the most effective solutions will be identified by a combination of brilliant hypothesizing and blind luck, by iterative examination of the effectiveness of each change, and by a winnowing of many possibilities to the broadly enacted few. True understanding of how best to structure and incentivize science will emerge slowly and will never be finished. That is how science works. The key to fostering a robust metascience that evaluates and improves practices is that the stakeholders of science must not embrace the status quo, but instead pursue self-examination continuously for improvement and self-correction of the scientific process itself.” | concepts[108] |
Full text — tagged
A manifesto for reproducible scienceterm Abstract Improving the reliability and efficiency of scientific researchterm will increase the credibility of the published scientific literatureterm and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific processterm: methodsterm, reporting and disseminationterm, reproducibilityterm, evaluationterm and incentivesterm. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluationterm and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparencyterm, reproducibilityterm and efficiency of scientific researchterm.
Subject terms: Social sciences What proportion of published research is likely to be false? Low sample sizeterm, small effect sizesterm, data dredgingterm (also known as P -hackingterm), conflicts of interestterm, large numbers of scientists working competitively in silos without combining their efforts, and so on, may conspire to dramatically increase the probability that a published finding is incorrect 1 . The field of metascienceterm — the scientific study of science itselfterm — is flourishing and has generated substantial empirical evidence for the existence and prevalence of threats to efficiency in knowledge accumulationterm (refs 2 , 3 , 4 , 5 , 6 , 7 ; Fig. 1 ).
Threats to reproducible scienceterm. An idealized version of the hypothetico-deductive model of the scientific methodterm is shown. Various potential threats to this model exist (indicated in red), including lack of replicationterm 5 , hypothesizing after the results are known (HARKingterm) 7 , poor study designterm, low statistical powerterm 2 , analytical flexibilityterm 51 , P -hackingterm 4 , publication biasterm 3 and lack of data sharingterm 6 . Together these will serve to undermine the robustness of published researchterm, and may also impact on the ability of science to self-correct.
Data from many fields suggests reproducibilityterm is lower than is desirable 8 , 9 , 10 , 11 , 12 , 13 , 14 ; one analysis estimates that 85% of biomedical research efforts are wasted 14 , while 90% of respondents to a recent survey in Nature agreed that there is a ‘reproducibilityterm crisis’ 15 . Whether ‘crisis’ is the appropriate term to describe the current state or trajectory of science is debatable, but accumulated evidence indicates that there is substantial room for improvement with regard to research practices to maximize the efficiency of the research community's use of the public's financial investment in research.
Here we propose a series of measures that we believe will improve research efficiency and robustness of scientific findings by directly targeting specific threats to reproducible scienceterm. We argue for the adoption, evaluationterm and ongoing improvement of these measures to optimize the pace and efficiency of knowledge accumulationterm. The measures are organized into the following categories 16 : methodsterm, reporting and disseminationterm, reproducibilityterm, evaluationterm and incentivesterm. They are not intended to be exhaustive, but provide a broad, practical and evidence-based set of actions that can be implemented by researchers, institutions, journals and funders. The measures and their current implementation are summarized in Table 1 .
A manifesto for reproducible scienceterm.
The problem A hallmark of scientific creativity is the ability to see novel and unexpected patterns in data. John Snow's identification of links between cholera and water supply 17 , Paul Broca's work on language lateralization 18 and Jocelyn Bell Burnell's discovery of pulsars 19 are examples of breakthroughs achieved by interpreting observations in a new way. However, a major challenge for scientists is to be open to new and important insights while simultaneously avoiding being misled by our tendency to see structure in randomness. The combination of apopheniaterm (the tendency to see patterns in random data), confirmation biasterm (the tendency to focus on evidence that is in line with our expectations or favoured explanation) and hindsight biasterm (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusionsterm 20 . Thomas Levenson documents the example of astronomers who became convinced they had seen the fictitious planet Vulcan because their contemporary theories predicted its existence 21 . Experimenter effectsterm are an example of this kind of bias 22 .
Over-interpretation of noise is facilitated by the extent to which data analysisterm is rapid, flexible and automated 23 . In a high-dimensional datasetterm, there may be hundreds or thousands of reasonable alternative approaches to analysing the same data 24 , 25 . For example, in a systematic reviewterm·mth of functional magnetic resonance imaging (fMRI) studies, Carp showed that there were almost as many unique analytical pipelinesterm as there were studies 26 . If several thousand potential analytical pipelinesterm can be applied to high-dimensional data, the generation of false-positive findingsterm is highly likely. For example, applying almost 7,000 analytical pipelinesterm to a single fMRI dataset resulted in over 90% of brain voxels showing significant activation in at least one analysis 27 .
During data analysisterm it can be difficult for researchers to recognize P -hackingterm 28 or data dredgingterm because confirmation and hindsight biasesterm can encourage the acceptance of outcomes that fit expectations or desires as appropriate, and the rejection of outcomes that do not as the result of suboptimal designs or analyses. Hypotheses may emerge that fit the data and are then reported without indication or recognition of their post hoc origin 7 . This, unfortunately, is not scientific discovery, but self-deception 29 . Uncontrolled, it can dramatically increase the false discovery rateterm. We need measures to counter the natural tendency of enthusiastic scientists who are motivated by discovery to see patterns in noise.
Methodsterm In this section we describe measures that can be implemented when performing research (including, for example, study design, methodsterm, statistics, and collaboration).
Protecting against cognitive biasesterm. There is a substantial literature on the difficulty of avoiding cognitive biasesterm. An effective solution to mitigate self-deception and unwanted biases is blindingterm. In some research contexts, participants and data collectors can be blinded to the experimental condition that participants are assigned to, and to the research hypotheses, while the data analyst can be blinded to key parts of the data. For example, during data preparation and cleaning, the identity of experimental conditions or the variable labels can be masked so that the output is not interpretable in terms of the research hypothesis. In some physical sciences this approach has been extended to include deliberate perturbations in or masking of data to allow data preparation (for example, identification of outliers) to proceed without the analyst being able to see the corresponding results 30 . Pre-registrationterm of the study design, primary outcome(s) and analysis plan (see ‘Promoting study pre-registrationterm’ section’, below) is a highly effective form of blindingterm because the data do not exist and the outcomes are not yet known.
Improving methodological trainingterm. Research designterm and statistical analysisterm are mutually dependent. Common misperceptionsterm, such as the interpretation of P values 31 , limitations of null-hypothesis significance testing 32 , the meaning and importance of statistical power 2 , the accuracy of reported effect sizes 33 , and the likelihood that a sample size that generated a statistically significant finding will also be adequate to replicate a true finding 34 , could all be addressed through improved statistical training. Similarly, basic design principles are important, such as blindingterm to reduce experimenter bias, randomizationterm or counterbalancingterm to control for confounding, and the use of within-subjects designsterm, where possible, to maximize power. However, integrative training in research practices that can protect oneself against cognitive biasesterm and the effects of distorted incentivesterm is arguably more important. Moreover, statistical and methodological best practices are under constant revision and improvement, so that senior as well as junior researchers need continuing methodological educationterm, not least because much training of early-career researchers is informal and flows from their supervisor or mentor. A failure to adopt advances in methodology — such as the very slow progress in increasing statistical power 35 , 36 — may be partly a function of failing to inculcate a continuing professional education and development ethic.
Without formal requirements for continuing education, the most effective solutions may be to develop educational resources that are accessible, easy-to-digest and immediately and effectively applicable to research (for example, brief, web-based modules for specific topics, and combinations of modules that are customized for particular research applications). A modular approach simplifies the process of iterative updating of those materials. Demonstration software and hands-on examples may also make the lessons and implications particularly tangible to researchers at any career stage: the Experimental Design Assistant ( https://eda.nc3rs.org.uk ) supports research designterm for whole animal experiments, while P -hacker ( http://shinyapps.org/apps/p-hacker/ ) shows just how easy it is to generate apparently statistically significant findings by exploiting analytic flexibility.
Implementing independent methodological supportterm. The need for independent methodological supportterm is well-established in some areas — many clinical trials, for example, have multidisciplinary trial steering committees to provide advice and oversee the design and conduct of the trial. The need for these committees grew out of the well-understood financial conflicts of interestterm that exist in many clinical trials. The sponsor of a trial may be the company manufacturing the product, and any intentional or unintentional influence can distort the study design, analysis and interpretation of results for the ultimate financial benefit of the manufacturer at the cost of the accuracy of the science and the health benefit to the consumers 37 , 38 . Non-financial conflicts of interestterm also exist, such as the beliefs and preconceptions of individual scientists and the stakes that researchers have in obtaining publishable results in order to progress their career 39 , 40 . Including independent researchers (particularly methodologists with no personal investment in a research topic) in the design, monitoring, analysis or interpretation of research outcomes may mitigate some of those influences, and can be done either at the level of the individual research project or through a process facilitated by a funding agency (see Box 1 ).
Encouraging collaboration and team scienceterm. Studies of statistical power persistently find it to be below (sometimes well below) 50%, across both time and the different disciplines studied 2 , 35 , 36 . Low statistical powerterm increases the likelihood of obtaining both false-positive and false-negative resultsterm 2 , meaning that it offers no advantage if the purpose is to accumulate knowledge. Despite this, low-powered research persists because of dysfunctional incentivesterm, poor understanding of the consequences of low power, and lack of resources to improve powerterm. Team scienceterm is a solution to the latter problem — instead of relying on the limited resources of single investigators, distributed collaborationterm across many study sites facilitates high-powered designsterm and greater potential for testing generalizability across the settings and populations sampled. This also brings greater scope for multiple theoretical and disciplinary perspectives, and a diverse range of research cultures and experiences, to be incorporated into a research project.
Multi-centre and collaborative efforts have a long and successful tradition in fields such as randomized controlled trials in some areas of clinical medicine, and in genetic association analyses, and have improved the robustness of the resulting research literatures. Multi-site collaborative projectsterm have also been advocated for other types of research, such as animal studies 41 – 43 , in an effort to maximize their power, enhance standardization, and optimize transparencyterm and protection from biases. The Many Labsterm projects illustrate this potential in the social and behavioural sciences, with dozens of laboratories implementing the same research protocol to obtain highly precise estimates of effect sizes, and evaluate variability across samples and settings 44 , 45 . It is also possible, and desirable, to incorporate a team scienceterm ethos into student training (see Box 2 ).
CHDI Foundation — a privately-funded non-profit drug-development organization targeting Huntington's disease — convened a working group in 2013 to identify practical and viable steps that could be taken to help ensure the rigor of their research 76 . One concrete product of this meeting was the establishment of the Independent Statistical Standing Committee (ISSC; http://chdifoundation.org/independent-statistical-standing-committee/ ) designed to provide independent, unbiased and objective evaluationterm and expert advice regarding all aspects of experimental design and statistics. CHDI has made this resource available to the wider Huntington's disease research community on a priority basis. The ISSC is comprised of individuals with specific expertise in research designterm and statistics. Critically, committee members are not themselves engaged in Huntington's disease research, and have no investment in study results, or other conflicts of interestterm. The committee provides a number of services, including (but not limited to) provision of expert assistance in developing protocols and statistical analysisterm plans, and evaluationterm of prepared study protocolsterm. Their oversight and input, particularly at the study design stage, may mitigate low statistical powerterm, inadequate study design, and flexibility in data analysisterm and reporting 6 , 71 . As recently highlighted, “asking questions at the design stage can save headaches at the analysis stage: careful data collection can greatly simplify analysis and make it more rigorous” 77 .
Student assessment requirements, and limited access to populations of interest, may hinder extensive collaboration within a single institution, but it could be achieved across multiple institutions in the form of a distributed student project. Under this model, academics and students from several institutions would form a consortium, collaboratively develop a research question, protocol and analysis plan, and publicly pre-register it prior to data collection. The protocol would be implemented by each student at each participating centre, and the resulting data pooled for analysis. Consortium meetings before and after data collection could be used to integrate training in research designterm, while offering opportunities for creative input from the students. Conclusions based on results would be mutually agreed in preparation for wider disseminationterm, using inclusive authorship conventions such as those adopted by genetic consortia. Students would learn rigorous research methodsterm through active participation in research that is sufficiently well designed and conducted to be genuinely meaningful. Critically, collaborative team scienceterm would be instilled at an early stage of training.
The Collaborative Replicationsterm and Education Project (CREP; https://osf.io/wfc6u/ ) is an example of this concept in psychology, albeit in a more centralized form. A coordinating team identifies recently published research that could be replicated in the context of a semester-long undergraduate course on research methodsterm. A central commons provides the materials and guidance to incorporate the replicationsterm into projects or classes, and the data collected across sites are aggregated into manuscripts for publicationterm. The Pipeline 78 and Many Labsterm 44 , 45 projects also offer opportunities to contribute to large-scale replicationterm efforts with coordinated data collection across many locations simultaneously.
Reporting and disseminationterm In this section we describe measures that can be implemented when communicating research (including, for example, reporting standards, study pre-registrationterm, and disclosing conflicts of interestterm).
Promoting study pre-registrationterm. Pre-registrationterm of study protocolsterm for randomized controlled trials in clinical medicine has become standard practice 46 . In its simplest form it may simply comprise the registration of the basic study design, but it can also include a detailed pre-specification of the study procedures, outcomes and statistical analysisterm plan. It was introduced to address two problems: publication biasterm and analytical flexibilityterm (in particular outcome switchingterm in the case of clinical medicine). Publication biasterm 47 , also known as the file drawer problemterm 48 , refers to the fact that many more studies are conducted than published. Studies that obtain positive and novel results are more likely to be published than studies that obtain negative resultsterm or report replicationsterm of prior results 47 , 49 , 50 . The consequence is that the published literature indicates stronger evidence for findings than exists in reality. Outcome switchingterm refers to the possibility of changing the outcomes of interest in the study depending on the observed results. A researcher may include ten variables that could be considered outcomes of the research, and — once the results are known — intentionally or unintentionally select the subset of outcomes that show statistically significant results as the outcomes of interest. The consequence is an increase in the likelihood that reported results are spurious by leveraging chance, while negative evidence gets ignored. This is one of several related research practices that can inflate spurious findings when analysis decisions are made with knowledge of the observed data, such as selection of models, exclusion rules and covariates. Such data-contingent analysis decisionsterm constitute what has become known as P -hackingterm 51 , and pre-registrationterm can protect against all of these.
The strongest form of pre-registrationterm involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research. In principle, this addresses publication biasterm by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switchingterm, and P -hackingterm more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction between data-independent confirmatory researchterm that is important for testing hypotheses, and data-contingent exploratory researchterm that is important for generating hypotheses.
While pre-registrationterm is now common in some areas of clinical medicine (due to requirements by journals and regulatory bodies, such as the Food and Drug Administration in the United States and the European Medicines Agency in the European Union), it is rare in the social and behavioural sciences. However, support for study pre-registrationterm is increasing; websites such as the Open Scienceterm Framework ( http://osf.io/ ) and AsPredicted ( http://AsPredicted.org/ ) offer services to pre-register studies, the Preregistrationterm Challenge offers education and incentivesterm to conduct pre-registered research ( http://cos.io/prereg ), and journals are adopting the Registered Reportsterm publishing format 52 , 53 to encourage pre-registrationterm and add results-blind peer reviewterm (see Box 3 ).
Improving the quality of reporting. Pre-registrationterm will improve discoverability of research, but discoverability does not guarantee usability. Poor usability reflects difficulty in evaluating what was done, in reusing the methodology to assess reproducibilityterm, and in incorporating the evidence into systematic reviewsterm·mth and meta-analyses. Improving the quality and transparencyterm in the reporting of research is necessary to address this. The Transparencyterm and Openness Promotion (TOP) guidelines offer standards as a basis for journals and funders to incentivize or require greater transparencyterm in planning and reporting of research 54 . TOP provides principles for how transparencyterm and usability can be increased, while other guidelines provide concrete steps for how to maximize the quality of reporting in particular areas. For example, the Consolidated Standards of Reporting Trials (CONSORT) statement provides guidance for clear, complete and accurate reporting of randomized controlled trials 55 – 57 . Over 300 reporting guidelinesterm now exist for observational studies, prognostic studies, predictive models, diagnostic tests, systematic reviewsterm·mth and meta-analyses in humans, a large variety of studies using different laboratory methodsterm, and animal studies. The Equator Network ( http://www.equator-network.org/ ) aggregates these guidelines to improve discoverability 58 . There are also guidelines for improving the reporting of research planning; for example, the Preferred Reporting Items for Systematic Reviewsterm·mth and Meta-analyses (PRISMA) statement for reporting of systematic reviewsterm·mth and meta-analyses 59 , and PRISMA-P for protocols of systematic reviewsterm·mth 60 . The Preregistrationterm Challenge workflow and the pre-registrationterm recipe for social-behavioural research 61 also illustrate guidelines for reporting research plans.
The success of reporting guidelinesterm depends on their adoption and effective use. The social and behavioural sciences are behind the biomedical sciences in their adoption of reporting guidelinesterm for research, although with rapid adoption of the TOP guidelinesterm and related developments by journals and funders that gap may be closing. However, improved reporting may be insufficient on its own to maximize research qualityterm. Reporting guidelinesterm are easily perceived by researchers as bureaucratic exercises rather than means of improving research and reporting. Even with pre-registrationterm of clinical trials, one study observed that just 13% of trials published outcomes completely consistent with the pre-registered commitments. Most publicationsterm of the trials did not report pre-registered outcomes and added new outcomes that were not part of the registered design (see www.COMPare-trials.org ). Franco and colleagues observed similar findings in psychology 62 ; using protocol pre-registrationsterm and public data from the Time-sharing Experiments for the Social Sciences project ( http://www.tessexperiments.org/ ), they found that 40% of published reports failed to mention one or more of the experimental conditions of the experiments, and approximately 70% of published reports failed to mention one or more of the outcome measures included in the study. Moreover, outcome measures that were not included were much more likely to be negative resultsterm and associated with smaller effect sizes than outcome measures that were included.
The positive outcome of reporting guidelinesterm is that they make it possible to detect and study these behaviours and their impact. Otherwise, these behaviours are simply unknowable in any systematic way. The negative outcome is the empirical evidence that reporting guidelinesterm may be necessary, but will not alone be sufficient, to address reporting biases. The impact of guidelines and how best to optimize their use and impact will be best assessed by randomized trials (see Box 4 ).
The Registered Reportsterm (RR) initiative seeks to eliminate various forms of bias in hypothesis-driven research 52 , 53 , and in particular, the evaluationterm of a study based on the results. Unlike conventional journal articles, RRs split the peer review processterm into two stages, before and after results are known. At the first stage, reviewers and editors assess a detailed protocol that includes the study rationale, procedure and a detailed analysis plan. Following favourable reviewsterm·mth (and probably revision to meet strict methodological standards), the journal offers in-principle acceptanceterm: publicationterm of study outcomes is guaranteed provided the authors adhere to the approved protocol, the study meets pre-specified quality checks, and conclusions are appropriately evidence-bound. Once the study is completed, the authors resubmit a complete manuscript that includes the results and discussion. The article is published at the end of this two-stage process. By accepting articles before results are known, RRs prevent publication biasterm. By reviewing the hypotheses and analysis plans in advance, RRs should also help neutralize P- hacking and HARKingterm (hypothesizing after the results are known) by authors, and CARKing (critiquing after the results are known) by reviewers with their own investments in the research outcomes, although empirical evidence will be required to confirm that this is the case.
Perhaps the most commonly voiced objection to RRs is that the format somehow limits exploration or creativity by requiring authors to adhere to a pre-specified methodology. However, RRs place no restrictions on creative analysis practices or serendipity. Authors are free to report the outcomes of any unregistered exploratory analyses, provided such tests are clearly labelled as post hoc . Thus, the sole requirement is that exploratory outcomes are identified transparently as exploratory (for a list of frequently asked questions see https://cos.io/rr/#faq ). Of course, RRs are not intended for research that is solely exploratory.
As of November 2016, RRs have been adopted by over 40 journals, including Nature Human Behaviour , covering a wide range of life, social and physical sciences (for a curated list see https://cos.io/rr/#journals ). The concept also opens the door to alternative forms of research funding that place a premium on transparencyterm and reproducibilityterm. For example, authors could submit a detailed proposal before they have funding for their research. Following simultaneous reviewterm·mth by both the funder and the journal, the strongest proposals would be offered financial support by the funder and in-principle acceptanceterm for publicationterm by the journal ( https://cos.io/rr/#funders ).
In medicine there is strong evidence for the effectiveness of CONSORT guidelines — journals that do not endorse the CONSORT statementterm show poorer reporting quality compared with endorsing journals 79 . For the ARRIVE (Animal Research: Reporting of In Vivo Experiments) guidelines 80 , studies comparing the reporting of ARRIVE items in specific fields of research before and after the guidelines were published report mixed results 81 – 83 . A randomized controlled trial is in progress to assess the impact of mandating a completed ARRIVE checklist with manuscript submissions on the quality of reporting in published articles ( https://ecrf1.clinicaltrials.ed.ac.uk/iicarus ). The success of these efforts will require journals and funders to adopt guidelines and support the community's iterative evaluationterm and improvement cycle.
Reproducibilityterm In this section we describe measures that can be implemented to support verification of research (including, for example, sharing data and methodsterm).
Promoting transparencyterm and open scienceterm. Science is a social enterprise: independent and collaborative groups work to accumulate knowledge as a public good. The credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology appliedterm, the data acquiredterm, and the process of methodology implementationterm, data analysisterm and outcome interpretationterm. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidenceterm. However, without transparencyterm, claims only achieve credibility based on trustterm in the confidence or authority of the originator. Transparencyterm is superior to trustterm.
Open scienceterm refers to the process of making the content and process of producing evidence and claims transparent and accessible to others. Transparencyterm is a scientific ideal, and adding ‘open’ should therefore be redundant. In reality, science often lacks openness: many published articles are not available to people without a personal or institutional subscription, and most data, materials and code supporting research outcomes are not made accessible, for example, in a public repository (refs 63 , 64 ; Box 5 ).
Very little of the research process (for example, study protocolsterm, analysis workflows, peer reviewterm) is accessible because, historically, there have been few opportunities to make it accessible even if one wanted to do so. This has motivated calls for open accessterm, open dataterm and open workflowsterm (including analysis pipelines), but there are substantial barriers to meeting these ideals, including vested financial interests (particularly in scholarly publishing) and few incentivesterm for researchers to pursue open practices. For example, current incentive structuresterm promote the publicationterm of ‘clean’ narratives, which may require the incomplete reporting of study procedures or results. Nevertheless, change is occurring. The TOP guidelinesterm 54 , 65 promote open practices, while an increasing number of journals and funders require open practices (for example, open dataterm), with some offering their researchers free, immediate open-access publicationterm with transparent post-publication peer reviewterm (for example, the Wellcome Trustterm, with the launch of Wellcome Open Research). Policies to promote open scienceterm can include reporting guidelinesterm or specific disclosure statements (see Box 6 ). At the same time, commercial and non-profit organizations are building new infrastructure such as the Open Scienceterm Framework to make transparencyterm easy and desirable for researchers.
Sharing data in public repositoriesterm offers field-wide advantages in terms of accountabilityterm, data longevity, efficiency and quality (for example, reanalyses may detect crucial mistakes or even data fabrication) 84 . Unfortunately, many scientific disciplines, including most of those devoted to the study of human behaviour, do not have a culture that values open dataterm 6 . In the past, data sharingterm has rarely been enforced or facilitated. Recent initiatives, however, aim to change the normative culture. Hopefully, these initiatives will change the culture on data sharingterm. Once accepted as the norm, we doubt that data sharingterm will ever go out of fashion.
Transparencyterm and Openness Promotion (TOP) In 2015, NosekpersName and colleagues 54 proposed author guidelines to help journals and funders adopt transparencyterm and reproducibilityterm policies. As of November 2016 there were 757 journal and 64 organization signatories to the TOP guidelinesterm. For example, the journal Science decided to “publish papers only if the data used in the analysis are available to any researcher for purposes of reproducing or extending the analysis” 65 and the conglomerate of Springer Nature journals adopted similar data-sharing policies.
Badges to acknowledge open-science practices The Center for Open Scienceterm has suggested that journals assign a badge to articles with open dataterm (as well as to other open practices such as pre-registrationterm and open materials). The main purpose of the badges is to signal that the journal values these practices. The journal Psychological Science has adopted these badges, and there is evidence that the open data badgeterm has had a positive effect, increasing data sharingterm by more than tenfold ( Fig. 2 ).
The Peer Reviewers' Openness Initiative Researchers who sign this initiative ( https://opennessinitiative.org ) pledge that as reviewers they will not offer comprehensive reviewterm·mth for any manuscript that does not make data publicly available without a clear reason 85 .
Requirements from funding agencies In recent years, prominent funding agencies such as Research Councils UK in the United Kingdom and the National Institutes of Health (NIH) and National Science Foundation (NSF) in the United States have increased pressure on researchers to share data. For instance, the 2015 NIH Public Access Plan ( https://grants.nih.gov/grants/NIH-Public-Access-Plan.pdf ) states: “NIH intends to make public access to digital scientific data the standard for all NIH-funded research”. Since 2010, NSF requires submission of a data-management plan that stipulates how data will be stored and shared.
Full disclosureterm refers to the process of describing in full the study design and data collected that underlie the results reported, rather than a curated version of the design, and/or a subset of the data collected. The need for disclosure is clear: in order to adequately evaluate resultsterm we need to know how they were obtained. For example, the informational value of a dependent variable exhibiting an effect of interest is different if only one variable was collected or if fifteen were. The probability of a single variable achieving P < 0.05 just by chance is 5%, but the probability of one of fifteen variables achieving P < 0.05 is 54% 1 . It is obvious that cherry-picking one from fifteen variables invalidates the results unless it is clear that this has happened. If readers know, then they can adjust their interpretation accordingly. From this simple fact it follows that if authors do not tell us whether they collected one or fifteen variables readers cannot evaluate their research 51 .
The simplest form of disclosure is for authors to assure readers via an explicit statement in their article that they are disclosing the data fully. This can be seen as a simple item of reporting guidance where extra emphasis is placed on some aspects that are considered most essential to disclose. For example, including the following 21-word statement: “We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study” 86 . Alternatively, a more complex, but also more enforceable and accountable process is for journals to require explicit and specific disclosure statements. The journal Psychological Science , for example, now requires authors to “Confirm that (a) the total number of excluded observations and (b) the reasons for making these exclusions have been reported in the Methodterm section(s)” 87 .
Evaluationterm In this section we describe measures that can be implemented when evaluating research (including, for example, peer reviewterm).
Diversifying peer reviewterm. For most of the history of scientific publishing, two functions have been confounded — evaluationterm and disseminationterm. Journals have provided disseminationterm via sorting and delivering content to the research community, and gatekeeping via peer reviewterm to determine what is worth disseminating. However, with the advent of the internet, individual researchers are no longer dependent on publishers to bind, print and mail their research to subscribers. Disseminationterm is now easy and can be controlled by researchers themselves. For example, preprint servicesterm (arXiv for some physical sciences, bioRxiv and PeerJ for the life sciences, engrXiv for engineering, PsyArXiv for psychology, and SocArXiv and the Social Science Research Network (SSRN) for the social sciences) facilitate easy sharing, sorting and discovery of research prior to publicationterm. This dramatically accelerates the disseminationterm of information to the research community.
With increasing ease of disseminationterm, the role of publishers as a gatekeeper is declining. Nevertheless, the other role of publishing — evaluationterm — remains a vital part of the research enterprise. Conventionally, a journal editor will select a limited number of reviewers to assess the suitability of a submission for a particular journal. However, more diverse evaluation processesterm are now emerging, allowing the collective wisdom of the scientific community to be harnessed 66 . For example, some preprint servicesterm support public comments on manuscripts, a form of pre-publication reviewterm·mth that can be used to improve the manuscript. Other services, such as PubMed Commons and PubPeer, offer public platforms to comment on published works facilitating post-publication peer reviewterm. At the same time, some journals are trialling ‘results-free’ reviewterm·mth, where editorial decisions to accept are based solely on reviewterm·mth of the rationale and study methodsterm alone (that is, results-blind) 67 .
Both pre- and post-publication peer review mechanismsterm dramatically accelerate and expand the evaluation processterm 68 . By sharing preprints, researchers can obtain rapid feedback on their work from a diverse community, rather than waiting several months for a few reviewsterm·mth in the conventional, closed peer review processterm. Using post-publication services, reviewers can make positive and critical commentary on articles instantly, rather than relying on the laborious, uncertain and lengthy process of authoring a commentary and submitting it to the publishing journal for possible publicationterm, eventually.
As public forms of pre- and post-publication reviewterm·mth, these new services introduce the potential for new forms of credit and reputation enhancement 69 . In the conventional model, peer reviewterm is done privately, anonymously and purely as a service. With public commenting systems, a reviewer that chooses to be identifiable may gain (or lose) reputation based on the quality of reviewterm·mth. There are a number of possible and perceived risks of non-anonymous reviewing that reviewers must consider and research must evaluate, but there is evidence that open peer reviewterm improves the quality of reviewsterm·mth received 70 . The opportunity for accelerated scholarly communication may both improve the pace of discovery and diversify the means of being an active contributor to scientific discourse.
Incentivesterm Publicationterm is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean resultsterm are more likely to be published than negative resultsterm, replicationsterm and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy 40 . These incentivesterm ultimately increase the likelihood of false positivesterm in the published literature 71 . Shifting the incentivesterm therefore offers an opportunity to increase the credibility and reproducibilityterm of published results. For example, with simulations, MunafòpersName and Higginson developed an optimality model that predicted the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study 72 . This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structuresterm, by considering more of a researcher's output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviourterm that would ultimately improve the scientific value of researchterm.
Funders, publishers, societies, institutions, editors, reviewers and authors all contribute to the cultural norms that create and sustain dysfunctional incentivesterm. Changing the incentivesterm is therefore a problem that requires a coordinated effort by all stakeholders to alter reward structuresterm. There will always be incentivesterm for innovative outcomes — those who discover new things will be rewarded more than those who do not. However, there can also be incentivesterm for efficiency and effectiveness — those who conduct rigorous, transparent and reproducible researchterm could be rewarded more than those who do not. There are promising examples of effective interventions for nudging incentivesterm. For example, journals are adopting badges to acknowledge open practices ( Fig. 2 ), Registered Reportsterm as a results-blind publishing model (see Box 3 ) and TOP guidelinesterm to promote openness and transparencyterm. Funders are also adopting transparencyterm requirements, and piloting funding mechanisms to promote reproducibilityterm such as the Netherlands Organisation for Scientific Researchterm (NWO) and the US National Science Foundation's Directorate of Social, Behavioral and Economic Sciences, both of which have announced funding opportunities for replicationterm studies. Institutions are wrestling with policy and infrastructure adjustments to promote data sharingterm, and there are hints of open-science practices becoming part of hiring and performance evaluationterm (for example, http://www.nicebread.de/open-science-hiring-practices/ ). Collectively, and at scale, such efforts can shift incentivesterm such that what is good for the scientist is also good for science — rigorous, transparent and reproducible researchterm practices producing credible results.
The impact of introducing badges for data sharingterm. In January 2014, the journal Psychological Science (PSCI) introduced badges for articles with open dataterm. Immediately afterwards, the proportion of articles with open dataterm increased steeply, and by October 2015, 38% of articles in Psychological Science had open dataterm. For comparison journals ( Clinical Psychological Science (CPS), Developmental Psychology (DP), Journal of Experimental Psychology: Learning, Memory and Cognition (JEPLMC) and Journal of Personality and Social Psychology (JPSP)) the proportion of articles with open dataterm remained uniformly low. Figure adapted from ref. 75 , PLoS.
Conclusion The challenges to reproducible scienceterm are systemic and cultural, but that does not mean they cannot be met. The measures we have described constitute practical and achievable steps toward improving rigor and reproducibilityterm. All of them have shown some effectiveness, and are well suited to wider adoption, evaluationterm and improvement. Equally, these proposals are not an exhaustive list; there are many other nascent and maturing ideas for making research practices more efficient and reliable 73 . Offering a solution to a problem does not guarantee its effectiveness, and making changes to cultural norms and incentivesterm can spur additional behavioural changes that are difficult to anticipate. Some solutions may be ineffective or even harmful to the efficiency and reliability of scienceterm, even if conceptually they appear sensible.
The field of metascienceterm (or metaresearch) is growing rapidly, with over 2,000 relevant publicationsterm accruing annually 16 . Much of that literature constitutes the evaluationterm of existing practices and the identification of alternative approaches. What was previously taken for granted may be questioned, such as widely used statistical methodsterm; for example, the most popular methodsterm and software for spatial extent analysis in fMRI imaging were recently shown to produce unacceptably high false-positive rates 74 . Proposed solutionsterm may also give rise to other challenges; for example, while replicationterm is a hallmark for reinforcing trustterm in scientific results, there is uncertainty about which studies deserve to be replicated and what would be the most efficient replicationterm strategies. Moreover, a recent simulation suggests that replicationterm alone may not suffice to rid us of false resultsterm 71 .
These cautions are not a rationale for inaction. Reproducible research practices are at the heart of sound research and integral to the scientific methodterm. How best to achieve rigorous and efficient knowledge accumulationterm is a scientific question; the most effective solutions will be identified by a combination of brilliant hypothesizing and blind luck, by iterative examination of the effectiveness of each change, and by a winnowing of many possibilities to the broadly enacted few. True understanding of how best to structure and incentivize science will emerge slowly and will never be finished. That is how science works. The key to fostering a robust metascienceterm that evaluates and improves practices is that the stakeholders of science must not embrace the status quoterm, but instead pursue self-examination continuously for improvement and self-correction of the scientific process itselfterm.
As Richard Feynman said, “The first principle is that you must not fool yourself – and you are the easiest person to fool.”
Acknowledgements M.R.M. is a member of the UK Centre for Tobacco Control Studies, a UKCRC Public Health Research Centre of Excellence. Funding from the British Heart Foundation, Cancer Research UK, Economic and Social Research Council, Medical Research Council, and the National Institute for Health Research, under the auspices of the UK Clinical Research Collaboration, is gratefully acknowledged. This work was supported by the Medical Research Council Integrative Epidemiology Unit at the University of Bristol (MC_UU_12013/6). D.V.M.B. is funded by a Wellcome Trustterm Principal Research Fellowship and Programme (grant number 082498/Z/07/Z). N.P.d.S. is employed by the NC3Rs, which is primarily funded by the UK government. J.P.A.I. is funded by an unrestricted gift from S. O'Donnell and B. O'Donnell to the Stanford Prevention Research Center. METRICS is supported by a grant by the Laura and John Arnold Foundation. The authors are grateful to Don van den Bergh for preparing Fig. 2 .
Competing interests M.R.M, together with C.D.C and D.V.M.B., has received funding from the BBSRC (grant number BB/N019660/1) to convene a workshop on advanced methodsterm for reproducible scienceterm, and is chair of the CHDI Foundation Independent Statistical Standing Committee. B.A.N. is executive director of the non-profit Center for Open Scienceterm with a mission to increase openness, integrity and reproducibilityterm of research. N.P.d.S. leads the NC3Rs programme of work on experimental design, which developed the ARRIVE guidelines and Experimental Design Assistant. J.J.W. is director, experimental design, at CHDI Management/CHDI Foundation, a non-profit biomedical research organization exclusively dedicated to developing therapeutics for Huntington's disease. The other authors declare no competing interests.
References