standOff. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote shown on the same row, so the borrowing can be checked against the thing borrowed from without leaving this page. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.| relation to other work | claim_relations | how this paper stands to another: agrees, contradicts, extends, qualifies |
| claim type | claim_type | the shape of the claim: existence, mechanism, method, critique, prediction, null |
| concept edge | concept_edges | an asserted relation between two concepts, with the sentence asserting it |
| concept | concepts | an idea the paper works with, quoted where the paper uses it |
| content tag | content_tags | a finding, definition, quantity, statistic or variable named in the paper |
| contribution | contribution | what the paper claims to add |
| control | controls | an experiment run to eliminate a rival explanation |
| limitation | limitations | what the paper says its evidence cannot support |
| mechanism | mechanisms | how the reported effect is said to come about |
| method | method | how the paper produced its evidence |
| object of study | object_of_study | what the paper is about — the thing it investigates |
| phenomenon | phenomenon | the effect or regularity under study |
| research question | research_question | the question the paper sets itself |
| sense | senses | a distinct use of a term, quoted where the paper fixes it |
The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.
Nat Med Nat Med 981 npgopen 9502015 Nature Medicine 1078-8956 1546-170X pmc-is-collection-domain yes pmc-collection-title Nature Portfolio PMC7598943 PMC7598943.1 7598943 7598943 32908283 10.1038/s41591-020-1034-x 1034 1 Consensus Statement Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension Liu Xiaoxuan 1 2 3 4 5 Cruz Rivera Samantha 5 6 7 http://orcid.org/0000-0003-2434-4206 Moher David 8 9 http://orcid.org/0000-0002-1856-837X Calvert Melanie J. 4 5 6 7 10 11 12 http://orcid.org/0000-0001-7849-0087 Denniston Alastair K. a.denniston@bham.ac.uk 2 3 4 5 6 13 The SPIRIT-AI and CONSORT-AI Working Group Chan An-Wen 14 Darzi Ara 15 16 Holmes Christopher 17 18 Yau Christopher 17 19 Ashrafian Hutan 15 16 Deeks Jonathan J. 7 10 Ferrante di Ruffano Lavinia 7 Faes Livia 20 Keane Pearse A. 13 Vollmer Sebastian J. 17 21 for the SPIRIT-AI and CONSORT-AI Steering Group Lee Aaron Y. 22 Jonas Adrian 23 Esteva Andre 24 Beam Andrew L. 25 Chan An-Wen 14 Panico Maria Beatrice 26 Lee Cecilia S. 22 Haug Charlotte 27 Kelly Christopher J. 28 Yau Christopher 17 19 Mulrow Cynthia 29 Espinoza Cyrus 30 Fletcher John 31 Paltoo Dina 32 Manna Elaine 33 Price Gary 34 Collins Gary S. 35 Harvey Hugh 36 Matcham James 37 Monteiro Joao 38 ElZarrad M. Khair 39 Ferrante di Ruffano Lavinia 7 Oakden-Rayner Luke 40 McCradden Melissa 41 Keane Pearse A. 13 Savage Richard 42 Golub Robert 43 Sarkar Rupa 44 Rowley Samuel 45 for the SPIRIT-AI and CONSORT-AI Consensus Group 1 0000 0000 9168 0080 grid.436474.6 Moorfields Eye Hospital NHS Foundation Trust, London, UK 2 0000 0004 1936 7486 grid.6572.6 Academic Unit of Ophthalmology, Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK 3 0000 0004 0376 6589 grid.412563.7 University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK 4 grid.507332.0 Health Data Research UK, London, UK 5 0000 0004 1936 7486 grid.6572.6 Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK 6 0000 0004 1936 7486 grid.6572.6 Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, University of Birmingham, Birmingham, UK 7 0000 0004 1936 7486 grid.6572.6 Institute of Applied Health Research, University of Birmingham, Birmingham, UK 8 0000 0000 9606 5108 grid.412687.e Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada 9 0000 0001 2182 2255 grid.28046.38 School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Canada 10 0000 0004 0376 6589 grid.412563.7 National Institute of Health Research Birmingham Biomedical Research Centre, University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK 11 National Institute of Health Research Applied Research Collaborative West Midlands, Coventry, UK 12 0000 0004 0376 6589 grid.412563.7 National Institute of Health Research Surgical Reconstruction and Microbiology Centre, University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK 13 0000000121901201 grid.83440.3b NIHR Biomedical Research Center at Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology, London, UK 14 0000 0001 2157 2938 grid.17063.33 Department of Medicine, Women’s College Research Institute, Women’s College Hospital, University of Toronto, Toronto, Ontario Canada 15 0000 0001 2113 8111 grid.7445.2 Patient Safety Translational Research Centre, Imperial College London, London, UK 16 0000 0001 2113 8111 grid.7445.2 Institute of Global Health Innovation, Imperial College London, London, UK 17 0000 0004 5903 3632 grid.499548.d Alan Turing Institute, London, UK 18 0000 0004 1936 8948 grid.4991.5 Department of Statistics and Nuffield Department of Medicine, University of Oxford, Oxford, UK 19 0000000121662407 grid.5379.8 University of Manchester, Manchester, UK 20 0000 0000 8587 8621 grid.413354.4 Department of Ophthalmology, Cantonal Hospital Lucerne, Lucerne, Switzerland 21 0000 0000 8809 1613 grid.7372.1 University of Warwick, Coventry, UK 22 0000000122986657 grid.34477.33 Department of Ophthalmology, University of Washington, Seattle, WA USA 23 0000 0004 1794 1878 grid.416710.5 The National Institute for Health and Care Excellence, London, UK 24 Salesforce Research, San Francisco, CA USA 25 000000041936754X grid.38142.3c Harvard T.H. Chan School of Public Health, Boston, MA USA 26 grid.57981.32 Medicines and Healthcare products Regulatory Agency, London, UK 27 0000 0001 1034 2272 grid.431129.c New England Journal of Medicine, Waltham, MA USA 28 Google Health, London, UK 29 Annals of Internal Medicine, Philadelphia, PA USA 30 Patient Partner, Birmingham, UK 31 British Medical Journal, London, UK 32 0000 0001 2297 5165 grid.94365.3d National Institutes of Health, Bethesda, MD USA 33 Patient Partner, London, UK 34 0000 0004 1936 7486 grid.6572.6 Patient Partner, Centre for Patient Reported Outcome Research, Institute of Applied Health Research, University of Birmingham, Birmingham, UK 35 0000 0004 1936 8948 grid.4991.5 Centre for Statistics in Medicine, University of Oxford, Oxford, UK 36 Hardian Health, London, UK 37 0000 0004 5929 4381 grid.417815.e AstraZeneca, Cambridge, UK 38 Nature Research, New York, NY USA 39 0000 0001 2243 3366 grid.417587.8 Food and Drug Administration, Silver Spring, MD USA 40 Australian Institute for Machine Learning, North Terrace, Adelaide, Australia 41 0000 0004 0473 9646 grid.42327.30 The Hospital for Sick Children, Toronto, Canada 42 PinPoint Data Science, Leeds, UK 43 0000 0004 4647 675X grid.413701.0 Journal of the American Medical Association, Chicago, IL USA 44 The Lancet Group, London, UK 45 0000000122478951 grid.14105.31 Medical Research Council, London, UK 9 9 2020 2020 26 9 365785 1364 1374 24 4 2020 23 7 2020 09 09 2020 12 11 2020 29 03 2024 © The Author(s) 2020 https://creativecommons.org/licenses/by/4.0/ Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ . The CONSORT 2010 statement provides minimum guidelines for reporting randomized trials. Its widespread use has been instrumental in ensuring transparency in the evaluation of new interventions. More recently, there has been a growing recognition that interventions involving artificial intelligence (AI) need to undergo rigorous, prospective evaluation to demonstrate impact on health outcomes. The CONSORT-AI (Consolidated Standards of Reporting Trials–Artificial Intelligence) extension is a new reporting guideline for clinical trials evaluating interventions with an AI component. It was developed in parallel with its companion statement for clinical trial protocols: SPIRIT-AI (Standard Protocol Items: Recommendations for Interventional Trials–Artificial Intelligence). Both guidelines were developed through a staged consensus process involving literature review and expert consultation to generate 29 candidate items, which were assessed by an international multi-stakeholder group in a two-stage Delphi survey (103 stakeholders), agreed upon in a two-day consensus meeting (31 stakeholders) and refined through a checklist pilot (34 participants). The CONSORT-AI extension includes 14 new items that were considered sufficiently important for AI interventions that they should be routinely reported in addition to the core CONSORT 2010 items. CONSORT-AI recommends that investigators provide clear descriptions of the AI intervention, including instructions and skills required for use, the setting in which the AI intervention is integrated, the handling of inputs and outputs of the AI intervention, the human–AI interaction and provision of an analysis of error cases. CONSORT-AI will help promote transparency and completeness in reporting clinical trials for AI interventions. It will assist editors and peer reviewers, as well as the general readership, to understand, interpret and critically appraise the quality of clinical trial design and risk of bias in the reported outcomes.
The CONSORT-AI and SPIRIT-AI extensions improve the transparency of clinical trial design and trial protocol reporting for artificial intelligence interventions.
Subject terms
Clinical trial design Technology https://doi.org/10.13039/100004440 Wellcome Trust (Wellcome) https://doi.org/10.13039/100012338 Alan Turing Institute pmc-status-qastatus 0 pmc-status-live yes pmc-status-embargo no pmc-status-released yes pmc-prop-open-access yes pmc-prop-olf no pmc-prop-manuscript no pmc-prop-legally-suppressed no pmc-prop-has-pdf yes pmc-prop-has-supplement yes pmc-prop-pdf-only no pmc-prop-suppress-copyright no pmc-prop-is-real-version no pmc-prop-is-scanned-article no pmc-prop-preprint no pmc-prop-in-epmc yes pmc-license-ref CC BY issue-copyright-statement © The Author(s), under exclusive licence to Springer Nature America, Inc. 2020 Main
Randomized controlled trials (RCTs) are considered the gold-standard experimental design for providing evidence of the safety and efficacy of an intervention 1 , 2 . Trial results, if adequately reported, have the potential to inform regulatory decisions, clinical guidelines and health policy. It is therefore crucial that RCTs are reported with transparency and completeness so that readers can critically appraise the trial methods and findings and assess the presence of bias in the results 3 – 5 .
The CONSORT statement provides evidence-based recommendations to improve the completeness of the reporting of RCTs. The statement was first introduced in 1996 and has since been widely endorsed by medical journals internationally 5 . Over the past two decades, it has undergone two updates and has demonstrated a substantial positive impact on the quality of RCT reports 6 , 7 . The most recent CONSORT 2010 statement provides a 25-item checklist of the minimum reporting content applicable to all RCTs, but it recognizes that certain interventions may require extension or elaboration of these items. Several such extensions exist 8 – 13 .
AI is an area of enormous interest with strong drivers to accelerate new interventions through to publication, implementation and market 14 . While AI systems have been researched for some time, recent advances in deep learning and neural networks have gained considerable interest for their potential in health applications. Examples of such applications are wide ranging and include AI systems for screening and triage 15 , 16 , diagnosis 17 – 20 ,prognostication 21 , 22 , decision support 23 and treatment recommendation 24 . However, in the most recent cases, published evidence has consisted of in silico, early-phase validation. research_questionlimitationsclaim_relationsIt has been recognized that most recent AI studies are inadequately reported and existing reporting guidelines do not fully cover potential sources of bias specific to AI systems 25 . The welcome emergence of RCTs seeking to evaluate newer interventions based on, or including, an AI component (called ‘AI interventions’ here) 23 , 26 – 31 has similarly been met with concerns about the design and reporting 25 , 32 – 34 . This has highlighted the need to provide reporting guidance that is ‘fit for purpose’ in this domain.
CONSORT-AI (as part of the SPIRIT-AI and CONSORT-AI initiative) is an international initiative supported by CONSORT and the EQUATOR (Enhancing the Quality and Transparency of Health Research) Network to evaluate the existing CONSORT 2010 statement and to extend or elaborate this guidance where necessary, to support the reporting of clinical trials for AI interventions 35 , 36 . It is complementary to the SPIRIT-AI statement, which aims to promote high-quality protocol reporting for AI trials. This Consensus Statement describes the methods used to identify and evaluate candidate items and gain consensus. In addition, it also provides the CONSORT-AI checklist, which includes the new extension items and their accompanying explanations.
Methods
The SPIRIT-AI and CONSORT-AI extensions were simultaneously developed for clinical trial protocols and trial reports. An announcement for the SPIRIT-AI and CONSORT-AI initiative was published in October 2019 (ref. 35 ), and the two guidelines were registered as reporting guidelines under development on the EQUATOR library of reporting guidelines in May 2019. Both guidelines were developed in accordance with the EQUATOR Network’s methodological framework 37 . The SPIRIT-AI and CONSORT-AI Steering Group, consisting of 15 international experts, was formed to oversee the conduct and methodology of the study. Definitions of key terms are provided in the glossary (Box 1 ).
Box 1 Glossary Artificial Intelligence The science of developing computer systems which can perform tasks normally requiring human intelligence.
concepts↗sensescontent_tags↗AI intervention A health intervention that relies upon an AI/ML component to serve its purpose.
CONSORT Consolidated Standards of Reporting Trials.
content_tags↗CONSORT-AI extension item An additional checklist item to address AI-specific content that is not adequately covered by CONSORT 2010.
Class-activation map Class-activation maps are particularly relevant to image classification AI interventions. Class-activation maps are visualizations of the pixels that had the greatest influence on predicted class, by displaying the gradient of the predicted outcome from the model with respect to the input. They are also referred to as ‘saliency maps’ or ‘heat maps’.
mechanismsconcepts↗concepts↗sensesconcept_edgescontent_tagscontent_tags↗Health outcome Measured variables in the trial that are used to assess the effects of an intervention.
Human–AI interaction The process of how users (humans) interact with the AI intervention, for the AI intervention to function as intended.
Clinical outcome Measured variables in the trial which are used to assess the effects of an intervention.
Delphi study A research method that derives the collective opinions of a group through a staged consultation of surveys, questionnaires, or interviews, with an aim to reach consensus at the end.
sensescontent_tags↗Development environment The clinical and operational settings from which the data used for training the model is generated. This includes all aspects of the physical setting (such as geographical location, physical environment), operational setting (such as integration with an electronic record system, installation on a physical device) and clinical setting (such as primary, secondary and/or tertiary care, patient disease spectrum).
Fine-tuning Modifications or additional training performed on the AI intervention model, done with the intention of improving its performance.
Input data The data that need to be presented to the AI intervention to allow it to serve its purpose.
Machine learning A field of computer science concerned with the development of models/algorithms that can solve specific tasks by learning patterns from data, rather than by following explicit rules. It is seen as an approach within the field of AI.
Operational environment The environment in which the AI intervention will be deployed, including the infrastructure required to enable the AI intervention to function.
Output data The predicted outcome given by the AI intervention based on modeling of the input data. The output data can be presented in different forms, including a classification (including diagnosis, disease severity or stage, or recommendation such as referability), a probability, a class activation map, etc. The output data typically provide additional clinical information and/or trigger a clinical decision.
mechanismsconcepts↗sensescontent_tagscontent_tags↗content_tags↗Performance error Instances in which the AI intervention fails to perform as expected. This term can describe different types of failures, and it is up to the investigator to specify what should be considered a performance error, preferably based on prior evidence. This can range from small decreases in accuracy (compared to expected accuracy) to erroneous predictions or the inability to produce an output, in certain cases.
mechanismssensescontent_tagsSPIRIT Standard Protocol Items: Recommendations for Interventional Trials.
SPIRIT-AI An additional checklist item to address AI-specific content that is not adequately covered by SPIRIT 2013.
SPIRIT-AI elaboration item Additional considerations to an existing SPIRIT 2013 item when applied to AI interventions.
Ethical approval
This study was approved by the ethical review committee at the University of Birmingham, UK (ERN_19-1100). Participant information was provided to Delphi participants electronically before survey completion and before the consensus meeting. Delphi participants provided electronic informed consent, and written consent was obtained from consensus meeting participants.
Literature review and candidate item generation
An initial list of candidate items for the SPIRIT-AI and CONSORT-AI checklists was generated through review of the published literature and consultation with the Steering Group and known international experts. A search was performed on 13 May 2019 using the terms ‘artificial intelligence’, ‘machine learning’ and ‘deep learning’ to identify existing clinical trials for AI interventions listed within the US National Library of Medicine’s clinical trial registry (ClinicalTrials.gov). limitationscontrolscontent_tagscontent_tagsThere were 316 registered trials, of which 62 were completed and 7 had published results 30 , 38 – 43 . Two studies were reported with reference to the CONSORT statement 30 , 42 , and one study provided an unpublished trial protocol 42 . The Operations Team (X.L., S.C.R., M.J.C. and A.K.D.) identified AI-specific considerations from these studies and reframed them as candidate reporting items. The candidate items were also informed by findings from a previous systematic review that evaluated the diagnostic accuracy of deep-learning systems for medical imaging 25 . After consultation with the Steering Group and additional international experts ( n = 19), 29 candidate items were generated, 26 of which were relevant for both SPIRIT-AI and CONSORT-AI and 3 of which were relevant only for CONSORT-AI. The Operations Team mapped these items to the corresponding SPIRIT and CONSORT items, revising the wording and providing explanatory text as required to contextualize the items. These items were included in subsequent Delphi surveys.
Delphi consensus process
In September 2019, 169 key international experts were invited to participate in the online Delphi survey to vote upon the candidate items and suggest additional items. Experts were identified and contacted via the Steering Group and were allowed one round of ‘snowball’ recruitment in which contacted experts could suggest additional experts. In addition, individuals who made contact following publication of the announcement were included 35 . The Steering Group agreed that individuals with expertise in clinical trials and AI and machine learning (ML), as well as key users of the technology, should be well represented in the consultation. Stakeholders included healthcare professionals, methodologists, statisticians, computer scientists, industry representatives, journal editors, policy makers, health ‘informaticists’, experts in law and ethics, regulators, patients and funders. Participant characteristics are described in Supplementary Table 1 . Two online Delphi surveys were conducted. DelphiManager software (version 4.0), developed and maintained by the COMET (Core Outcome Measures in Effectiveness Trials) initiative, was used to undertake the e-Delphi survey. Participants were given written information about the study and were asked to provide their level of expertise within the fields of (i) AI/ML, and (ii) clinical trials. Each item was presented for consideration (26 for SPIRIT-AI and 29 for CONSORT-AI). Participants were asked to vote on each item using a 9-point scale, as follows: 1–3, not important; 4–6, important but not critical; and 7–9, important and critical. Respondents provided separate ratings for SPIRIT-AI and CONSORT-AI. There was an option to opt out of voting for each item, and each item included space for free text comments. At the end of the Delphi survey, participants had the opportunity to suggest new items. method↗controlscontent_tags103 responses were received for the first Delphi round, and 91 responses (88% of participants from round one) were received for the second round. The results of the Delphi survey informed the subsequent international consensus meeting. 12 new items were proposed by the Delphi study participants and were added for discussion at the consensus meeting. Data collected during the Delphi survey were anonymized, and item-level results were presented at the consensus meeting for discussion and voting.
The two-day consensus meeting took place in January 2020 and was hosted by the University of Birmingham, UK, to seek consensus on the content of SPIRIT-AI and CONSORT-AI. 31 international stakeholders from among the Delphi survey participants were invited to discuss the items and vote on their inclusion. Participants were selected to achieve adequate representation from all the stakeholder groups. 41 items were discussed in turn, comprising the 29 items generated in the initial literature review and item-generation phase (26 items relevant to both SPIRIT-AI and CONSORT-AI; 3 items relevant only to CONSORT-AI) and the 12 new items proposed by participants during the Delphi surveys. Each item was presented to the consensus group, alongside its score from the Delphi exercise (median and interquartile ranges) and any comments made by Delphi participants related to that item. Consensus-meeting participants were invited to comment on the importance of each item and whether the item should be included in the AI extension. In addition, participants were invited to comment on the wording of the explanatory text accompanying each item and the position of each item relative to the SPIRIT 2013 and CONSORT 2010 checklists. After open discussion of each item and the option to adjust wording, an electronic vote took place, with the option to include or exclude the item. controlscontent_tagsAn 80% threshold for inclusion was pre-specified and deemed reasonable by the Steering Group to demonstrate majority consensus. Each stakeholder voted anonymously using Turning Point voting pads (Turning Technologies, version 8.7.2.14).
Checklist pilot
Following the consensus meeting, attendees were given the opportunity to make final comments on the wording and agree that the updated SPIRIT-AI and CONSORT-AI items reflected discussions from the meeting. The Operations Team assigned each item as an extension or elaboration item on the basis of a decision tree and produced a penultimate draft of the SPIRIT-AI and CONSORT-AI checklists (Supplementary Fig. 1 ). A pilot of the penultimate checklists was conducted with 34 participants to ensure clarity of wording. Experts participating in the pilot included the following: (a) Delphi participants who did not attend the consensus meeting, and (b) external experts who had not taken part in the development process but who had reached out to the Steering Group after the Delphi study commenced. Final changes were made on wording only to improve clarity for readers, by the Operations Team (Supplementary Fig. 2 ).
Recommendations
CONSORT-AI checklist items and explanation
object_of_studycontributionclaim_typeclaim_relationscontent_tagsThe CONSORT-AI extension recommends that 14 new checklists items be added to the existing CONSORT 2010 statement (11 extensions and 3 elaborations). These items were considered sufficiently important for clinical-trial reports for AI interventions that they should be routinely reported in addition to the core CONSORT 2010 checklist items. Table 1 lists the CONSORT-AI items. Table 1 CONSORT-AI checklist
Section CONSORT 2010 item a CONSORT-AI item Addressed on page number b
Title and abstract
Title and Abstract 1a Identification as a randomized trial in the title CONSORT-AI 1a,b Elaboration (i) Indicate that the intervention involves artificial intelligence/machine learning in the title and/or abstract and specify the type of model.
1b Structured summary of trial design, methods, results, and conclusions (for specific guidance see CONSORT for abstracts) (ii) State the intended use of the AI intervention within the trial in the title and/or abstract.
Introduction
Background and objectives 2a Scientific background and explanation of rationale CONSORT-AI 2a (i) Extension Explain the intended use of the AI intervention in the context of the clinical pathway, including its purpose and its intended users (for example, healthcare professionals, patients, public).
2b Specific objectives or hypotheses
Methods
Trial design 3a Description of trial design (such as parallel, factorial) including allocation ratio
3b Important changes to methods after trial commencement (such as eligibility criteria), with reasons
Participants 4a Eligibility criteria for participants CONSORT-AI 4a (i) Elaboration State the inclusion and exclusion criteria at the level of participants.
CONSORT-AI 4a (ii) Extension State the inclusion and exclusion criteria at the level of the input data.
4b Settings and locations where the data were collected CONSORT-AI 4b Extension Describe how the AI intervention was integrated into the trial setting, including any onsite or offsite requirements.
Interventions 5 The interventions for each group with sufficient details to allow replication, including how and when they were actually administered CONSORT-AI 5 (i) Extension State which version of the AI algorithm was used.
CONSORT-AI 5 (ii) Extension Describe how the input data were acquired and selected for the AI intervention.
CONSORT-AI 5 (iii) Extension Describe how poor quality or unavailable input data were assessed and handled.
CONSORT-AI 5 (iv) Extension Specify whether there was human–AI interaction in the handling of the input data, and what level of expertise was required of users.
CONSORT-AI 5 (v) Extension Specify the output of the AI intervention
CONSORT-AI 5 (vi) Extension Explain how the AI intervention’s outputs contributed to decision-making or other elements of clinical practice.
Outcomes 6a Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed
6b Any changes to trial outcomes after the trial commenced, with reasons
Sample size 7a How sample size was determined
7b When applicable, explanation of any interim analyses and stopping guidelines
Randomization
Sequence generation 8a Method used to generate the random allocation sequence
8b Type of randomization; details of any restriction (such as blocking and block size)
Allocation concealment mechanism 9 Mechanism used to implement the random allocation sequence (such as sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned
Implementation 10 Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions
Blinding 11a If done, who was blinded after assignment to interventions (for example, participants, care providers, those assessing outcomes) and how
11b If relevant, description of the similarity of interventions
Statistical methods 12a Statistical methods used to compare groups for primary and secondary outcomes
12b Methods for additional analyses, such as subgroup analyses and adjusted analyses
Results
Participant flow (a diagram is strongly recommended) 13a For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analyzed for the primary outcome
13b For each group, losses and exclusions after randomization, together with reasons
Recruitment 14a Dates defining the periods of recruitment and follow-up
14b Why the trial ended or was stopped
Baseline data 15 A table showing baseline demographic and clinical characteristics for each group
Numbers analyzed 16 For each group, number of participants (denominator) included in each analysis and whether the analysis was by original assigned groups
Outcomes and estimation 17a For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval)
17b For binary outcomes, presentation of both absolute and relative effect sizes is recommended
Ancillary analyses 18 Results of any other analyses performed, including subgroup analyses and adjusted analyses, distinguishing pre-specified from exploratory
Harms 19 All important harms or unintended effects in each group (for specific guidance see CONSORT for harms) CONSORT-AI 19 Extension Describe results of any analysis of performance errors and how errors were identified, where applicable. If no such analysis was planned or done, justify why not.
Discussion
Limitations 20 Trial limitations, addressing sources of potential bias, imprecision, and, if relevant, multiplicity of analyses
Generalizability 21 Generalizability (external validity, applicability) of the trial findings
Interpretation 22 Interpretation consistent with results, balancing benefits and harms, and considering other relevant evidence
Other Information
Registration 23 Registration number and name of trial registry
Protocol 24 Where the full trial protocol can be accessed, if available
Funding 25 Sources of funding and other support (such as supply of drugs), role of funders CONSORT-AI 25 Extension State whether and how the AI intervention and/or its code can be accessed, including any restrictions to access or re-use.
a We strongly recommend reading this statement in conjunction with the CONSORT 2010 Explanation and Elaboration for important clarifications on all the items. b Indicates page numbers to be completed by authors during protocol development.
The 14 items below passed the threshold of 80% for inclusion at the consensus meeting. CONSORT-AI 2a, CONSORT-AI 5 (ii) and CONSORT-AI 19 each resulted from the merging of two items after discussion with the consensus group. CONSORT-AI 4a (i) and (ii) was split into two items for clarity and was voted upon separately. mechanismscontent_tagscontent_tagsCONSORT-AI 5(iii) did not fulfill the criteria for inclusion on the basis of its initial wording (77% vote to include); however, after extensive discussion and rewording, the consensus group unanimously supported a re-vote, at which point it passed the inclusion threshold (97% to include). The Delphi and voting results for each included and excluded item are described in Supplementary Table 2 .
Title and abstract
CONSORT-AI 1a,b (i) Elaboration: Indicate that the intervention involves artificial intelligence/machine learning in the title and/or abstract and specify the type of model
Explanation
Indicating in the title and/or abstract of the trial report that the intervention involves a form of AI is encouraged, as it immediately identifies the intervention as an AI/ML intervention and also serves to facilitate indexing and searching of the trial report. mechanismscontent_tagsThe title should be understandable by a wide audience; therefore, a broader umbrella term such as ‘artificial intelligence’ or ‘machine learning’ is encouraged. More-precise terms should be used in the abstract, rather than the title, unless they are broadly recognized as being a form of AI/ML. Specific terminology relating to the model type and architecture should be detailed in the abstract.
CONSORT-AI 1a,b (ii) Elaboration: State the intended use of the AI intervention within the trial in the title and/or abstract
Explanation
Describe the intended use of the AI intervention in the trial report title and/or abstract. This should describe the purpose of the AI intervention and the disease context 26 , 44 . Some AI interventions may have multiple intended uses, or the intended use may evolve over time. Therefore, documenting this allows readers to understand the intended use of the algorithm at the time of the trial.
Introduction
CONSORT-AI 2a (i) Extension: Explain the intended use for the AI intervention in the context of the clinical pathway, including its purpose and its intended users (for example, healthcare professionals, patients, public)
Explanation
In order to clarify how the AI intervention is intended to fit into a clinical pathway, a detailed description of its role should be included in the background of the trial report. AI interventions may be designed to interact with different users, including healthcare professionals, patients and the public, and their roles can be wide-ranging (for example, the same AI intervention could theoretically be replacing, augmenting or adjudicating components of clinical decision-making). Clarifying the intended use of the AI intervention and its intended user helps readers understand the purpose for which the AI intervention was evaluated in the trial.
Methods
CONSORT-AI 4a (i) Elaboration: State the inclusion and exclusion criteria at the level of participants
Explanation
The inclusion and exclusion criteria should be defined at the participant level as per usual practice in non-AI interventional trial reports (Fig. 1 ). This is distinct from the inclusion and exclusion criteria made at the input-data level, which is addressed in item 4a (ii). Fig. 1 CONSORT 2010 flow diagram — adapted for AI clinical trials.
CONSORT-AI 4a (i): State the inclusion and exclusion criteria at the level of participants. CONSORT-AI 4a (ii): State the inclusion and exclusion criteria at the level of the input data. CONSORT 13b (core CONSORT item): For each group, losses and exclusions after randomization, together with reasons.
CONSORT-AI 4a (ii) Extension: State the inclusion and exclusion criteria at the level of the input data
Explanation
‘Input data’ refers to the data required by the AI intervention to serve its purpose (for example, for a breast-cancer diagnostic system, the input data could be the unprocessed or vendor-specific post-processing mammography scan upon which a diagnosis is being made; for an early-warning system, the input data could be physiological measurements or laboratory results from the electronic health record). The trial report should pre-specify if there were minimum requirements for the input data (such as image resolution, quality metrics or data format) that determined pre-randomization eligibility. It should specify when, how and by whom this was assessed. For example, if a participant met the eligibility criteria for lying flat for a CT scan as per item 4a (i), but the scan quality was compromised (for any given reason) to such a level that it was deemed unfit for use by the AI system, this should be reported as an exclusion criterion at the input-data level. Note that where input data are acquired after randomization, any exclusion is considered to be from the analysis, not from enrollment (CONSORT item 13b and Fig. 1 ).
CONSORT-AI 4b Extension: Describe how the AI intervention was integrated into the trial setting, including any onsite or offsite requirements
Explanation
There are limitations to the generalizability of AI algorithms, one of which is when they are used outside of their development environment 45 , 46 . AI systems are dependent on their operational environment, and the report should provide details of the hardware and software requirements to allow technical integration of the AI intervention at each study site. For example, it should be stated if the AI intervention required vendor-specific devices, if there was specialized computing hardware at each site, or if the site had to support cloud integration, particularly if this was vendor specific. If any changes to the algorithm were required at each study site as part of the implementation procedure (such as fine-tuning the algorithm on local data), then this process should also be clearly described.
CONSORT-AI 5 (i) Extension: State which version of the AI algorithm was used
Explanation
Similar to other forms of software as a medical device, AI systems are likely to undergo multiple iterations and updates during their lifespan. It is therefore important to specify which version of the AI system was used in the clinical trial, whether this is the same as the version evaluated in previous studies that have been used to justify the study rationale, and whether the version changed during the conduct of the trial. If applicable, the report should describe what has changed between the relevant versions and the rationales for the changes. Where available, the report should include a regulatory marking reference, such as an unique device identifier, that requires a new identifier for updated versions of the device 47 .
CONSORT-AI 5 (ii) Extension: Describe how the input data were acquired and selected for the AI intervention
Explanation
The measured performance of any AI system may be critically dependent on the nature and quality of the input data 48 . A description of the input-data handling, including acquisition, selection and pre-processing before analysis by the AI system, should be provided. Completeness and transparency of this description is integral to the replicability of the intervention beyond the clinical trial in real-world settings. It also helps readers identify whether input-data-handling procedures were standardized across trial sites.
CONSORT-AI 5 (iii) Extension: Describe how poor-quality or unavailable input data were assessed and handled
Explanation
As with CONSORT-AI 4a (ii), ‘input data’ refers to the data required by the AI intervention to serve its purpose. As discussed in item 4a (ii), the performance of AI systems may be compromised as a result of poor quality or missing input data 49 (for example, excessive movement artifact on an electrocardiogram). The trial report should report the amount of missing data, as well as how this was identified and handled. The report should also specify if there was a minimum standard required for the input data and, where this standard was not achieved, how this was handled (including the impact on, or any changes to, the participant care pathway).
Poor quality or unavailable data can also affect non-AI interventions. For example, sub-optimal quality of a scan could affect a radiologist’s ability to interpret it and make a diagnosis. It is therefore important that this information is reported equally in the control intervention, where relevant. If this minimum quality standard was different from the inclusion criteria for input data used to assess eligibility pre-randomization, this should be stated.
CONSORT-AI 5 (iv) Extension: Specify whether there was human–AI interaction in the handling of the input data, and what level of expertise was required of users
Explanation
A description of the human–AI interface and the requirements for successful interaction when input data are handled should be provided — for example, clinician-led selection of regions of interest from a histology slide that is then interpreted by an AI diagnostic system 50 , or an endoscopist’s selection of a colonoscopy video clips as input data for an algorithm designed to detect polyps 28 . A description of any user training provided and instructions for how users should handle the input data provides transparency and replicability of trial procedures. Poor clarity on the human–AI interface may lead to lack of a standard approach and may carry ethical implications, particularly in the event of harm 51 , 52 . For example, it may become unclear whether an error case occurred due to human deviation from the instructed procedure, or if it was an error made by the AI system.
CONSORT-AI 5 (v) Extension: Specify the output of the AI intervention
Explanation
The output of the AI intervention should be clearly specified in the trial report. For example, an AI system may output a diagnostic classification or probability, a recommended action, an alarm alerting to an event, an instigated action in a closed-loop system (such as titration of drug infusions) or another output. The nature of the AI intervention’s output has direct implications on its usability and how it may lead to downstream actions and outcomes.
CONSORT-AI 5 (vi) Extension: Explain how the AI intervention’s outputs contributed to decision-making or other elements of clinical practice
Explanation
Since health outcomes may also critically depend on how humans interact with the AI intervention, the report should explain how the outputs of the AI system were used to contribute to decision-making or other elements of clinical practice. This should include adequate description of downstream interventions that can affect outcomes. As with CONSORT-AI 5 (iv), any effects of human–AI interaction on the outputs should be described in detail, including the level of expertise required to understand the outputs and any training and/or instructions provided for this purpose. content_tagsFor example, a skin cancer detection system that produced a percentage likelihood as its output should be accompanied by an explanation of how this output was interpreted and acted upon by the user, specifying both the intended pathways (for example, skin lesion excision if the diagnosis is positive) and the thresholds for entry to these pathways (for example, skin lesion excision if the diagnosis is positive and the probability is >80%). The information produced by comparator interventions should be similarly described, alongside an explanation of how such information was used to arrive at clinical decisions on patient management, where relevant. Any discrepancy in how decision-making occurred versus how it was intended to occur (that is, as specified in the trial protocol) should be reported.
Results
CONSORT-AI 19 Extension: Describe results of any analysis of performance errors and how errors were identified, where applicable. If no such analysis was planned or done, explain why not
Explanation
Reporting performance errors and failure case analysis is especially important for AI interventions. AI systems can make errors that may be hard to foresee but that, if allowed to be deployed at scale, could have catastrophic consequences 53 . Therefore, reporting cases of error and defining risk-mitigation strategies are important for informing when, and for which populations, the intervention can be safely implemented. The results of any performance-error analysis should be reported and the implications of the results should be discussed.
Other information
CONSORT-AI 25 Extension: State whether and how the AI intervention and/or its code can be accessed, including any restrictions to access or re-use
Explanation
The trial report should make it clear whether and how the AI intervention and/or its code can be accessed or re-used. This should include details about the license and any restrictions to access.
Discussion
CONSORT-AI is a new reporting-guideline extension developed through international multi-stakeholder consensus. It aims to promote transparent reporting of AI intervention trials and is intended to facilitate critical appraisal and evidence synthesis. The extension items added in CONSORT-AI address a number of issues specific to the implementation and evaluation of AI interventions, which should be considered alongside the core CONSORT 2010 checklist and other CONSORT extensions 54 . phenomenonlimitationscontent_tagsIt is important to note that these are minimum requirements and there may be value in including additional items not included in the checklists in the report or in supplementary materials (Supplementary Table 2 ).
In both CONSORT-AI and its companion project SPIRIT-AI, a major emphasis was the addition of several new items related to the intervention itself and its application in the clinical context. Items 5 (i)–5 (vi) were added to address AI-specific considerations in descriptions of the intervention. Specific recommendations were made pertinent to AI systems related to algorithm version, input and output data, integration into trial settings, expertise of the users and protocol for acting upon the AI system’s recommendations. It was agreed that these details are critical for independent evaluation or replication of the trial. Journal editors reported that despite the importance of these items, they are currently often missing from trial reports at the time of submission for publication, which provides further weight for their inclusion as specifically listed extension items.
A recurrent focus of the Delphi comments and consensus group discussion was the safety of AI systems. concepts↗concepts↗concept_edgesThis was in recognition that AI systems, unlike other health interventions, can unpredictably yield errors that are not easily detectable or explainable by human judgement. For example, changes to medical imaging that are invisible, or appear random, to the human eye may change the likelihood of the diagnostic output entirely 55 , 56 . The concern is that given the theoretical ease with which AI systems could be deployed at scale, any unintended harmful consequences could be catastrophic. CONSORT-AI item 19, which requires specification of any plans to analyze performance errors, was added to emphasize the importance of anticipating systematic errors made by the algorithm and their consequences. Beyond this, investigators should also be encouraged to explore differences in performance and error rates across population subgroups. It has been shown that AI systems may be systematically biased toward different outputs, which may lead to different or even unfair treatment, on the basis of extant features 53 , 57 – 59 .
limitationsconcepts↗concepts↗sensesconcept_edgescontent_tagsThe topic of ‘continuously evolving’ AI systems (also known as ‘continuously adapting’ or ‘continuously learning’ AI systems) was discussed at length during the consensus meeting, but it was agreed that this be excluded from CONSORT-AI. These are AI systems with the ability to continuously train on new data, which may cause changes in performance over time. The group noted that, while of interest, this field is relatively early in its development without tangible examples in healthcare applications, and that it would not be appropriate for it to be included in CONSORT-AI at this stage 60 . This topic will be monitored and will be revisited in future iterations of CONSORT-AI. It is worth noting that incremental software changes, whether continuous or iterative, intentional or unintentional, could have serious consequences on safety performance after deployment. It is therefore of vital importance that such changes be documented and identified by software version and that a robust post-deployment surveillance plan is in place.
This study is set in the current context of AI in health; therefore, several limitations should be noted. First, there are relatively few published interventional trials in the field of AI for healthcare; therefore, the discussions and decisions made during this study were not always supported by existing examples of completed trials. This arises from our stated aim of addressing the issues of poor reporting in this field as early as possible, recognizing the strong drivers in the field and the specific challenges of study design and reporting for AI. As the science and study of AI evolves, we welcome collaboration with investigators to co-evolve these reporting standards to ensure their continued relevance. limitationscontrolsconcepts↗concepts↗concept_edgescontent_tagsSecond, the literature search of AI RCTs used terminology such as ‘artificial intelligence’, ‘machine learning’ and ‘deep learning’, but not terms such as ‘clinical decision support systems’ or ‘expert systems’, which were more commonly used in the 1990s for technologies underpinned by AI systems and share risks similar to those of recent examples 61 . It is likely that such systems, if published today, would be indexed under ‘artificial intelligence’ or ‘machine learning’; however, clinical decision support systems were not actively discussed during this consensus process. Third, the initial candidate items list was generated by a relatively small group of experts consisting of SteeringGroup members and additional international experts; however, additional items from the wider Delphi group were taken forward for consideration by the consensus group, and no new items were suggested during the consensus meeting or post-meeting evaluation.
As with the CONSORT statement, the CONSORT-AI extension is intended as a minimum reporting guidance, and there are additional AI-specific considerations for trial reports that may warrant consideration (Supplementary Table 2 ). This extension is aimed particularly at investigators and readers reporting or appraising clinical trials; however, it may also serve as useful guidance for developers of AI interventions in earlier validation stages of an AI system. Investigators seeking to report studies developing and validating the diagnostic and predictive properties of AI models should refer to TRIPOD-ML (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis–Machine Learning) and STARD-AI (Standards for Reporting Diagnostic Accuracy Studies–Artificial Intelligence), both of which are currently under development 32 , 62 . Other potentially relevant guidelines, which are agnostic to study design, are registered with the EQUATOR Network 63 . The CONSORT-AI extension is expected to encourage careful early planning of AI interventions for clinical trials and this, in conjunction with SPIRIT-AI, should help to improve the quality of trials for AI interventions. The development of the CONSORT-AI guidance does not include additional items within the discussion section of trial reports. The guidance provided by CONSORT 2010 on trial limitations, generalizability and interpretation were deemed to be translatable to trials for AI interventions.
There is also recognition that AI is a rapidly evolving field, and there will be the need to update CONSORT-AI as the technology, and newer applications for it, develop. Currently, most applications of AI involve disease detection, diagnosis and triage, and this is likely to have influenced the nature and prioritization of items within CONSORT-AI. As wider applications that utilize ‘AI as therapy’ emerge, it will be important to continue to evaluate CONSORT-AI in the light of such studies. Additionally, advances in computational techniques and the ability to integrate them into clinical workflows will bring new opportunities for innovation that benefits patients. However, they may be accompanied by new challenges around study design and reporting. In order to ensure transparency, minimize potential biases and promote the trustworthiness of the results and the extent to which they may be generalizable, the SPIRIT-AI and CONSORT-AI Steering Group will continue to monitor the need for updates.
Supplementary information
Supplementary Information Supplementary Figures 1–2, Supplementary Tables 1–2, Supplementary Note.
Peer review information Javier Carmona was the primary editor on this article, and managed its editorial process and peer review in collaboration with the rest of the editorial team.
A list of authors and their affiliations appears at the end of the paper.
Supplementary information
is available for this paper at 10.1038/s41591-020-1034-x.
Acknowledgements
We thank the participants who were involved in the Delphi study and Pilot study (Supplementary Note); E. Marston (University of Birmingham, UK) for strategic support; and C. Radovanovic (University Hospitals Birmingham NHS Foundation Trust, UK) and A. Walker (University of Birmingham, UK) for administrative support. The views expressed in this publication are those of the authors, Delphi participants and stakeholder participants and may not represent the views of the broader stakeholder group or host institution. This work was funded by a Wellcome Trust Institutional Strategic Support Fund: Digital Health Pilot Grant, Research England (part of UK Research and Innovation), Health Data Research UK and the Alan Turing Institute. The study was sponsored by the University of Birmingham, UK. The study funders and sponsors had no role in the design and conduct of the study; collection, management, analysis and interpretation of the data; preparation, review or approval of the manuscript; or decision to submit the manuscript for publication. M.J.C. is a National Institute for Health Research (NIHR) Senior Investigator and receives funding from the National Institute for Health Research (NIHR) Birmingham Biomedical Research Centre; the NIHR Surgical Reconstruction and Microbiology Research Centre and NIHR ARC West Midlands at the University of Birmingham and University Hospitals Birmingham NHS Foundation Trust; Health Data Research UK; Innovate UK (part of UK Research and Innovation); the Health Foundation; Macmillan Cancer Support; and UCB Pharma. A.D. and J.D. are also NIHR Senior Investigators. The views expressed in this article are those of the author(s) and not necessarily those of the NIHR, or the Department of Health and Social Care. S.J.V. receives funding from the Engineering and Physical Sciences Research Council, UK Research and Innovation (UKRI), Accenture, Warwick Impact Fund, Health Data Research UK and European Regional Development Fund. S.R. is an employee of the Medical Research Council (UKRI). D.M. is supported by a University of Ottawa Research Chair. M.K.E. is supported by the U.S. Food and Drug Administration (FDA), and D.P. is supported in part by the Office of the Director at the National Library of Medicine (NLM), US National Institutes of Health (NIH). A.B. is supported by a National Institutes of Health (NIH) award 7K01HL141771-02. This article may not be consistent with NIH and/or FDA’s views or policies. It reflects only the views and opinions of the authors.
Author contributions
Concept and design, and acquisition, analysis and interpretation of data, all authors; drafting of the manuscript, X.L., S.C.R., D.M., M.J.C. and A.K.D.; obtaining of funding, M.J.C., C.Y., C.H. and A.K.D. The SPIRIT-AI and CONSORT-AI Working Group consists of two groups that have been key in the development of the guidelines: the SPIRIT-AI and CONSORT-AI Steering Group, which was responsible for overseeing the consensus process and guidelines development methodology (Alastair K. Denniston, An-Wen Chan, Ara Darzi, Christopher Holmes, Christopher Yau, David Moher, Hutan Ashrafian, Jonathan J. Deeks, Lavinia Ferrante di Ruffano, Livia Faes, Melanie J. Calvert, Pearse A. Keane, Samantha Cruz Rivera, Sebastian J. Vollmer and Xiaoxuan Liu); and the SPIRIT-AI and CONSORT-AI Consensus Group, which was responsible for reaching consensus on the content and wording of the items within the checklists (Aaron Y. Lee, Adrian Jonas, Andre Esteva, Andrew L. Beam, An-Wen Chan, Maria Beatrice Panico, Cecilia S. Lee, Charlotte Haug, Christopher J. Kelly, Christopher Yau, Cynthia Mulrow, Cyrus Espinoza, David Moher, Dina Paltoo, Elaine Manna, Gary Price, Gary S Collins, Hugh Harvey, James Matcham, Joao Monteiro, John Fletcher, M. Khair ElZarrad, Lavinia Ferrante Di Ruffano, Luke Oakden-Rayner, Melanie J. Calvert, Melissa McCradden, Pearse A. Keane, Richard Savage, Robert Golub, Rupa Sarkar and Samuel Rowley).
Data availability
Data requests should be made to the corresponding author and release will be subject to consideration by the SPIRIT-AI and CONSORT-AI Steering Group.
Competing interests
M.J.C. has received personal fees from Astellas, Takeda, Merck, Daiichi Sankyo, Glaukos, GlaxoSmithKline and the Patient-Centered Outcomes Research Institute (PCORI) outside the submitted work. P.A.K. is a consultant for DeepMind Technologies, Roche, Novartis and Apellis, and has received speaker fees or travel support from Bayer, Allergan, Topcon and Heidelberg Engineering. C.J.K. is an employee of Google and owns Alphabet stock. A.E. is an employee of Salesforce CRM. R.S. is an employee of Pinpoint Science. J. Matcham was an employee of AstraZeneca at the time of this study. J. Monteiro is Chief Editor of the journal Nature Medicine ; he has recused himself from any aspect of decision-making on this manuscript and played no part in the assignment of this manuscript to in-house editors or peer reviewers, and was also separated and blinded from the editorial process from submission inception to decision.
Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; none of 4 relations here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.
| facet | tag | field | conf. | whose words |
|---|---|---|---|---|
| object_of_study | A CONTROLLED VOCABULARY BUILT BY VOTE - 14 required disclosures for a trial of an AI intervention, chosen from 41 candidates by a two-round Delphi of 103 stakeholders and a 31-person consensus meeting with a pre-specified 80% inclusion threshold | object_of_study | the paper's words | |
| research_question | which AI-specific facts must appear in a clinical trial report before that report can be critically appraised, and which of them command consensus | research_question | the paper's words | |
| contribution | the CONSORT-AI checklist: 11 extension and 3 elaboration items added to CONSORT 2010, with a glossary defining the terms they use | contribution | the paper's words | |
| claim_type | method | claim_type | the paper's words | |
| phenomenon | reporting completeness as something that must be defined term by term, voted on, and published as a list | phenomenon | the paper's words | |
| method | expert-consensus also in: expert-consensus | method | the paper's words | |
| mechanisms | THE GLOSSARY DEFINES TWO TERMS IDENTICALLY. `Health outcome: Measured variables in the trial **THAT** are used to assess the effects of an intervention.` `Clinical outcome: Measured variables in the trial **WHICH** are used to assess the effects of an intervention.` Word-diffed rather than eyeballed: the only difference is `that` vs `which`. Two entries, one definition, in a vocabulary whose purpose is precision | mechanisms[0] | 0.95 | the tagger's words |
| mechanisms | AND ONE TERM'S DEFINITION HANDS THE DEFINING BACK TO THE READER: `Performance error: Instances in which the AI intervention fails to perform as expected. This term can describe different types of failures, and it is up to the investigator to specify what should be considered A PERFORMANCE ERROR, preferably based on prior evidence.` A deliberately open slot in a controlled vocabulary - the opposite move to a threshold, and made on purpose | mechanisms[1] | 0.95 | the tagger's words |
| mechanisms | AND THE `SPIRIT-AI` ENTRY CARRIES A DIFFERENT TERM'S DEFINITION: it reads `An additional checklist item to address AI-specific content that is not adequately covered by SPIRIT 2013` - which is the `CONSORT-AI extension item` definition with SPIRIT and 2013 substituted, word for word by difflib. `SPIRIT-AI EXTENSION ITEM` does not appear anywhere in the paper, while `SPIRIT-AI elaboration item` sits in the next entry. THE INITIATIVE'S NAME HAS BEEN GIVEN ITS OWN CHECKLIST ITEM'S DEFINITION | mechanisms[2] | 0.95 | the tagger's words |
| mechanisms | AND A THRESHOLD HELD WHILE THE ITEM MOVED UNTIL IT PASSED: `An 80% threshold for inclusion was pre-specified`, and `CONSORT-AI 5(iii) did not fulfill the criteria for inclusion on the basis of its initial wording (77% vote to include); however, after extensive discussion and rewording, the consensus group unanimously supported a re-vote, at which point it passed the inclusion threshold (97% to include)`. Both numbers published | mechanisms[3] | 0.95 | the tagger's words |
| mechanisms | AND THE GUIDELINE PRESCRIBES TERM GRANULARITY BY POSITION IN THE PAPER: `The title should be understandable by a wide audience; therefore, A BROADER UMBRELLA TERM such as artificial intelligence or machine learning is encouraged. More-precise terms should be used in the abstract, rather than the title`. The same system is to be named at two levels of abstraction depending on where the name appears | mechanisms[4] | 0.95 | the tagger's words |
| limitations | THE VOCABULARY OF THE ERA DECIDED WHAT THE SEARCH COULD FIND, AND THEY SAY SO: `the literature search of AI RCTs used terminology such as artificial intelligence, machine learning and deep learning, BUT NOT terms such as clinical decision support systems or EXPERT SYSTEMS, WHICH WERE MORE COMMONLY USED IN THE 1990s for technologies underpinned by AI systems and share risks similar to those of recent examples. It is likely that such systems, if published today, would be indexed under artificial intelligence OR machine learning`. A renaming across decades, named as a limitation of the guideline's own evidence base | limitations[0] | 0.95 | the tagger's words |
| limitations | A CONCEPT WAS EXCLUDED BY VOTE AND THE REASON PUBLISHED: `The topic of CONTINUOUSLY EVOLVING AI systems (ALSO KNOWN AS continuously adapting or continuously learning AI systems) was discussed at length during the consensus meeting, but it was agreed that this be excluded from CONSORT-AI ... this field is relatively early in its development WITHOUT TANGIBLE EXAMPLES in healthcare applications`. A published statement of what the vocabulary deliberately does not cover | limitations[1] | 0.95 | the tagger's words |
| limitations | and the checklist is explicitly a floor, not a ceiling: `these are minimum requirements and there may be value in including additional items not included in the checklists in the report or in supplementary materials` | limitations[2] | 0.95 | the tagger's words |
| limitations | THE EVIDENCE BASE WAS SEVEN PAPERS: `There were 316 REGISTERED TRIALS, of which 62 were completed and 7 HAD PUBLISHED RESULTS. Two studies were reported with reference to the CONSORT statement, and one study provided an unpublished trial protocol`. A reporting standard written mostly in advance of the literature it governs | limitations[3] | 0.95 | the tagger's words |
| limitations | and the reason the extension exists at all is a stated failure of the existing vocabulary: `most recent AI studies are inadequately reported and existing reporting guidelines do not fully cover potential sources of bias specific to AI systems` | limitations[4] | 0.95 | the tagger's words |
| controls | a pre-specified 80% inclusion threshold at the consensus meeting, with anonymous electronic voting eliminates: items entering the checklist on the strength of a loud minority - and it bound at least once: 'CONSORT-AI 5(iii) did not fulfill the criteria for inclusion on the basis of its initial wording (77%)' · outcome: held | controls[0] | 0.95 | the tagger's words |
| controls | a two-stage Delphi survey of 103 then 91 international stakeholders before the consensus meeting eliminates: a checklist reflecting only the 15-person Steering Group - and 12 items came from the wider group · outcome: held | controls[1] | 0.95 | the tagger's words |
| controls | a systematic search of the trial registry for existing AI RCTs to ground the candidate items in published practice eliminates: items invented without evidence - but it barely bound: 'There were 316 registered trials, of which 62 were completed and 7 had published results' · outcome: partial | controls[2] | 0.95 | the tagger's words |
| controls | search terms covering the older names for the same technologies - 'clinical decision support systems', 'expert systems' eliminates: nothing, because they were not used: 'the literature search ... used terminology such as artificial intelligence, machine learning and deep learning, BUT NOT terms such as clinical decision support systems or expert systems' · outcome: failed | controls[3] | 0.95 | the tagger's words |
| concepts | Health outcome also in: Health outcome | concepts[0] | 0.95 | the paper's words |
| concepts | Clinical outcome also in: Clinical outcome | concepts[1] | 0.95 | the paper's words |
| concepts | Performance error also in: Performance error | concepts[2] | 0.95 | the paper's words |
| concepts | AI intervention also in: AI intervention | concepts[3] | 0.95 | the paper's words |
| concepts | AI systems also in: AI systems | concepts[4] | 0.95 | the paper's words |
| concepts | other health interventions also in: other health interventions | concepts[5] | 0.95 | the paper's words |
| concepts | clinical decision support systems also in: clinical decision support systems | concepts[6] | 0.95 | the paper's words |
| concepts | artificial intelligence also in: artificial intelligence | concepts[7] | 0.95 | the paper's words |
| concepts | continuously evolving also in: continuously evolving | concepts[8] | 0.95 | the paper's words |
| concepts | continuously adapting also in: continuously adapting | concepts[9] | 0.95 | the paper's words |
| senses | `Health outcome` and `Clinical outcome` - TWO GLOSSARY ENTRIES WITH ONE DEFINITION, differing only in `that` versus `which`. Either they are the same concept named twice, or a distinction the authors intended did not survive into the text | senses[0] | 0.95 | the tagger's words |
| senses | `Performance error` - DEFINED AS DELIBERATELY UNDEFINED: `it is up to the INVESTIGATOR to specify what should be considered a performance error, preferably based on prior evidence`. The vocabulary names the slot and refuses to fill it | senses[1] | 0.95 | the tagger's words |
| senses | `SPIRIT-AI` - AS PRINTED IN THE GLOSSARY, THE DEFINITION OF `SPIRIT-AI extension item`, not of the initiative. The correct term is absent from the paper entirely | senses[2] | 0.95 | the tagger's words |
| senses | `AI intervention` - the load-bearing term, defined by dependence rather than by composition: `A health intervention that relies upon an AI/ML component to serve its purpose` | senses[3] | 0.95 | the tagger's words |
| senses | `Development environment` versus `Operational environment` - the pair that carries the generalisation claim: the settings the training data came from, against the settings the system is deployed into | senses[4] | 0.95 | the tagger's words |
| senses | `continuously evolving AI systems` - `also known as ‘continuously adapting’ or ‘continuously learning’ AI systems`. Three names for one concept, recorded in the sentence that excludes it from the vocabulary | senses[5] | 0.95 | the tagger's words |
| claim_relations | extends → the CONSORT 2010 statement ('schulz2010-consort' in this corpus), whose 25-item checklist these 14 items are added to rather than replacing | claim_relations[0] | 0.95 | the tagger's words |
| claim_relations | contrasts-with → the existing reporting guidelines, which the authors say 'do not fully cover potential sources of bias specific to AI systems' | claim_relations[1] | 0.95 | the tagger's words |
| concept_edges | continuously evolving —same-as→ continuously adapting strength of the relation: 0.95 · note: `The topic of ‘continuously evolving’ AI systems (also known as ‘continuously adapting’ or ‘continuously learning’ AI systems)`. Three names for one concept, printed in the sentence that votes it out of the vocabulary also in: continuously evolving · continuously adapting | concept_edges[0] | 0.95 | the tagger's words |
| concept_edges | clinical decision support systems —renamed-across-fields→ artificial intelligence strength of the relation: 0.9 · note: 'terms such as clinical decision support systems or EXPERT SYSTEMS, WHICH WERE MORE COMMONLY USED IN THE 1990s for technologies underpinned by AI systems ... It is likely that such systems, if published today, would be indexed under artificial intelligence OR machine learning'. THE AXIS IS TIME, NOT FIELD - the corpus's edge vocabulary has no type for that, so this is the closest available and the note carries the correction
Audit 2026-09-28: DOUBTFUL - read against its span and kept, but older CDSS papers would now be indexed under AI; renamed-across-fields describes a change over time, not across fields. | concept_edges[1] | 0.95 | the tagger's words |
| concept_edges | AI systems —distinct-from→ other health interventions strength of the relation: 0.95 · note: 'AI systems, unlike other health interventions, can unpredictably yield errors that are not easily detectable or explainable by human judgement'. The stated reason the existing checklist needed extending at all also in: AI systems · other health interventions | concept_edges[2] | 0.95 | the tagger's words |
| concept_edges | Health outcome —same-as→ Clinical outcome strength of the relation: 0.85 · note: NO CUE ATTESTS THIS EDGE AND NONE COULD. The identity is not stated anywhere in the paper - it is a property of two definition STRINGS, which differ by 'that' versus 'which' and in nothing else (verified with difflib, not by reading). A cue list can establish that a sentence contains a word; this relation is carried by the artefact's structure, and confidence is 0.85 rather than 0.95 because the authors may have intended a distinction that did not survive into the text
Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the glossary gives the two terms word-for-word identical definitions; same-as is inferred, not stated. also in: Health outcome · Clinical outcome | concept_edges[3] | 0.85 | the tagger's words |
| content_tags | THE GLOSSARY DEFINES TWO TERMS IDENTICALLY: `Health outcome` and `Clinical outcome` are `Measured variables in the trial THAT / WHICH are used to assess the effects of an intervention`. Word-diffed, the only difference is `that` vs `which` | content_tags.findings[0] | 0.95 | the tagger's words |
| content_tags | AND ONE ENTRY REFUSES TO DEFINE ITS TERM ON PURPOSE: `it is up to the investigator to specify what should be considered a performance error, preferably based on prior evidence`. A named, deliberately open slot in a controlled vocabulary | content_tags.findings[1] | 0.95 | the tagger's words |
| content_tags | the glossary defines SPIRIT-AI as an additional checklist item for AI-specific content not covered by SPIRIT 2013 | content_tags.findings[2] | 0.95 | the tagger's words |
| content_tags | CONSORT-AI 5(iii) first failed inclusion (77%), then passed at 97% after rewording and a re-vote | content_tags.findings[3] | 0.95 | the tagger's words |
| content_tags | TERM GRANULARITY IS PRESCRIBED BY POSITION IN THE PAPER: `A BROADER UMBRELLA TERM such as artificial intelligence or machine learning is encouraged` in the title, and `More-precise terms should be used in the abstract, rather than the title` | content_tags.findings[4] | 0.95 | the tagger's words |
| content_tags | THE VOCABULARY OF THE ERA DECIDED WHAT THE SEARCH FOUND, AND THEY SAY SO: the search used `artificial intelligence`, `machine learning`, `deep learning` `BUT NOT terms such as clinical decision support systems or EXPERT SYSTEMS, WHICH WERE MORE COMMONLY USED IN THE 1990s ... It is likely that such systems, if published today, would be indexed under artificial intelligence` | content_tags.findings[5] | 0.95 | the tagger's words |
| content_tags | A CONCEPT WAS EXCLUDED BY VOTE WITH THE REASON PUBLISHED: `continuously evolving AI systems ... it was agreed that this be excluded from CONSORT-AI ... relatively early in its development WITHOUT TANGIBLE EXAMPLES in healthcare applications` | content_tags.findings[6] | 0.95 | the tagger's words |
| content_tags | there were 316 registered trials, 62 completed and 7 with published results | content_tags.findings[7] | 0.95 | the tagger's words |
| content_tags | and the checklist is a floor, stated as one: `these are minimum requirements and there may be value in including additional items not included in the checklists` | content_tags.findings[8] | 0.95 | the tagger's words |
| content_tags | AND A DECISION THRESHOLD IS THE WORKED EXAMPLE OF WHAT MUST BE REPORTED: `skin lesion excision if the diagnosis is positive and the probability is >80%` - the guideline asks for the cut point that turns an output into an action | content_tags.findings[9] | 0.95 | the tagger's words |
| content_tags | `A health intervention that relies upon an AI/ML component to serve its purpose` also in: AI intervention | content_tags.definitions[0] | 0.95 | the tagger's words |
| content_tags | `Instances in which the AI intervention fails to perform as expected ... it is up to the investigator to specify what should be considered a performance error, preferably based on prior evidence` also in: Performance error | content_tags.definitions[1] | 0.95 | the tagger's words |
| content_tags | `Measured variables in the trial THAT are used to assess the effects of an intervention` - and `Clinical outcome` is the same sentence with `which` also in: Health outcome | content_tags.definitions[2] | 0.95 | the tagger's words |
| content_tags | `The clinical and operational settings from which the data used for training the model is generated`, spanning physical, operational and clinical setting also in: Development environment | content_tags.definitions[3] | 0.95 | the tagger's words |
| content_tags | `An additional checklist item to address AI-specific content that is not adequately covered by CONSORT 2010` also in: CONSORT-AI extension item | content_tags.definitions[4] | 0.95 | the tagger's words |
| content_tags | 80 — the pre-specified vote share required for an item to enter the checklist | content_tags.quantities[0] | 0.95 | the tagger's words |
| content_tags | 77 — the vote CONSORT-AI 5(iii) received on its initial wording - below the threshold, after which the item was reworded and re-voted at 97% | content_tags.quantities[1] | 0.95 | the tagger's words |
| content_tags | 14 — new checklist items added to CONSORT 2010 - 11 extensions and 3 elaborations | content_tags.quantities[2] | 0.95 | the tagger's words |
| content_tags | 103 — first-round Delphi responses; 91 in the second round, 88% of round one | content_tags.quantities[3] | 0.95 | the tagger's words |
| content_tags | 7 — AI trials with published results at the time of the search, from 316 registered and 62 completed - the whole published evidence base for the checklist | content_tags.quantities[4] | 0.95 | the tagger's words |
| content_tags | the outcome CONSORT-AI item 19 requires an analysis of, and the one term the glossary leaves the investigator to define also in: Performance error | content_tags.variables[0] | 0.95 | the tagger's words |
Tags that could not be placed in the text:
What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.