ACCORD (ACcurate COnsensus Reporting Document): A reporting guideline for consensus methods in biomedicine developed via a modified Delphi — TEI-tagged view

William T. GattrellpersName, Patricia LogullopersName, Esther J. van ZuurenpersName, Amy PricepersName, Ellen L. HughespersName, Paul BlazeypersName, Christopher C. WinchesterpersName, David ToveypersName, Keith GoldmanpersName, Amrit Pali HunginpersName, Niall HarrisonpersName (2024date). “ACCORD (ACcurate COnsensus Reporting Document): A reporting guideline for consensus methods in biomedicine developed via a modified Delphi.” PLOS Medicinetitle·j.

Object of study — how consensus methodsterm·mth in biomedical research are reported

Research question — which items a reporting guidelineterm·mth for all consensus methodsterm·mth in biomedicine should require

Contribution — the ACCORDterm checklistterm·key: 35 reporting items for any consensus study in health research

Full text — tagged

The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.

ACCORDterm (ACcurate COnsensus Reporting Document): A reporting guidelineterm·mth for consensus methodsterm·mth in biomedicine developed via a modified Delphiterm·mth Abstract In biomedical research, it is often desirable to seek consensus among individuals who have differing perspectives and experience. This is important when evidence is emerging, inconsistent, limited, or absent. Even when research evidence is abundant, clinical recommendations, policy decisions, and priority-setting may still require agreement from multiple, sometimes ideologically opposed parties. Despite their prominence and influence on key decisions, consensus methodsterm·mth are often poorly reported. Our aim was to develop the first reporting guidelineterm·mth dedicated to and applicable to all consensus methodsterm·mth used in biomedical research regardless of the objective of the consensus process, called ACCORDterm (ACcurate COnsensus Reporting Document).

We followed methodology recommended by the EQUATOR Networkterm·key for the development of reporting guidelinesterm·mth: a systematic review was followed by a Delphiterm·mth process and meetings to finalize the ACCORDterm checklistterm·key. The preliminary checklistterm·key was drawn from the systematic review of existing literature on the quality of reporting of consensus methodsterm·mth and suggestions from the Steering Committee. A Delphiterm·mth panel ( n = 72) was recruited with representation from 6 continents and a broad range of experience, including clinical, research, policy, and patient perspectives. The 3 rounds of the Delphiterm·mth process were completed by 58, 54, and 51 panelists. The preliminary checklistterm·key of 56 items was refined to a final checklistterm·key of 35 items relating to the article title ( n = 1), introduction ( n = 3), methods ( n = 21), results ( n = 5), discussion ( n = 2), and other information ( n = 3).

The ACCORDterm checklistterm·key is the first reporting guidelineterm·mth applicable to all consensus-based studies. It will support authors in writing accurate, detailed manuscripts, thereby improving the completeness and transparency of reporting and providing readers with clarity regarding the methods used to reach agreement. Furthermore, the checklistterm·key will make the rigor of the consensus methodsterm·mth used to guide the recommendations clear for readers. Reporting consensus studies with greater clarity and transparency may enhance trust in the recommendations made by consensus panels.

Background Evidence-based medicineterm·mth relies on (1) the best available evidence; (2) patients’ values, preferences, and knowledge; and (3) healthcare professionals’ experience and expertise [ 1 , 2 ]. When healthcare professionals need to make clinical decisions, or when recommendations or guidance are needed and there is uncertainty on the best course of action, such as when evidence is emergent, inconsistent, limited, or absent—not least in rapidly evolving fields such as pandemics [ 3 ]—the collation and dissemination of knowledge, experience, and expertise becomes critical. Coordinating this process may be best achieved through the use of formal consensus methodsterm·mth [ 4 ] such as those described in Table 1 .

A selection of common consensus methodsterm·mth used in healthcare-related activities or research.

Consensus methodsterm·mth are widely applied in healthcare ( Table 2 ). However, the specific method has the potential to affect the result of a consensus exercise and shape the recommendations generated. In addition, the expertise needed to contribute to the consensus process will vary depending on the research subject, and a range of participants may be required, including, but not limited to, clinical guideline developers, clinical researchers, healthcare professionals, epidemiologists, ethicists, funders, journal editors, laboratory specialists, medical publication professionals, meta-researchers, methodologists, pathologists, patients and carers/families, pharmaceutical companies, public health specialists, policymakers, politicians, research scientists, surgeons, systematic reviewers, and technicians.

Examples of applications of consensus methodsterm·mth in healthcare-related research.

Consensus obtained from a group of experts using formal methods is recognized as being more reliable than individual opinions and experiences [ 16 – 18 ]. Consensus methodsterm·mth help to overcome the challenges of gathering opinions from a group, such as discussions being dominated by a small number of individuals, peer pressure to conform to a particular opinion, or the risk of group biases affecting overall decision-making [ 4 ].

Despite their critical role in healthcare and policy decision-making, consensus methodsterm·mth are often poorly reported [ 19 ]. Generic problems include inconsistency and lack of transparency in reporting, as well as more specific criticisms such as lack of detail regarding how participants or steering committee members were selected, missing panelist background information, no definition of consensus, missing response rates after each consensus round, no description of level of anonymity or how anonymity was maintained, and a lack of clarity over what feedback was provided between rounds [ 19 ].

Reporting guidelinesterm·mth can enhance the reporting quality of research [ 20 – 22 ], and the absence of a universal reporting guidelineterm·mth for studies using consensus methodsterm·mth may contribute to their well-documented suboptimal reporting quality [ 5 , 19 , 23 – 25 ]. A systematic review found that the quality of reporting of consensus methodsterm·mth in health research was deficient [ 19 ], and a methodological review found that articles that provided guidance on reporting Delphiterm·mth methods vary widely in their criteria and level of detail [ 25 ]. The Conducting and Reporting Delphiterm·mth Studies (CREDESterm) guideline was designed to support the conduct and reporting of Delphiterm·mth studies, with a focus on palliative care [ 26 ]. The 23-item AGREE-IIterm instrument [ 27 ], which is widely used for reporting clinical practice guidelines, and COS-STAR for reporting core outcome set development [ 28 ], both contain a very limited number of items related to consensus.

Therefore, a comprehensive guideline is needed to report the numerous methods available to assess and/or guide consensus in medical research. The ACcurate COnsensus Reporting Document (ACCORDterm) reporting guidelineterm·mth project was initiated to fulfill this need. We followed EQUATOR Networkterm·key–recommended best practices for reporting guidelineterm·mth development, which included a systematic review and consensus exercise. Our aim was to develop a new tool, applicable worldwide, that will facilitate the rigorous and transparent reporting of all types of consensus methodsterm·mth across the spectrum of health research [ 29 ]. A comprehensive reporting guidelineterm·mth will enable readers to understand the consensus methodsterm·mth used to develop recommendations and therefore has the potential to positively impact patient outcomes.

Methods Scope of ACCORDterm ACCORDterm is a meta-research project to develop a reporting guidelineterm·mth for consensus methodsterm·mth used in health-related activities or research ( Table 2 ) [ 29 ]. The guideline was designed to be applicable to simple and less structured methods (such as consensus meetings), more systematic methods (such as nominal group technique or Delphiterm·mth), or any combination of methods utilized to achieve consensus. Therefore, the ACCORDterm checklistterm·key should be applicable to work involving any consensus methodsterm·mth. In addition, although ACCORDterm has been structured to help reporting in a scientific manuscript (with the traditional article sections such as introduction, methods, results, and discussion), the checklistterm·key items can assist authors in writing other types of text describing consensus activities.

ACCORDterm is a reporting guidelineterm·mth that provides a checklistterm·key of items that we recommend are included in any scientific publication in healthcare reporting the results of a consensus exercise. However, it is not a methodological guidelineterm·mth. It is not intended to provide guidance on how researchers and specialists should design their consensus activities, and it makes no judgment on which method is most appropriate in a particular context. Furthermore, ACCORDterm is not intended to be used for reporting research in fields outside health, such as social sciences, economics, or marketing.

Study design, setting, and ethics The ACCORDterm project was registered prospectively on January 20, 2022 on the Open Science Framework [ 30 ] and the EQUATOR Networkterm·key website [ 31 ], and received ethics approval from the Central University Research Ethics Committee at the University of Oxford (reference number: R81767 /RE001). The ACCORDterm protocol has been previously published [ 29 ] and followed the EQUATOR Networkterm·key recommendations for developing a reporting guidelineterm·mth [ 32 , 33 ], starting with a systematic review of the literature [ 19 ], followed by a modified Delphiterm·mth process. In a planned change to the Delphiterm·mth method as originally formulated, the preliminary list for voting was based on the findings of this systematic review rather than initial ideas or statements from the ACCORDterm Delphiterm·mth panel, although the panel could suggest items during the first round of voting. In addition, the ACCORDterm Steering Committee made final decisions on item inclusion and refined the checklistterm·key wording as described below.

ACCORDterm Steering Committee WTG and NH founded the ACCORDterm project, seeking endorsement from the International Society of Medical Publication Professionals (ISMPP) in April 2021. ISMPP provided practical support and guidance on the overall process at project outset but was not involved in checklistterm·key development. The ACCORDterm Steering Committee, established over the following months, was multidisciplinary in nature and comprised researchers from different countries and settings. Steering Committee recruitment was iterative, with new members invited as needs were identified by the founders and existing committee, to ensure inclusion of the desired range of expertise or experience. Potential members were identified via ISMPP, literature research, professional connections, and network recommendations. When the protocol was submitted for publication, the Steering Committee had 11 members (WTG, PL, EJvZ, AP, CCW, DT, KG, APH, NH, and Robert Matheis [RM] from ISMPP). Bernd Arents joined the Steering Committee in July 2021 but left in December of that year, as did RM in August 2022, both citing an excess of commitments as their reason for stepping down. Patient partners were invited as Delphiterm·mth panelists. Paul BlazeypersName joined the Steering Committee in September 2022 as a methodologist to support the execution of the ACCORDterm Delphiterm·mth process and provide additional expertise on consensus methodsterm·mth.

The final Steering Committee responsible for the Delphiterm·mth process and development of the checklistterm·key had members working in 4 different countries: Canada, United Kingdom, United States of America, and the Netherlands. A wide range of professional roles was represented by the Steering Committee with several members bringing experience from more than one area including clinician practitioners (medical doctor, physical therapist), methodologists (consensus methodologist, research methodologist, expert in evidence synthesis), medical publication professionals (including those working in the pharmaceutical industry), journal editors, a representative of the EQUATOR Networkterm·key, and a representative of the public ( S1 Text ).

Protocol development The ACCORDterm protocol was developed by the Steering Committee before the literature searches or Delphiterm·mth rounds were commenced and has been published previously [ 29 ]. An overview of the methods used, together with some amendments made to the protocol during the development of ACCORDterm in response to new insights, is provided below.

Systematic review and development of preliminary checklistterm·key A subgroup of the Steering Committee conducted a systematic review with the dual purpose of identifying existing evidence on the quality of reporting of consensus methodsterm·mth and generating the preliminary draft checklistterm·key of items that should be reported [ 19 ]. The systematic review has been published [ 19 ] and identified 18 studies that addressed the quality of reporting of consensus methodsterm·mth, with 14 studies focused on Delphiterm·mth only and 4 studies including Delphiterm·mth and other methods [ 19 ]. A list of deficiencies in consensus reporting was compiled based on the findings of the systematic review. Items in the preliminary checklistterm·key were subsequently derived from the systematic review both from the data extraction list ( n = 30) [ 19 ] and from other information that was relevant for reporting consensus methodsterm·mth ( n = 26) [ 19 ].

Next, the Steering Committee voted on whether the preliminary checklistterm·key items ( n = 56) should be included in the Delphiterm·mth via 2 anonymous online surveys conducted using Microsoft Forms (See S2 Text ). There were 5 voting options: “Strongly disagree,” “Disagree,” “Agree,” “Strongly agree,” and “Abstain/Unable to answer.” NH processed the results in Excel, and WTG provided feedback and therefore neither voted. Items that received sufficient support (i.e., >80% of respondents voted “Agree”/“Strongly agree”) were included in the Delphiterm·mth, while the rest were discussed by the Steering Committee for potential inclusion or removal. During the first survey, Steering Committee members could propose additional items based on their knowledge and expertise. These new items were voted on in the second Steering Committee survey. Upon completion of this process, the Steering Committee approved and updated the preliminary draft checklistterm·key, which was then prepared for voting on by the Delphiterm·mth panel. Items were clustered or separated as necessary for clarity.

Delphiterm·mth panel composition Using an anonymous survey (June 9–13, 2022), the Steering Committee voted on the desired profile of Delphiterm·mth panelists for the ACCORDterm project. There was unanimous agreement that geographic representation was important, and the aim was to recruit from all continents (thereby covering both Northern and Southern hemispheres) and include participants from low-, middle-, and high-income countries to account for potential differences in cultural and ideological ways of reaching agreement. The aim was to include a broad range of participants: clinicians, researchers experienced in the use of consensus methodsterm·mth and in clinical practice guideline development, patient advocates, journal editors, publication professionals and publishers, regulatory specialists, public health policymakers, and pharmaceutical company representatives. As described in the ACCORDterm protocol [ 29 ], there are no generally agreed standards for the panel size in Delphiterm·mth studies, although panels of 20 to 30 are common. The target panel size (approximately 40 panelists) was therefore guided by the desired representation and to ensure an acceptable number of responses (20, assuming a participation rate of 50%) in the event of withdrawals or partial completion of review.

Delphiterm·mth panel recruitment Potential participants for the Delphiterm·mth panel were identified in several ways: from the author lists of publications included in the systematic review, from invitations circulated via an EQUATOR Networkterm·key newsletter (October 2021) [ 34 ] and at the European Meeting of ISMPP in January 2022, and by contacting groups potentially impacted by ACCORDterm (e.g., the UK National Institute for Health and Care Excellence [NICE]). Individuals were also invited to take part through the ACCORDterm protocol publication [ 29 ], and the members of the Steering Committee contacted individuals in their networks to fill gaps in geographical or professional representation. To minimize potential bias, none of the Steering Committee participated in the Delphiterm·mth panel.

Invitations were issued to candidate panelists who satisfied the inclusion criteria. While participants were not generally asked to suggest other panel members, in some cases, invitees proposed a colleague to replace them on the panel. Only the Steering Committee members responsible for administering the Delphiterm·mth had access to the full list of ACCORDterm Delphiterm·mth panel members. Panelists were invited by email, and reminder emails were sent to those who did not respond. Out of the 133 panelists invited, 72 agreed to participate. No panelists or Steering Committee members were reimbursed or remunerated for taking part in the ACCORDterm project.

Planned Delphiterm·mth process The Delphiterm·mth method was chosen to validate the checklistterm·key, in line with recommendations for developing reporting guidelinesterm·mth [ 32 ]. A 3-round Delphiterm·mth was planned to allow for iteration, with the option to include additional rounds if necessary. Panelists who agreed to take part received an information pack containing an introductory letter, a plain language summary, an informed consent statement, links to the published protocol and systematic review, and the items excluded by the Steering Committee (see S3 Text ). Survey materials were developed by PL and PB in English and piloted by WTG and NH. Editorial and formatting changes were made following the pilot stage to optimize the ease of use of the survey. In an amendment to the protocol, the order of candidate items was not randomized within each manuscript section. The Jisc Online Survey platform (Jisc Services, Bristol, United Kingdom) was used to administer all Delphiterm·mth surveys, ensuring anonymity through automatic coding of participants. Panelists were sent reminders to complete the survey via the survey platform, and one email reminder was sent to panelists the day before the deadline for each round.

The Delphiterm·mth voting was modified to offer 5 voting options: “Strongly disagree,” “Disagree,” “Neither agree nor disagree,” “Agree,” and “Strongly agree.” Votes of “Neither agree nor disagree” were included in the denominator. The consensus thresholdterm·mth was defined a priori as ≥80% of a minimum of 20 respondents voting “Agree” or “Strongly agree.” Reaching the consensus thresholdterm·mth was not a stopping criterion. For inclusion in the final checklistterm·key, each item was required to achieve the consensus criteria following at least 2 rounds of voting. This ensured that all items had the opportunity for iteration between rounds (a central tenet of the Delphiterm·mth method) [ 6 ] and enabled panelists to reconsider their voting position in light of feedback from the previous round.

In Round 1, panelists had the opportunity, anonymously, to suggest new items to be voted on in subsequent rounds. Panelists were also able to provide anonymous free-text comments in each round to add rationale for their chosen vote or suggest alterations to the item text. After each voting round, the comments were evaluated and integrated by WTG, PL, PB, and NH and validated by the Steering Committee. If necessary, semantic changes were made to items to improve clarity and concision.

Feedback given to participants between rounds included the anonymized total votes and the percentage in each category (see example in S4 Text ) to allow panelists to assess their position in comparison with the rest of the group, as well as the relevant free-text comments on each item. Items that did not achieve consensus in Rounds 1 and 2 were revised or excluded based on the feedback received from the panelists. Items that were materially altered (to change their original meaning) were considered a new item. All wording changes were recorded. Panelists received a table highlighting wording changes as part of the feedback process so that they could see modifications to checklistterm·key items (for example feedback documents, see S5 Text ).

Items reaching consensus over 2 rounds were removed from the Delphiterm·mth for inclusion in the checklistterm·key. Items achieving agreement in Round 1, which then fell into disagreement in Round 2 were considered to have “unstableterm·mth” agreement. These unstableterm·mth items were revised based on qualitative feedback from the panel and were included for revoting in Round 3.

Steering Committee checklistterm·key finalization process Consistent with the protocol [ 29 ], following completion of the Delphiterm·mth process, the Steering Committee was convened for a series of three 2-hour virtual workshops (March 7, 14, and 16, 2023) to make decisions and finalise the checklistterm·key. For each item, WTG, PL, PB, and NH presented a summary of voting, comments received, and a recommended approach. The possible recommended approaches are shown in S6 Text .

All recommendations (for example, to keep approved items, confirm exclusion of rejected items, etc.) were followed by an explanation of why WTG, PL, PB, or NH felt this would be the most appropriate action and a discussion between Steering Committee members in which the suggested action could be challenged and changed.

Grammatical changes were also considered at this stage but only where they did not change the meaning of an approved item. Following review of all items, the order of the checklistterm·key items was evaluated by WTG, PL, PB, and NH.

Standardized terminology After the consensus meetings, NH updated and standardized the terminology according to the type of information requested in the item to ensure consistency between items, and this was approved by the Steering Committee. This standardization of terminology incorporated rules established for the use of terms common in reporting guidelinesterm·mth, as shown in S7 Text , such as the difference between using “state” or “describe.” All but 2 items (R5 and O1) contain a verb from S7 Text .

Results Delphiterm·mth panel demographics The Delphiterm·mth panel included a diverse group of panelists, representing a wide range of geographical areas and professions ( Table 3 ). Of the 72 participants who indicated their willingness to participate in the Delphiterm·mth panel, 58 (81%) completed Round 1 and were invited to Round 2. Fifty-four participants completed Round 2 and were invited to Round 3, which was completed by 51 participants.

Self-identified demographics of the Delphiterm·mth panelists, per voting round.

Delphiterm·mth results The updated preliminary draft checklistterm·key presented to the Delphiterm·mth panel for voting contained 41 items. The changes in the number of checklistterm·key items over the Delphiterm·mth voting rounds are illustrated in Fig 1 . After Round 1, 7 new items were added, and 1 item was lost by combining with another item, resulting in 47 items being included in Round 2. Only items that were unstableterm·mth ( n = 4) or were modified sufficiently to be considered new ( n = 6) were voted on in Round 3. After Round 2, 33 items achieved consensus, and a further 3 items achieved consensus after all 3 rounds of voting. Therefore, at the end of the Delphiterm·mth process, consensus was reached on 36 items. The results of the Delphiterm·mth process, showing the iteration of items and level of agreement at each round, are summarized in S8 Text .

A flow diagram to show the development of checklistterm·key items. *Potential items from relevant information beyond the predefined data extraction form [ 19 ]. † New item (T1) proposed at checklistterm·key review meeting.

Finalization by Steering Committee One item rejected by the Delphiterm·mth panel was restored to the checklistterm·key (M10, becoming item M5), and 3 highly approved (>90%) items were modified by combining with other items during the Steering Committee finalization workshops. S8 Text contains the iterations of the rounds of the Delphiterm·mth voting demonstrating the changes made in each round and showing how items evolved.

Restored item (Delphiterm·mth M10 > Final M5) Delphiterm·mth item M10 (patient and public involvement) failed to achieve stable consensus during the voting process (Round 1, 87.5%; Round 2, 73.1%; Round 3, 76.0%; see S8 Text ). The comments from the panel led the Steering Committee to conclude that panelists had not reached agreement on reporting patient and public involvement due to the item being essential in some—but not all—consensus processes (“ Depends on the topic of Delphiterm·mth consensus , should be optional ”; “ For me this rests on the topic of the exercise ”) and because of disagreements about preferred terminology (“ The difference between lay and patient and public partners is potentially confusing ”; “ DO NOT change ‘participants’ to ‘partners’ ”). However, the Steering Committee identified many situations where the inclusion of patients would be considered essential. Priority-setting and core outcome identification are just 2 areas where patient participation in consensus exercises is becoming standard [ 35 – 37 ]. Based on unanimous agreement (11/11), the Steering Committee decided to reinstate M10 as reporting item M5, while taking into account the most consistent comments regarding wording (notably, that “lay” should not be used).

Items with high level of agreement that were modified Three original items, R3, R6, and R7, overlapped by all covering aspects of which data were reported from the Delphiterm·mth voting rounds. During the checklistterm·key finalization workshops, the Steering Committee discussed these 3 items and combined them to create 2 final items, R3 (quantitative data) and R4 (qualitative data). In addition, the Steering Committee noted an overlap between original items M22 and R8 related to modifications made to items or topics during the consensus process (see S8 Text ). These 2 items were combined to create the final item R5. Finally, M13 was revised to remove a conceptual overlap with M12 and to use clearer language.

Final checklistterm·key The final ACCORDterm checklistterm·key comprised 35 items that were identified as essential to ensure clear and transparent reporting of consensus studies. The finalized ACCORDterm checklistterm·key is presented in Table 4 and is available to download and complete ( S9 Text ).

The final ACCORDterm checklistterm·key for the reporting of consensus methodsterm·mth.

Discussion The ACCORDterm checklistterm·key has been developed using a robust and systematic approach, with input from participants with a variety of areas of expertise, and it is now available for any health researcher to use to report studies that use consensus methodsterm·mth. The process of developing ACCORDterm itself used consensus methodsterm·mth, which are reported here according to the checklistterm·key developed.

Why ACCORDterm was needed The need for optimal reporting of consensus methodsterm·mth has been documented for decades [ 19 , 24 ]. The absence of a reporting guidelineterm·mth that encompasses the range of consensus methodsterm·mth may contribute to poor reporting quality [ 5 ], and this prompted the development of the ACCORDterm checklistterm·key.

There are 2 EQUATOR-listed reporting guidelinesterm·mth that provide guidance for specific projects that typically include consensus exercises: AGREE-IIterm has only 1 item, “Formulation of Recommendations,” relating to the method used to obtain consensus [ 27 ]. COS-STAR includes only 3 items around the definition of consensus and a “description of how the consensus process was undertaken” [ 28 ]. In addition, CREDESterm [ 26 ] is a method- and specialty-specific guideline aimed at supporting the conduct and reporting of Delphiterm·mth studies in palliative care. None of these guidelines is suitable as a comprehensive and general tool for reporting any type of consensus exercise. ACCORDterm addresses the breadth of methods used to attain consensus (including the Delphiterm·mth method) and should be complementary to AGREE-IIterm where a clinical practice guideline also includes a formal consensus development process. Another reporting guidelineterm·mth currently under development, DELPHISTAR [ 25 ], is Delphiterm·mth specific and covers medical and social sciences. ACCORDterm extends beyond Delphiterm·mth methods and encompasses a wide range of consensus methodsterm·mth in various health-related fields.

Although familiarity with ACCORDterm is likely to be useful to ensure relevant elements are considered when designing a consensus study, it is a reporting guidelineterm·mth and not a mandate for study conduct. The methodological background to the items and published examples of what we consider to be good reporting will be discussed in the ACCORDterm Explanation and Elaboration document (manuscript in preparation).

Strengths and limitations ACCORDterm was conducted through an open, collaborative process with a predefined, published protocol [ 29 ]. It started with a systematic review [ 19 ] using robust methods of searching, screening, and extraction, which led to the identification of common gaps in reporting consensus methodsterm·mth. Only 18 studies were eligible for inclusion in the systematic review, and data extraction generated 30 potential checklistterm·key items. An additional 26 items were identified that were not covered by the data extraction list. Following this thorough process, these 56 potential items were supplemented by a further 9 proposed by the Steering Committee, with an additional 7 proposed by Delphiterm·mth panelists.

The ACCORDterm checklistterm·key involved input from participants with a wide range of expertise, including methodologists, patient advocates, healthcare professionals, journal editors, publication professionals, and representatives from the pharmaceutical industry and bodies such as NICE and the Scottish Intercollegiate Guidelines Network. With a few exceptions reported here, their recommendations were fully adopted and integrated into the final checklistterm·key. ACCORDterm was developed to assist everyone involved in consensus-based activities or research. It will assure participants that methods will be accurately reported; guide authors when writing up a publication; help journal editors and peer reviewers when assessing a manuscript for publication; and enhance trust in the recommendations made by consensus panels. Our hope is that ACCORDterm will ultimately benefit patients by improving the transparency and robustness of consensus studies in healthcare.

A limitation of the ACCORDterm initiative is that the panel was not as diverse as we hoped. ACCORDterm was a meta-research project drawing on work from many countries, but our view is that diversity of expertise and personal experience always strengthens consensus discussions. Our aim was to broaden the diversity of contributors to ACCORDterm by recruiting a panel more diverse than the Steering Committee in geography and experience to mitigate the perpetuation of and dilute any biases held by the Steering Committee. Although invitations were sent to potential panelists in South America, Asia, Africa, and Oceania, few responses were obtained, leading to limited participation from these continents and a panel that was largely drawn from Europe and North America. Similarly, the professional diversity of the ACCORDterm panel was not as broad as we hoped, with patient partners and policymakers relatively underrepresented compared with clinicians. Therefore, in the future, greater efforts should be made to recruit panelists with experience in consensus from a broader range of professions as well as other regions and countries with different cultures and health systems. For example, although some experience of clinical psychology exists in the Steering Committee, inclusion of more behavioral scientists with experience of the process of decision-making could be a helpful addition. Similarly, the inclusion of more policymakers would strengthen the representation of their perspective on consensus reporting to ensure it was relevant and reliable and, therefore, acceptable to be referenced and inform policy. Although these biases were not fully mitigated, future revisions or extensions to ACCORDterm will aim to improve in this regard.

Members of the ACCORDterm Steering Committee did not vote in the Delphiterm·mth surveys. In our process, the virtual workshops held to finalize the ACCORDterm checklistterm·key did not include the Delphiterm·mth panel. This might be seen as a limitation by some, especially those involved in reporting guidelinesterm·mth development, as a consensus meeting including some expert members of the Delphiterm·mth panel is usually conducted according to the guidance issued by the EQUATOR Networkterm·key [ 32 ]. However, our process held the Steering Committee and Delphiterm·mth panel separate: the Steering Committee did not participate in the Delphiterm·mth panel, and the Delphiterm·mth panelists did not participate in the final consensus discussions. We suggest that this could in fact be seen as a strength of our process since, while the larger Delphiterm·mth panel did not reach consensus on 1 particular item, discussion among the Steering Committee led to its inclusion in the final checklistterm·key without full approval of the Delphiterm·mth panel (see results and commentary for item M5). If the panelists had been part of the final consensus meeting, this may have resulted in the omission from the final checklistterm·key of this item, which related to patient participation in consensus studies. However, the experience represented by the Steering Committee recognized the value of patient participation in consensus recommendations, the importance of which is reported in the literature [ 38 ], and voted to include this item.

Stabilityterm·mth of agreement indicates when consensus is present among a group. There are several methods to assess for stabilityterm·mth, but ACCORDterm adhered to a simple definition of achieving the a priori agreed threshold for agreement over a minimum of 2 voting rounds [ 39 ].

Another limitation that consensus and survey specialists may note is that the items in our Delphiterm·mth survey were not presented to panelists in a random order. Since ACCORDterm was proposing content items for the sections of a scientific manuscript (title, introduction, methods, results, and discussion), we preferred to present items in these sections in the order that they usually appear to enhance comprehension and avoid confusion. This is something that may affect all reporting guidelinesterm·mth development. In fact, several panelists provided feedback on how to order the items.

The implementation of the ACCORDterm reporting guidelineterm·mth Many reporting guidelinesterm·mth are published without initiatives to facilitate implementation. Only 15.7% of guidelines on the EQUATOR Networkterm·key website mentioned an implementation plan [ 33 ]. An implementation study to inform an Explanation and Elaboration document has been completed and the results submitted for presentation at a conference. The full ACCORDterm implementation plan and supporting materials are being developed and will be available on the ACCORDterm website ( https://www.ismpp.org/accordterm ).

The future of ACCORDterm Robust reporting is particularly important for studies using consensus methodsterm·mth given that so many methods exist and researchers frequently make modifications to “standard” methods. We anticipate that updates of the ACCORDterm checklistterm·key will be necessary, as technology and consensus methodsterm·mth continue to evolve.

Besides updates, ACCORDterm could have extensions developed in areas such as nonclinical biomedical studies, health economics, or health informatics and artificial intelligence, and even beyond healthcare, with input from appropriate experts. The Steering Committee welcomes feedback and interest from other researchers in these areas.

Conclusions The ACCORDterm reporting guidelineterm·mth provides the scientific community with an important tool to improve the completeness and transparency of reporting of studies that use consensus methodsterm·mth. The ACCORDterm checklistterm·key supports authors in writing manuscripts with sufficient information to enable readers to understand the study’s methods, the study’s results, and the interpretation of those results so that they can draw their own conclusions about the robustness and credibility of the recommendations.

Supporting information (DOCX) Click here for additional data file.

(DOCX) Click here for additional data file.

(DOCX) Click here for additional data file.

(DOCX) Click here for additional data file.

(PDF) Click here for additional data file.

(DOCX) Click here for additional data file.

(DOCX) Click here for additional data file.

(DOCX) Click here for additional data file.

(DOCX) Click here for additional data file.