standOff. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote shown on the same row, so the borrowing can be checked against the thing borrowed from without leaving this page. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.| claim type | claim_type | the shape of the claim: existence, mechanism, method, critique, prediction, null |
| concept edge | concept_edges | an asserted relation between two concepts, with the sentence asserting it |
| concept | concepts | an idea the paper works with, quoted where the paper uses it |
| content tag | content_tags | a finding, definition, quantity, statistic or variable named in the paper |
| contribution | contribution | what the paper claims to add |
| limitation | limitations | what the paper says its evidence cannot support |
| method | method | how the paper produced its evidence |
| object of study | object_of_study | what the paper is about — the thing it investigates |
| phenomenon | phenomenon | the effect or regularity under study |
| research question | research_question | the question the paper sets itself |
| sense | senses | a distinct use of a term, quoted where the paper fixes it |
Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study.
Abstract
phenomenonPoor terminological consistency for core concepts in implementation science has been widely noted as an obstacle to effective meta-analyses. This inconsistency is also a barrier for those seeking guidance from the research literature when developing and planning implementation initiatives. concepts↗concepts↗concepts↗concept_edgesThe Expert Recommendations for Implementing Change (ERIC) study aims to address one area of terminological inconsistency: discrete implementation strategies involving one process or action used to support a practice change. The present report is on the second stage of the ERIC project that focuses on providing initial validation of the compilation of 73 implementation strategies that were identified in the first phase.
Purposive sampling was used to recruit a panel of experts in implementation science and clinical practice ( N = 35). These key stakeholders used concept mapping sorting and rating activities to place the 73 implementation strategies into similar groups and to rate each strategy’s relative importance and feasibility. Multidimensional scaling analysis provided a quantitative representation of the relationships among the strategies, all but one of which were found to be conceptually distinct from the others. content_tagsHierarchical cluster analysis supported organizing the 73 strategies into 9 categories. The ratings data reflect those strategies identified as the most important and feasible.
contributionThis study provides initial validation of the implementation strategies within the ERIC compilation as being conceptually distinct. The categorization and strategy ratings of importance and feasibility may facilitate the search for, and selection of, strategies that are best suited for implementation efforts in a particular setting.
The online version of this article (doi:10.1186/s13012-015-0295-0) contains supplementary material, which is available to authorized users.
Keywords: Concept mapping, Implementation research, Implementation strategies, Mental health, US Department of Veterans Affairs
Background
concepts↗concepts↗concept_edgesConcerns about poor consistency in implementation science (IS) terminology have led researchers to characterize the field as a “Tower of Babel” [ 1 ]. Inconsistent terminology complicates literature searches, and researchers have found that search strategy yield and precision indices for implementation and quality improvement studies are moderate at best [ 2 , 3 ]. This limits meta-analytic and replication efforts aimed at rigorously evaluating the effectiveness of implementation strategies and the value of existing literature for those enacting implementation initiatives. The science and practice of implementation would be greatly facilitated by a parsimonious nomenclature of conceptually distinct implementation strategies [ 2 , 4 – 7 ].
Recently, Powell et al. [ 8 ] reviewed the health and mental health literature (including 41 compilations and reviews) and proposed a compilation of 68 discrete implementation strategies involving one action or process. This compilation served as the starting point for a subsequent multi-stage project called Expert Recommendations for Implementing Change (ERIC) [ 9 ]. The ERIC project’s first stage involved expert panelists ( N = 71) using a modified Delphi process to revise the compilation, which resulted in an updated compilation of 73 discrete implementation strategies [ 10 ].
object_of_studyThe aim of the ERIC project’s second stage, presented here, was to obtain preliminary validation of the compilation of 73 implementation strategies by studying the relationships between the strategies and obtaining relative importance and feasibility ratings for each strategy. research_questionThe study of the relationships among the strategies supports the evaluation of whether the strategies are conceptually distinct from one another as well as how the strategies can be organized into conceptually relevant groupings. The former can also serve the practical purpose of making it easier for stakeholders to consider the range of implementation strategies by thematic cluster. The importance and feasibility ratings for the strategies provide insight into the perceived applicability of the strategies. It is of general interest which strategies have relatively high and low ratings by experts.
Method
method↗A purposive sampling procedure was used to recruit an expert panel of implementation science and clinical experts ( N = 35) to participate in concept mapping and rating tasks [ 9 , 10 ]. A detailed description of procedures has been published [ 9 ], and a summary is provided here. concepts↗Concept mapping is a mixed-method procedure for engaging stakeholder groups in a structured conceptualization process [ 11 ]. This process supports visually representing the relationships among a set of related concepts and empirically clustering them into conceptually distinct categories and rating them on multiple dimensions.
The Concept Systems Global MAX™ [ 12 ] web platform was used for the panel’s sorting and rating tasks and data analysis. A more detailed introduction to concept mapping can be found in Trochim and Kane [ 13 ]. For the sorting task, participants were asked to sort virtual cards for each of the 73 strategies, accompanied by their definitions, into piles as they deemed appropriate. Participants were asked to rate each strategy for importance and feasibility ranging from 1 (relatively unimportant/not at all feasible) to 5 (extremely important/extremely feasible). These global ratings were prefaced by the following instructions: “Please select a number from 1 to 5 for each discrete implementation strategy to provide a rating in terms of how important (feasible) you think it is. Keep in mind that we are looking for relative importance (feasibility), use all the values in the rating scale to make distinctions.” Participants were able to select which set of activities they wanted to do first and were also able to work on the sorting and rating activities over multiple online sessions, at their convenience, before submitting their responses.
Multidimensional scaling and hierarchical cluster analyses were conducted to produce visual representations of the relationships among the strategies. Descriptive statistics for the importance and feasibility ratings were calculated. Each strategy’s importance and feasibility score was plotted on a graph. concepts↗The resulting scatterplot was divided into four quadrants or “Go-zones” (e.g., I, II, III, IV) using the mean of each dimension. For example, quadrant I contains strategies that have values above the means for both dimensions. The Go-zone quadrants column in Table 1 reflects the combined relative importance and feasibility for each strategy.
A summary of the 73 implementation strategies, organized by cluster with mean importance and feasibility ratings
Results
Experts who participated in the concept mapping and rating tasks and were affiliated with academic or healthcare institutions in the United States ( n = 34) or in Canada ( n = 1). Thirty-two of the 35 experts provided valid sorts (>75 % of strategies sorted), and 30 provided importance and feasibility ratings for all strategies. Sixty-three percent of participants had exclusive expertise in IS, 29 % were experts in both IS and clinical practice, and 8 % indicated clinical practice expertise only. Sixty-nine percent of participants had some affiliation with the US Department of Veterans Affairs (VA), most of whom also held academic appointments in social science or health-related schools or departments.
Figure 1 presents a point map that visually represents the relationships among the 73 implementation strategies, with each point on the map representing a strategy. The strategies are numbered to aid in cross-referencing the spatial relationships of the points on the map with their labels enumerated in Table 1 . claim_typeAll but two strategies were sorted as being conceptually distinct. Strategies #66 ( Use capitated payments ) and #70 ( Use other payment schemes ) were always sorted together. Two other strategies were proximal to one another though they were sorted together by only 4 of 32 panelists (#35 Identify and prepare champions and #57 Recruit, designate, and train for leadership ), indicating that they are more similar in how they relate with other strategies on the map, than they are directly similar to one another.
Point and cluster map of all 73 strategies identified in the ERIC process. The map reflects the product of an expert panel (valid response n = 32) sorting 73 discrete implementation strategies into groupings by similarity with each strategy being depicted by a yellow dot and accompanied by a number supporting cross-referencing to the strategies enumerated in Table 1 . sensesSpatial distances reflect how frequently the strategies were sorted together as similar. In general, the closer two points are together, the more frequently those strategies were sorted together. Strategies distal from one another were infrequently, if at all, sorted together. These spatial relationships are relative to the sorting data obtained in this study, and distances do not reflect an absolute relationship (i.e., a 5-mm distance in the present map does not reflect the same relationship as a 5-mm distance on a map from a different data set). The legend provides the label for each of the nine clusters of strategies. Dotted lines within the Develop stakeholder interrelationships cluster indicate how two separate clusters were merged into one large cluster due to conceptual similarity among their items. Dotted lines extending between other clusters archive the reassignment of strategies from their original cluster to a neighboring cluster to which there was a better conceptual fit (i.e., strategies #48, #58, and #62)
The final clusters were developed over 3 weeks of deliberations by the ERIC investigative team. A 13 cluster starting point was selected because it is one standard deviation above the mean number of clusters typically obtained in concept mapping [ 14 ]. In this study, 69 % of respondents sorted statements into 13 or fewer piles. We sequentially reviewed cluster merges and achieved consensus to merge clusters down to nine conceptually distinct clusters. For example, two clusters shown in pale green at the center bottom in Fig. 1 (separated by dashed lines) were merged to form a single cluster labeled Develop stakeholder interrelationships , as the original clusters were judged as not sufficiently conceptually distinct.
When the team reviewed the clusters for conceptual clarity, three proposals came forward to move individual strategies to neighboring clusters. First, #62 ( Start a dissemination organization ) was moved from the Engage consumers cluster to the Change infrastructure cluster, as it was judged more similar to infrastructure support for a practice change than engaging consumers. Second, #48 ( Organize clinician implementation team meetings ) was moved to the Develop stakeholder interrelationships cluster from Adapt and tailor to the context , as the former has greater interpersonal focus than the latter. And finally, #58 ( Remind clinicians ) was moved to the Support clinicians cluster from Provide interactive assistance because it is more administrative than interactive in focus. Unanimous consensus was reached for the final cluster arrangements. Additional file 1 provides a cluster-by-cluster visual tour of the concept map.
A multi-step process was used to determine labels for the final clusters. The list began with labels provided by expert panel members for their clusters that were most similar to the final cluster solutions. This list was supplemented with highly descriptive labels identified from the investigative team’s meeting minutes from cluster solution deliberations. Proposed criteria for developing cluster labels (Table 2 ) were introduced for team comment by one of the authors (LJD) along with suggested label revisions. These criteria were helpful in structuring iterative discussion among team members, the result of which was voted upon by the team and unanimously adopted.
Guidelines for cluster labels
Table 1 presents a summary of the 73 implementation strategies, organized by cluster with mean importance and feasibility ratings. content_tagsThere was a strong relationship ( r = 0.7) between the feasibility and importance ratings, meaning that most strategies fell within either quadrant I (high importance and feasibility) or III (low importance/feasibility). However, there were still a number of strategies that were viewed as important but not as feasible (12 %, e.g., Access new funding ), or feasible but less important (15 %, e.g., Remind clinicians ). Clusters of strategies that are more immediate and concrete and are potentially more in the control of those tasked with supporting change (e.g., Use evaluative and iterative strategies , Train and educate stakeholders ) tended to have higher importance and feasibility ratings. Clusters that are more strategic, but also potentially involve changing well-established systems (e.g., Change infrastructure , Utilize financial strategies ), tended to have lower ratings. Figure 2 presents a graphic of the Go-zone data.
Go-zone plot for all 73 strategies based on expert ratings. Note . The range of the x and y axes reflect the mean values obtained for all 73 of the discrete implementation strategies for each of the rating scales. The plot is divided into quadrants on the basis of the overall mean values for each of the rating scales. Quadrant labels are depicted with roman numerals next to the plot. Strategies in quadrant I fall above the mean for both the importance and the feasibility ratings. Thus, these strategies are those where there was the highest consensus regarding their relative high importance and feasibility. Conversely, quadrant III reflects the strategies where there was consensus regarding their relative low importance and feasibility. Quadrants II and IV reflect strategies that were relatively high in feasibility or importance, respectively, but low on the other rating scale
Discussion
Results from this study provide initial validation for viewing the 73 implementation strategies as conceptually distinct. Cluster analyses of the concept mapping data support grouping strategies into nine clusters which have practical heuristic value for those looking to the ERIC compilation of implementation strategies for guidance. The importance and feasibility ratings for the strategies supported the formation of Go-zone quadrants that can be used to help decision makers prioritize which strategies to use when planning an implementation initiative.
While the concept mapping strategy used in this study represents a strong methodological approach to evaluating whether the 73 implementation strategies are conceptually distinct and organizing them by theme and potential applicability (i.e., Go-zone analysis), there are notable limitations. Recruitment had been restricted to the time zones within the continental United States to minimize scheduling conflicts for elements of the ERIC project that required real time interactions among participants. limitationsThus, all but one of the 35 participants were from the United States, and 69 % had some affiliation with the VA. While concept maps with 30 or more participants are considered to be highly reliable [ 14 ], if stakeholders from outside the United States had practice contexts that alter the perceptions of these strategies interrelationships, or the ratings of their perceived importance and feasibility, different results may be obtained.
Acknowledgements
This project is funded through the US Department of Veterans Affairs Veterans Health Administration Mental Health Quality Enhancement Research Initiative (QLP 55–025). The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the US government. Additionally, TJW received support from the VA Office of Academic Affiliations Advanced Fellowships Program in Health Services Research and Development at the Center for Mental Healthcare & Outcomes Research, and BJP received support from the National Institute of Mental Health (F31 MH098478), the Doris Duke Charitable Foundation (Fellowship for the Promotion of Child Well-Being), and the Fahs-Beck Fund for Research and Experimentation. MJC received support from the VISN 4 Mental Illness Research, Education, and Clinical Center. MJC received support from the VISN 4 Mental Illness Research, Education, and Clinical Center at the VA Pittsburgh Healthcare System.
We would like to acknowledge the contributions of each member of the expert panel: Greg Aarons, University of California, San Diego; Mark Bauer, Harvard University and US Department of Veterans Affairs; Laurel Copeland, Scott and White Healthcare and US Department of Veterans Affairs; Torrey Creed, University of Pennsylvania; Laura Damschroder, University of Michigan and US Department of Veterans Affairs; Rani Elwy, Boston University and US Department of Veterans Affairs; Bradford Felker, University of Washington and US Department of Veterans Affairs; Hildi Hagedorn, University of Minnesota and US Department of Veterans Affairs; Alison Hamilton, University of California, Los Angeles and US Department of Veterans Affairs; Kathy Henderson, US Department of Veterans Affairs; Timothy Hogan, University of Massachusetts and US Department of Veterans Affairs; Kelly Koerner, Evidence-Based Practice Institute; Sarah Krein, University of Michigan and US Department of Veterans Affairs; Julie Kreyenbuhl, University of Maryland and US Department of Veterans Affairs; Sara Landes, University of Washington and US Department of Veterans Affairs; Cara Lewis, Indiana University-Bloomington; Julie Lowery, University of Michigan and US Department of Veterans Affairs; Aaron Lyon, University of Washington; Natalie Maples, University of Texas Health Science Center San Antonio; Alan McGuire, Indiana University-Purdue University Indianapolis and US Department of Veterans Affairs; Amanda Midboe, US Department of Veterans Affairs; Laura Murray, Johns Hopkins University; Princess Osei-Bonsu, US Department of Veterans Affairs; Richard Owen, University of Arkansas for Medical Sciences and US Department of Veterans Affairs; Louise Parker, University of Massachusetts Boston; Enola Proctor, Washington University in St. Louis; Mona Ritchie, US Department of Veterans Affairs; Angie Rollins, US Department of Veterans Affairs and Indiana University Purdue University Indianapolis; Anju Sahay, US Department of Veterans Affairs; Anne Sales, University of Michigan and US Department of Veterans Affairs; Sharon Straus, University of Toronto; Carol VanDeusen Lukas, Boston University and US Department of Veterans Affairs; Dawn Velligan, University of Texas Health Science Center San Antonio; Kara Zivin, University of Michigan and US Department of Veterans Affairs; and Cynthia Zubritsky, University of Pennsylvania.
Abbreviations
Expert Recommendations for Implementing Change
US Department of Veterans Affairs
Additional file
A cluster-by-cluster visual tour of the concept map.
Footnotes
Contributor Information
Thomas J. Waltz, Email: twaltz1@emich.edu
Byron J. Powell, Email: byronjpowell@gmail.com
Monica M. Matthieu, Email: mmatthie@slu.edu
Laura J. Damschroder, Email: laura.damschroder@va.gov
Matthew J. Chinman, Email: chinman@rand.org
Jeffrey L. Smith, Email: jeffrey.smith6@va.gov
Enola K. Proctor, Email: ekp@wustl.edu
JoAnn E. Kirchner, Email: joann.kirchner@va.gov
References
Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; 2 of 2 relations here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.
This record holds no relations between claims, controls or mechanisms. A layer is empty when the tagger recorded nothing in it — which is not the same as the paper having nothing to record.
| facet | tag | field | conf. | whose words |
|---|---|---|---|---|
| object_of_study | the relationships among 73 discrete implementation strategies and their rated importance and feasibility | object_of_study | the paper's words | |
| research_question | whether the implementation strategies are conceptually distinct and how they can be grouped | research_question | the paper's words | |
| contribution | initial validation of the ERIC implementation strategies as conceptually distinct | contribution | the paper's words | |
| claim_type | existence | claim_type | the paper's words | |
| phenomenon | poor terminological consistency for core concepts in implementation science | phenomenon | the paper's words | |
| method | expert-consensus also in: expert-consensus | method | the paper's words | |
| limitations | Nearly all panelists were from the United States and most were affiliated with the VA. | limitations[0] | 0.8 | the tagger's words |
| concepts | Concept mapping also in: Concept mapping | concepts[0] | 0.85 | the paper's words |
| concepts | discrete implementation strategies also in: discrete implementation strategies | concepts[1] | 0.8 | the paper's words |
| concepts | Expert Recommendations for Implementing Change | concepts[2] | 0.7 | the paper's words |
| concepts | ERIC also in: ERIC | concepts[3] | 0.85 | the paper's words |
| concepts | Go-zones also in: Go-zones | concepts[4] | 0.7 | the paper's words |
| concepts | implementation science also in: implementation science | concepts[5] | 0.7 | the paper's words |
| concepts | IS also in: IS | concepts[6] | 0.85 | the paper's words |
| senses | strategy similarity - read off the map as distance, from how often panelists put two strategies in one pile | senses[0] | 0.75 | the tagger's words |
| concept_edges | Expert Recommendations for Implementing Change —same-as→ ERIC the relation itself is separately evidenced · strength of the relation: 0.9 · the relation stated: The Expert Recommendations for Implementing Change (ERIC) study aims to address one area of terminological inconsistency: discrete implementation strategies involving one process or action used to support a practice change. | concept_edges[0] | 0.9 | the tagger's words |
| concept_edges | implementation science —same-as→ IS the relation itself is separately evidenced · strength of the relation: 0.9 · the relation stated: Concerns about poor consistency in implementation science (IS) terminology have led researchers to characterize the field as a “Tower of Babel” [ 1 ]. also in: implementation science · IS | concept_edges[1] | 0.9 | the tagger's words |
| content_tags | Feasibility and importance ratings were strongly related, so most strategies fell in the high-high or low-low quadrant. | content_tags.findings[0] | 0.8 | the tagger's words |
| content_tags | categories into which cluster analysis organised the 73 strategies | content_tags.quantities[0] | 0.8 | the tagger's words |
Tags that could not be placed in the text:
What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.