Thomas J. WaltzpersName, Byron J. PowellpersName, Monica M. MatthieupersName, Laura J. DamschroderpersName, Matthew J. ChinmanpersName, Jeffrey L. SmithpersName, Enola K. ProctorpersName, JoAnn E. KirchnerpersName (2015date). “Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study..” Implementation Sciencetitle·j.
Object of study — the relationships among 73 discrete implementation strategiesterm·dom and their rated importanceterm·key and feasibilityterm·key
Research question — whether the implementation strategiesterm·key are conceptually distinct and how they can be grouped
Contribution — initial validation of the ERICterm·dom implementation strategiesterm·key as conceptually distinct
phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | poor terminological consistency for core concepts in implementation science — “Poor terminological consistency for core concepts in implementation science has been widely noted as an obstacle to effective meta-analyses.” | |
| claim-type | existence — “All but two strategies were sorted as being conceptually distinct.” | |
| method — evidence | — “A purposive sampling procedure was used to recruit an expert panel of implementation science and clinical experts ( N = 35) to participate in concept mapping and rating tasks [ 9 , 10 ].” | |
| object of study — evidence | — “The aim of the ERIC project’s second stage, presented here, was to obtain preliminary validation of the compilation of 73 implementation strategies by studying the relationships between the strategies and obtaining relative importance and feasibility ratings for each strategy.” | |
| research question — evidence | — “The study of the relationships among the strategies supports the evaluation of whether the strategies are conceptually distinct from one another as well as how the strategies can be organized into conceptually relevant groupings.” | |
| contribution — evidence | — “This study provides initial validation of the implementation strategies within the ERIC compilation as being conceptually distinct.” | |
| sense | strategy similarity - read off the map as distance, from how often panelists put two strategies in one pile — “Spatial distances reflect how frequently the strategies were sorted together as similar.” | senses[0] |
| relation | Expert Recommendations for Implementing Change —same-as→ ERIC — The Expert Recommendations for Implementing Change (ERIC) study aims to address one area of terminological inconsistency: discrete implementation strategies involving one process or action used to support a practice change. — “The Expert Recommendations for Implementing Change (ERIC) study aims to address one area of terminological inconsistency: discrete implementation strategies involving one process or action used to support a practice change.” | concept_edges[0] |
| weight | Expert Recommendations for Implementing Change —same-as→ ERIC — 0.9 — “The Expert Recommendations for Implementing Change (ERIC) study aims to address one area of terminological inconsistency: discrete implementation strategies involving one process or action used to support a practice change.” | concept_edges[0] |
| relation | implementation science —same-as→ IS — Concerns about poor consistency in implementation science (IS) terminology have led researchers to characterize the field as a “Tower of Babel” [ 1 ]. — “Concerns about poor consistency in implementation science (IS) terminology have led researchers to characterize the field as a “Tower of Babel” [ 1 ].” | concept_edges[1] |
| weight | implementation science —same-as→ IS — 0.9 — “Concerns about poor consistency in implementation science (IS) terminology have led researchers to characterize the field as a “Tower of Babel” [ 1 ].” | concept_edges[1] |
| replication | original | |
| limitation | Nearly all panelists were from the United States and most were affiliated with the VA. — “Thus, all but one of the 35 participants were from the United States, and 69 % had some affiliation with the VA.” | limitations[0] |
finding · quantity · variable · definition · key-term — grounded in the full text| finding | Feasibility and importance ratings were strongly related, so most strategies fell in the high-high or low-low quadrant. — “There was a strong relationship ( r = 0.7) between the feasibility and importance ratings, meaning that most strategies fell within either quadrant I (high importance and feasibility) or III (low importance/feasibility).” | content_tags.findings[0] |
| quantity | 9 — — “Hierarchical cluster analysis supported organizing the 73 strategies into 9 categories.” | content_tags.quantities[0] |
| key-terms | implementation strategies, concept mapping, implementation science, feasibility, importance, cluster analysis, multidimensional scaling, Delphi, expert panel, Veterans Affairs |
| concept | Concept mapping — a mixed-method procedure for engaging stakeholder groups in a structured conceptualization process — “Concept mapping is a mixed-method procedure for engaging stakeholder groups in a structured conceptualization process [ 11 ].” | concepts[0] |
| concept | discrete implementation strategies — one process or action used to support a practice change — “The Expert Recommendations for Implementing Change (ERIC) study aims to address one area of terminological inconsistency: discrete implementation strategies involving one process or action used to support a practice change.” | concepts[1] |
| concept | Expert Recommendations for Implementing Change — not defined in this paper's own words; the multi-stage project refining a compilation of implementation strategies — “The Expert Recommendations for Implementing Change (ERIC) study aims to address one area of terminological inconsistency: discrete implementation strategies involving one process or action used to support a practice change.” | concepts[2] |
| concept | ERIC — Expert Recommendations for Implementing Change — “The Expert Recommendations for Implementing Change (ERIC) study aims to address one area of terminological inconsistency: discrete implementation strategies involving one process or action used to support a practice change.” | concepts[3] |
| concept | Go-zones — not defined in this paper's own words; quadrants of the importance-by-feasibility plot split at each mean — “The resulting scatterplot was divided into four quadrants or “Go-zones” (e.g., I, II, III, IV) using the mean of each dimension.” | concepts[4] |
| concept | implementation science — not defined in this paper's own words; the study of methods for bringing research findings into routine practice — “Concerns about poor consistency in implementation science (IS) terminology have led researchers to characterize the field as a “Tower of Babel” [ 1 ].” | concepts[5] |
| concept | IS — implementation science — “Concerns about poor consistency in implementation science (IS) terminology have led researchers to characterize the field as a “Tower of Babel” [ 1 ].” | concepts[6] |
Full text — tagged
Use of concept mappingterm to characterize relationships among implementation strategiesterm·key and assess their feasibilityterm·key and importanceterm·key: results from the Expert Recommendations for Implementing Changeterm·dom (ERICterm·dom) study.
Abstract Poor terminological consistency for core concepts in implementation scienceterm·dom has been widely noted as an obstacle to effective meta-analyses. This inconsistency isterm·dom also a barrier for those seeking guidance from the research literature when developing and planning implementation initiatives. The Expert Recommendations for Implementing Changeterm·dom (ERICterm·dom) study aims to address one area of terminological inconsistency: discrete implementation strategiesterm·dom involving one process or action used to support a practice change. The present report isterm·dom on the second stage of the ERICterm·dom project that focuses on providing initial validation of the compilation of 73 implementation strategiesterm·key that were identified in the first phase.
Purposive sampling was used to recruit a panel of experts in implementation scienceterm·dom and clinical practice ( N = 35). These key stakeholders used concept mappingterm sorting and rating activities to place the 73 implementation strategiesterm·key into similar groups and to rate each strategy’s relative importanceterm·key and feasibilityterm·key. Multidimensional scalingterm·key analysis provided a quantitative representation of the relationships among the strategies, all but one of which were found to be conceptually distinct from the others. Hierarchical cluster analysisterm·key supported organizing the 73 strategies into 9 categories. The ratings data reflect those strategies identified as the most important and feasible.
This study provides initial validation of the implementation strategiesterm·key within the ERICterm·dom compilation as being conceptually distinct. The categorization and strategy ratings of importanceterm·key and feasibilityterm·key may facilitate the search for, and selection of, strategies that are best suited for implementation efforts in a particular setting.
The online version of this article (doi:10.1186/s13012-015-0295-0) contains supplementary material, which isterm·dom available to authorized users.
Keywords: Concept mappingterm, Implementation research, Implementation strategiesterm·key, Mental health, US Department of Veterans Affairsterm·key Background Concerns about poor consistency in implementation scienceterm·dom (ISterm·dom) terminology have led researchers to characterize the field as a “Tower of Babel” [ 1 ]. Inconsistent terminology complicates literature searches, and researchers have found that search strategy yield and precision indices for implementation and quality improvement studies are moderate at best [ 2 , 3 ]. This limits meta-analytic and replication efforts aimed at rigorously evaluating the effectiveness of implementation strategiesterm·key and the value of existing literature for those enacting implementation initiatives. The science and practice of implementation would be greatly facilitated by a parsimonious nomenclature of conceptually distinct implementation strategiesterm·key [ 2 , 4 – 7 ].
Recently, PowellpersName et al. [ 8 ] reviewed the health and mental health literature (including 41 compilations and reviews) and proposed a compilation of 68 discrete implementation strategiesterm·dom involving one action or process. This compilation served as the starting point for a subsequent multi-stage project called Expert Recommendations for Implementing Changeterm·dom (ERICterm·dom) [ 9 ]. The ERICterm·dom project’s first stage involved expert panelists ( N = 71) using a modified Delphiterm·key process to revise the compilation, which resulted in an updated compilation of 73 discrete implementation strategiesterm·dom [ 10 ].
The aim of the ERICterm·dom project’s second stage, presented here, was to obtain preliminary validation of the compilation of 73 implementation strategiesterm·key by studying the relationships between the strategies and obtaining relative importanceterm·key and feasibilityterm·key ratings for each strategy. The study of the relationships among the strategies supports the evaluation of whether the strategies are conceptually distinct from one another as well as how the strategies can be organized into conceptually relevant groupings. The former can also serve the practical purpose of making it easier for stakeholders to consider the range of implementation strategiesterm·key by thematic cluster. The importanceterm·key and feasibilityterm·key ratings for the strategies provide insight into the perceived applicability of the strategies. It isterm·dom of general interest which strategies have relatively high and low ratings by experts.
Method A purposive sampling procedure was used to recruit an expert panelterm·key of implementation scienceterm·dom and clinical experts ( N = 35) to participate in concept mappingterm and rating tasks [ 9 , 10 ]. A detailed description of procedures has been published [ 9 ], and a summary isterm·dom provided here. Concept mappingterm isterm·dom a mixed-method procedure for engaging stakeholder groups in a structured conceptualization process [ 11 ]. This process supports visually representing the relationships among a set of related concepts and empirically clustering them into conceptually distinct categories and rating them on multiple dimensions.
The Concept Systems Global MAX™ [ 12 ] web platform was used for the panel’s sorting and rating tasks and data analysis. A more detailed introduction to concept mappingterm can be found in Trochim and Kane [ 13 ]. For the sorting task, participants were asked to sort virtual cards for each of the 73 strategies, accompanied by their definitions, into piles as they deemed appropriate. Participants were asked to rate each strategy for importanceterm·key and feasibilityterm·key ranging from 1 (relatively unimportant/not at all feasible) to 5 (extremely important/extremely feasible). These global ratings were prefaced by the following instructions: “Please select a number from 1 to 5 for each discrete implementation strategy to provide a rating in terms of how important (feasible) you think it isterm·dom. Keep in mind that we are looking for relative importanceterm·key (feasibilityterm·key), use all the values in the rating scale to make distinctions.” Participants were able to select which set of activities they wanted to do first and were also able to work on the sorting and rating activities over multiple online sessions, at their convenience, before submitting their responses.
Multidimensional scalingterm·key and hierarchical cluster analyses were conducted to produce visual representations of the relationships among the strategies. Descriptive statistics for the importanceterm·key and feasibilityterm·key ratings were calculated. Each strategy’s importanceterm·key and feasibilityterm·key score was plotted on a graph. The resulting scatterplot was divided into four quadrants or “Go-zonesterm” (e.g., I, II, III, IV) using the mean of each dimension. For example, quadrant I contains strategies that have values above the means for both dimensions. The Go-zoneterm quadrants column in Table 1 reflects the combined relative importanceterm·key and feasibilityterm·key for each strategy.
A summary of the 73 implementation strategiesterm·key, organized by cluster with mean importanceterm·key and feasibilityterm·key ratings Results Experts who participated in the concept mappingterm and rating tasks and were affiliated with academic or healthcare institutions in the United States ( n = 34) or in Canada ( n = 1). Thirty-two of the 35 experts provided valid sorts (>75 % of strategies sorted), and 30 provided importanceterm·key and feasibilityterm·key ratings for all strategies. Sixty-three percent of participants had exclusive expertise in ISterm·dom, 29 % were experts in both ISterm·dom and clinical practice, and 8 % indicated clinical practice expertise only. Sixty-nine percent of participants had some affiliation with the US Department of Veterans Affairsterm·key (VA), most of whom also held academic appointments in social science or health-related schools or departments.
Figure 1 presents a point map that visually represents the relationships among the 73 implementation strategiesterm·key, with each point on the map representing a strategy. The strategies are numbered to aid in cross-referencing the spatial relationships of the points on the map with their labels enumerated in Table 1 . All but two strategies were sorted as being conceptually distinct. Strategies #66 ( Use capitated payments ) and #70 ( Use other payment schemes ) were always sorted together. Two other strategies were proximal to one another though they were sorted together by only 4 of 32 panelists (#35 Identify and prepare champions and #57 Recruit, designate, and train for leadership ), indicating that they are more similar in how they relate with other strategies on the map, than they are directly similar to one another.
Point and cluster map of all 73 strategies identified in the ERICterm·dom process. The map reflects the product of an expert panelterm·key (valid response n = 32) sorting 73 discrete implementation strategiesterm·dom into groupings by similarity with each strategy being depicted by a yellow dot and accompanied by a number supporting cross-referencing to the strategies enumerated in Table 1 . Spatial distances reflect how frequently the strategies were sorted together as similar. In general, the closer two points are together, the more frequently those strategies were sorted together. Strategies distal from one another were infrequently, if at all, sorted together. These spatial relationships are relative to the sorting data obtained in this study, and distances do not reflect an absolute relationship (i.e., a 5-mm distance in the present map does not reflect the same relationship as a 5-mm distance on a map from a different data set). The legend provides the label for each of the nine clusters of strategies. Dotted lines within the Develop stakeholder interrelationships cluster indicate how two separate clusters were merged into one large cluster due to conceptual similarity among their items. Dotted lines extending between other clusters archive the reassignment of strategies from their original cluster to a neighboring cluster to which there was a better conceptual fit (i.e., strategies #48, #58, and #62) The final clusters were developed over 3 weeks of deliberations by the ERICterm·dom investigative team. A 13 cluster starting point was selected because it isterm·dom one standard deviation above the mean number of clusters typically obtained in concept mappingterm [ 14 ]. In this study, 69 % of respondents sorted statements into 13 or fewer piles. We sequentially reviewed cluster merges and achieved consensus to merge clusters down to nine conceptually distinct clusters. For example, two clusters shown in pale green at the center bottom in Fig. 1 (separated by dashed lines) were merged to form a single cluster labeled Develop stakeholder interrelationships , as the original clusters were judged as not sufficiently conceptually distinct.
When the team reviewed the clusters for conceptual clarity, three proposals came forward to move individual strategies to neighboring clusters. First, #62 ( Start a dissemination organization ) was moved from the Engage consumers cluster to the Change infrastructure cluster, as it was judged more similar to infrastructure support for a practice change than engaging consumers. Second, #48 ( Organize clinician implementation team meetings ) was moved to the Develop stakeholder interrelationships cluster from Adapt and tailor to the context , as the former has greater interpersonal focus than the latter. And finally, #58 ( Remind clinicians ) was moved to the Support clinicians cluster from Provide interactive assistance because it isterm·dom more administrative than interactive in focus. Unanimous consensus was reached for the final cluster arrangements. Additional file 1 provides a cluster-by-cluster visual tour of the concept map.
A multi-step process was used to determine labels for the final clusters. The list began with labels provided by expert panelterm·key members for their clusters that were most similar to the final cluster solutions. This list was supplemented with highly descriptive labels identified from the investigative team’s meeting minutes from cluster solution deliberations. Proposed criteria for developing cluster labels (Table 2 ) were introduced for team comment by one of the authors (LJD) along with suggested label revisions. These criteria were helpful in structuring iterative discussion among team members, the result of which was voted upon by the team and unanimously adopted.
Guidelines for cluster labels Table 1 presents a summary of the 73 implementation strategiesterm·key, organized by cluster with mean importanceterm·key and feasibilityterm·key ratings. There was a strong relationship ( r = 0.7) between the feasibilityterm·key and importanceterm·key ratings, meaning that most strategies fell within either quadrant I (high importanceterm·key and feasibilityterm·key) or III (low importanceterm·key/feasibilityterm·key). However, there were still a number of strategies that were viewed as important but not as feasible (12 %, e.g., Access new funding ), or feasible but less important (15 %, e.g., Remind clinicians ). Clusters of strategies that are more immediate and concrete and are potentially more in the control of those tasked with supporting change (e.g., Use evaluative and iterative strategies , Train and educate stakeholders ) tended to have higher importanceterm·key and feasibilityterm·key ratings. Clusters that are more strategic, but also potentially involve changing well-established systems (e.g., Change infrastructure , Utilize financial strategies ), tended to have lower ratings. Figure 2 presents a graphic of the Go-zoneterm data.
Go-zoneterm plot for all 73 strategies based on expert ratings. Note . The range of the x and y axes reflect the mean values obtained for all 73 of the discrete implementation strategiesterm·dom for each of the rating scales. The plot isterm·dom divided into quadrants on the basis of the overall mean values for each of the rating scales. Quadrant labels are depicted with roman numerals next to the plot. Strategies in quadrant I fall above the mean for both the importanceterm·key and the feasibilityterm·key ratings. Thus, these strategies are those where there was the highest consensus regarding their relative high importanceterm·key and feasibilityterm·key. Conversely, quadrant III reflects the strategies where there was consensus regarding their relative low importanceterm·key and feasibilityterm·key. Quadrants II and IV reflect strategies that were relatively high in feasibilityterm·key or importanceterm·key, respectively, but low on the other rating scale Discussion Results from this study provide initial validation for viewing the 73 implementation strategiesterm·key as conceptually distinct. Cluster analyses of the concept mappingterm data support grouping strategies into nine clusters which have practical heuristic value for those looking to the ERICterm·dom compilation of implementation strategiesterm·key for guidance. The importanceterm·key and feasibilityterm·key ratings for the strategies supported the formation of Go-zoneterm quadrants that can be used to help decision makers prioritize which strategies to use when planning an implementation initiative.
While the concept mappingterm strategy used in this study represents a strong methodological approach to evaluating whether the 73 implementation strategiesterm·key are conceptually distinct and organizing them by theme and potential applicability (i.e., Go-zoneterm analysis), there are notable limitations. Recruitment had been restricted to the time zones within the continental United States to minimize scheduling conflicts for elements of the ERICterm·dom project that required real time interactions among participants. Thus, all but one of the 35 participants were from the United States, and 69 % had some affiliation with the VA. While concept maps with 30 or more participants are considered to be highly reliable [ 14 ], if stakeholders from outside the United States had practice contexts that alter the perceptions of these strategies interrelationships, or the ratings of their perceived importanceterm·key and feasibilityterm·key, different results may be obtained.
Acknowledgements This project isterm·dom funded through the US Department of Veterans Affairsterm·key Veterans Health Administration Mental Health Quality Enhancement Research Initiative (QLP 55–025). The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairsterm·key or the US government. Additionally, TJW received support from the VA Office of Academic Affiliations Advanced Fellowships Program in Health Services Research and Development at the Center for Mental Healthcare & Outcomes Research, and BJP received support from the National Institute of Mental Health (F31 MH098478), the Doris Duke Charitable Foundation (Fellowship for the Promotion of Child Well-Being), and the Fahs-Beck Fund for Research and Experimentation. MJC received support from the VISN 4 Mental Illness Research, Education, and Clinical Center. MJC received support from the VISN 4 Mental Illness Research, Education, and Clinical Center at the VA Pittsburgh Healthcare System.
We would like to acknowledge the contributions of each member of the expert panelterm·key: Greg Aarons, University of California, San Diego; Mark Bauer, Harvard University and US Department of Veterans Affairsterm·key; Laurel Copeland, Scott and White Healthcare and US Department of Veterans Affairsterm·key; Torrey Creed, University of Pennsylvania; Laura DamschroderpersName, University of Michigan and US Department of Veterans Affairsterm·key; Rani Elwy, Boston University and US Department of Veterans Affairsterm·key; Bradford Felker, University of Washington and US Department of Veterans Affairsterm·key; Hildi Hagedorn, University of Minnesota and US Department of Veterans Affairsterm·key; Alison Hamilton, University of California, Los Angeles and US Department of Veterans Affairsterm·key; Kathy Henderson, US Department of Veterans Affairsterm·key; Timothy Hogan, University of Massachusetts and US Department of Veterans Affairsterm·key; Kelly Koerner, Evidence-Based Practice Institute; Sarah Krein, University of Michigan and US Department of Veterans Affairsterm·key; Julie Kreyenbuhl, University of Maryland and US Department of Veterans Affairsterm·key; Sara Landes, University of Washington and US Department of Veterans Affairsterm·key; Cara Lewis, Indiana University-Bloomington; Julie Lowery, University of Michigan and US Department of Veterans Affairsterm·key; Aaron Lyon, University of Washington; Natalie Maples, University of Texas Health Science Center San Antonio; Alan McGuire, Indiana University-Purdue University Indianapolis and US Department of Veterans Affairsterm·key; Amanda Midboe, US Department of Veterans Affairsterm·key; Laura Murray, Johns Hopkins University; Princess Osei-Bonsu, US Department of Veterans Affairsterm·key; Richard Owen, University of Arkansas for Medical Sciences and US Department of Veterans Affairsterm·key; Louise Parker, University of Massachusetts Boston; Enola ProctorpersName, Washington University in St. Louis; Mona Ritchie, US Department of Veterans Affairsterm·key; Angie Rollins, US Department of Veterans Affairsterm·key and Indiana University Purdue University Indianapolis; Anju Sahay, US Department of Veterans Affairsterm·key; Anne Sales, University of Michigan and US Department of Veterans Affairsterm·key; Sharon Straus, University of Toronto; Carol VanDeusen Lukas, Boston University and US Department of Veterans Affairsterm·key; Dawn Velligan, University of Texas Health Science Center San Antonio; Kara Zivin, University of Michigan and US Department of Veterans Affairsterm·key; and Cynthia Zubritsky, University of Pennsylvania.
Abbreviations Expert Recommendations for Implementing Changeterm·dom US Department of Veterans Affairsterm·key Additional file A cluster-by-cluster visual tour of the concept map.
Footnotes Contributor Information Thomas J. WaltzpersName, Email: twaltz1@emich.edu Byron J. PowellpersName, Email: byronjpowell@gmail.com Monica M. MatthieupersName, Email: mmatthie@slu.edu Laura J. DamschroderpersName, Email: laura.damschroderpersName@va.gov Matthew J. ChinmanpersName, Email: chinmanpersName@rand.org Jeffrey L. SmithpersName, Email: jeffrey.smith6@va.gov Enola K. ProctorpersName, Email: ekp@wustl.edu JoAnn E. KirchnerpersName, Email: joann.kirchnerpersName@va.gov References