RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review — TEI-tagged view

Russell E. GlasgowpersName, Samantha M. HardenpersName, Bridget GagliopersName, Borsika RabinpersName, Matthew Lee SmithpersName, Gwenndolyn C. PorterpersName, Marcia G. OrypersName, Paul A. EstabrookspersName (2019date). “RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review.” Frontiers in Public Healthtitle·j.

Object of study — the 20-year evolution, application, and prospective use of the RE-AIM planning and evaluation framework

Research question — How has RE-AIM evolved from its original dimensions, how is it now applied, and what directions should guide its future use?

Contribution — a 20-year synthesis that clarifies RE-AIM's levels and dimensions, documents shifts toward adaptationterm·dom, costterm·dom, qualitative and pragmatic use, and proposes contextual, iterative, and integrated future directions

Full text — tagged

The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.

RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Reviewterm·mth Abstract The RE-AIM planning and evaluation framework was conceptualized two decades ago. As one of the most frequently applied implementation frameworksterm·mth, RE-AIM has now been cited in over 2,800 publications. This paper describes the application and evolution of RE-AIM as well as lessons learned from its use. RE-AIM has been applied most often in public health and health behavior change research, but increasingly in more diverse content areas and within clinical, community, and corporate settings. We discuss challenges of using RE-AIM while encouraging a more pragmatic use of key dimensions rather than comprehensive applications of all elements. Current foci of RE-AIM include increasing the emphasis on costterm·dom and adaptationsterm·dom to programs and expanding the use of qualitative methods to understand “how” and “why” results came about. The framework will continue to evolve to focus on contextual and explanatory factors related to RE-AIM outcomes, package RE-AIM for use by non-researchers, and integrate RE-AIM with other pragmatic and reporting frameworks.

Introduction The RE-AIM frameworkterm·mth ( 1 ) was developed to address the issue that the translation of scientific advances into practiceterm·dom, and especially into public health impactterm·dom and policy, have been slow and inequitable ( 2 – 6 ). RE-AIM and other models ( 7 ) have helped balance the traditional focus on internal over external validityterm·dom. Unique features of RE-AIM include an explicit focus on issues, dimensions, and steps in the design, dissemination, and implementationterm·dom process that can either facilitate or impede success in achieving broad and equitable population-based impactterm·dom.

The seminal RE-AIM paper ( 1 ) has been cited over 2,800 times, and the RE-AIM frameworkterm·mth has been applied to study planning or evaluation in over 450 publications ( 7 ). RE-AIM is one of the most frequently used frameworks for planning and evaluation of grant applications at most of the leading U.S. health and medical research agencies ( 8 ) and has been used widely (nationally and internationally) ( 9 , 10 ) and across populations, settings, and health conditions ( 11 – 24 ). Generally, RE-AIM does seem to translate and be useful in the different countries and cultures in which use has been reported. Some international applications include low- and middle-income countries ( 25 ) including Australia ( 26 – 30 ), the Netherlands ( 31 – 34 ), and Brazil ( 35 , 36 ). One interesting application was the use of RE-AIM to help plan and evaluate interventions to reduce the use of coal-fired indoor cook stoves in Africa ( 37 ). In this article, we summarize the history of the RE-AIM frameworkterm·mth, discusses current applications of RE-AIM for research and practice, and outline opportunities for future application.

Historical Perspectives The dimensions of the RE-AIM frameworkterm·mth were originally introduced to encourage scientists to be more transparent and consider internal and external validityterm·dom across pilot, efficacy, effectivenessterm·dom, demonstration, and translational research ( 23 , 38 ). Most peer reviewed publications previously emphasized efficacy, leaving researchers, and practitioners with little information about the generalizability of the intervention context, implementationterm·dom personnel and conditions, and findings. The main goal from its conception was to improve assessment and reporting along the five RE-AIM dimensionsterm·dom, not necessarily intervening to improve all dimensions (see Table 1 ).

The RE-AIM dimensionsterm·dom: definitions, evolution, and examples from the literature.

The RE-AIM dimensionsterm·dom include reachterm·dom (R), effectivenessterm·dom (E), and maintenanceterm·dom (M)–which operate at the individual-levelterm·dom (i.e., those who are intended to benefit), and adoptionterm·dom (A), implementationterm·dom (I), and maintenanceterm·dom (M), which focus on the staff and setting levelsterm·dom. Setting-level RE-AIM factors are often multi-level and address context and external validityterm·dom issues important to population impactterm·dom. For example, settings may include clinics, schools, or worksitesterm nested within communities or larger systemsterm, and within these settings are nested clinicians, teachers, or human resources staffterm responsible for implementationterm·dom.

All RE-AIM dimensionsterm·dom are complex, but implementationterm·dom currently has the most indices. It focuses on fidelity to an interventionterm·dom: the extent to which the program is implemented consistentlyterm·dom across different settings, staff, and patients. It also includes adaptationsterm·dom made ( 58 ) and coststerm·dom from multiple stakeholder perspectives ( 59 ). Maintenanceterm·dom has indices at the individual- (long-term effectivenessterm·dom) and setting-level (sustainability after original research funded is completedterm·dom).

The framework's operational components have been increasingly applied over the years. For example, in the past, studies reported participant characteristics that differed between study conditions or between those retained and those lost to follow-up. However, studies using RE-AIM compared the representativenessterm·dom of individuals who enrolled in a study to the characteristics of the intended population. These comparisons used in RE-AIM studies increased understanding about access, awareness, appropriateness, and likely generalizability of recruitment strategiesterm·mth and intervention approaches.

In the past, clinical effectivenessterm·dom research focused relatively narrowly on physiologic outcomes. RE-AIM expanded this focus to multiple factors that impact public health . This approach to assessing broader impacts aided in understanding comprehensive effects of a program on quality of life, including unintended consequences (e.g., increasing health inequity or the social stigma of labeling someone with a chronic condition).

There have been several literature reviewsterm·mth on use of RE-AIM ( 11 , 42 , 52 , 60 – 62 ). The most comprehensive reviewsterm·mth have spanned the literature from 2000 to 2012 or 2015 ( 10 , 42 , 52 ). Notably, these reviewsterm·mth of different content areas reached similar conclusions: that adoptionterm·dom and maintenanceterm·dom, as well as representativenessterm·dom across individual- and organizational-levels, were reported far less frequently. They identified frequent issues with confusing different dimensions, in particular reachterm·dom (at the individual-levelterm·dom) and adoptionterm·dom (at the setting-level). These observations are not limited to the United States alone.

To enhance development and application of the framework, several scientists contributed to a RE-AIM research consortium funded by the Robert Wood Johnson Foundation ( 63 – 66 ). This work led to the development of a website, www.re-aim.org , in 2004 ( 64 ). The website serves as a repository of various resources and tools including self-quizzes, checklists, figures, tables, measures, tips for using RE-AIM, and increasingly, other social media tools. These are available to facilitate the operationalization and application of RE-AIM across diverse interventions, settings, and populations. To enhance a dialogue within the broader research community, monthly webinars are held about RE-AIM related issues; archived recordings are available on the website ( www.re-aim.org ).

From Past to Present Below, we summarize five general areas currently being examined using RE-AIM ( Table 1 ). The first is to understand and maximize the potential of RE-AIM to assess adaptationsterm·dom prior to, during, and after program implementationterm·dom ( 54 , 55 , 67 ). Adaptationsterm·dom naturally occur during the implementationterm·dom of programs ( 68 ). Mittman et al. ( 69 ) suggest that instead of ignoring or suppressing this phenomenon, we should find ways to document and assess these changes. Recent Patient-Centered Outcomes Research Institute (PCORI) Methodology Standards ( 70 ) suggest that adaptationsterm·dom should be systematically documented. RE-AIM has great potential to provide guidance about documenting adaptationsterm·dom. It can also provide guidance about how to evaluate the impact of these adaptationsterm·dom, as well as their purpose ( 58 ). RE-AIM has been used to expand the widely known Stirman framework ( 71 ) on adaptationsterm·dom with additional components to address “who, what, why, where, and when” questions ( 67 ). RE-AIM considers adaptationsterm·dom in a longitudinal, multi-method, and multi-level manner and includes data collection at multiple time points and from multiple stakeholders, using multiple data collection approaches ( 54 , 55 , 67 ).

Second, there has been a recent focus on more qualitative RE-AIM assessmentsterm·mth. Most evaluations and uses of the framework have emphasized descriptive or quantitative data, often focusing on key aspects such as the percentage of potentially eligible persons or settings that participate. A qualitative focus as presented by Holtrop et al. ( 72 ) can enhance understanding about what happenedterm·dom as well as the “how” and the “why.”

Third, we recognized the need for more pragmatic uses of the frameworkterm·mth rather than trying to comprehensively assess all RE-AIM dimensionsterm·dom in all applications, especially when not having many evaluation resources ( 67 , 73 ). All studies or evaluations, and particularly those without large evaluation budgets, do not need to assess all components of RE-AIM. Rather, they should address those components most valued and appropriate for their particular question, setting, stakeholders, and stage of research. An a priori decision should be made, however, to select the dimensions on which to focus for evaluation and on which to use for planning and improvement (i.e., beyond the evaluation scope). In some cases, the decision to capture all five dimensions is made a priori to understand individual impacts, contextual implications, and feasibility of ongoing data collection. This is demonstrated in two recent applications of RE-AIM before, during, and after program implementationterm·dom—and on limited funds ( 74 , 75 ). Both applications highlighted the need for stakeholder buy-interm·dom ( 54 ) and operationalization ( 67 ) of each dimension that holds value for these stakeholders.

Fourth, assessment of coststerm·dom, from the perspective of multiple stakeholders and across the various RE-AIM dimensionsterm·dom, is another area of emphasis ( 5 ). Building upon earlier work by Ritzwoller et al. ( 76 ), recent RE-AIM costterm·dom assessments have focused on the multilevel nature of implementationterm·dom, different stakeholder perspectives, and costterm·dom estimates for replicating a program or policy in different settings. Rhodes et al. ( 59 ) have provided templates to assess coststerm·dom at the patient-, staff-, clinic-, and organizational-levels. Coststerm·dom to deliver programs are associated with activities to address and enhance each RE-AIM dimensionterm·dom ( Figure 1 ). In the future, we anticipate more consistent reporting of coststerm·dom and burden and more frequent comparative effectivenessterm·dom research about cost-effective methods to enhance value and various RE-AIM dimensionsterm·dom.

Revised, enhanced RE-AIM/PRISMterm·mth 2019 model.

Finally, GlasgowpersName et al. ( 77 ) have recently advocated for an extension of RE-AIM concepts and dimensions, termed an Expanded CONSORT Figureterm·mth to enhance transparent reporting, and potentially, replication. The goal is to expand the CONSORT reporting criteria required for randomized studies ( 78 ) to (a) include factors related to setting and staff level participation and representativenessterm·dom, which begin before individual participants are recruited, and (b) extend the temporal focus beyond the end of a study. The expanded CONSORT figureterm·mth and a related downloadable template summarize issues of exclusion and inclusion criteria for settings (e.g., communities or healthcare networks) and delivery staff, [e.g., evaluating the percent and characteristics of settings and staff that participate or do not (adoptionterm·dom)], reasons for participation or non-participation, and intervention sustainability after project support ends ( 79 ).

Based on these observations, we have developed a new RE-AIM figureterm·mth to highlight the various changes to the model, as well as new emphases, including explicit inclusion of coststerm·dom and adaptationsterm·dom, as shown in Figure 1 . The figure also emphasizes key multi-level contextual factorsterm·dom (both the internal and external context) that influence RE-AIM outcomes as discussed below. Two crosscutting issues are: (a) that it is critical that there is alignmentterm·dom across setting and context, the intervention and implementationterm·dom strategies; and (b) it is important to include qualitative assessments to determine how and why various RE-AIM outcomes are produced.

What Will the Future Bring?

With the historical context and current application of RE-AIM in mind, we outline five future directions for researchers and practitioners interested in using RE-AIM.

First, there have been recent calls to more explicitly describe strategies and context ( 80 , 81 ) as well as test mediating relationships between implementationterm·dom strategies and implementationterm·dom outcomes ( 82 ). We also see this as an emerging area for RE-AIM. The most well-articulated attempt to do this so far is the Practical, Robust, Implementation, and Sustainability Model (PRISM)term·mth ( 83 , 84 ) that focuses on specific contextual factors from external macro-level factors such as policies, guidelines, and incentives, to more local organizational-level factors. It focuses on the fit between the characteristics of an intervention (i.e., Rogers' constructs of relative advantage, complexity, compatibility, observability, trialability, and costterm·dom) ( 85 ) and the particular intervention and implementationterm·dom system. A somewhat unique factor of PRISMterm·mth is its focus on enhancing setting-level maintenanceterm·dom characteristics by addressing the “implementation and sustainability infrastructureterm·dom”—including job requirementsterm·dom, ongoing audit and feedbackterm·mth, and institutionalization of intervention activitiesterm·dom ( 84 , 86 ).

Second, mixed-methodsterm·mth should be used across framework components to identify explanatory processesterm·dom across RE-AIM dimensionsterm·dom. To date, quantitative measures alone have been insufficient to strongly predict dissemination (reachterm·dom and adoptionterm·dom), implementationterm·dom, and maintenanceterm·dom outcomes. Using mixed-methodsterm·mth approaches can help identify factors that are causally related to different RE-AIM outcomes in different situations ( 72 ). Qualitative information integrated with newer predictive modeling approaches should provide more detailed guidance about actions that can be taken to enhance outcomes by addressing empirically-derived causal relationships.

Third, we encourage more iterative applications of RE-AIM and use of the framework during the implementationterm·dom period, not just for initial planning and summative evaluation. Rapid, iterative use and analysisterm·mth of brief practical measures of RE-AIM factors can inform adaptationsterm·dom ( 55 , 66 , 71 ). In brief, RE-AIM can be used as part of a participatory approach (EstabrookspersName et al., under reviewterm·mth), to determine which dimensions should be assessed, described, or targeted for intervention. For example, a recruitment strategy may need to be adapted over the life-course of an intervention (to improve reachterm·dom) or a new training strategy may be employed (to improve adoptionterm·dom and implementationterm·dom). While often examined and interpreted independently, these adaptationsterm·dom can work together to be empirically robust and practically meaningful.

Another issue to be addressed is use of RE-AIM by non-researchers and groups such as state health programs or program evaluators without substantial funds (grants/contracts). Using RE-AIM in low-resource and real-world settings can be challenging but successful ( 2 ). Preliminary findings assessing such use are that RE-AIM is used widely, and seems to be relatively intuitive, but there are challenges implementing it at a detailed level and assessing all components. The development of user-friendly tools and aidsterm·mth using human centered designterm·mth, as well as more examples of the application of RE-AIM for such users is an important future direction.

Finally, we think there is great opportunity for RE-AIM to be used in combination with other approaches such as the Pragmatic Explanatory Continuum Indicator Summary (PRECIS) model ( 87 , 88 ), where RE-AIM factors can be combined with the PRECIS-2 dimensionsterm·mth to determine how pragmatic a study is and how generalizable it is likely to be. Such use is illustrated in a recent systematic reviewterm·mth by Luoma et al. ( 89 ), who demonstrated how reviewsterm·mth can simultaneously summarize effectivenessterm·dom (using Cochrane-type criteria) and pragmatism (using a combination of PRECIS-2 and RE-AIM factors). RE-AIM and its Expanded CONSORT extension could also be integrated with the Standards for Reporting Implementation Studies (StaRI)term·mth ( 90 ) or other dissemination and implementationterm·dom (D&I) research reporting criteria.

Conclusion RE-AIM has been applied in research and practice for 20 years. Although its original components have remained, much has been modified and evolved to address emerging issues such as adaptationterm·dom and dissemination coststerm·dom. We expect that RE-AIM will continue to evolve to better address and enhance its original purpose—to increase the prevalence of relevant research that can be applied broadly across a wide variety of populations and settings to achieve a large, equitable, and replicable public health impactterm·dom. We invite researchers and practitioners to contribute to the expanded use of RE-AIM before, during, and after intervention delivery.

Author Contributions All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

Conflict of Interest Statement The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.