Kathryn SkivingtonpersName, Lynsay MatthewspersName, Sharon Anne SimpsonpersName, Peter CraigpersName, Janis BairdpersName, Jane M BlazebypersName, Kathleen Anne BoydpersName, Neil CraigpersName, David P FrenchpersName, Emma McIntoshpersName, Mark PetticrewpersName, Jo Rycroft-MalonepersName, Martin WhitepersName, Laurence MoorepersName (2021date). “A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance..” BMJtitle·j.
Object of study — the updated MRC-NIHR framework for developing and evaluating complex interventionsterm·key
Research question — what complex interventionterm·key research should ask beyond whether an intervention works
Contribution — a pluralist framework of four research perspectives, four phases and six core elements
phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | evidence from tightly controlled efficacy trials often fails to carry over into everyday practice — “Efficacy trials of relatively uncomplicated interventions in tightly controlled conditions, where research questions are answered with great certainty, will always be important, but translation of the evidence into the diverse settings of everyday practice is often highly problematic.” | |
| claim-type | method — “The framework divides complex intervention research into four phases: development or identification of the intervention, feasibility, evaluation, and implementation ( fig 1 ).” | |
| method — evidence | — “The updated Framework for Developing and Evaluating Complex Interventions is the culmination of a process that included four stages:” | |
| object of study — evidence | — “This article introduces the framework and summarises the main messages for producers and users of evidence.” | |
| research question — evidence | — “Complex intervention research goes beyond asking whether an intervention works in the sense of achieving its intended outcome—to asking a broader range of questions” | |
| contribution — evidence | — “The new framework therefore takes a pluralistic approach and identifies four perspectives that can be used to guide the design and conduct of complex intervention research: efficacy, effectiveness, theory based, and systems ( table 2 ).” | |
| mechanism | interventions reshape their context Interventions provoke responses from others and shift norms, so their effects change over time. — “As the examples in table 1 show, interventions can modify the contexts in which they are implemented, by eliciting responses from other agents, or by changing behavioural norms or exposure to risk, so that their effects will also vary over time.” | mechanisms[0] |
| mechanism | peer contact in cohesive communities A peer-led smoking intervention worked best where peer supporters met their peers often. — “For example, in a cluster randomised trial of ASSIST (a peer led, smoking prevention intervention), researchers found that the intervention worked particularly well in cohesive communities that were served by one secondary school where peer supporters were in regular contact with their peers” | mechanisms[1] |
| extends | MRC framework (2000) and guidance (2006) on complex interventions — “The previous framework and guidance were based on a paradigm in which the salient question was to identify whether an intervention was effective.” | claim_relations[0] |
| builds-on | INDEX study guidance on intervention development — “Guidance on intervention development has recently been developed through the INDEX study 4 ; although here we highlight that complex intervention research does not always begin with new or researcher led interventions.” | claim_relations[1] |
| sense | complexity (from the intervention) - many parts, targeted behaviours or levels, or allowed flexibility — “An intervention might be considered complex because of properties of the intervention itself, such as the number of components involved; the range of behaviours targeted” | senses[0] |
| sense | complexity (from context) - arising from how an intervention interacts with its circumstances — “Complexity might also arise through interactions between the intervention and its context” | senses[1] |
| sense | complex intervention (systems view) - an event within a complex adaptive system — “With this framing, complex interventions can helpfully be considered as events in systems.” | senses[2] |
| sense | evaluation (new framework) - more than whether it works: wider impact, theory, context and system change — “The new framework defines evaluation as going beyond asking whether an intervention works (in the sense of achieving its intended outcome)” | senses[3] |
| relation | SMART trials —same-as→ sequential multiple assignment randomised trials — Extensions to standard designs of randomised controlled trials (including adaptive designs, SMART trials (sequential multiple assignment randomised trials), n-of-1 trials, and hybrid effectiveness-implementation designs) are important areas of methods development to improve the efficiency of complex intervention research. — “Extensions to standard designs of randomised controlled trials (including adaptive designs, SMART trials (sequential multiple assignment randomised trials), n-of-1 trials, and hybrid effectiveness-implementation designs) are important areas of methods development to improve the efficiency of complex intervention research.” | concept_edges[0] |
| weight | SMART trials —same-as→ sequential multiple assignment randomised trials — 0.9 — “Extensions to standard designs of randomised controlled trials (including adaptive designs, SMART trials (sequential multiple assignment randomised trials), n-of-1 trials, and hybrid effectiveness-implementation designs) are important areas of methods development to improve the efficiency of complex intervention research.” | concept_edges[0] |
| relation | cost consequence analysis —contrasted-with→ cost utility analysis — Broad ranging approaches such as cost benefit analysis or cost consequence analysis, which seek to capture the full range of health and non-health costs and benefits across different sectors, 51 will often be more suitable for an economic evaluation of a complex intervention than narrower approaches such as cost effectiveness or cost utility analysis. — “Broad ranging approaches such as cost benefit analysis or cost consequence analysis, which seek to capture the full range of health and non-health costs and benefits across different sectors, 51 will often be more suitable for an economic evaluation of a complex intervention than narrower approaches such as cost effectiveness or cost utility analysis.” | concept_edges[1] |
| weight | cost consequence analysis —contrasted-with→ cost utility analysis — 0.65 — “Broad ranging approaches such as cost benefit analysis or cost consequence analysis, which seek to capture the full range of health and non-health costs and benefits across different sectors, 51 will often be more suitable for an economic evaluation of a complex intervention than narrower approaches such as cost effectiveness or cost utility analysis.” | concept_edges[1] |
| relation | Context —causes→ policy implementation — For example, the evaluation of the Breastfeeding In Groups intervention found that the context of the different localities (eg, staff morale and suitable premises) influenced policy implementation and was an explanatory factor in why breastfeeding rates increased in some intervention localities and declined in others. — “For example, the evaluation of the Breastfeeding In Groups intervention found that the context of the different localities (eg, staff morale and suitable premises) influenced policy implementation and was an explanatory factor in why breastfeeding rates increased in some intervention localities and declined in others.” | concept_edges[2] |
| weight | Context —causes→ policy implementation — 0.6 — “For example, the evaluation of the Breastfeeding In Groups intervention found that the context of the different localities (eg, staff morale and suitable premises) influenced policy implementation and was an explanatory factor in why breastfeeding rates increased in some intervention localities and declined in others.” | concept_edges[2] |
| replication | not-applicable | |
| limitation | Use and impact of the framework have not been monitored in the past. — “Monitoring the use of the framework and evaluating its acceptability and impact is important but has been lacking in the past.” | limitations[0] |
| limitation | The wider approach yields answers that carry caveats and assumptions. — “This approach could help improve understanding and identify important implications for decision makers, albeit with caveats, assumptions, and limitations.” | limitations[1] |
| limitation | Some of the methods advocated have little track record yet. — “A bolder approach is needed—to include methods and perspectives where experience is still quite limited, but where we, supported by our workshop participants and respondents to our consultations, believe there is an urgent need to make progress.” | limitations[2] |
finding · quantity · variable · definition · key-term — grounded in the full text| finding | Involving service users in the PARADES programme improved the intervention, its evaluation and dissemination. — “Involving service users with lived experiences of bipolar disorder had many benefits, for example, it enhanced the intervention but also improved the evaluation and dissemination methods.” | content_tags.findings[0] |
| finding | The NHS diabetes prevention programme had no explicit theoretical basis or logic model. — “Another example is a recent analysis of the implicit theory of the NHS diabetes prevention programme, involving analysis of documentation by NHS England and four providers, showing that there was no explicit theoretical basis for the programme, and no logic model showing how the intervention was expected to work.” | content_tags.findings[1] |
| quantity | 36 — — “A full day expert workshop, in May 2018, of 36 participants to discuss the topics identified in the gap analysis” | content_tags.quantities[0] |
| quantity | 52 — — “An open consultation on a draft of the framework in April 2019, whereby we sought stakeholder opinion by advertising via social media, email lists and other networks for written feedback (52 detailed responses were received from stakeholders internationally)” | content_tags.quantities[1] |
| key-terms | complex interventions, programme theory, context, stakeholders, process evaluation, economic evaluation |
| concept | Context — Context=any feature of the circumstances in which an intervention is conceived, developed, evaluated, and implemented — “Context can be considered as both dynamic and multi-dimensional. Key dimensions include physical, spatial, organisational, social, cultural, political, or economic features of the healthcare, health system, or public health contexts in which interventions are implemented.” | concepts[0] |
| concept | Programme theory — Programme theory describes how an intervention is expected to lead to its effects and under what conditions. It articulates the key components of the intervention and how they interact — “Programme theory can be used to promote shared understanding of the intervention among diverse stakeholders, and to identify key uncertainties and research questions.” | concepts[1] |
| concept | mechanisms of change — where mechanisms are the causal links between intervention components and outcomes — “The importance of interactions between the intervention and its context emphasises the value of identifying mechanisms of change, where mechanisms are the causal links between intervention components and outcomes; and contextual factors, which determine and shape whether and how outcomes are generated.” | concepts[2] |
| concept | Stakeholders — Stakeholders include those individuals who are targeted by the intervention or policy, those involved in its development or delivery, or those whose personal or professional interests are affected — “Meaningful engagement with appropriate stakeholders at each phase of the research is needed to maximise the potential of developing or identifying an intervention that is likely to have positive impacts on health and to enhance prospects of achieving changes in policy or practice.” | concepts[3] |
| concept | Evaluability assessment — Evaluability assessment involves collaboration with stakeholders to reach agreement on the expected outcomes of the intervention, the data that could be collected to assess processes and outcomes, and the options for designing the evaluation. — “Evaluability assessment involves collaboration with stakeholders to reach agreement on the expected outcomes of the intervention, the data that could be collected to assess processes and outcomes, and the options for designing the evaluation.” | concepts[4] |
| concept | Economic evaluation — Economic evaluation—the comparative analysis of alternative courses of action in terms of both costs (resource use) and consequences (outcomes, effects) — “Economic evaluation—the comparative analysis of alternative courses of action in terms of both costs (resource use) and consequences (outcomes, effects)—should be a core component of all phases of intervention research.” | concepts[5] |
| concept | refinement — refinement=the process of fine tuning or making changes to the intervention once a preliminary version (prototype) has been developed — “Refinements should be guided by the programme theory, with acceptable boundaries agreed and specified at the beginning of each research phase, and with transparent reporting of the rationale for change.” | concepts[6] |
| concept | Process evaluation — not defined in this paper's own words; a study run beside a trial that asks how delivery, mechanisms and setting shape results — “Process evaluation can answer questions around fidelity and quality of implementation (eg, what is implemented and how?), mechanisms of change (eg, how does the delivered intervention produce change?), and context (eg, how does context affect implementation and outcomes?).” | concepts[7] |
| concept | SMART trials — sequential multiple assignment randomised trials — “Extensions to standard designs of randomised controlled trials (including adaptive designs, SMART trials (sequential multiple assignment randomised trials), n-of-1 trials, and hybrid effectiveness-implementation designs) are important areas of methods development to improve the efficiency of complex intervention research.” | concepts[8] |
| concept | sequential multiple assignment randomised trials — not defined in this paper's own words; a trial design that re-randomises participants at set decision points — “Extensions to standard designs of randomised controlled trials (including adaptive designs, SMART trials (sequential multiple assignment randomised trials), n-of-1 trials, and hybrid effectiveness-implementation designs) are important areas of methods development to improve the efficiency of complex intervention research.” | concepts[9] |
| concept | cost consequence analysis — not defined in this paper's own words; a broad economic method listing all costs and outcomes across sectors side by side — “Broad ranging approaches such as cost benefit analysis or cost consequence analysis, which seek to capture the full range of health and non-health costs and benefits across different sectors, 51 will often be more suitable for an economic evaluation of a complex intervention than narrower approaches such as cost effectiveness or cost utility analysis.” | concepts[10] |
| concept | cost utility analysis — not defined in this paper's own words; a narrower economic method valuing outcomes in a single health utility unit — “Broad ranging approaches such as cost benefit analysis or cost consequence analysis, which seek to capture the full range of health and non-health costs and benefits across different sectors, 51 will often be more suitable for an economic evaluation of a complex intervention than narrower approaches such as cost effectiveness or cost utility analysis.” | concepts[11] |
| concept | policy implementation — not defined in this paper's own words; how a policy-led intervention is put into practice in a given locality — “For example, the evaluation of the Breastfeeding In Groups intervention found that the context of the different localities (eg, staff morale and suitable premises) influenced policy implementation and was an explanatory factor in why breastfeeding rates increased in some intervention localities and declined in others.” | concepts[12] |
Full text — tagged
The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.
A new framework for developing and evaluating complex interventionsterm·key: update of Medical Research Council guidance.
Abstract The UK Medical Research Council’s widely used guidance for developing and evaluating complex interventionsterm·key has been replaced by a new framework, commissioned jointly by the Medical Research Council and the National Institute for Health Research, which takes account of recent developments in theory and methods and the need to maximise the efficiency, use, and impact of research.
Complex interventionsterm·key are commonly used in the health and social care services, public health practice, and other areas of social and economic policy that have consequences for health. Such interventions are delivered and evaluated at different levels, from individual to societal levels. Examples include a new surgical procedure, the redesign of a healthcare programme, and a change in welfare policy. The UK Medical Research Council (MRC) published a framework for researchers and research funders on developing and evaluating complex interventionsterm·key in 2000 and revised guidance in 2006. 1 2 3 Although these documents continue to be widely used and are now accompanied by a range of more detailed guidance on specific aspects of the research process, 4 5 6 7 8 several important conceptual, methodological and theoretical developments have taken place since 2006. These developments have been included in a new framework commissioned by the National Institute of Health Research (NIHR) and the MRC. 9 The framework aims to help researchers work with other stakeholdersterm·dom to identify the key questions about complex interventionsterm·key, and to design and conduct research with a diversity of perspectives and appropriate choice of methods.
Complex interventionterm·key research can take an efficacy, effectiveness, theory based, and/or systems perspective, the choice of which is based on what is known already and what further evidence would add most to knowledge Complex interventionterm·key research goes beyond asking whether an intervention works in the sense of achieving its intended outcome—to asking a broader range of questions (eg, identifying what other impact it has, assessing its value relative to the resources required to deliver it, theorising how it works, taking account of how it interacts with the contextterm·dom in which it is implemented, how it contributes to system change, and how the evidence can be used to support real world decision making) A trade-off exists between precise unbiased answers to narrow questions and more uncertain answers to broader, more complex questions; researchers should answer the questions that are most useful to decision makers rather than those that can be answered with greater certainty Complex interventionterm·key research can be considered in terms of phases, although these phases are not necessarily sequential: development or identification of an intervention, assessment of feasibility of the intervention and evaluation design, evaluation of the intervention, and impactful implementation At each phase, six core elements should be considered to answer the following questions: How does the intervention interact with its contextterm·dom? What is the underpinning programme theoryterm·dom? How can diverse stakeholderterm·dom perspectives be included in the research? What are the key uncertainties? How can the intervention be refined? What are the comparative resource and outcome consequences of the intervention?
The answers to these questions should be used to decide whether the research should proceed to the next phase, return to a previous phase, repeat a phase, or stop Development of the Framework for Developing and Evaluating Complex Interventionsterm·key The updated Framework for Developing and Evaluating Complex Interventionsterm·key is the culmination of a process that included four stages: A gap analysis to identify developments in the methods and practice since the previous framework was published A full day expert workshop, in May 2018, of 36 participants to discuss the topics identified in the gap analysis An open consultation on a draft of the framework in April 2019, whereby we sought stakeholderterm·dom opinion by advertising via social media, email lists and other networks for written feedback (52 detailed responses were received from stakeholdersterm·dom internationally) Redraft using findings from the previous stages, followed by a final expert review.
We also sought stakeholderterm·dom views at various interactive workshops throughout the development of the framework: at the annual meetings of the Society for Social Medicine and Population Health (2018), the UK Society for Behavioural Medicine (2017, 2018), and internationally at the International Congress of Behavioural Medicine (2018). The entire process was overseen by a scientific advisory group representing the range of relevant NIHR programmes and MRC population health investments. The framework was reviewed by the MRC-NIHR Methodology Research Programme Advisory Group and then approved by the MRC Population Health Sciences Group in March 2020 before undergoing further external peer and editorial review through the NIHR Journals Library peer review process. More detailed information and the methods used to develop this new framework are described elsewhere. 9 This article introduces the framework and summarises the main messages for producers and users of evidence.
What are complex interventionsterm·key?
An intervention might be considered complex because of properties of the intervention itself, such as the number of components involved; the range of behaviours targeted; expertise and skills required by those delivering and receiving the intervention; the number of groups, settings, or levels targeted; or the permitted level of flexibility of the intervention or its components. For example, the Links Worker Programme was an intervention in primary care in Glasgow, Scotland, that aimed to link people with community resources to help them “live well” in their communities. It targeted individual, primary care (general practitioner (GP) surgery), and community levels. The intervention was flexible in that it could differ between primary care GP surgeries. In addition, the Link Workers did not support just one specific health or wellbeing issue: bereavement, substance use, employment, and learning difficulties were all included. 10 11 The complexity of this intervention had implications for many aspects of its evaluation, such as the choice of appropriate outcomes and processes to assess.
Flexibility in intervention delivery and adherence might be permitted to allow for variation in how, where, and by whom interventions are delivered and received. Standardisation of interventions could relate more to the underlying process and functions of the intervention than on the specific form of components delivered. 12 For example, in surgical trials, protocols can be designed with flexibility for intervention delivery. 13 Interventions require a theoretical deconstruction into components and then agreement about permissible and prohibited variation in the delivery of those components. This approach allows implementation of a complex interventionterm·key to vary across different contextsterm·dom yet maintain the integrity of the core intervention components. Drawing on this approach in the ROMIO pilot trial, core components of minimally invasive oesophagectomy were agreed and subsequently monitored during main trial delivery using photography. 14 Complexity might also arise through interactions between the intervention and its contextterm·dom, by which we mean “any feature of the circumstances in which an intervention is conceived, developed, implemented and evaluated.” 6 15 16 17 Much of the criticism of and extensions to the existing framework and guidance have focused on the need for greater attention on understanding how and under what circumstances interventions bring about change. 7 15 18 The importance of interactions between the intervention and its contextterm·dom emphasises the value of identifying mechanisms of changeterm·dom, where mechanisms are the causal links between intervention components and outcomes; and contextual factors, which determine and shape whether and how outcomes are generated. 19 Thus, attention is given not only to the design of the intervention itself but also to the conditions needed to realise its mechanisms of changeterm·dom and/or the resources required to support intervention reach and impact in real world implementation. For example, in a cluster randomised trial of ASSIST (a peer led, smoking prevention intervention), researchers found that the intervention worked particularly well in cohesive communities that were served by one secondary school where peer supporters were in regular contact with their peers—a key contextual factor consistent with diffusion of innovation theory, which underpinned the intervention design. 20 A process evaluationterm·mth conducted alongside a trial of robot assisted surgery identified key contextual factors to support effective implementation of this procedure, including engaging staff at different levels and surgeons who would not be using robot assisted surgery, whole team training, and an operating theatre of suitable size. 21 With this framing, complex interventionsterm·key can helpfully be considered as events in systems. 16 Thinking about systems helps us understand the interaction between an intervention and the contextterm·dom in which it is implemented in a dynamic way. 22 Systems can be thought of as complex and adaptive, 23 characterised by properties such as emergence, feedback, adaptation, and self-organisation ( table 1 ).
Properties and examples of complex adaptive systems For complex interventionterm·key research to be most useful to decision makers, it should take into account the complexity that arises both from the intervention’s components and from its interaction with the contextterm·dom in which it is being implemented.
Research perspectives The previous framework and guidance were based on a paradigm in which the salient question was to identify whether an intervention was effective. Complex interventionterm·key research driven primarily by this question could fail to deliver interventions that are implementable, cost effective, transferable, and scalable in real world conditions. To deliver solutions for real world practice, complex interventionterm·key research requires strong and early engagement with patients, practitioners, and policy makers, shifting the focus from the “binary question of effectiveness” 26 to whether and how the intervention will be acceptable, implementable, cost effective, scalable, and transferable across contextsterm·dom. In line with a broader conception of complexity, the scope of complex interventionterm·key research needs to include the development, identification, and evaluation of whole system interventions and the assessment of how interventions contribute to system change. 22 27 The new framework therefore takes a pluralistic approach and identifies four perspectives that can be used to guide the design and conduct of complex interventionterm·key research: efficacy, effectiveness, theory based, and systems ( table 2 ).
Research perspectives Although each research perspective prompts different types of research question, they should be thought of as overlapping rather than mutually exclusive. For example, theory based and systems perspectives to evaluation can be used in conjunction, 33 while an effectiveness evaluation can draw on a theory based or systems perspective through an embedded process evaluationterm·mth to explore how and under what circumstances outcomes are achieved. 34 35 36 Most complex health intervention research so far has taken an efficacy or effectiveness perspective and for some research questions these perspectives will continue to be the most appropriate. However, some questions equally relevant to the needs of decision makers cannot be answered by research restricted to an efficacy or effectiveness perspective. A wider range and combination of research perspectives and methods, which answer questions beyond efficacy and effectiveness, need to be used by researchers and supported by funders. Doing so will help to improve the extent to which key questions for decision makers can be answered by complex interventionterm·key research. Example questions include: Will this effective intervention reproduce the effects found in the trial when implemented here?
Is the intervention cost effective?
What are the most important things we need to do that will collectively improve health outcomes?
In the absence of evidence from randomised trials and the infeasibility of conducting such a trial, what does the existing evidence suggest is the best option now and how can this be evaluated?
What wider changes will occur as a result of this intervention?
How are the intervention effects mediated by different settings and contextsterm·dom?
Phases and core elements of complex interventionterm·key research The framework divides complex interventionterm·key research into four phases: development or identification of the intervention, feasibility, evaluation, and implementation ( fig 1 ). A research programme might begin at any phase, depending on the key uncertainties about the intervention in question. Repeating phases is preferable to automatic progression if uncertainties remain unresolved. Each phase has a common set of core elements—considering contextterm·dom, developing and refining programme theoryterm·dom, engaging stakeholdersterm·dom, identifying key uncertainties, refining the intervention, and economic considerations. These elements should be considered early and continually revisited throughout the research process, and especially before moving between phases (for example, between feasibility testing and evaluation).
Framework for developing and evaluating complex interventionsterm·key. Contextterm·dom=any feature of the circumstances in which an intervention is conceived, developed, evaluated, and implemented; programme theoryterm·dom=describes how an intervention is expected to lead to its effects and under what conditions—the programme theoryterm·dom should be tested and refined at all stages and used to guide the identification of uncertainties and research questions; stakeholdersterm·dom=those who are targeted by the intervention or policy, involved in its development or delivery, or more broadly those whose personal or professional interests are affected (that is, who have a stake in the topic)—this includes patients and members of the public as well as those linked in a professional capacity; uncertainties=identifying the key uncertainties that exist, given what is already known and what the programme theoryterm·dom, research team, and stakeholdersterm·dom identify as being most important to discover—these judgments inform the framing of research questions, which in turn govern the choice of research perspective; refinementterm·mth=the process of fine tuning or making changes to the intervention once a preliminary version (prototype) has been developed; economic considerations=determining the comparative resource and outcome consequences of the interventions for those people and organisations affected Core elements Contextterm·dom The effects of a complex interventionterm·key might often be highly dependent on contextterm·dom, such that an intervention that is effective in some settings could be ineffective or even harmful elsewhere. 6 As the examples in table 1 show, interventions can modify the contextsterm·dom in which they are implemented, by eliciting responses from other agents, or by changing behavioural norms or exposure to risk, so that their effects will also vary over time. Contextterm·dom can be considered as both dynamic and multi-dimensional. Key dimensions include physical, spatial, organisational, social, cultural, political, or economic features of the healthcare, health system, or public health contextsterm·dom in which interventions are implemented. For example, the evaluation of the Breastfeeding In Groups intervention found that the contextterm·dom of the different localities (eg, staff morale and suitable premises) influenced policy implementationterm·dom and was an explanatory factor in why breastfeeding rates increased in some intervention localities and declined in others. 37 Programme theoryterm·dom Programme theoryterm·dom describes how an intervention is expected to lead to its effects and under what conditions. It articulates the key components of the intervention and how they interact, the mechanisms of the intervention, the features of the contextterm·dom that are expected to influence those mechanisms, and how those mechanisms might influence the contextterm·dom. 38 Programme theoryterm·dom can be used to promote shared understanding of the intervention among diverse stakeholdersterm·dom, and to identify key uncertainties and research questions. Where an intervention (such as a policy) is developed by others, researchers still need to theorise the intervention before attempting to evaluate it. 39 Best practice is to develop programme theoryterm·dom at the beginning of the research project with involvement of diverse stakeholdersterm·dom, based on evidence and theory from relevant fields, and to refine it during successive phases. The EPOCH trial tested a large scale quality improvement programme aimed at improving 90 day survival rates for patients undergoing emergency abdominal surgery; it included a well articulated programme theoryterm·dom at the outset, which supported the tailoring of programme delivery to local contextsterm·dom. 40 The development, implementation, and post-study reflection of the programme theoryterm·dom resulted in suggested improvements for future implementation of the quality improvement programme.
A refined programme theoryterm·dom is an important evaluation outcome and is the principal aim where a theory based perspective is taken. Improved programme theoryterm·dom will help inform transferability of interventions across settings and help produce evidence and understanding that is useful to decision makers. In addition to full articulation of programme theoryterm·dom, it can help provide visual representations—for example, using a logic model, 41 42 43 realist matrix, 44 or a system map, 45 with the choice depending on which is most appropriate for the research perspective and research questions. Although useful, any single visual representation is unlikely to sufficiently articulate the programme theoryterm·dom—it should always be articulated well within the text of publications, reports, and funding applications.
Stakeholdersterm·dom Stakeholdersterm·dom include those individuals who are targeted by the intervention or policy, those involved in its development or delivery, or those whose personal or professional interests are affected (that is, all those who have a stake in the topic). Patients and the public are key stakeholdersterm·dom. Meaningful engagement with appropriate stakeholdersterm·dom at each phase of the research is needed to maximise the potential of developing or identifying an intervention that is likely to have positive impacts on health and to enhance prospects of achieving changes in policy or practice. For example, patient and public involvement 46 activities in the PARADES programme, which evaluated approaches to reduce harm and improve outcomes for people with bipolar disorder, were wide ranging and central to the project. 47 Involving service users with lived experiences of bipolar disorder had many benefits, for example, it enhanced the intervention but also improved the evaluation and dissemination methods. Service users involved in the study also had positive outcomes, including more settled employment and progression to further education. Broad thinking and consultation is needed to identify a diverse range of appropriate stakeholdersterm·dom.
The purpose of stakeholderterm·dom engagement will differ depending on the contextterm·dom and phase of the research, but is essential for prioritising research questions, the co-development of programme theoryterm·dom, choosing the most useful research perspective, and overcoming practical obstacles to evaluation and implementation. Researchers should nevertheless be mindful of conflicts of interest among stakeholdersterm·dom and use transparent methods to record potential conflicts of interest. Research should not only elicit stakeholderterm·dom priorities, but also consider why they are priorities. Careful consideration of the appropriateness and methods of identification and engagement of stakeholdersterm·dom is needed. 46 48 Key uncertainties Many questions could be answered at each phase of the research process. The design and conduct of research need to engage pragmatically with the multiple uncertainties involved and offer a flexible and emergent approach to exploring them. 15 Therefore, researchers should spend time developing the programme theoryterm·dom, clearly identifying the remaining uncertainties, given what is already known and what the research team and stakeholdersterm·dom identify as being most important to determine. Judgments about the key uncertainties inform the framing of research questions, which in turn govern the choice of research perspective.
Efficacy trials of relatively uncomplicated interventions in tightly controlled conditions, where research questions are answered with great certainty, will always be important, but translation of the evidence into the diverse settings of everyday practice is often highly problematic. 27 For intervention research in healthcare and public health settings to take on more challenging evaluation questions, greater priority should be given to mixed methods, theory based, or systems evaluation that is sensitive to complexity and that emphasises implementation, contextterm·dom, and system fit. This approach could help improve understanding and identify important implications for decision makers, albeit with caveats, assumptions, and limitations. 22 Rather than maintaining the established tendency to prioritise strong research designs that answer some questions with certainty but are unsuited to resolving many important evaluation questions, this more inclusive, deliberative process could place greater value on equivocal findings that nevertheless inform important decisions where evidence is sparse.
Intervention refinementterm·mth Within each phase of complex interventionterm·key research and on transition from one phase to another, the intervention might need to be refined, on the basis of data collected or development of programme theoryterm·dom. 4 The feasibility and acceptability of interventions can be improved by engaging potential intervention users to inform refinementsterm·mth. For example, an online physical activity planner for people with diabetes mellitus was found to be difficult to use, resulting in the tool providing incorrect personalised advice. To improve usability and the advice given, several iterations of the planner were developed on the basis of interviews and observations. This iterative process led to the refined planner demonstrating greater feasibility and accuracy. 49 Refinementsterm·mth should be guided by the programme theoryterm·dom, with acceptable boundaries agreed and specified at the beginning of each research phase, and with transparent reporting of the rationale for change. Scope for refinementterm·mth might also be limited by the policy or practice contextterm·dom. Refinementterm·mth will be rare in the evaluation phase of efficacy and effectiveness research, where interventions will ideally not change or evolve within the course of the study. However, between the phases of research and within systems and theory based evaluation studies, refinementterm·mth of interventions in response to accumulated data or as an adaptive and variable response to contextterm·dom and system change are likely to be desirable features of the intervention and a key focus of the research.
Economic considerations Economic evaluationterm·mth—the comparative analysis of alternative courses of action in terms of both costs (resource use) and consequences (outcomes, effects)—should be a core component of all phases of intervention research. Early engagement of economic expertise will help identify the scope of costs and benefits to assess in order to answer questions that matter most to decision makers. 50 Broad ranging approaches such as cost benefit analysis or cost consequence analysisterm·mth, which seek to capture the full range of health and non-health costs and benefits across different sectors, 51 will often be more suitable for an economic evaluationterm·mth of a complex interventionterm·key than narrower approaches such as cost effectiveness or cost utility analysisterm·mth. For example, evaluation of the New Orleans Intervention Model for infants entering foster care in Glasgow included short and long term economic analysis from multiple perspectives (the UK’s health service and personal social services, public sector, and wider societal perspectives); and used a range of frameworks, including cost utility and cost consequence analysisterm·mth, to capture changes in the intersectoral costs and outcomes associated with child maltreatment. 52 53 The use of multiple economic evaluationterm·mth frameworks provides decision makers with a comprehensive, multi-perspective guide to the cost effectiveness of the New Orleans Intervention Model.
Phases Developing or identifying a complex interventionterm·key Development refers to the whole process of designing and planning an intervention, from initial conception through to feasibility, pilot, or evaluation study. Guidance on intervention development has recently been developed through the INDEX study 4 ; although here we highlight that complex interventionterm·key research does not always begin with new or researcher led interventions. For example: A key source of intervention development might be an intervention that has been developed elsewhere and has the possibility of being adapted to a new contextterm·dom. Adaptation of existing interventions could include adapting to a new population, to a new setting, 54 55 or to target other outcomes (eg, a smoking prevention intervention being adapted to tackle substance misuse and sexual health). 20 56 57 A well developed programme theoryterm·dom can help identify what features of the antecedent intervention(s) need to be adapted for different applications, and the key mechanisms that should be retained even if delivered slightly differently. 54 58 Policy or practice led interventions are an important focus of evaluation research. Again, uncovering the implicit theoretical basis of an intervention and developing a programme theoryterm·dom is essential to identifying key uncertainties and working out how the intervention might be evaluated. This step is important, even if rollout has begun, because it supports the identification of mechanisms of changeterm·dom, important contextual factors, and relevant outcome measures. For example, researchers evaluating the UK soft drinks industry levy developed a bounded conceptual system map to articulate their understanding (drawing on stakeholderterm·dom views and document review) of how the intervention was expected to work. This system map guided the evaluation design and helped identify data sources to support evaluation. 45 Another example is a recent analysis of the implicit theory of the NHS diabetes prevention programme, involving analysis of documentation by NHS England and four providers, showing that there was no explicit theoretical basis for the programme, and no logic model showing how the intervention was expected to work. This meant that the justification for the inclusion of intervention components was unclear. 59 Intervention identification and intervention development represent two distinct pathways of evidence generation, 60 but in both cases, the key considerations in this phase relate to the core elements described above.
Feasibility A feasibility study should be designed to assess predefined progression criteria that relate to the evaluation design (eg, reducing uncertainty around recruitment, data collection, retention, outcomes, and analysis) or the intervention itself (eg, around optimal content and delivery, acceptability, adherence, likelihood of cost effectiveness, or capacity of providers to deliver the intervention). If the programme theoryterm·dom suggests that contextual or implementation factors might influence the acceptability, effectiveness, or cost effectiveness of the intervention, these questions should be considered.
Despite being overlooked or rushed in the past, the value of feasibility testing is now widely accepted with key terms and concepts well defined. 61 62 Before initiating a feasibility study, researchers should consider conducting an evaluability assessmentterm·mth to determine whether and how an intervention can usefully be evaluated. Evaluability assessmentterm·mth involves collaboration with stakeholdersterm·dom to reach agreement on the expected outcomes of the intervention, the data that could be collected to assess processes and outcomes, and the options for designing the evaluation. 63 The end result is a recommendation on whether an evaluation is feasible, whether it can be carried out at a reasonable cost, and by which methods. 64 Economic modelling can be undertaken at the feasibility stage to assess the likelihood that the expected benefits of the intervention justify the costs (including the cost of further research), and to help decision makers decide whether proceeding to a full scale evaluation is worthwhile. 65 Depending on the results of the feasibility study, further work might be required to progressively refine the intervention before embarking on a full scale evaluation.
Evaluation The new framework defines evaluation as going beyond asking whether an intervention works (in the sense of achieving its intended outcome), to a broader range of questions including identifying what other impact it has, theorising how it works, taking account of how it interacts with the contextterm·dom in which it is implemented, how it contributes to system change, and how the evidence can be used to support decision making in the real world. This implies a shift from an exclusive focus on obtaining unbiased estimates of effectiveness 66 towards prioritising the usefulness of information for decision making in selecting the optimal research perspective and in prioritising answerable research questions.
A crucial aspect of evaluation design is the choice of outcome measures or evidence of change. Evaluators should work with stakeholdersterm·dom to assess which outcomes are most important, and how to deal with multiple outcomes in the analysis with due consideration of statistical power and transparent reporting. A sharp distinction between one primary outcome and several secondary outcomes is not necessarily appropriate, particularly where the programme theoryterm·dom identifies impacts across a range of domains. Where needed to support the research questions, prespecified subgroup analyses should be carried out and reported. Even where such analyses are underpowered, they should be included in the protocol because they might be useful for subsequent meta-analyses, or for developing hypotheses for testing in further research. Outcome measures could capture changes to a system rather than changes in individuals. Examples include changes in relationships within an organisation, the introduction of policies, changes in social norms, or normalisation of practice. Such system level outcomes include how changing the dynamics of one part of a system alters behaviours in other parts, such as the potential for displacement of smoking into the home after a public smoking ban.
A helpful illustration of the use of system level outcomes is the evaluation of the Delaware Young Health Program—an initiative to improve the health and wellbeing of young people in Delaware, USA. The intervention aimed to change underlying system dynamics, structures, and conditions, so the evaluation identified systems oriented research questions and methods. Three systems science methods were used: group model building and viable systems model assessment to identify underlying patterns and structures; and social network analysis to evaluate change in relationships over time. 67 Researchers have many study designs to choose from, and different designs are optimally suited to consider different research questions and different circumstances. 68 Extensions to standard designs of randomised controlled trials (including adaptive designs, SMART trialsterm·mth (sequential multiple assignment randomised trialsterm·mth), n-of-1 trials, and hybrid effectiveness-implementation designs) are important areas of methods development to improve the efficiency of complex interventionterm·key research. 69 70 71 72 Non-randomised designs and modelling approaches might work best if a randomised design is not practical, for example, in natural experiments or systems evaluations. 5 73 74 A purely quantitative approach, using an experimental design with no additional elements such as a process evaluationterm·mth, is rarely adequate for complex interventionterm·key research, where qualitative and mixed methods designs might be necessary to answer questions beyond effectiveness. In many evaluations, the nature of the intervention, the programme theoryterm·dom, or the priorities of stakeholdersterm·dom could lead to a greater focus on improving theories about how to intervene. In this view, effect estimates are inherently contextterm·dom bound, so that average effects are not a useful guide to decision makers working in different contextsterm·dom. Contextualised understandings of how an intervention induces change might be more useful, as well as details on the most important enablers and constraints on its delivery across a range of settings. 7 Process evaluationterm·mth can answer questions around fidelity and quality of implementation (eg, what is implemented and how?), mechanisms of changeterm·dom (eg, how does the delivered intervention produce change?), and contextterm·dom (eg, how does contextterm·dom affect implementation and outcomes?). 7 Process evaluationterm·mth can help determine why an intervention fails unexpectedly or has unanticipated consequences, or why it works and how it can be optimised. Such findings can facilitate further development of the intervention programme theoryterm·dom. 75 In a theory based or systems evaluation, there is not necessarily such a clear distinction between process and outcome evaluation as there is in an effectiveness study. 76 These perspectives could prioritise theory building over evidence production and use case study or simulation methods to understand how outcomes or system behaviour are generated through intervention. 74 77 Implementation Early consideration of implementation increases the potential of developing an intervention that can be widely adopted and maintained in real world settings. Implementation questions should be anticipated in the intervention programme theoryterm·dom, and considered throughout the phases of intervention development, feasibility testing, process, and outcome evaluation. Alongside implementation specific outcomes (such as reach or uptake of services), attention to the components of the implementation strategy, and contextual factors that support or hinder the achievement of impacts, are key. Some flexibility in intervention implementation might support intervention transferability into different contextsterm·dom (an important aspect of long term implementation 78 ), provided that the key functions of the programme are maintained, and that the adaptations made are clearly understood. 8 In the ASSIST study, 20 a school based, peer led intervention for smoking prevention, researchers considered implementation at each phase. The intervention was developed to have minimal disruption on school resources; the feasibility study resulted in intervention refinementsterm·mth to improve acceptability and improve reach to male students; and in the evaluation (cluster randomised controlled trial), the intervention was delivered as closely as possible to real world implementation. Drawing on the process evaluationterm·mth, the implementation included an intervention manual that identified critical components and other components that could be adapted or dropped to allow flexible implementation while achieving delivery of the key mechanisms of changeterm·dom; and a training manual for the trainers and ongoing quality assurance built into rollout for the longer term.
In a natural experimental study, evaluation takes place during or after the implementation of the intervention in a real world contextterm·dom. Highly pragmatic effectiveness trials or specific hybrid effectiveness-implementation designs also combine effectiveness and implementation outcomes in one study, with the aim of reducing time for translation of research on effectiveness into routine practice. 72 79 80 Implementation questions should be included in economic considerations during the early stages of intervention and study development. How the results of economic analyses are reported and presented to decision makers can affect whether and how they act on the results. 81 A key consideration is how to deal with interventions across different sectors, where those paying for interventions and those receiving the benefits of them could differ, reducing the incentive to implement an intervention, even if shown to be beneficial and cost effective. Early engagement with appropriate stakeholdersterm·dom will help frame appropriate research questions and could anticipate any implementation challenges that might arise. 82 Conclusions One of the motivations for developing this new framework was to answer calls for a change in research priorities, towards allocating greater effort and funding to research that can have the optimum impact on healthcare or population health outcomes. The framework challenges the view that unbiased estimates of effectiveness are the cardinal goal of evaluation. It asserts that improving theories and understanding how interventions contribute to change, including how they interact with their contextterm·dom and wider dynamic systems, is an equally important goal. For some complex interventionterm·key research problems, an efficacy or effectiveness perspective will be the optimal approach, and a randomised controlled trial will provide the best design to achieve an unbiased estimate. For others, alternative perspectives and designs might work better, or might be the only way to generate new knowledge to reduce decision maker uncertainty.
What is important for the future is that the scope of intervention research is not constrained by an unduly limited set of perspectives and approaches that might be less risky to commission and more likely to produce a clear and unbiased answer to a specific question. A bolder approach is needed—to include methods and perspectives where experience is still quite limited, but where we, supported by our workshop participants and respondents to our consultations, believe there is an urgent need to make progress. This endeavour will involve mainstreaming new methods that are not yet widely used, as well as undertaking methodological innovation and development. The deliberative and flexible approach that we encourage is intended to reduce research waste, 83 maximise usefulness for decision makers, and increase the efficiency with which complex interventionterm·key research generates knowledge that contributes to health improvement.
Monitoring the use of the framework and evaluating its acceptability and impact is important but has been lacking in the past. We encourage research funders and journal editors to support the diversity of research perspectives and methods that are advocated here and to seek evidence that the core elements are attended to in research design and conduct. We have developed a checklist to support the preparation of funding applications, research protocols, and journal publications. 9 This checklist offers one way to monitor impact of the guidance on researchers, funders, and journal editors.
We recommend that the guidance is continually updated, and future updates continue to adopt a broad, pluralist perspective. Given its wider scope, and the range of detailed guidance that is now available on specific methods and topics, we believe that the framework is best seen as meta-guidance. Further editions should be published in a fluid, web based format, and more frequently updated to incorporate new material, further case studies, and additional links to other new resources.