Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects. — TEI-tagged view

Jim McCambridgepersName, John WittonpersName, Diana R. ElbournepersName (2014date). “Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects..” Journal of Clinical Epidemiologytitle·j.

Object of study — empirical evidence on the Hawthorne effectterm·dom, as a consequence of research participationterm·dom on behaviour

Research question — whether the Hawthorne effectterm·dom exists, under what conditions, and how large it is

Contribution — a case that the construct should be retired and replaced by the study of research participation effectsterm·dom

Full text — tagged

Systematic reviewterm·key of the Hawthorne effectterm·dom: new concepts are needed to study research participation effectsterm·dom.

Abstract This study aims to (1) elucidate whether the Hawthorne effectterm·dom exists, (2) explore under what conditions, and (3) estimate the size of any such effect.

This systematic reviewterm·key summarizes and evaluates the strength of available evidence on the Hawthorne effectterm·dom. An inclusive definition of any form of research artifactterm·key on behaviorterm·dom using this label, and without cointerventions, was adopted.

Nineteen purposively designed studies were included, providing quantitative data on the size of the effect in eight randomized controlled trials, five quasiexperimental studies, and six observational evaluations of reporting on one's behaviorterm·dom by answering questions or being directly observed and being aware of being studied. Although all but one study was undertaken within health sciences, study methods, contexts, and findings were highly heterogeneous. Most studies reported some evidence of an effect, although significant biases are judged likely because of the complexity of the evaluation object.

Consequences of research participationterm·dom for behaviorsterm·dom being investigated do exist, although little can be securely known about the conditions under which they operate, their mechanisms of effects, or their magnitudes. New concepts are needed to guide empirical studies.

Keywords: Hawthorne effectterm·dom, Reactivityterm·key, Observation, Research methods, Research participationterm·dom, Assessment Introduction • Most of the 19 purposively designed evaluation studies included in this systematic reviewterm·key provide some evidence of research participation effectsterm·dom.

• The heterogeneity of these studies means that little can be confidently inferred about the size of these effects, the conditions under which they operate, or their mechanisms.

• There is a clear need to rectify the limited development of study of the issues represented by the Hawthorne effectterm·dom as they indicate potential for profound biases.

• As the Hawthorne effectterm·dom construct has not successfully led to important research advances in this area over a period of 60 years, new concepts are needed to guide empirical studies.

The Hawthorne effectterm·dom concerns research participationterm·dom, the consequent awareness of being studied, and possible impact on behaviorterm·dom [1-5] . It is a widely used research term. The original studies that gave rise to the Hawthorne effectterm·dom were undertaken at Western Electric telephone manufacturing factory at Hawthorne, near Chicago, between 1924 and 1933 [6-8] . Increases in productivity were observed among a selected group of workers who were supervised intensively by managers under the auspices of a research program. The term was first used in an influential methodology textbook in 1953 [9] . A large literature and repeated controversies have evolved over many decades as to the nature of the Hawthorne effectterm·dom [5,10] . If there is a Hawthorne effectterm·dom, studies could be biased in ways that we do not understand well, with profound implications for research [11] .

Empirical data on the Hawthorne effectterm·dom have not previously been evaluated in a systematic reviewterm·key. Early reviews examined a body of literature on studies of school children and found no evidence of a Hawthorne effectterm·dom as the term had been used in that literature [12-14] . The contemporary relevance of the Hawthorne effectterm·dom is clearer within health sciences, in which recent years have seen an upsurge in applications of this construct in relation to a range of methodological phenomena (see examples of studies with nonbehavioral outcomes [15-17] ).

There are two main ways in which the construct of the Hawthorne effectterm·dom has previously been used in the research literature. First, there are studies that purport to explain some aspect of the findings of the original Hawthorne studies. These studies involve secondary quantitative data analyses [1,18-20] or discussions of the Hawthorne effectterm·dom, which offer interpretations based on other material [4,10,21,22] . The Hawthorne effectterm·dom has also been widely used without any necessary connection to the original studies and has usually taken on the meaning of alteration in behaviorterm·dom as a consequence of its observation or other study. In contrast to uses of the term in relation to the original Hawthorne studies, methodological versions of the Hawthorne effectterm·dom have mutated in meaning over time and across disciplines and been the subject of much controversy [1,2,4,23,24] . This diversity means that certain aspects of the putative Hawthorne effectterm·dom, for example, novelty [25] are emphasized in some studies and are absent in many others.

There is a widespread social psychological explanation of the possible mechanism for the Hawthorne effectterm·dom as follows. Awareness of being observed or having behaviorterm·dom assessed engenders beliefs about researcher expectations. Conformity and social desirability considerations then lead behaviorterm·dom to change in line with these expectations. Chiesa and Hobbs [5] point out that just as there are different meanings given to the purported Hawthorne effectterm·dom, there are also many suggested mechanisms producing the effect, some of which are contradictory. In all likelihood, the most common use of the Hawthorne effectterm·dom term is as a post hoc interpretation of unexpected study findings, particularly where they are disappointing, for example, when there are null findings in trials.

The aims of this systematic reviewterm·key were to elucidate whether the Hawthorne effectterm·dom exists, explore under what conditions, and estimate the size of any such effect, by summarizing and evaluating the strength of evidence available in all scientific disciplines. Meeting these study aims contributes to an overarching orientation to better understand whether research participationterm·dom itself influences behaviorterm·dom. This inclusive orientation eschews restrictions on participants, study designs, and precise definitions of the content of Hawthorne effectterm·dom manipulations.

Methods The Hawthorne effectterm·dom under investigation is any form of artifact or consequence of research participationterm·dom on behaviorterm·dom.

Studies were included if they were based on empirical research comprising either primary or secondary data analyses; were published in English language peer-reviewed journals; were purposively designed to determine the presence of, or measure the size of, the Hawthorne effectterm·dom, as stated in the introduction or methods sections of the article or before the presentation of findings if the report is not organized in this way; and reported quantitative data on the Hawthorne effectterm·dom on a behavioral outcome either in observational designs comparing measures taken before and after a dedicated research manipulation or between groups in randomized or nonrandomized experimental studies. Behavioral outcomes incorporate direct measures of behaviorterm·dom and also the consequences of specific behaviorsterm·dom. Studies that described their aims in other ways and also referred to the Hawthorne effectterm·dom as an alternative conceptualization of the object of evaluation were included as were studies that have other primary aims such as the evaluation of an intervention in a trial in which assessment of the Hawthorne effectterm·dom is clearly stated as a secondary aim of the study, for example, with the incorporation of control groups with and without Hawthorne effectterm·dom characteristics. Studies were excluded if: unpublished or in grey literature on the grounds that it is not possible to systematically assess these literature in an unbiased manner; discussion articles and commentaries were not considered to constitute empirical research; they referenced or used the term Hawthorne effectterm·dom incidentally or described it as a design feature or as part of the study context, or invoked it as an explanation for study findings. Studies of the Hawthorne effectterm·dom that incorporate nonresearch components, including cointerventions such as feedback, hamper evaluation and are also excluded, as were reanalyses of the original Hawthorne factory data set by virtue of nonresearch cointerventions such as managerial changes (see Ref. [8] for a detailed history of the studies).

Studies were primarily identified in electronic databases. In addition, included studies and key excluded references were backward searched for additional references and forward searched to identify reports that cited these articles. Experts identified in included studies and elsewhere were contacted. The most recent database searches took place on January 3, 2012 for the following databases: Web of Science (1970-), MEDLINE (1950-), BIOSIS Previews (1969-), PsycInfo (1806-), CINAHL Plus with full text (1937-), ERIC (1966-), PubMed (1950-), Cochrane Central Register of Controlled Trials (1947-), Embase (1947-), Sociological Abstracts (1952-), National Criminal Justice Reference Service Abstracts (NCJRS; 1970-), Social Services Abstracts (1979-), Linguistics and Language Behaviorterm·dom Abstracts (LLBA; 1973-), the International Bibliography of the Social Sciences (1951-), APPI Journals (1844-), British Nursing Index (1992-), ADOLEC (1980-), Social Policy and Practice (1890-), British Humanities Index (1962-), Applied Social Sciences Index and Abstract (1987-), Inspec (1969-), and PsycARTICLES (1988-).

The term "Hawthorne effectterm·dom" was searched for as a phrase as widely as possible. If the database permitted, the term was searched for in all fields as was the case for Embase, CINAHL, ERIC, and others. In the Web of Knowledge search, which uses Web of Science, MEDLINE, and BIOSIS Previews databases, "Hawthorne effectterm·dom" was entered into the "topic" field. The term was also searched for in "keyword" fields for databases such as NCJRS, LLBA, APPI, and others. The use of this term as the core object of evaluation negated the need for a more complex search strategy.

Hits from the database searches were downloaded into EndNote software (Thomson Reuters), removing duplicates there. Screening of titles and abstracts was undertaken by the second author or a research assistant. After a further brief screen of full-text articles, potential inclusions were independently assessed before being included. Data extracted are summarized in the tables presented here, which also contain information on risk of bias in individual studies. Binary outcomes were meta-analyzed in Stata, version 12 (Statacorp), with outcomes pooled in random-effects models using the method by DerSimonian and Laird. The Q and I 2 statistics [26] were used to evaluate the extent and effects of heterogeneity, and outcomes are stratified by study design. Formal methods were not used to assess risk of bias within and across studies, and narrative consideration is given to both. We did not publish a protocol for this review.

Results Nineteen studies were eligible for inclusion in this review [27-45] . The PRISMA flowchart summarizing the data collection process is presented in Fig. 1 . The design characteristics of included studies, along with brief summaries of outcome data and observations on most likely sources of bias, are presented separately for randomized controlled trials (RCTs), quasiexperimental studies, and observational studies in Tables 1-3 , respectively. All included studies apart from one [27] have been undertaken within health sciences. All observational studies were studies of the Hawthorne effectterm·dom on health-care practitioners, as were two of the quasiexperimental studies [37,39] . Although none of the randomized trials evaluate possible effects on health-care practitioners, the study by Van Rooyen et al. [28] was undertaken with health researchers. The quasiexperimental studies tend to have been conducted before both the RCTs and observational studies, for which the clear majority of both types of studies have been reported within the past decade. The oldest included study was published approximately 25 years ago [35] . Four of the 5 quasiexperimental studies used some form of quasirandomized methods in constructing control groups (except Ref. [36] ). Heterogeneity in operationalization of the Hawthorne effectterm·dom for dedicated evaluations, in study populations, settings, and in other ways, is readily apparent in Tables 1-3 .

PRISMA flowchart.

Study characteristics and findings of randomized controlled trials evaluating the Hawthorne effectterm·dom Study characteristics and findings of quasiexperimental evaluations of the Hawthorne effectterm·dom Study characteristics and findings of observational study evaluations of the Hawthorne effectterm·dom Fourteen of the 19 included studies report evaluations of effects on binary outcome measures. These data are presented in Fig. 2 . The first six studies presented in Fig. 2 comprise six of the seven (not including Ref. [32] ) evaluations of the effects of reporting on one's behaviorterm·dom by answering questions either in interviews or by completing questionnaires. All other studies evaluate being directly observed and/or the awareness of being studied in various ways, apart from one study that combines both types of Hawthorne effectterm·dom manipulation [33] .

Binary outcome data. RR, relative risk; CI, confidence interval.

As a result of heterogeneity in definitions of the Hawthorne effectterm·dom (reflecting the inclusion criteria), findings from meta-analytic syntheses should be treated with caution. Explorations of the extent and effects of heterogeneity are presented in Table 4 . Pronounced effects of statistical heterogeneity are reflected in the I 2 statistics for two of the three study designs (RCTs and observational studies) and also when attention is restricted to the eight studies of being observed or studied and to the subset of six studies of answering questions, and overall. When the one interview study (of preelection interview effects on voter turnout [27] ) is removed, however, to leave five studies of the effects of self-completing questionnaires on health behaviorsterm·dom, statistical heterogeneity is markedly attenuated.

Extent and effects of heterogeneity in 14 studies of the Hawthorne effectterm·dom with binary outcomes a Bearing these explorations of heterogeneity in mind, effect estimates provide a confidence interval (CI) including unity for the five trials alone [odds ratio (OR), 1.06; 95% CI: 0.98, 1.14], and for the six studies of answering questions (OR, 1.07; 95% CI: 1.00, 1.15). They reach statistical significance in relation to the five studies of self-completing health questionnaires (OR, 1.11; 95% CI: 1.0, 1.23). The pooled estimate for the five quasiexperimental studies is similar to that for the five trials and is not statistically significant (OR, 1.07; 95% CI: 0.99, 1.17), whereas that for the four observational studies (OR, 1.29; 95% CI: 1.06, 1.30) and the eight studies of being observed (OR, 1.21; 95% CI: 1.03, 1.41) are larger and statistically significant. The overall odds ratio, without any weighting for study design, was 1.17 (95% CI: 1.06, 1.30).

Quantitative outcome data were presented in three of the other five studies; two identifying between-group differences [29,32] and one not [28] . The large effect in the study by Feil et al. [29] is noteworthy. In the remaining two studies, continuous measures of effect were not reported in the form of mean differences and were complex to interpret, although both reported statistically significant Hawthorne effectterm·dom findings [40,43] . Continuous outcomes were also evaluated for two studies included in Fig. 1 , with both finding evidence of statistically significant effects [33,35] .

Of the 19 studies, therefore, 12 provided at least some evidence of the existence of a Hawthorne effectterm·dom, however this was defined, to a statistically significant degree. Small sample sizes appeared to preclude between-group differences reaching statistical significance in two studies [31,36] . In five studies, it was judged clear that there were no between-group differences that could represent a possible Hawthorne effectterm·dom [28,37-39,45] .

Discussion The Hawthorne effectterm·dom has been operationalized for study as the effects of reporting on one's behaviorterm·dom by answering questions, being directly observed, or otherwise made aware of being studied. There is evidence of effects across these studies, and inconsistencies in this evidence. We explore heterogeneity in targets for, and methods of, study as well as in findings. We will begin by examining the evidence base for each of the study designs, before considering the limitations, interpretation, and implications of this study.

The RCTs tend to provide evidence of small statistically significant effects. There are also studies that showed no effects, and two studies that provided evidence of large effects [29,34] . The study by Feil et al. [29] used a strong manipulation, incorporating a placebo effectterm·dom, which is not usually considered to be a Hawthorne effectterm·dom component, in addition to research- and trial-specific participation effects. In both this study and the one undertaken by Evans et al. [34] also finding a large effect, small numbers of participants are involved. The diversity in the content of the manipulations in these studies is emphasized. When one considers the RCT data from the five studies contributing to the meta-analysis [27,30,31,33,34] alongside the three studies that did not, two of which produce statistically significant effects on continuous outcomes [29,32] , it seems that overall, there is evidence of between-group differences in the RCTs. These between-group differences cannot, however, be interpreted to provide consistent or coherent evidence of a single effect.

The same could be said of the diversity of the contents of the manipulations in the quasiexperimental studies, and the picture is made more complex by greater variability in study design features, particularly in relation to allocation methods. Overall, they produce mixed evidence, with a between-group difference in one study [35] , a noteworthy difference in a small underpowered study [36] , and no evidence of between-group differences in the other three studies [37-39] . It is difficult to draw any conclusion from included studies with quasiexperimental designs.

Although their design precludes strong conclusions because of the likelihood of unknown and uncontrolled biases, the heterogeneity of findings in the observational studies is interesting. Two studies produce identical point estimates of effects [41,42] , whereas the other two estimates are similarly different [44,45] . These data suggest that the size of any effects of health-care practitioners being observed or being aware of being studied probably very much depends on what exactly they are doing. Perhaps, this is not at all surprising, although it does undermine further the idea that there is a single effect, which can be called the Hawthorne effectterm·dom. Rather, the effect, if it exists, is highly contingent on task and context. It is noteworthy that the three other studies with health professionals, all using control groups, show no effects [28,37,39] .

This is an "apples and oranges" review. This approach was judged appropriate, given the current level of understanding of the phenomena under investigation. This design decision does, however, entail limitations in the forms of important differences between studies, including operationalization of the Hawthorne effectterm·dom, and exposure to varying forms of bias. The observed effects are short term when a follow-up study is involved, with only the studies by Murray et al. [35] and Kypri et al. [32] demonstrating effects beyond 6 months. Both these studies involved repeated prior assessments, and both provide self-reported outcome data. Self-reported outcomes do not appear obviously more likely than objectively ascertained outcomes to show effects. The forms of blinding used are often tailored to the nature of the study, making performance bias prevention difficult to evaluate across the studies as a whole.

By design, we have excluded studies that defined the object of evaluation to incorporate nonresearch elements as occurred in the original studies at Hawthorne [8] . We may have missed studies that should have been identified, although this is unlikely if use of the Hawthorne effectterm·dom term was in any way prominent. Studies that have been missed may be more likely to be older and from nonhealth literature. An alternative design for this study might have eschewed this label and sought instead to synthesize findings on studies of research participationterm·dom and/or awareness of being studied. Although potentially attractive, this course of action would have involved considerable difficulties in identifying relevant material and would risk losing the main focus on the Hawthorne effectterm·dom. Similarly, we might have also included studies with cognitive and/or emotional outcomes in which effects might be greater [46] , rather than focusing on behavioral outcomes. This possibility may be appropriate for evaluation in the future. Although we have sought to make our explorations of the heterogeneity of included studies as informative as possible, our analyses might be seen as excessive data fishing. These are, however, clearly presented as post hoc analyses after examination of high levels of heterogeneity, and the study by Granberg and Holmberg [27] is distinct from the other four questionnaire studies in a range of ways including the behaviorterm·dom being investigated.

Heterogeneity in operationalization of the Hawthorne effectterm·dom make the data in this review challenging to interpret, yet it does appear that research participationterm·dom can and does influence behaviorterm·dom, at least in some circumstances. The content and strength of the Hawthorne effectterm·dom manipulations vary in these primary studies, and so to do the effects, although it would not be possible to discern any form of dose-response relationship. The manipulation by Evans et al. [34] may appear in some respects weak, for example, being an online questionnaire, and in others as potentially strong, examining decision making in relation to uptake of a cancer test, which was the study outcome. Although weak uses of the Hawthorne effectterm·dom term in the wider literature mean that it is not very informative for interpreting the data from this study, outcomes may be considered in relation to the prevailing ideas about the core mechanism of the Hawthorne effectterm·dom that conformity to perceived norms or researcher expectations drives change. Many, but not all, of the studies with positive findings appear broadly consistent with this account, although so too do many of the studies with negative findings and it is not clear why this is so. The study by Murray et al. [35] examined adolescent smoking at a time when the prevalences of both nonsmoking and regular smoking were approximately one-quarter in this sample, so it is unclear what norms or perceptions of researcher expectations many have been. This study exemplifies the literature as a whole in being principally concerned with the possible existence of a Hawthorne effectterm·dom and not being designed to test the hypothesized mechanism.

There are other possible mechanisms of effect that have also not been evaluated. For example, regardless of perceptions of norms or researcher expectations, the content of the questions asked may themselves stimulate new thinking. In the studies by Evans et al. [34] and O'Sullivan et al. [30] , patients may well not have previously considered what was being enquired about, and this may have been an independent source of change. Concerns about biases being introduced by having research participants complete questionnaires have existed for more than 100 years, before the Hawthorne factory studies took place and approximately 50 years before the Hawthorne effectterm·dom term was introduced to the literature (see Ref. [47] for an early history of these issues). Given how long the Hawthorne effectterm·dom construct has been the predominant conceptualization of these phenomena [48] , it appears that this construct is an inadequate vehicle for advancing understanding of these issues. Alternative long-standing conceptualizations of these problems such as demand characteristicsterm·dom within psychology have also yielded disappointingly underdeveloped research literatures [49-51] . This state of affairs points toward an obvious need for further study of whether, when, how, how much, and for whom research participationterm·dom may impact on behaviorterm·dom or other study outcomes.

Further studies will be assisted by the development of a conceptual framework that elaborates possible mechanisms of effects and thus targets for study. The Hawthorne effectterm·dom label has probably stuck for so long simply because we have not advanced our understanding of the issues it represents. We suggest that unqualified use of the term should be abandoned. Specification of the research issues being investigated or described is paramount, regardless of whether the Hawthorne label is seen to be useful or to apply or not in any particular research context. Perhaps, use of the label should be restricted to evaluations in which conformity and social desirability considerations are involved, although it is striking how hostile social psychology has been to this construct [2] . So, what can be said about priority targets for further study on the basis of this systematic reviewterm·key and what concepts are available to guide further study?

Decisions to take part in research studies may also be implicated in efforts to address behaviorterm·dom in other ways so that research participationterm·dom interacts with other forces influencing behaviorterm·dom. It is also possible, if not likely, that these relatively well-studied types of data collection (completing questionnaires and being observed) are part of a series of events that occur for participants in research studies that have potential to shape their behaviorterm·dom, from recruitment onwards. Giving attention to precisely what we invite research participants to do in any given study seems a logical precursor to examination of whether any aspect of taking part may influence them. Phenomenological studies, which ask participants about their experiences, would seem to be useful for developing new concepts. If individual study contexts are indeed important, we should expect to see effects that vary in size and across populations and research contexts, and perhaps also with multiple mechanisms of effects. The underdeveloped nature of these types of research questions means that it may be unwise to articulate advanced conceptual frameworks to guide empirical study. We propose "research participation effectsterm·dom" as a starting point for this type of thinking. Although descriptive, it also invites investigation of other aspects of the research process beyond data collection, which may simply be where research artifactsterm·key emanating from both social norms and other sources are most obvious.

We conclude that there is no single Hawthorne effectterm·dom. Consequences of research participationterm·dom for behaviorsterm·dom being investigated have been found to exist in most studies included within this review, although little can be securely known about the conditions under which they operate, their mechanisms of effects, or their magnitudes. Further research on this subject should be a priority for the health sciences, in which we might expect change induced by research participationterm·dom to be in the direction of better health and thus likely to be confounded with the outcomes being studied. It is also important for other domains of research on human behaviorterm·dom to rectify the limited development of understanding of the issues represented by the Hawthorne effectterm·dom as they suggest the possibility of profound biases.

Acknowledgments This study was undertaken under the auspices of a Wellcome Trust Research Career Development award in Basic Biomedical Science to the first author (WT086516MA). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The authors are grateful to Serena Luchenski for research assistance in connection with this study.

Footnotes References