What is a pilot or feasibility study? A review of current practice and editorial policy

What is a pilot or feasibility study? A review of current practice and editorial policy Mubashir Arain Michael J. Campbell Cindy Cooper Gillian Lancaster

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

What is a pilot or feasibility study? A review of current practice and editorial policy 2010 BMC Medical Research Methodology 10.1186/1471-2288-10-67 arain2010-what-is-a-pilot-or-feasibility-stu arain2010-what-is-a-pilot-or-feasibility-stu.txt, the project's staged copy 6d613a860454b3e3d1f02c2097f9055f8377abe9 cc-by statistics

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

59 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them TWO LABELS AND WHAT THEY BUY - 'pilot' and 'feasibility' measured across 54 published studies, 12 registered studies, three funding bodies and seven journal editors, six years before the same team's framework abandoned the distinction they recommend here when it is legitimate to label a study a 'pilot' or a 'feasibility' study, and what a reviewer should expect to find inside one evidence from four independent sources - published papers, a registered-study database, funder policy and journal editors - that the two labels are used inconsistently, plus a recommendation to adopt the NETSCC definitions critique a label that authors reach for to protect an underpowered study and that editors read as a warning, attached to studies whose promised follow-up happens eight times in ninety reviewreview THE LABEL IS A SHIELD FOR AUTHORS AND A LIABILITY AT JOURNALS, IN ONE PAPER: `small studies with all the trappings of a major study ... may be labeled a `PILOT` because they do not have the power to test clinically meaningful hypotheses. The authors of such studies perhaps hope that reviewers will regard a `PILOT` more favourably than a small clinical trial` - against `In general they were loathe to publish studies described as `PILOT``. The same word is worth more to the author and less to the editor AND THE NAMING ACT IS GIVEN AN ETHICS RULING: `It is generally accepted that small, underpowered clinical trials are unethical. Thus it is not an excuse to label such a study as a pilot and hope to make it ethical.` A sentence about what a word cannot do - the corpus has definitions that decide funding and diagnosis, and this is the first that decides whether a study should have been run at all AND THE PROMISE IS KEPT EIGHT TIMES IN NINETY: `Lancaster et al found that 50% of pilot studies reported the intention of further work yet we identified only 8 (8.8%) which were followed up by a major study` - and `one would have expected about 45 ... whereas we only found 8`. The defining property of a pilot study is that a bigger study follows, and it does not AND THE REASON GIVEN INVERTS THE PURPOSE: `if the pilot shows significant results then researchers may not find it necessary to conduct the main trial. In addition if the results are unfavorable or the authors find an unfeasible procedure, the main study is less likely to be considered useful.` A positive result cancels the trial and a negative result cancels the trial - which leaves no outcome under which the study does what its name says AND THREE PRIOR AUTHORITIES TREAT THE SECOND WORD THREE WAYS: Arnold et al `made useful distinctions between: pilot work which is any background research ..., a pilot study which has specific hypotheses, objectives and methodology and a pilot trial which is a stand-alone pilot study and includes a randomization procedure. They excluded feasibility studies from their consideration` - while `Thabane et al ... INCLUDED `FEASIBILITY` OR `VANGUARD` studies but did not distinguish them from pilot studies`. Excluded, conflated, or split three ways AND THE LABELS DO TRACK THE METHOD, IN THE PUBLISHED LITERATURE: `We found that studies labeled `FEASIBILITY` were conducted with more flexible methodology compared to those labeled `PILOT` ... `pilot` studies were reported with more rigorous methodological components like sample size estimation, randomization and control group selection than studies labeled `feasibility`.` Four of six components differ significantly by keyword alone AND THEY DO NOT, IN THE REGISTERED-STUDY DATABASE: `both pilot and feasibility studies included trials run as complete studies, including randomization, but with sample sizes smaller than would be intended in the full study and the distinction between the two terms was not clear-cut.` Two data sources in one paper, opposite answers to whether the words mean anything - the published literature separates them by keyword and the registry does not, and the paper reports both AND THE DEFINITION THIS PAPER RECOMMENDS IS THE ONE ITS OWN AUTHORS LATER ABANDON: `We found the NETSCC definition to be the most helpful since it distinguishes between these types of study and to most closely mirror what investigators are doing.` Michael J. Campbell and Gillian A. Lancaster are authors here and on eldridge2016, which opens `We initially adopted MUTUALLY EXCLUSIVE DEFINITIONS ... based on ... THE DEFINITIONS USED BY THE NIHR` and reports the panel voting them down 14-1 AND THE RECOMMENDED DEFINITION DRAWS ITS LINE AT THE OUTCOME: `Crucially, feasibility studies do not evaluate the outcome of interest; that is left to the main study. If a feasibility study is a small randomised controlled trial, it need not have a primary outcome and the usual sort of power calculation is not normally undertaken.` A class defined by what it must NOT measure AND THE OTHER HALF DEFINES BY RESEMBLANCE: `A Pilot Study is a version of the main study that is run in miniature to test whether the components of the main study can all work together ... It will therefore resemble the main study in many respects.` One class bounded by an omission, its sibling by a likeness - and eldridge2016 later keeps the likeness as the only differentia AND PRACTICE IGNORED THE 2004 RECOMMENDATION IT WAS GIVEN: `Some authors suggest that analysis of pilot studies should mainly be descriptive, as hypothesis testing requires a powered sample size which is usually not available ... However most of the pilot interventional studies in this review contained a control group and the authors performed and reported hypothesis testing ... Observed practice is not necessarily good practice.` A repeat review measuring whether its own field took the advice, and finding it did not AND A CONSENT CONSEQUENCE FALLS OUT OF THE DEFINITION: `Participants in such studies should be informed that they are in a pilot study and that there may not be a further larger study.` If the defining promise is kept 8 times in 90, then what a participant is told about why they are enrolled depends on a definition nobody in the field shares THE EDITOR SURVEY IS FIVE REPLIES OUT OF SEVEN AND THE AUTHORS SAY SO TWICE: `The editors of five (out of seven) medical journals responded`, and later `The views of the journal editors, albeit from a small sample, were not particularly encouraging`. Four of the five had no written policy at all, so the finding is about absence of policy as much as about opinion AND THE REGISTRY ARM IS TWELVE STUDIES: of 34 identified, `Only 12 studies were interventional pilot/feasibility studies which included testing of some component of the research process. Of these 5 were referred to as `feasibility`, 6 as `pilot` and 1 as both` - and the component counts that follow (17 across 6 pilots against 6 across 5 feasibility studies) rest on those twelve AND THE PAPER'S OWN NUMBER CELLS ARE UNRELIABLE, WHICH IS NOT OUR DAMAGE: Table 3 reports participant ranges of `[8, 187777]` and `[5, 12774614]` - twelve million participants in a feasibility study - and Table 2 gives `76 [8, 1299]` under a header saying (IQR) while the prose says `median 76, INTER-QUARTILE RANGE (42, 216)`. Checked against the publisher's own XML: each is a SINGLE `<td>` cell, so the fusion is in the source. No quantity from Tables 2-4 is tagged on this record; the percentages ARE arithmetically consistent with their counts and only the participant-range cells are broken AND THE FIELD'S PUBLICATION FILTER WORKS AGAINST THE EVIDENCE THE PAPER WANTS: `Pilot studies, by their nature, will not produce `SIGNIFICANT` (i.e P < 0.05) RESULTS. We believe that publishing the results of well conducted pilot or feasibility studies is important for research, IRRESPECTIVE OF OUTCOME. There is an increasing awareness that PUBLISHING ONLY `SIGNIFICANT` results can lead to considerably error.` A class of study defined so that it cannot meet the criterion its journals apply repeating a prior review's search terms, journals and time window exactly, and reporting the two periods side by sideeliminates: attributing a change in practice to a change in the reviewer's method · outcome: held adding a registered-study database arm, because 'as many pilot studies are never published' the journal literature is a filtered sampleeliminates: publication bias in the sample of studies whose labels are being compared · outcome: partial canvassing the journal editors and funding bodies directly rather than inferring policy from what was publishedeliminates: reading an editorial policy off the papers that survived it · outcome: partial pilot trialpilot trial pilot studypilot study pilot studiespilot studies RCTsRCTs feasibility studiesfeasibility studies the outcome of interestthe outcome of interest pilotpilot small clinical trialsmall clinical trial feasibilityfeasibility `pilot` (the label, as authors use it) - reached for when a study `DOES NOT HAVE THE POWER to test clinically meaningful hypotheses`, in the hope that reviewers `WILL REGARD A `PILOT` more favourably than a small clinical trial` `pilot study` (NETSCC, recommended here) - `a version of the main study that is run in miniature to test whether the components of the main study can all work together`; internal if its data join the final analysis, external if set aside `feasibility study` (NETSCC) - research done before a main study to estimate design parameters, which `Crucially ... do not evaluate the outcome of interest` `pilot work` / `pilot study` / `pilot trial` (Arnold et al) - a three-way split: any background research, a study with hypotheses and methodology, and a stand-alone pilot study with randomisation replicates → Lancaster et al's 2004 review of pilot studies in the same seven journals for 2000-01, repeated here for 2007-08 to test whether its recommendations changed practice critiques → the reporting of pilot studies with hypothesis testing and control arms, against the 2004 recommendation that their analysis be mainly descriptive critiques → Thabane et al's checklist, which `INCLUDED `FEASIBILITY` OR `VANGUARD` studies but did not distinguish them from pilot studies` and was `opinion based and not supported by a review of current practice` pilot trial —is-a→ pilot studythe relation itself is separately evidenced · strength of the relation: 0.95 · note: 'They made useful distinctions between: pilot work which is any background research to inform a future study, a pilot study which has specific hypotheses, objectives and methodology and a pilot trial which is a stand-alone pilot study and includes a randomization procedure.' Three terms sharing one word, nested, from a review that then 'excluded feasibility studies from their consideration' entirely - so the nesting is complete only within the half of the vocabulary they kept · the relation stated: a pilot trial is a stand-alone pilot study that additionally includes a randomisation procedurepilot trialpilot study pilot studies —distinct-from→ RCTsthe relation itself is separately evidenced · strength of the relation: 0.95 · note: 'We have shown that pilot studies have different objectives to RCTs and these should be clearly described.' The consequence is a consent one and the paper draws it: 'Participants in such studies should be informed that they are in a pilot study and that there may not be a further larger study' - which matters because the follow-up happened 8 times in 90 · the relation stated: pilot studies have different OBJECTIVES from the trials they precede, and the paper's complaint is that they are reported as though they did notpilot studiesRCTs pilot —contrasted-with→ small clinical trialthe relation itself is separately evidenced · strength of the relation: 0.95 · note: `The authors of such studies perhaps hope that reviewers will regard a `PILOT` more favourably than a small clinical trial.` An edge between two names for one object, where the difference is in the READER not the study - and the editor survey finds the bet is wrong: `In general they were loathe to publish studies described as `PILOT`` · the relation stated: the same study under two names - and the paper's opening claim is that authors choose the first expecting a softer review Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the contrast is what authors 'perhaps hope' reviewers will see; attributed and hedged.pilotsmall clinical trial feasibility —contrasted-with→ pilotthe relation itself is separately evidenced · strength of the relation: 0.95 · note: `We found that studies labeled `FEASIBILITY` were conducted with more flexible methodology compared to those labeled `PILOT` ... `pilot` studies were reported with more rigorous methodological components like sample size estimation, randomization and control group selection.` And the registry arm of the same paper says the opposite: `the distinction between the two terms was not clear-cut`. Both are true - the group means separate and the individual cases do not - and eldridge2016 later reports only the second · the relation stated: studies keyworded 'feasibility' are run with more flexible methodology than those keyworded 'pilot' - the labels predict the design at the group levelfeasibilitypilot small underpowered trials may be labelled 'pilot', perhaps in the hope that reviewers view them more favourably than a small trialsmall underpowered trials may be labelled 'pilot', perhaps in the hope that reviewers view them more favourably than a small trial while 50% of pilot studies reportedly intend further work, only 8 (8.8%) were followed by a major studywhile 50% of pilot studies reportedly intend further work, only 8 (8.8%) were followed by a major study AND BOTH OUTCOMES CANCEL THE MAIN TRIAL: `if the pilot shows significant results then researchers may not find it necessary to conduct the main trial ... if the results are unfavorable ... the main study is less likely to be considered useful`AND BOTH OUTCOMES CANCEL THE MAIN TRIAL: `if the pilot shows significant results then researchers may not find it necessary to conduct the main trial ... if the results are unfavorable ... the main study is less likely to be considered useful` both pilot and feasibility studies included complete small randomised trials, and the distinction between the terms was not clear-cutboth pilot and feasibility studies included complete small randomised trials, and the distinction between the terms was not clear-cut AND THE NAMING ACT GETS AN ETHICS RULING: `it is not an excuse to label such a study as a pilot and hope to make it ethical`AND THE NAMING ACT GETS AN ETHICS RULING: `it is not an excuse to label such a study as a pilot and hope to make it ethical` the authors found the NETSCC definition most helpful because it distinguishes pilot from feasibility studiesthe authors found the NETSCC definition most helpful because it distinguishes pilot from feasibility studies AND A CONSENT CONSEQUENCE: `PARTICIPANTS ... should be informed that they are in a pilot study and that there may not be a further larger study`AND A CONSENT CONSEQUENCE: `PARTICIPANTS ... should be informed that they are in a pilot study and that there may not be a further larger study` `a version of the main study that is run in miniature to test whether the components of the main study can all work together ... It will therefore resemble the main study in many respects`pilot study `pieces of research done before a main study ... used to estimate important parameters ... Crucially, feasibility studies do not evaluate the outcome of interest`feasibility study and 2 of those 8 appeared in the same journal as the pilotvalue source note: THE VALUE DRAWS ON THE PAPER BEYOND THIS QUOTE: part of it is printed elsewhere in the paper, not in this quote ('Only 8 out of 90 pilot studies identified by the earlier'). The 45 is half of 90, from the 50% the quote reports.pilot studies followed by a main study the pilot label attaches to studies testing more of the main trial at oncemethodological components under investigation, registry arm `Most of the other journals do not encourage the publication of pilot studies because they consider them less rigorous than main studies`journal editors responding, and their policies `Pilot studies, by their nature, will not produce `SIGNIFICANT` (i.e P < 0.05) RESULTS` - and most editors `do not encourage the publication of pilot studies`. The definition and the publication filter are in direct conflict, and the paper says so in the same paragraph as `publishing only `SIGNIFICANT` results can lead to considerably error`a class of study defined so that it cannot meet its journals' criterion the only variable in the comparison - studies were split by which of two words appeared in the title or abstract, and then compared on sample size estimation, hypothesis testing, randomisation, blinding, control group and stated intent to continuethe keyword

43 bmcmrm BMC Medical Research Methodology BMC Med Res Methodol BMC PMC2912920 2912920 2912920 20637084 10.1186/1471-2288-10-67 What is a pilot or feasibility study? A review of current practice and editorial policy Arain Mubashir 1 Campbell Michael J 1 ✉ Cooper Cindy L 1 Lancaster Gillian A 2 1 Health Services Research, ScHARR, University of Sheffield, Regent Court, Regent St Sheffield S1 4DA, UK 2 Department of Mathematics and Statistics, University of Lancaster LA1 4YF, UK ✉ Corresponding author.

16 7 2010 10 67 67 31 7 2010 Copyright ©2010 Arain et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract

Background

In 2004, a review of pilot studies published in seven major medical journals during 2000-01 recommended that the statistical analysis of such studies should be either mainly descriptive or focus on sample size estimation, while results from hypothesis testing must be interpreted with caution. We revisited these journals to see whether the subsequent recommendations have changed the practice of reporting pilot studies. We also conducted a survey to identify the methodological components in registered research studies which are described as 'pilot' or 'feasibility' studies. We extended this survey to grant-awarding bodies and editors of medical journals to discover their policies regarding the function and reporting of pilot studies.

Methods

Papers from 2007-08 in seven medical journals were screened to retrieve published pilot studies. Reports of registered and completed studies on the UK Clinical Research Network (UKCRN) Portfolio database were retrieved and scrutinized. Guidance on the conduct and reporting of pilot studies was retrieved from the websites of three grant giving bodies and seven journal editors were canvassed.

Results

54 pilot or feasibility studies published in 2007-8 were found, of which 26 (48%) were pilot studies of interventions and the remainder feasibility studies. The majority incorporated hypothesis-testing (81%), a control arm (69%) and a randomization procedure (62%). Most (81%) pointed towards the need for further research. Only 8 out of 90 pilot studies identified by the earlier review led to subsequent main studies. Twelve studies which were interventional pilot/feasibility studies and which included testing of some component of the research process were identified through the UKCRN Portfolio database. There was no clear distinction in use of the terms 'pilot' and 'feasibility'. Five journal editors replied to our entreaty. In general they were loathe to publish studies described as 'pilot'.

Conclusion

Pilot studies are still poorly reported, with inappropriate emphasis on hypothesis-testing. Authors should be aware of the different requirements of pilot studies, feasibility studies and main studies and report them appropriately. Authors should be explicit as to the purpose of a pilot study. The definitions of feasibility and pilot studies vary and we make proposals here to clarify terminology.

status released display-pdf yes is-olf no is-manuscript no is-preprint no is-journal-matter no is-scanned no is-retracted no Received 2010 May 20; Accepted 2010 Jul 16; Collection date 2010.

Background

A brief definition is that a pilot study is a 'small study for helping to design a further confirmatory study'[ 1 ]. A very useful discussion of exactly what is a pilot study has been given by Thabane et al. [ 2 ] Such kinds of study may have various purposes such as testing study procedures, validity of tools, estimation of the recruitment rate, and estimation of parameters such as the variance of the outcome variable to calculate sample size etc. In pharmacological trials they may be referred to as 'proof of concept' or Phase I or Phase II studies. It has become apparent to us when reviewing research proposals that small studies with all the trappings of a major study, such as randomization and hypothesis testing may be labeled a 'pilot' because they do not have the power to test clinically meaningful hypotheses. The authors of such studies perhaps hope that reviewers will regard a 'pilot' more favourably than a small clinical trial. This lead us to ask when it is legitimate to label a study as a 'pilot' or 'feasibility' study, and what features should be included in these types of studies.

Lancaster et al [ 3 ] conducted a review of seven major medical journals in 2000-1 to produce evidence regarding the components of pilot studies for randomized controlled trials. Their search included both 'pilot' and 'feasibility' studies as keywords. They reported certain recommendations: having clear objectives in a pilot study, inappropriateness of mixing pilot data with main research study, using mainly descriptive statistics obtained and caution regarding the use of hypothesis testing for conclusions. Arnold et al [ 1 ] recently reviewed pilot studies particularly related to critical care medicine by searching the literature from 1997 to 2007. They provided narrative descriptions of some pilot papers particularly those describing critical care medicine procedures. They pointed out that few pilot trials later evolved into subsequent published major trials. They made useful distinctions between: pilot work which is any background research to inform a future study, a pilot study which has specific hypotheses, objectives and methodology and a pilot trial which is a stand-alone pilot study and includes a randomization procedure. They excluded feasibility studies from their consideration.

Thabane et al [ 2 ] gave a checklist of what they think should be included in a pilot study. They included 'feasibility' or 'vanguard' studies but did not distinguish them from pilot studies. They provided a good discussion on how to interpret a pilot study. They stress that not only the outcome or surrogate outcome for the subsequent main study should be described but also that a pilot study should have feasibility outcomes which should be clearly defined and described. Their article was opinion based and not supported by a review of current practice.

The objective of this paper is to provide writers and reviewers of research proposals with evidence from a variety of sources for which components they should expect, and which are unnecessary or unhelpful, in a study which is labeled as a pilot or feasibility study. To do this we repeated Lancaster et al's [ 3 ] review for current papers see if there has been any change in how pilot studies were reported since their study. As many pilot studies are never published we also identified pilot studies which were registered with the UK Clinical Research Network (UKCRN) Portfolio Database. This aims to be a "complete picture of the clinical research which is currently taking place across the UK". All studies included have to have been peer reviewed through a formal independent process. We examined the websites of some grant giving bodies to find their definition of a pilot study and their funding policy toward them. Finally we contacted editors of leading medical journals to discover their policy of accepting studies described as 'pilot' or 'feasibility'.

Methods

Literature survey

MEDLINE, Web of Science and university library data bases were searched for the years 2007-8 using the same key words "Pilot" or "Feasibility" as used by Lancaster et al. [ 3 ]. We reviewed the same four general medicine journals: the British Medical Journal (BMJ), Lancet, the New England Journal of Medicine (NEJM) and the Journal of American Medical Association (JAMA) and the same three specialist journals: British Journal of Surgery (BJS), British Journal of Cancer (BJC), British Journal of Obstetrics and Gynecology (BJOG). We excluded review papers. The full text of the relevant papers was obtained. GL reviewed 20 papers and classified them into groups as described in her original paper [ 3 ]. Subsequently MA, in discussion with MC, designed a data extraction form to classify the papers. We changed one category from GL's original paper. We separated the category 'Phase I/II trials' from the 'Piloting new treatment, technique, combination of treatments' category. We then classified the remaining paper into the categories described in Table 1 . The total number of research papers by journal was obtained by searching journal article with abstracts (excluding reviews) using Pubmed. We searched citations to see whether the pilot studies identified by Lancaster et al [ 3 ] eventually led to main trials.

Table 1 Literature search using key words "Pilot" OR "Feasibility"

Journal Name

BMJ Lancet NEJM JAMA BJS BJC BJOG Total

2007-8

Original articles 6 5 5 1 10 16 10 54 1 (1.6%)

Pilot or feasibility study in preparation for a trial 0 3 3 0 2 1 3 12

Piloting new technique, combination of treatments 1 0 0 0 4 2 6 13

Phase I, II trials 0 1 1 0 0 7 0 9

Piloting screening program 2 0 0 1 1 1 0 5

Piloting guidelines, educational package, patient care strategy 3 1 0 0 2 3 1 10

Laboratory testing of activity of compounds 0 0 1 0 1 2 0 4

Total research papers 292 379 444 383 338 1084 398 3318

2000-1 2

Original articles 11 17 3 7 9 33 10 90 (2.0%)

Total research papers 372 1115 434 619 396 1132 381 4449

1 excluded Review = 8, Commentaries = 4, News = 3, Indirectly referring to previous pilot = 9

2 from Lancaster et al [ 1 ]

Portfolio database review

The (UKCRN) Portfolio Database was searched for the terms 'feasibility' or 'pilot' in the title or research summary. Duplicate cases and studies classified as 'observational' were omitted. From the remaining studies those classified as 'closed' were selected to exclude studies which may not have started or progressed. Data were extracted directly from the research summary of the database or where that was insufficient the principle investigator was contacted for related publications or study protocols.

Editor and funding agency survey

We wrote to the seven medical journal editors of the same journals used by Lancaster et al. [ 3 ], (BMJ, Lancet, NEJM, JAMA. BJS, BJC and BJOG) and looked at the policies of three funding agencies (British Medical Research Council, Research for Patient Benefit and NETSCC (National Institute for Health Research Trials and Studies Coordinating Centre). We wished to explore whether there was any specified policy of the journal for publishing pilot trials and how the editors defined a pilot study. We also wished to see if there was funding for pilot studies.

Results

Literature survey

Initially 77 papers were found in the target journals for 2007-8 but 23 were review papers or commentaries or indirectly referred to the word "pilot" or "feasibility" and were not actually pilot studies leaving a total of 54 papers. Table 1 shows the results by journal and by type of study and also shows the numbers reported by Lancaster et al. [ 3 ] for 2000-01 in the same medical journals. There was a decrease in the proportion of pilot studies published over the period of time, however the difference was not statistically significant (2.0% vs 1.6%; X 2 = 1.6, P = 0.2). It is noticeable that the Phase I or Phase II studies are largely confined to the cancer journals.

Lancaster et al [ 3 ] found that 50% of pilot studies reported the intention of further work yet we identified only 8 (8.8%) which were followed up by a major study. Of these 2 (25%) were published in the same journal as the pilot.

Twenty-six of the studies found in 2007-8 were described as pilot or feasibility studies for randomized clinical trials (RCTs) including Phase II studies. Table 2 gives the numbers of studies which describe specific components of RCTs. Sample size calculations were performed and reported in 9 (36%) of the studies. Hypothesis testing and performing inferential statistics to report significant results was observed in 21 (81%) of pilot studies. The processes of blinding was observed in only 5 (20%) although the randomization procedure was applied or tested in 16 (62%) studies. Similarly a control group was assigned in most of the studies (n = 18; 69%). As many as 21 (81%) of pilot studies suggested the need for further investigation of the tested drug or procedure and did not report conclusive results on the basis of their pilot data. The median number of participants was 76, inter-quartile range (42, 216).

Table 2 Literature survey: Frequency of methodological components appearing in pilot or feasibility studies of interventions (n = 26 1 ) in 2007-8

n (%)

Sample size calculation 9 (35%)

Hypothesis testing 21 (81%)

Randomization 16 (62%)

Blinding 5 (19%)

Control group 18 (69%)

Further study suggested 21 (81%)

Median number of participants (IQR) 76 [8, 1299]

1 Pilot studies = 14 Feasibility studies = 12

Of the 54 studies in 2007-8, a total of 20 were described as 'pilot' and 34 were described as 'feasibility' studies. Table 3 contrasts those which were identified by the keyword 'pilot' with those identified by 'feasibility'. Those using 'pilot' were more likely to have a pre-study sample size estimate, to use randomization and to use a control group. In the 'pilot' group 16(80%) suggested further study, in contrast to 15 (44%) in the 'feasibility' group.

Table 3 Literature survey: Comparison of studies (n = 54) using the key words feasibility or pilot

Study components Pilot n = 20 Feasibility n = 34 Chi-squared 1 P-value

Sample size 7 (35%) 3 (8%) 5.7 0.028

Hypothesis testing 14 (70%) 25 (74%) 0.78 0.51

Randomization 11 (55%) 8 (24%) 5.5 0.037

Blinding 3 (15%) 3 (9%) 0.48 0.39

Control group 13 (65%) 11 (32%) 5.4 0.020

Further study suggested 16 (80%) 15 (44%) 6.6 0.012

Number of participants Median (IQR) [Range] 62.5 (31, 189) [8, 187777] 125.5 (36, 1005) [5, 12774614] -1.04* 0.29

1 1 degree of freedom

* z-statistic (Mann-Whitney test)

Portfolio database review

A total of 34 studies were identified using the term 'feasibility' or 'pilot' in the title or research summary which were prospective interventional studies and were closed, i.e. not currently running and available for analysis. Only 12 studies were interventional pilot/feasibility studies which included testing of some component of the research process. Of these 5 were referred to as 'feasibility', 6 as 'pilot' and 1 as both 'feasibility' and 'pilot' (Table 4 ).

Table 4 Portfolio database survey: comparison of components in studies termed pilot or feasibility

Pilot n = 6 Feasibility n = 5 Both n = 1

Methods related

Miniature RCT 4 3 0

Testing recruitment 4 1 0

Determining sample size/numbers available 3 0 1

Resources 1 0 0

Randomization 4 1 0

Outcome measures 2 4 2

Data collection 1 0 0

Follow up/dropout 2 0 0

Intervention related

Clinical outcomes 3 1 1

Dose/efficacy/safety 0 1 0

Acceptability 2 0 1

Feasibility 3 0 1

The methodological components tested within these studies were: estimation of sample size; number of subjects eligible; resources (e.g. cost), time scale; population-related (e.g. exclusion criteria), randomisation process/acceptability; data collection systems/forms; outcome measures; follow-up (response rates, adherence); overall design; whole trial feasibility. In addition to one or more of these, some studies also looked at clinical outcomes including: feasibility/acceptability of intervention; dose, efficacy and safety of intervention.

The results are shown in Table 4 . Pilot studies alone included estimation of sample size for a future bigger study and tested a greater number of components in each study. The majority of the pilots and the feasibility studies ran the whole study 'in miniature' as it would be in the full study, with or without randomization.

As an example of a pilot study consider 'CHOICES: A pilot patient preference randomised controlled trial of admission to a Women's Crisis House compared with psychiatric hospital admissions' http://www.iop.kcl.ac.uk/projects/default.aspx?id=10290 . This study looked at multiple components of a potential bigger study. It aimed to determine the proportion of women unwilling to be randomised, the feasibility of a patient preference RCT design, the outcome and cost measures to determine which outcome measures to use, the recruitment and drop out rates; and to estimate the levels of outcome variability to calculate sample sizes for the main study. It also intended to develop a user focused and designed instrument which is the outcome from the study. The sample size was 70.

Editor and funding agency survey

The editors of five (out of seven) medical journals responded to our request for information regarding publishing policy for pilot studies. Four of the journals did not have a specified policy about publishing pilot studies and mostly reported that pilot trials cannot be published if the standard is lower than a full clinical trial requirement. The Lancet has started creating space for preliminary phase I trials and set a different standard for preliminary studies. Most of the other journals do not encourage the publication of pilot studies because they consider them less rigorous than main studies. Nevertheless some editors accepted pilot studies for publication by compromising only on the requirement for a pre-study sample size calculation. All other methodological issued were considered as important as for the full trials, such as trial registration, randomization, hypothesis testing, statistical analysis and reporting according to the CONSORT guidelines.

All three funding bodies made a point to note that pilot and feasibility studies would be considered for funding. Thabane et al [ 2 ] provided a list of websites which define pilot or feasibility studies. We considered the NETSCC definition to be most helpful and to most closely mirror what investigators are doing and it is given below.

NETSCC definition of pilot and feasibility studies http://www.netscc.ac.uk/glossary/

Feasibility Studies

Feasibility Studies are pieces of research done before a main study. They are used to estimate important parameters that are needed to design the main study. For instance:

• standard deviation of the outcome measure, which is needed in some cases to estimate sample size,

• willingness of participants to be randomised,

• willingness of clinicians to recruit participants,

• number of eligible patients,

• characteristics of the proposed outcome measure and in some cases feasibility studies might involve designing a suitable outcome measure,

• follow-up rates, response rates to questionnaires, adherence/compliance rates, ICCs in cluster trials, etc.

Feasibility studies for randomised controlled trials may not themselves be randomised. Crucially, feasibility studies do not evaluate the outcome of interest; that is left to the main study.

If a feasibility study is a small randomised controlled trial, it need not have a primary outcome and the usual sort of power calculation is not normally undertaken. Instead the sample size should be adequate to estimate the critical parameters (e.g. recruitment rate) to the necessary degree of precision.

Pilot studies

A Pilot Study is a version of the main study that is run in miniature to test whether the components of the main study can all work together. It is focused on the processes of the main study, for example to ensure recruitment, randomisation, treatment, and follow-up assessments all run smoothly. It will therefore resemble the main study in many respects. In some cases this will be the first phase of the substantive study and data from the pilot phase may contribute to the final analysis; this can be referred to as an internal pilot. Alternatively at the end of the pilot study the data may be analysed and set aside, a so-called external pilot.

Discussion

In our repeat of Lancaster et al's study [ 3 ] we found that the reporting of pilot studies was still poor. It is generally accepted that small, underpowered clinical trials are unethical [ 4 ]. Thus it is not an excuse to label such a study as a pilot and hope to make it ethical. We have shown that pilot studies have different objectives to RCTs and these should be clearly described. Participants in such studies should be informed that they are in a pilot study and that there may not be a further larger study.

It is helpful to make a more formal distinction between a 'pilot' and a 'feasibility' study. We found that studies labeled 'feasibility' were conducted with more flexible methodology compared to those labeled 'pilot'. For example the term 'feasibility' has been used for large scale studies such as a screening programme applied at a population level to determine the initial feasibility of the programme. On the other hand 'pilot' studies were reported with more rigorous methodological components like sample size estimation, randomization and control group selection than studies labeled 'feasibility'. We found the NETSCC definition to be the most helpful since it distinguishes between these types of study.

In addition it was observed that most of the pilot studies report their results as inconclusive, with the intention of conducting a further, larger study. In contrast, several of the feasibility studies did not admit such an intention. On the basis of their intention one would have expected about 45 of the studies identified by Lancaster et al in 2000/1 to have been followed by a bigger study whereas we only found 8. This would reflect the opinion of most of the journal editors and experts who responded to our survey, who felt that pilot studies rarely act as a precursor for a bigger study. The main reason given was that if the pilot shows significant results then researchers may not find it necessary to conduct the main trial. In addition if the results are unfavorable or the authors find an unfeasible procedure, the main study is less likely to be considered useful. Our limited review of funding bodies was encouraging. Certainly when reviewing grant applications, we have found it helpful to have the results of a pilot study included in the bid. We think that authors of pilots studies should be explicit as to their purpose, e.g. to test a new procedure in preparation for a clinical trial. We also think that authors of proposals for pilot studies should be more explicit as to the criteria which lead to further studies being abandoned, and that this should be an important part of the proposal.

In the Portfolio Database review, only pilot studies cited an intention to estimate sample size calculations for future studies and the majority of pilot studies were full studies run with smaller sample sizes to test out a number of methodological components and clinical outcomes simultaneously. In comparison the feasibility studies tended to focus on fewer methodological components within individual studies. For example, the 6 pilot studies reported the intention to evaluate a total of 17 methodological components whereas in the 5 feasibility studies a total of only 6 methodological components were specifically identified as being under investigation (Table 4 ). However, both pilot and feasibility studies included trials run as complete studies, including randomization, but with sample sizes smaller than would be intended in the full study and the distinction between the two terms was not clear-cut.

Another reason for conducting a pilot study is to provide information to enable a sample size calculation in a subsequent main study. However since pilot studies tend to be small, the results should be interpreted with caution [ 5 ]. Only a small proportion of published pilot studies reported pre-study sample size calculations. Most journal editors reported that a sample size calculation is not a mandatory criterion for publishing pilot studies and suggested that it should not be done.

Some authors suggest that analysis of pilot studies should mainly be descriptive,[ 3 , 6 ] as hypothesis testing requires a powered sample size which is usually not available in pilot studies. In addition, inferential statistics and testing hypothesis for effectiveness require a control arm which may not be present in all pilot studies. However most of the pilot interventional studies in this review contained a control group and the authors performed and reported hypothesis testing for one or more variables. Some tested the effectiveness of an intervention and others just performed statistical testing to discover any important associations in the study variables. Observed practice is not necessarily good practice and we concur with Thabane et al [ 2 ] that any testing of an intervention needs to be reported cautiously.

The views of the journal editors, albeit from a small sample, were not particularly encouraging and reflected the experience of Lancaster et al [ 3 ]. Pilot studies, by their nature, will not produce 'significant' (i.e P < 0.05) results. We believe that publishing the results of well conducted pilot or feasibility studies is important for research, irrespective of outcome.. There is an increasing awareness that publishing only 'significant' results can lead to considerably error [ 7 ]. The journals we considered were all established, paper journals and perhaps the newer electronic journals will be more willing to consider the publication of the results from these types of studies.

We may expect that trials will increasingly be used to evaluate 'complex interventions'[ 8 , 9 ]. The MRC guidelines [ 8 ] explicitly suggest that preliminary studies, including pilots, be used prior to any major trial which seeks to evaluate a package of interventions (such as an educational course), rather than a single intervention (such as a drug). Thus it is likely that reviewers will be increasingly asked to pronounce on these and will require guidance as to how to review them.

Conclusions

We conclude that pilot studies are still poorly reported, with inappropriate emphasis on hypothesis-testing. We believe authors should be aware of the different requirements of pilot studies and feasibility studies and report them appropriately. We found that in practice the definitions of feasibility and pilot studies are not distinct and vary between health research funding bodies and we suggest use of the NETSCC definition to clarify terminology.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

MA reviewed the papers of 2000/1 and those of 2007/8 under the supervision of MC and helped to draft the manuscript. MC conceived of the study, and participated in its design and coordination and drafted the manuscript. CC conducted the portfolio database study and commented on the manuscript. GA conducted the original study, reviewed 20 papers and commented on the manuscript. All authors read and approved the final manuscript.

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1471-2288/10/67/prepub

Contributor Information

Mubashir Arain, Email: mp08ma@sheffield.ac.uk.

Michael J Campbell, Email: m.j.campbell@sheffield.ac.uk.

Cindy L Cooper, Email: c.l.cooper@sheffield.ac.uk.

Gillian A Lancaster, Email: g.lancaster@lancaster.ac.uk.