Mubashir ArainpersName, Michael J. CampbellpersName, Cindy CooperpersName, Gillian LancasterpersName (2010date). “What is a pilot or feasibility study? A review of current practice and editorial policy.” BMC Medical Research Methodologytitle·venue.
Object of study — TWO LABELS AND WHAT THEY BUY - 'pilotterm' and 'feasibilityterm' measured across 54 published studies, 12 registered studies, three funding bodies and seven journal editors, six years before the same team's framework abandoned the distinction they recommend here
Research question — when it is legitimate to label a study a 'pilotterm' or a 'feasibilityterm' study, and what a reviewer should expect to find inside one
Contribution — evidence from four independent sources - published papers, a registered-study database, funder policy and journal editors - that the two labels are used inconsistently, plus a recommendation to adopt the NETSCC definitions
phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | a label that authors reach for to protect an underpowered study and that editors read as a warning, attached to studies whose promised follow-up happens eight times in ninety — “Lancaster et al [ 3 ] found that 50% of pilot studies reported the intention of further work yet we identified only 8 (8.8%) which were followed up by a major study. Of these 2 (25%) were published in the same journal as the pilot.” | |
| claim-type | critique — “Some authors suggest that analysis of pilot studies should mainly be descriptive,[ 3 , 6 ] as hypothesis testing requires a powered sample size which is usually not available in pilot studies. In addition, inferential statistics and testing hypothesis for effectiveness require a control arm which may not be present in all pilot studies. However most of the pilot interventional studies in this review contained a control group and the authors performed and reported hypothesis testing for one or more variables. Some tested the effectiveness of an intervention and others just performed statistical testing to discover any important associations in the study variables. Observed practice is not necessarily good practice and we concur with Thabane et al [ 2 ] that any testing of an intervention needs to be reported cautiously.” | |
| method — evidence | — “The editors of five (out of seven) medical journals responded to our request for information regarding publishing policy for pilot studies. Four of the journals did not have a specified policy about publishing pilot studies and mostly reported that pilot trials cannot be published if the standard is lower than a full clinical trial requirement. The Lancet has started creating space for preliminary phase I trials and set a different standard for preliminary studies. Most of the other journals do not encourage the publication of pilot studies because they consider them less rigorous than main studies.” | |
| object of study — evidence | — “Twelve studies which were interventional pilot/feasibility studies and which included testing of some component of the research process were identified through the UKCRN Portfolio database. There was no clear distinction in use of the terms 'pilot' and 'feasibility'. Five journal editors replied to our entreaty. In general they were loathe to publish studies described as 'pilot'.” | |
| research question — evidence | — “It has become apparent to us when reviewing research proposals that small studies with all the trappings of a major study, such as randomization and hypothesis testing may be labeled a 'pilot' because they do not have the power to test clinically meaningful hypotheses. The authors of such studies perhaps hope that reviewers will regard a 'pilot' more favourably than a small clinical trial.” | |
| contribution — evidence | — “We conclude that pilot studies are still poorly reported, with inappropriate emphasis on hypothesis-testing. We believe authors should be aware of the different requirements of pilot studies and feasibility studies and report them appropriately. We found that in practice the definitions of feasibility and pilot studies are not distinct and vary between health research funding bodies and we suggest use of the NETSCC definition to clarify terminology.” | |
| mechanism | THE LABEL IS A SHIELD FOR AUTHORS AND A LIABILITY AT JOURNALS, IN ONE PAPER: `small studies with all the trappings of a major study ... may be labeled a `PILOT` because they do not have the power to test clinically meaningful hypotheses. The authors of such studies perhaps hope that reviewers will regard a `PILOT` more favourably than a small clinical trial` - against `In general they were loathe to publish studies described as `PILOT``. The same word is worth more to the author and less to the editor — “It has become apparent to us when reviewing research proposals that small studies with all the trappings of a major study, such as randomization and hypothesis testing may be labeled a 'pilot' because they do not have the power to test clinically meaningful hypotheses. The authors of such studies perhaps hope that reviewers will regard a 'pilot' more favourably than a small clinical trial.” | mechanisms[0] |
| mechanism | AND THE NAMING ACT IS GIVEN AN ETHICS RULING: `It is generally accepted that small, underpowered clinical trials are unethical. Thus it is not an excuse to label such a study as a pilot and hope to make it ethical.` A sentence about what a word cannot do - the corpus has definitions that decide funding and diagnosis, and this is the first that decides whether a study should have been run at all — “It is generally accepted that small, underpowered clinical trials are unethical [ 4 ]. Thus it is not an excuse to label such a study as a pilot and hope to make it ethical.” | mechanisms[1] |
| mechanism | AND THE PROMISE IS KEPT EIGHT TIMES IN NINETY: `Lancaster et al found that 50% of pilot studies reported the intention of further work yet we identified only 8 (8.8%) which were followed up by a major study` - and `one would have expected about 45 ... whereas we only found 8`. The defining property of a pilot study is that a bigger study follows, and it does not — “Lancaster et al [ 3 ] found that 50% of pilot studies reported the intention of further work yet we identified only 8 (8.8%) which were followed up by a major study. Of these 2 (25%) were published in the same journal as the pilot.” | mechanisms[2] |
| mechanism | AND THE REASON GIVEN INVERTS THE PURPOSE: `if the pilot shows significant results then researchers may not find it necessary to conduct the main trial. In addition if the results are unfavorable or the authors find an unfeasible procedure, the main study is less likely to be considered useful.` A positive result cancels the trial and a negative result cancels the trial - which leaves no outcome under which the study does what its name says — “The main reason given was that if the pilot shows significant results then researchers may not find it necessary to conduct the main trial. In addition if the results are unfavorable or the authors find an unfeasible procedure, the main study is less likely to be considered useful.” | mechanisms[3] |
| mechanism | AND THREE PRIOR AUTHORITIES TREAT THE SECOND WORD THREE WAYS: Arnold et al `made useful distinctions between: pilot work which is any background research ..., a pilot study which has specific hypotheses, objectives and methodology and a pilot trial which is a stand-alone pilot study and includes a randomization procedure. They excluded feasibility studies from their consideration` - while `Thabane et al ... INCLUDED `FEASIBILITY` OR `VANGUARD` studies but did not distinguish them from pilot studies`. Excluded, conflated, or split three ways — “They made useful distinctions between: pilot work which is any background research to inform a future study, a pilot study which has specific hypotheses, objectives and methodology and a pilot trial which is a stand-alone pilot study and includes a randomization procedure. They excluded feasibility studies from their consideration.” | mechanisms[4] |
| mechanism | AND THE LABELS DO TRACK THE METHOD, IN THE PUBLISHED LITERATURE: `We found that studies labeled `FEASIBILITY` were conducted with more flexible methodology compared to those labeled `PILOT` ... `pilot` studies were reported with more rigorous methodological components like sample size estimation, randomization and control group selection than studies labeled `feasibility`.` Four of six components differ significantly by keyword alone — “We found that studies labeled 'feasibility' were conducted with more flexible methodology compared to those labeled 'pilot'. For example the term 'feasibility' has been used for large scale studies such as a screening programme applied at a population level to determine the initial feasibility of the programme. On the other hand 'pilot' studies were reported with more rigorous methodological components like sample size estimation, randomization and control group selection than studies labeled 'feasibility'.” | mechanisms[5] |
| mechanism | AND THEY DO NOT, IN THE REGISTERED-STUDY DATABASE: `both pilot and feasibility studies included trials run as complete studies, including randomization, but with sample sizes smaller than would be intended in the full study and the distinction between the two terms was not clear-cut.` Two data sources in one paper, opposite answers to whether the words mean anything - the published literature separates them by keyword and the registry does not, and the paper reports both — “However, both pilot and feasibility studies included trials run as complete studies, including randomization, but with sample sizes smaller than would be intended in the full study and the distinction between the two terms was not clear-cut.” | mechanisms[6] |
| mechanism | AND THE DEFINITION THIS PAPER RECOMMENDS IS THE ONE ITS OWN AUTHORS LATER ABANDON: `We found the NETSCC definition to be the most helpful since it distinguishes between these types of study and to most closely mirror what investigators are doing.` Michael J. Campbell and Gillian A. Lancaster are authors here and on eldridge2016, which opens `We initially adopted MUTUALLY EXCLUSIVE DEFINITIONS ... based on ... THE DEFINITIONS USED BY THE NIHR` and reports the panel voting them down 14-1 — “We found the NETSCC definition to be the most helpful since it distinguishes between these types of study. In addition it was observed that most of the pilot studies report their results as inconclusive, with the intention of conducting a further, larger study. In contrast, several of the feasibility studies did not admit such an intention.” | mechanisms[7] |
| mechanism | AND THE RECOMMENDED DEFINITION DRAWS ITS LINE AT THE OUTCOME: `Crucially, feasibility studies do not evaluate the outcome of interest; that is left to the main study. If a feasibility study is a small randomised controlled trial, it need not have a primary outcome and the usual sort of power calculation is not normally undertaken.` A class defined by what it must NOT measure — “Feasibility Studies are pieces of research done before a main study. They are used to estimate important parameters that are needed to design the main study. For instance: • standard deviation of the outcome measure, which is needed in some cases to estimate sample size, • willingness of participants to be randomised, • willingness of clinicians to recruit participants, • number of eligible patients, • characteristics of the proposed outcome measure and in some cases feasibility studies might involve designing a suitable outcome measure, • follow-up rates, response rates to questionnaires, adherence/compliance rates, ICCs in cluster trials, etc. Feasibility studies for randomised controlled trials may not themselves be randomised. Crucially, feasibility studies do not evaluate the outcome of interest; that is left to the main study.” | mechanisms[8] |
| mechanism | AND THE OTHER HALF DEFINES BY RESEMBLANCE: `A Pilot Study is a version of the main study that is run in miniature to test whether the components of the main study can all work together ... It will therefore resemble the main study in many respects.` One class bounded by an omission, its sibling by a likeness - and eldridge2016 later keeps the likeness as the only differentia — “A Pilot Study is a version of the main study that is run in miniature to test whether the components of the main study can all work together. It is focused on the processes of the main study, for example to ensure recruitment, randomisation, treatment, and follow-up assessments all run smoothly. It will therefore resemble the main study in many respects. In some cases this will be the first phase of the substantive study and data from the pilot phase may contribute to the final analysis; this can be referred to as an internal pilot. Alternatively at the end of the pilot study the data may be analysed and set aside, a so-called external pilot.” | mechanisms[9] |
| mechanism | AND PRACTICE IGNORED THE 2004 RECOMMENDATION IT WAS GIVEN: `Some authors suggest that analysis of pilot studies should mainly be descriptive, as hypothesis testing requires a powered sample size which is usually not available ... However most of the pilot interventional studies in this review contained a control group and the authors performed and reported hypothesis testing ... Observed practice is not necessarily good practice.` A repeat review measuring whether its own field took the advice, and finding it did not — “Some authors suggest that analysis of pilot studies should mainly be descriptive,[ 3 , 6 ] as hypothesis testing requires a powered sample size which is usually not available in pilot studies. In addition, inferential statistics and testing hypothesis for effectiveness require a control arm which may not be present in all pilot studies. However most of the pilot interventional studies in this review contained a control group and the authors performed and reported hypothesis testing for one or more variables. Some tested the effectiveness of an intervention and others just performed statistical testing to discover any important associations in the study variables. Observed practice is not necessarily good practice and we concur with Thabane et al [ 2 ] that any testing of an intervention needs to be reported cautiously.” | mechanisms[10] |
| mechanism | AND A CONSENT CONSEQUENCE FALLS OUT OF THE DEFINITION: `Participants in such studies should be informed that they are in a pilot study and that there may not be a further larger study.` If the defining promise is kept 8 times in 90, then what a participant is told about why they are enrolled depends on a definition nobody in the field shares — “We have shown that pilot studies have different objectives to RCTs and these should be clearly described. Participants in such studies should be informed that they are in a pilot study and that there may not be a further larger study.” | mechanisms[11] |
| replicates | Lancaster et al's 2004 review of pilot studies in the same seven journals for 2000-01, repeated here for 2007-08 to test whether its recommendations changed practice — “Lancaster et al [ 3 ] found that 50% of pilot studies reported the intention of further work yet we identified only 8 (8.8%) which were followed up by a major study. Of these 2 (25%) were published in the same journal as the pilot.” | claim_relations[0] |
| critiques | the reporting of pilot studies with hypothesis testing and control arms, against the 2004 recommendation that their analysis be mainly descriptive — “Some authors suggest that analysis of pilot studies should mainly be descriptive,[ 3 , 6 ] as hypothesis testing requires a powered sample size which is usually not available in pilot studies. In addition, inferential statistics and testing hypothesis for effectiveness require a control arm which may not be present in all pilot studies. However most of the pilot interventional studies in this review contained a control group and the authors performed and reported hypothesis testing for one or more variables. Some tested the effectiveness of an intervention and others just performed statistical testing to discover any important associations in the study variables. Observed practice is not necessarily good practice and we concur with Thabane et al [ 2 ] that any testing of an intervention needs to be reported cautiously.” | claim_relations[1] |
| critiques | Thabane et al's checklist, which `INCLUDED `FEASIBILITY` OR `VANGUARD` studies but did not distinguish them from pilot studies` and was `opinion based and not supported by a review of current practice` — “Thabane et al [ 2 ] gave a checklist of what they think should be included in a pilot study. They included 'feasibility' or 'vanguard' studies but did not distinguish them from pilot studies.” | claim_relations[2] |
| sense | `pilot` (the label, as authors use it) - reached for when a study `DOES NOT HAVE THE POWER to test clinically meaningful hypotheses`, in the hope that reviewers `WILL REGARD A `PILOT` more favourably than a small clinical trial` — “It has become apparent to us when reviewing research proposals that small studies with all the trappings of a major study, such as randomization and hypothesis testing may be labeled a 'pilot' because they do not have the power to test clinically meaningful hypotheses. The authors of such studies perhaps hope that reviewers will regard a 'pilot' more favourably than a small clinical trial.” | senses[0] |
| sense | `pilot study` (NETSCC, recommended here) - `a version of the main study that is run in miniature to test whether the components of the main study can all work together`; internal if its data join the final analysis, external if set aside — “A Pilot Study is a version of the main study that is run in miniature to test whether the components of the main study can all work together. It is focused on the processes of the main study, for example to ensure recruitment, randomisation, treatment, and follow-up assessments all run smoothly. It will therefore resemble the main study in many respects. In some cases this will be the first phase of the substantive study and data from the pilot phase may contribute to the final analysis; this can be referred to as an internal pilot. Alternatively at the end of the pilot study the data may be analysed and set aside, a so-called external pilot.” | senses[1] |
| sense | `feasibility study` (NETSCC) - research done before a main study to estimate design parameters, which `Crucially ... do not evaluate the outcome of interest` — “Feasibility Studies are pieces of research done before a main study. They are used to estimate important parameters that are needed to design the main study. For instance: • standard deviation of the outcome measure, which is needed in some cases to estimate sample size, • willingness of participants to be randomised, • willingness of clinicians to recruit participants, • number of eligible patients, • characteristics of the proposed outcome measure and in some cases feasibility studies might involve designing a suitable outcome measure, • follow-up rates, response rates to questionnaires, adherence/compliance rates, ICCs in cluster trials, etc. Feasibility studies for randomised controlled trials may not themselves be randomised. Crucially, feasibility studies do not evaluate the outcome of interest; that is left to the main study.” | senses[2] |
| sense | `pilot work` / `pilot study` / `pilot trial` (Arnold et al) - a three-way split: any background research, a study with hypotheses and methodology, and a stand-alone pilot study with randomisation — “They made useful distinctions between: pilot work which is any background research to inform a future study, a pilot study which has specific hypotheses, objectives and methodology and a pilot trial which is a stand-alone pilot study and includes a randomization procedure. They excluded feasibility studies from their consideration.” | senses[3] |
| control | repeating a prior review's search terms, journals and time window exactly, and reporting the two periods side by side — held — “Some authors suggest that analysis of pilot studies should mainly be descriptive,[ 3 , 6 ] as hypothesis testing requires a powered sample size which is usually not available in pilot studies. In addition, inferential statistics and testing hypothesis for effectiveness require a control arm which may not be present in all pilot studies. However most of the pilot interventional studies in this review contained a control group and the authors performed and reported hypothesis testing for one or more variables. Some tested the effectiveness of an intervention and others just performed statistical testing to discover any important associations in the study variables. Observed practice is not necessarily good practice and we concur with Thabane et al [ 2 ] that any testing of an intervention needs to be reported cautiously.” — rules out attributing a change in practice to a change in the reviewer's method — “Lancaster et al [ 3 ] found that 50% of pilot studies reported the intention of further work yet we identified only 8 (8.8%) which were followed up by a major study. Of these 2 (25%) were published in the same journal as the pilot.” | controls[0] |
| control | adding a registered-study database arm, because 'as many pilot studies are never published' the journal literature is a filtered sample — partial — “For example, the 6 pilot studies reported the intention to evaluate a total of 17 methodological components whereas in the 5 feasibility studies a total of only 6 methodological components were specifically identified as being under investigation” — rules out publication bias in the sample of studies whose labels are being compared — “However, both pilot and feasibility studies included trials run as complete studies, including randomization, but with sample sizes smaller than would be intended in the full study and the distinction between the two terms was not clear-cut.” | controls[1] |
| control | canvassing the journal editors and funding bodies directly rather than inferring policy from what was published — partial — “Pilot studies, by their nature, will not produce 'significant' (i.e P < 0.05) results. We believe that publishing the results of well conducted pilot or feasibility studies is important for research, irrespective of outcome.. There is an increasing awareness that publishing only 'significant' results can lead to considerably error [ 7 ].” — rules out reading an editorial policy off the papers that survived it — “The editors of five (out of seven) medical journals responded to our request for information regarding publishing policy for pilot studies. Four of the journals did not have a specified policy about publishing pilot studies and mostly reported that pilot trials cannot be published if the standard is lower than a full clinical trial requirement. The Lancet has started creating space for preliminary phase I trials and set a different standard for preliminary studies. Most of the other journals do not encourage the publication of pilot studies because they consider them less rigorous than main studies.” | controls[2] |
| weight | Lancaster et al's 2004 review of pilot studies in the same seven journals for 2000-01, repeated here for 2007-08 to test whether its recommendations changed practice — 0.95 — “Lancaster et al [ 3 ] found that 50% of pilot studies reported the intention of further work yet we identified only 8 (8.8%) which were followed up by a major study. Of these 2 (25%) were published in the same journal as the pilot.” | claim_relations[0] |
| weight | the reporting of pilot studies with hypothesis testing and control arms, against the 2004 recommendation that their analysis be mainly descriptive — 0.9 — “Some authors suggest that analysis of pilot studies should mainly be descriptive,[ 3 , 6 ] as hypothesis testing requires a powered sample size which is usually not available in pilot studies. In addition, inferential statistics and testing hypothesis for effectiveness require a control arm which may not be present in all pilot studies. However most of the pilot interventional studies in this review contained a control group and the authors performed and reported hypothesis testing for one or more variables. Some tested the effectiveness of an intervention and others just performed statistical testing to discover any important associations in the study variables. Observed practice is not necessarily good practice and we concur with Thabane et al [ 2 ] that any testing of an intervention needs to be reported cautiously.” | claim_relations[1] |
| weight | Thabane et al's checklist, which `INCLUDED `FEASIBILITY` OR `VANGUARD` studies but did not distinguish them from pilot studies` and was `opinion based and not supported by a review of current practice` — 0.85 — “Thabane et al [ 2 ] gave a checklist of what they think should be included in a pilot study. They included 'feasibility' or 'vanguard' studies but did not distinguish them from pilot studies.” | claim_relations[2] |
| note | pilot trial —is-a→ pilot study — 'They made useful distinctions between: pilot work which is any background research to inform a future study, a pilot study which has specific hypotheses, objectives and methodology and a pilot trial which is a stand-alone pilot study and includes a randomization procedure.' Three terms sharing one word, nested, from a review that then 'excluded feasibility studies from their consideration' entirely - so the nesting is complete only within the half of the vocabulary they kept — “They made useful distinctions between: pilot work which is any background research to inform a future study, a pilot study which has specific hypotheses, objectives and methodology and a pilot trial which is a stand-alone pilot study and includes a randomization procedure. They excluded feasibility studies from their consideration.” | concept_edges[0] |
| relation | pilot trial —is-a→ pilot study — a pilot trial is a stand-alone pilot study that additionally includes a randomisation procedure — “They made useful distinctions between: pilot work which is any background research to inform a future study, a pilot study which has specific hypotheses, objectives and methodology and a pilot trial which is a stand-alone pilot study and includes a randomization procedure. They excluded feasibility studies from their consideration.” — “They made useful distinctions between: pilot work which is any background research to inform a future study, a pilot study which has specific hypotheses, objectives and methodology and a pilot trial which is a stand-alone pilot study and includes a randomization procedure.” | concept_edges[0] |
| weight | pilot trial —is-a→ pilot study — 0.95 — “They made useful distinctions between: pilot work which is any background research to inform a future study, a pilot study which has specific hypotheses, objectives and methodology and a pilot trial which is a stand-alone pilot study and includes a randomization procedure. They excluded feasibility studies from their consideration.” | concept_edges[0] |
| note | pilot studies —distinct-from→ RCTs — 'We have shown that pilot studies have different objectives to RCTs and these should be clearly described.' The consequence is a consent one and the paper draws it: 'Participants in such studies should be informed that they are in a pilot study and that there may not be a further larger study' - which matters because the follow-up happened 8 times in 90 — “We have shown that pilot studies have different objectives to RCTs and these should be clearly described. Participants in such studies should be informed that they are in a pilot study and that there may not be a further larger study.” | concept_edges[2] |
| relation | pilot studies —distinct-from→ RCTs — pilot studies have different OBJECTIVES from the trials they precede, and the paper's complaint is that they are reported as though they did not — “We have shown that pilot studies have different objectives to RCTs and these should be clearly described. Participants in such studies should be informed that they are in a pilot study and that there may not be a further larger study.” — “We have shown that pilot studies have different objectives to RCTs and these should be clearly described.” | concept_edges[2] |
| weight | pilot studies —distinct-from→ RCTs — 0.95 — “We have shown that pilot studies have different objectives to RCTs and these should be clearly described. Participants in such studies should be informed that they are in a pilot study and that there may not be a further larger study.” | concept_edges[2] |
| audit | pilot —contrasted-with→ small clinical trial — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the contrast is what authors 'perhaps hope' reviewers will see; attributed and hedged. — “It has become apparent to us when reviewing research proposals that small studies with all the trappings of a major study, such as randomization and hypothesis testing may be labeled a 'pilot' because they do not have the power to test clinically meaningful hypotheses. The authors of such studies perhaps hope that reviewers will regard a 'pilot' more favourably than a small clinical trial.” | concept_edges[3] |
| note | pilot —contrasted-with→ small clinical trial — `The authors of such studies perhaps hope that reviewers will regard a `PILOT` more favourably than a small clinical trial.` An edge between two names for one object, where the difference is in the READER not the study - and the editor survey finds the bet is wrong: `In general they were loathe to publish studies described as `PILOT`` — “It has become apparent to us when reviewing research proposals that small studies with all the trappings of a major study, such as randomization and hypothesis testing may be labeled a 'pilot' because they do not have the power to test clinically meaningful hypotheses. The authors of such studies perhaps hope that reviewers will regard a 'pilot' more favourably than a small clinical trial.” | concept_edges[3] |
| relation | pilot —contrasted-with→ small clinical trial — the same study under two names - and the paper's opening claim is that authors choose the first expecting a softer review — “It has become apparent to us when reviewing research proposals that small studies with all the trappings of a major study, such as randomization and hypothesis testing may be labeled a 'pilot' because they do not have the power to test clinically meaningful hypotheses. The authors of such studies perhaps hope that reviewers will regard a 'pilot' more favourably than a small clinical trial.” — “The authors of such studies perhaps hope that reviewers will regard a 'pilot' more favourably than a small clinical trial.” | concept_edges[3] |
| weight | pilot —contrasted-with→ small clinical trial — 0.95 — “It has become apparent to us when reviewing research proposals that small studies with all the trappings of a major study, such as randomization and hypothesis testing may be labeled a 'pilot' because they do not have the power to test clinically meaningful hypotheses. The authors of such studies perhaps hope that reviewers will regard a 'pilot' more favourably than a small clinical trial.” | concept_edges[3] |
| note | feasibility —contrasted-with→ pilot — `We found that studies labeled `FEASIBILITY` were conducted with more flexible methodology compared to those labeled `PILOT` ... `pilot` studies were reported with more rigorous methodological components like sample size estimation, randomization and control group selection.` And the registry arm of the same paper says the opposite: `the distinction between the two terms was not clear-cut`. Both are true - the group means separate and the individual cases do not - and eldridge2016 later reports only the second — “We found that studies labeled 'feasibility' were conducted with more flexible methodology compared to those labeled 'pilot'. For example the term 'feasibility' has been used for large scale studies such as a screening programme applied at a population level to determine the initial feasibility of the programme. On the other hand 'pilot' studies were reported with more rigorous methodological components like sample size estimation, randomization and control group selection than studies labeled 'feasibility'.” | concept_edges[4] |
| relation | feasibility —contrasted-with→ pilot — studies keyworded 'feasibility' are run with more flexible methodology than those keyworded 'pilot' - the labels predict the design at the group level — “We found that studies labeled 'feasibility' were conducted with more flexible methodology compared to those labeled 'pilot'. For example the term 'feasibility' has been used for large scale studies such as a screening programme applied at a population level to determine the initial feasibility of the programme. On the other hand 'pilot' studies were reported with more rigorous methodological components like sample size estimation, randomization and control group selection than studies labeled 'feasibility'.” — “We found that studies labeled 'feasibility' were conducted with more flexible methodology compared to those labeled 'pilot'.” | concept_edges[4] |
| weight | feasibility —contrasted-with→ pilot — 0.95 — “We found that studies labeled 'feasibility' were conducted with more flexible methodology compared to those labeled 'pilot'. For example the term 'feasibility' has been used for large scale studies such as a screening programme applied at a population level to determine the initial feasibility of the programme. On the other hand 'pilot' studies were reported with more rigorous methodological components like sample size estimation, randomization and control group selection than studies labeled 'feasibility'.” | concept_edges[4] |
| outcome | control — held — “Lancaster et al [ 3 ] found that 50% of pilot studies reported the intention of further work yet we identified only 8 (8.8%) which were followed up by a major study. Of these 2 (25%) were published in the same journal as the pilot.” — “Some authors suggest that analysis of pilot studies should mainly be descriptive,[ 3 , 6 ] as hypothesis testing requires a powered sample size which is usually not available in pilot studies. In addition, inferential statistics and testing hypothesis for effectiveness require a control arm which may not be present in all pilot studies. However most of the pilot interventional studies in this review contained a control group and the authors performed and reported hypothesis testing for one or more variables. Some tested the effectiveness of an intervention and others just performed statistical testing to discover any important associations in the study variables. Observed practice is not necessarily good practice and we concur with Thabane et al [ 2 ] that any testing of an intervention needs to be reported cautiously.” | controls[0] |
| outcome | control — partial — “However, both pilot and feasibility studies included trials run as complete studies, including randomization, but with sample sizes smaller than would be intended in the full study and the distinction between the two terms was not clear-cut.” — “For example, the 6 pilot studies reported the intention to evaluate a total of 17 methodological components whereas in the 5 feasibility studies a total of only 6 methodological components were specifically identified as being under investigation” | controls[1] |
| outcome | control — partial — “The editors of five (out of seven) medical journals responded to our request for information regarding publishing policy for pilot studies. Four of the journals did not have a specified policy about publishing pilot studies and mostly reported that pilot trials cannot be published if the standard is lower than a full clinical trial requirement. The Lancet has started creating space for preliminary phase I trials and set a different standard for preliminary studies. Most of the other journals do not encourage the publication of pilot studies because they consider them less rigorous than main studies.” — “Pilot studies, by their nature, will not produce 'significant' (i.e P < 0.05) results. We believe that publishing the results of well conducted pilot or feasibility studies is important for research, irrespective of outcome.. There is an increasing awareness that publishing only 'significant' results can lead to considerably error [ 7 ].” | controls[2] |
| limitation | THE EDITOR SURVEY IS FIVE REPLIES OUT OF SEVEN AND THE AUTHORS SAY SO TWICE: `The editors of five (out of seven) medical journals responded`, and later `The views of the journal editors, albeit from a small sample, were not particularly encouraging`. Four of the five had no written policy at all, so the finding is about absence of policy as much as about opinion — “The editors of five (out of seven) medical journals responded to our request for information regarding publishing policy for pilot studies. Four of the journals did not have a specified policy about publishing pilot studies and mostly reported that pilot trials cannot be published if the standard is lower than a full clinical trial requirement. The Lancet has started creating space for preliminary phase I trials and set a different standard for preliminary studies. Most of the other journals do not encourage the publication of pilot studies because they consider them less rigorous than main studies.” | limitations[0] |
| limitation | AND THE REGISTRY ARM IS TWELVE STUDIES: of 34 identified, `Only 12 studies were interventional pilot/feasibility studies which included testing of some component of the research process. Of these 5 were referred to as `feasibility`, 6 as `pilot` and 1 as both` - and the component counts that follow (17 across 6 pilots against 6 across 5 feasibility studies) rest on those twelve — “For example, the 6 pilot studies reported the intention to evaluate a total of 17 methodological components whereas in the 5 feasibility studies a total of only 6 methodological components were specifically identified as being under investigation” | limitations[1] |
| limitation | AND THE PAPER'S OWN NUMBER CELLS ARE UNRELIABLE, WHICH IS NOT OUR DAMAGE: Table 3 reports participant ranges of `[8, 187777]` and `[5, 12774614]` - twelve million participants in a feasibility study - and Table 2 gives `76 [8, 1299]` under a header saying (IQR) while the prose says `median 76, INTER-QUARTILE RANGE (42, 216)`. Checked against the publisher's own XML: each is a SINGLE `<td>` cell, so the fusion is in the source. No quantity from Tables 2-4 is tagged on this record; the percentages ARE arithmetically consistent with their counts and only the participant-range cells are broken — “We conclude that pilot studies are still poorly reported, with inappropriate emphasis on hypothesis-testing. We believe authors should be aware of the different requirements of pilot studies and feasibility studies and report them appropriately. We found that in practice the definitions of feasibility and pilot studies are not distinct and vary between health research funding bodies and we suggest use of the NETSCC definition to clarify terminology.” | limitations[2] |
| limitation | AND THE FIELD'S PUBLICATION FILTER WORKS AGAINST THE EVIDENCE THE PAPER WANTS: `Pilot studies, by their nature, will not produce `SIGNIFICANT` (i.e P < 0.05) RESULTS. We believe that publishing the results of well conducted pilot or feasibility studies is important for research, IRRESPECTIVE OF OUTCOME. There is an increasing awareness that PUBLISHING ONLY `SIGNIFICANT` results can lead to considerably error.` A class of study defined so that it cannot meet the criterion its journals apply — “Pilot studies, by their nature, will not produce 'significant' (i.e P < 0.05) results. We believe that publishing the results of well conducted pilot or feasibility studies is important for research, irrespective of outcome.. There is an increasing awareness that publishing only 'significant' results can lead to considerably error [ 7 ].” | limitations[3] |
finding · quantity · variable · definition · key-term — grounded in the full text| finding | — “It has become apparent to us when reviewing research proposals that small studies with all the trappings of a major study, such as randomization and hypothesis testing may be labeled a 'pilot' because they do not have the power to test clinically meaningful hypotheses. The authors of such studies perhaps hope that reviewers will regard a 'pilot' more favourably than a small clinical trial.” | content_tags.findings[0] |
| finding | — “Lancaster et al [ 3 ] found that 50% of pilot studies reported the intention of further work yet we identified only 8 (8.8%) which were followed up by a major study. Of these 2 (25%) were published in the same journal as the pilot.” | content_tags.findings[1] |
| finding | — “The main reason given was that if the pilot shows significant results then researchers may not find it necessary to conduct the main trial. In addition if the results are unfavorable or the authors find an unfeasible procedure, the main study is less likely to be considered useful.” | content_tags.findings[2] |
| finding | — “However, both pilot and feasibility studies included trials run as complete studies, including randomization, but with sample sizes smaller than would be intended in the full study and the distinction between the two terms was not clear-cut.” | content_tags.findings[3] |
| finding | — “It is generally accepted that small, underpowered clinical trials are unethical [ 4 ]. Thus it is not an excuse to label such a study as a pilot and hope to make it ethical.” | content_tags.findings[4] |
| finding | — “We found the NETSCC definition to be the most helpful since it distinguishes between these types of study. In addition it was observed that most of the pilot studies report their results as inconclusive, with the intention of conducting a further, larger study. In contrast, several of the feasibility studies did not admit such an intention.” | content_tags.findings[5] |
| finding | — “We have shown that pilot studies have different objectives to RCTs and these should be clearly described. Participants in such studies should be informed that they are in a pilot study and that there may not be a further larger study.” | content_tags.findings[6] |
| quantity | 8 of 90 (8.8%), against about 45 expected from stated intentions — — “Lancaster et al [ 3 ] found that 50% of pilot studies reported the intention of further work yet we identified only 8 (8.8%) which were followed up by a major study. Of these 2 (25%) were published in the same journal as the pilot.” | content_tags.quantities[0] |
| value source note | quantity — THE VALUE DRAWS ON THE PAPER BEYOND THIS QUOTE: part of it is printed elsewhere in the paper, not in this quote ('Only 8 out of 90 pilot studies identified by the earlier'). The 45 is half of 90, from the 50% the quote reports. | content_tags.quantities[0] |
| quantity | 17 across 6 pilot studies against 6 across 5 feasibility studies — — “For example, the 6 pilot studies reported the intention to evaluate a total of 17 methodological components whereas in the 5 feasibility studies a total of only 6 methodological components were specifically identified as being under investigation” | content_tags.quantities[1] |
| quantity | 5 of 7 replied; 4 of those had no specified policy on publishing pilot studies — — “The editors of five (out of seven) medical journals responded to our request for information regarding publishing policy for pilot studies. Four of the journals did not have a specified policy about publishing pilot studies and mostly reported that pilot trials cannot be published if the standard is lower than a full clinical trial requirement. The Lancet has started creating space for preliminary phase I trials and set a different standard for preliminary studies. Most of the other journals do not encourage the publication of pilot studies because they consider them less rigorous than main studies.” | content_tags.quantities[2] |
| variable | the keyword — “We found that studies labeled 'feasibility' were conducted with more flexible methodology compared to those labeled 'pilot'. For example the term 'feasibility' has been used for large scale studies such as a screening programme applied at a population level to determine the initial feasibility of the programme. On the other hand 'pilot' studies were reported with more rigorous methodological components like sample size estimation, randomization and control group selection than studies labeled 'feasibility'.” | content_tags.variables[0] |
| definition | pilot study — “A Pilot Study is a version of the main study that is run in miniature to test whether the components of the main study can all work together. It is focused on the processes of the main study, for example to ensure recruitment, randomisation, treatment, and follow-up assessments all run smoothly. It will therefore resemble the main study in many respects. In some cases this will be the first phase of the substantive study and data from the pilot phase may contribute to the final analysis; this can be referred to as an internal pilot. Alternatively at the end of the pilot study the data may be analysed and set aside, a so-called external pilot.” | content_tags.definitions[0] |
| definition | feasibility study — “Feasibility Studies are pieces of research done before a main study. They are used to estimate important parameters that are needed to design the main study. For instance: • standard deviation of the outcome measure, which is needed in some cases to estimate sample size, • willingness of participants to be randomised, • willingness of clinicians to recruit participants, • number of eligible patients, • characteristics of the proposed outcome measure and in some cases feasibility studies might involve designing a suitable outcome measure, • follow-up rates, response rates to questionnaires, adherence/compliance rates, ICCs in cluster trials, etc. Feasibility studies for randomised controlled trials may not themselves be randomised. Crucially, feasibility studies do not evaluate the outcome of interest; that is left to the main study.” | content_tags.definitions[1] |
| concept | pilot trial — not defined in this paper's own words; Arnold et al's narrowest term: a stand-alone pilot study that includes randomisation — “They made useful distinctions between: pilot work which is any background research to inform a future study, a pilot study which has specific hypotheses, objectives and methodology and a pilot trial which is a stand-alone pilot study and includes a randomization procedure. They excluded feasibility studies from their consideration.” | concepts[0] |
| concept | pilot study — not defined in this paper's own words; a study with specific hypotheses, objectives and methodology run before a main study — “They made useful distinctions between: pilot work which is any background research to inform a future study, a pilot study which has specific hypotheses, objectives and methodology and a pilot trial which is a stand-alone pilot study and includes a randomization procedure. They excluded feasibility studies from their consideration.” | concepts[1] |
| concept | pilot studies — not defined in this paper's own words; the class as this review measures it across 54 papers and a registry — “We have shown that pilot studies have different objectives to RCTs and these should be clearly described. Participants in such studies should be informed that they are in a pilot study and that there may not be a further larger study.” | concepts[2] |
| concept | RCTs — not defined in this paper's own words; the definitive trials pilot studies are supposedly run in preparation for — “We have shown that pilot studies have different objectives to RCTs and these should be clearly described. Participants in such studies should be informed that they are in a pilot study and that there may not be a further larger study.” | concepts[3] |
| concept | feasibility studies — not defined in this paper's own words; NETSCC's class: research before a main study, estimating design parameters only — “Feasibility Studies are pieces of research done before a main study. They are used to estimate important parameters that are needed to design the main study. For instance: • standard deviation of the outcome measure, which is needed in some cases to estimate sample size, • willingness of participants to be randomised, • willingness of clinicians to recruit participants, • number of eligible patients, • characteristics of the proposed outcome measure and in some cases feasibility studies might involve designing a suitable outcome measure, • follow-up rates, response rates to questionnaires, adherence/compliance rates, ICCs in cluster trials, etc. Feasibility studies for randomised controlled trials may not themselves be randomised. Crucially, feasibility studies do not evaluate the outcome of interest; that is left to the main study.” | concepts[4] |
| concept | the outcome of interest — not defined in this paper's own words; the quantity a feasibility study is forbidden to evaluate — “Feasibility Studies are pieces of research done before a main study. They are used to estimate important parameters that are needed to design the main study. For instance: • standard deviation of the outcome measure, which is needed in some cases to estimate sample size, • willingness of participants to be randomised, • willingness of clinicians to recruit participants, • number of eligible patients, • characteristics of the proposed outcome measure and in some cases feasibility studies might involve designing a suitable outcome measure, • follow-up rates, response rates to questionnaires, adherence/compliance rates, ICCs in cluster trials, etc. Feasibility studies for randomised controlled trials may not themselves be randomised. Crucially, feasibility studies do not evaluate the outcome of interest; that is left to the main study.” | concepts[5] |
| concept | pilot — not defined in this paper's own words; the bare label, as a strategic object - what an author gains and an editor deducts — “It has become apparent to us when reviewing research proposals that small studies with all the trappings of a major study, such as randomization and hypothesis testing may be labeled a 'pilot' because they do not have the power to test clinically meaningful hypotheses. The authors of such studies perhaps hope that reviewers will regard a 'pilot' more favourably than a small clinical trial.” | concepts[6] |
| concept | small clinical trial — not defined in this paper's own words; what the same study would be called without the label — “It has become apparent to us when reviewing research proposals that small studies with all the trappings of a major study, such as randomization and hypothesis testing may be labeled a 'pilot' because they do not have the power to test clinically meaningful hypotheses. The authors of such studies perhaps hope that reviewers will regard a 'pilot' more favourably than a small clinical trial.” | concepts[7] |
| concept | feasibility — not defined in this paper's own words; the other bare label, attached in practice to more flexible methodology — “We found that studies labeled 'feasibility' were conducted with more flexible methodology compared to those labeled 'pilot'. For example the term 'feasibility' has been used for large scale studies such as a screening programme applied at a population level to determine the initial feasibility of the programme. On the other hand 'pilot' studies were reported with more rigorous methodological components like sample size estimation, randomization and control group selection than studies labeled 'feasibility'.” | concepts[8] |
| a class of study defined so that it cannot meet its journals' criterion | — “Pilot studies, by their nature, will not produce 'significant' (i.e P < 0.05) results. We believe that publishing the results of well conducted pilot or feasibility studies is important for research, irrespective of outcome.. There is an increasing awareness that publishing only 'significant' results can lead to considerably error [ 7 ].” | content_tags.statistics[0] |
Full text — tagged
The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.
43 bmcmrm BMC Medical Research Methodology BMC Med Res Methodol BMC PMC2912920 2912920 2912920 20637084 10.1186/1471-2288-10-67 What is a pilotterm or feasibility studyterm·def? A reviewterm·mth of current practice and editorial policy ArainpersName Mubashir 1 CampbellpersName Michael J 1 ✉ CooperpersName Cindy L 1 LancasterpersName Gillian A 2 1 Health Services Research, ScHARR, University of Sheffield, Regent Court, Regent St Sheffield S1 4DA, UK 2 Department of Mathematics and Statistics, University of LancasterpersName LA1 4YF, UK ✉ Corresponding author.
16 7 2010 10 67 67 31 7 2010 Copyright ©2010 ArainpersName et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract Background In 2004, a reviewterm·mth of pilot studiesterm published in seven major medical journals during 2000-01 recommended that the statistical analysis of such studies should be either mainly descriptive or focus on sample size estimation, while results from hypothesis testing must be interpreted with caution. We revisited these journals to see whether the subsequent recommendations have changed the practice of reporting pilot studiesterm. We also conducted a survey to identify the methodological components in registered research studies which are described as 'pilotterm' or 'feasibilityterm' studies. We extended this survey to grant-awarding bodies and editors of medical journals to discover their policies regarding the function and reporting of pilot studiesterm.
Methods Papers from 2007-08 in seven medical journals were screened to retrieve published pilot studiesterm. Reports of registered and completed studies on the UK Clinical Research Network (UKCRN) Portfolio database were retrieved and scrutinized. Guidance on the conduct and reporting of pilot studiesterm was retrieved from the websites of three grant giving bodies and seven journal editors were canvassed.
Results 54 pilotterm or feasibility studiesterm published in 2007-8 were found, of which 26 (48%) were pilot studiesterm of interventions and the remainder feasibility studiesterm. The majority incorporated hypothesis-testing (81%), a control arm (69%) and a randomization procedure (62%). Most (81%) pointed towards the need for further research. Only 8 out of 90 pilot studiesterm identified by the earlier reviewterm·mth led to subsequent main studies. Twelve studies which were interventional pilotterm/feasibility studiesterm and which included testing of some component of the research process were identified through the UKCRN Portfolio database. There was no clear distinction in use of the terms 'pilotterm' and 'feasibilityterm'. Five journal editors replied to our entreaty. In general they were loathe to publish studies described as 'pilotterm'.
Conclusion Pilot studiesterm are still poorly reported, with inappropriate emphasis on hypothesis-testing. Authors should be aware of the different requirements of pilot studiesterm, feasibility studiesterm and main studies and report them appropriately. Authors should be explicit as to the purpose of a pilot studyterm. The definitions of feasibilityterm and pilot studiesterm vary and we make proposals here to clarify terminology.
status released display-pdf yes is-olf no is-manuscript no is-preprint no is-journal-matter no is-scanned no is-retracted no Received 2010 May 20; Accepted 2010 Jul 16; Collection date 2010.
Background A brief definition is that a pilot studyterm is a 'small study for helping to design a further confirmatory study'[ 1 ]. A very useful discussion of exactly what is a pilot studyterm has been given by Thabane et al. [ 2 ] Such kinds of study may have various purposes such as testing study procedures, validity of tools, estimation of the recruitment rate, and estimation of parameters such as the variance of the outcome variable to calculate sample size etc. In pharmacological trials they may be referred to as 'proof of concept' or Phase I or Phase II studies. It has become apparent to us when reviewing research proposals that small studies with all the trappings of a major study, such as randomization and hypothesis testing may be labeled a 'pilotterm' because they do not have the power to test clinically meaningful hypotheses. The authors of such studies perhaps hope that reviewers will regard a 'pilotterm' more favourably than a small clinical trialterm. This lead us to ask when it is legitimate to label a study as a 'pilotterm' or 'feasibilityterm' study, and what features should be included in these types of studies.
LancasterpersName et al [ 3 ] conducted a reviewterm·mth of seven major medical journals in 2000-1 to produce evidence regarding the components of pilot studiesterm for randomized controlled trials. Their search included both 'pilotterm' and 'feasibilityterm' studies as keywords. They reported certain recommendations: having clear objectives in a pilot studyterm, inappropriateness of mixing pilotterm data with main research study, using mainly descriptive statistics obtained and caution regarding the use of hypothesis testing for conclusions. Arnold et al [ 1 ] recently reviewed pilot studiesterm particularly related to critical care medicine by searching the literature from 1997 to 2007. They provided narrative descriptions of some pilotterm papers particularly those describing critical care medicine procedures. They pointed out that few pilot trialsterm later evolved into subsequent published major trials. They made useful distinctions between: pilotterm work which is any background research to inform a future study, a pilot studyterm which has specific hypotheses, objectives and methodology and a pilot trialterm which is a stand-alone pilot studyterm and includes a randomization procedure. They excluded feasibility studiesterm from their consideration.
Thabane et al [ 2 ] gave a checklist of what they think should be included in a pilot studyterm. They included 'feasibilityterm' or 'vanguard' studies but did not distinguish them from pilot studiesterm. They provided a good discussion on how to interpret a pilot studyterm. They stress that not only the outcome or surrogate outcome for the subsequent main study should be described but also that a pilot studyterm should have feasibilityterm outcomes which should be clearly defined and described. Their article was opinion based and not supported by a reviewterm·mth of current practice.
The objective of this paper is to provide writers and reviewers of research proposals with evidence from a variety of sources for which components they should expect, and which are unnecessary or unhelpful, in a study which is labeled as a pilotterm or feasibility studyterm·def. To do this we repeated LancasterpersName et al's [ 3 ] reviewterm·mth for current papers see if there has been any change in how pilot studiesterm were reported since their study. As many pilot studiesterm are never published we also identified pilot studiesterm which were registered with the UK Clinical Research Network (UKCRN) Portfolio Database. This aims to be a "complete picture of the clinical research which is currently taking place across the UK". All studies included have to have been peer reviewed through a formal independent process. We examined the websites of some grant giving bodies to find their definition of a pilot studyterm and their funding policy toward them. Finally we contacted editors of leading medical journals to discover their policy of accepting studies described as 'pilotterm' or 'feasibilityterm'.
Methods Literature survey MEDLINE, Web of Science and university library data bases were searched for the years 2007-8 using the same key words "Pilotterm" or "Feasibilityterm" as used by LancasterpersName et al. [ 3 ]. We reviewed the same four general medicine journals: the British Medical Journal (BMJ), Lancet, the New England Journal of Medicine (NEJM) and the Journal of American Medical Association (JAMA) and the same three specialist journals: British Journal of Surgery (BJS), British Journal of Cancer (BJC), British Journal of Obstetrics and Gynecology (BJOG). We excluded reviewterm·mth papers. The full text of the relevant papers was obtained. GL reviewed 20 papers and classified them into groups as described in her original paper [ 3 ]. Subsequently MA, in discussion with MC, designed a data extraction form to classify the papers. We changed one category from GL's original paper. We separated the category 'Phase I/II trials' from the 'Piloting new treatment, technique, combination of treatments' category. We then classified the remaining paper into the categories described in Table 1 . The total number of research papers by journal was obtained by searching journal article with abstracts (excluding reviewsterm·mth) using Pubmed. We searched citations to see whether the pilot studiesterm identified by LancasterpersName et al [ 3 ] eventually led to main trials.
Table 1 Literature search using key words "Pilotterm" OR "Feasibilityterm" Journal Name BMJ Lancet NEJM JAMA BJS BJC BJOG Total 2007-8 Original articles 6 5 5 1 10 16 10 54 1 (1.6%) Pilotterm or feasibility studyterm·def in preparation for a trial 0 3 3 0 2 1 3 12 Piloting new technique, combination of treatments 1 0 0 0 4 2 6 13 Phase I, II trials 0 1 1 0 0 7 0 9 Piloting screening program 2 0 0 1 1 1 0 5 Piloting guidelines, educational package, patient care strategy 3 1 0 0 2 3 1 10 Laboratory testing of activity of compounds 0 0 1 0 1 2 0 4 Total research papers 292 379 444 383 338 1084 398 3318 2000-1 2 Original articles 11 17 3 7 9 33 10 90 (2.0%) Total research papers 372 1115 434 619 396 1132 381 4449 1 excluded Reviewterm·mth = 8, Commentaries = 4, News = 3, Indirectly referring to previous pilotterm = 9 2 from LancasterpersName et al [ 1 ] Portfolio database reviewterm·mth The (UKCRN) Portfolio Database was searched for the terms 'feasibilityterm' or 'pilotterm' in the title or research summary. Duplicate cases and studies classified as 'observational' were omitted. From the remaining studies those classified as 'closed' were selected to exclude studies which may not have started or progressed. Data were extracted directly from the research summary of the database or where that was insufficient the principle investigator was contacted for related publications or study protocols.
Editor and funding agency survey We wrote to the seven medical journal editors of the same journals used by LancasterpersName et al. [ 3 ], (BMJ, Lancet, NEJM, JAMA. BJS, BJC and BJOG) and looked at the policies of three funding agencies (British Medical Research Council, Research for Patient Benefit and NETSCC (National Institute for Health Research Trials and Studies Coordinating Centre). We wished to explore whether there was any specified policy of the journal for publishing pilot trialsterm and how the editors defined a pilot studyterm. We also wished to see if there was funding for pilot studiesterm.
Results Literature survey Initially 77 papers were found in the target journals for 2007-8 but 23 were reviewterm·mth papers or commentaries or indirectly referred to the word "pilotterm" or "feasibilityterm" and were not actually pilot studiesterm leaving a total of 54 papers. Table 1 shows the results by journal and by type of study and also shows the numbers reported by LancasterpersName et al. [ 3 ] for 2000-01 in the same medical journals. There was a decrease in the proportion of pilot studiesterm published over the period of time, however the difference was not statistically significant (2.0% vs 1.6%; X 2 = 1.6, P = 0.2). It is noticeable that the Phase I or Phase II studies are largely confined to the cancer journals.
LancasterpersName et al [ 3 ] found that 50% of pilot studiesterm reported the intention of further work yet we identified only 8 (8.8%) which were followed up by a major study. Of these 2 (25%) were published in the same journal as the pilotterm.
Twenty-six of the studies found in 2007-8 were described as pilotterm or feasibility studiesterm for randomized clinical trials (RCTsterm) including Phase II studies. Table 2 gives the numbers of studies which describe specific components of RCTsterm. Sample size calculations were performed and reported in 9 (36%) of the studies. Hypothesis testing and performing inferential statistics to report significant results was observed in 21 (81%) of pilot studiesterm. The processes of blinding was observed in only 5 (20%) although the randomization procedure was applied or tested in 16 (62%) studies. Similarly a control group was assigned in most of the studies (n = 18; 69%). As many as 21 (81%) of pilot studiesterm suggested the need for further investigation of the tested drug or procedure and did not report conclusive results on the basis of their pilotterm data. The median number of participants was 76, inter-quartile range (42, 216).
Table 2 Literature survey: Frequency of methodological components appearing in pilotterm or feasibility studiesterm of interventions (n = 26 1 ) in 2007-8 n (%) Sample size calculation 9 (35%) Hypothesis testing 21 (81%) Randomization 16 (62%) Blinding 5 (19%) Control group 18 (69%) Further study suggested 21 (81%) Median number of participants (IQR) 76 [8, 1299] 1 Pilot studiesterm = 14 Feasibility studiesterm = 12 Of the 54 studies in 2007-8, a total of 20 were described as 'pilotterm' and 34 were described as 'feasibilityterm' studies. Table 3 contrasts those which were identified by the keywordterm·var 'pilotterm' with those identified by 'feasibilityterm'. Those using 'pilotterm' were more likely to have a pre-study sample size estimate, to use randomization and to use a control group. In the 'pilotterm' group 16(80%) suggested further study, in contrast to 15 (44%) in the 'feasibilityterm' group.
Table 3 Literature survey: Comparison of studies (n = 54) using the key words feasibilityterm or pilot Studyterm components Pilotterm n = 20 Feasibilityterm n = 34 Chi-squared 1 P-value Sample size 7 (35%) 3 (8%) 5.7 0.028 Hypothesis testing 14 (70%) 25 (74%) 0.78 0.51 Randomization 11 (55%) 8 (24%) 5.5 0.037 Blinding 3 (15%) 3 (9%) 0.48 0.39 Control group 13 (65%) 11 (32%) 5.4 0.020 Further study suggested 16 (80%) 15 (44%) 6.6 0.012 Number of participants Median (IQR) [Range] 62.5 (31, 189) [8, 187777] 125.5 (36, 1005) [5, 12774614] -1.04* 0.29 1 1 degree of freedom * z-statistic (Mann-Whitney test) Portfolio database reviewterm·mth A total of 34 studies were identified using the term 'feasibilityterm' or 'pilotterm' in the title or research summary which were prospective interventional studies and were closed, i.e. not currently running and available for analysis. Only 12 studies were interventional pilotterm/feasibility studiesterm which included testing of some component of the research process. Of these 5 were referred to as 'feasibilityterm', 6 as 'pilotterm' and 1 as both 'feasibilityterm' and 'pilotterm' (Table 4 ).
Table 4 Portfolio database survey: comparison of components in studies termed pilotterm or feasibilityterm Pilotterm n = 6 Feasibilityterm n = 5 Both n = 1 Methods related Miniature RCTterm 4 3 0 Testing recruitment 4 1 0 Determining sample size/numbers available 3 0 1 Resources 1 0 0 Randomization 4 1 0 Outcome measures 2 4 2 Data collection 1 0 0 Follow up/dropout 2 0 0 Intervention related Clinical outcomes 3 1 1 Dose/efficacy/safety 0 1 0 Acceptability 2 0 1 Feasibilityterm 3 0 1 The methodological components tested within these studies were: estimation of sample size; number of subjects eligible; resources (e.g. cost), time scale; population-related (e.g. exclusion criteria), randomisation process/acceptability; data collection systems/forms; outcome measures; follow-up (response rates, adherence); overall design; whole trial feasibilityterm. In addition to one or more of these, some studies also looked at clinical outcomes including: feasibilityterm/acceptability of intervention; dose, efficacy and safety of intervention.
The results are shown in Table 4 . Pilot studiesterm alone included estimation of sample size for a future bigger study and tested a greater number of components in each study. The majority of the pilotsterm and the feasibility studiesterm ran the whole study 'in miniature' as it would be in the full study, with or without randomization.
As an example of a pilot studyterm consider 'CHOICES: A pilotterm patient preference randomised controlled trial of admission to a Women's Crisis House compared with psychiatric hospital admissions' http://www.iop.kcl.ac.uk/projects/default.aspx?id=10290 . This study looked at multiple components of a potential bigger study. It aimed to determine the proportion of women unwilling to be randomised, the feasibilityterm of a patient preference RCTterm design, the outcome and cost measures to determine which outcome measures to use, the recruitment and drop out rates; and to estimate the levels of outcome variability to calculate sample sizes for the main study. It also intended to develop a user focused and designed instrument which is the outcome from the study. The sample size was 70.
Editor and funding agency survey The editors of five (out of seven) medical journals responded to our request for information regarding publishing policy for pilot studiesterm. Four of the journals did not have a specified policy about publishing pilot studiesterm and mostly reported that pilot trialsterm cannot be published if the standard is lower than a full clinical trial requirement. The Lancet has started creating space for preliminary phase I trials and set a different standard for preliminary studies. Most of the other journals do not encourage the publication of pilot studiesterm because they consider them less rigorous than main studies. Nevertheless some editors accepted pilot studiesterm for publication by compromising only on the requirement for a pre-study sample size calculation. All other methodological issued were considered as important as for the full trials, such as trial registration, randomization, hypothesis testing, statistical analysis and reporting according to the CONSORT guidelines.
All three funding bodies made a point to note that pilotterm and feasibility studiesterm would be considered for funding. Thabane et al [ 2 ] provided a list of websites which define pilotterm or feasibility studiesterm. We considered the NETSCC definition to be most helpful and to most closely mirror what investigators are doing and it is given below.
NETSCC definition of pilotterm and feasibility studiesterm http://www.netscc.ac.uk/glossary/ Feasibility Studiesterm Feasibility Studiesterm are pieces of research done before a main study. They are used to estimate important parameters that are needed to design the main study. For instance: • standard deviation of the outcome measure, which is needed in some cases to estimate sample size, • willingness of participants to be randomised, • willingness of clinicians to recruit participants, • number of eligible patients, • characteristics of the proposed outcome measure and in some cases feasibility studiesterm might involve designing a suitable outcome measure, • follow-up rates, response rates to questionnaires, adherence/compliance rates, ICCs in cluster trials, etc.
Feasibility studiesterm for randomised controlled trials may not themselves be randomised. Crucially, feasibility studiesterm do not evaluate the outcome of interestterm; that is left to the main study.
If a feasibility studyterm·def is a small randomised controlled trial, it need not have a primary outcome and the usual sort of power calculation is not normally undertaken. Instead the sample size should be adequate to estimate the critical parameters (e.g. recruitment rate) to the necessary degree of precision.
Pilot studiesterm A Pilot Studyterm is a version of the main study that is run in miniature to test whether the components of the main study can all work together. It is focused on the processes of the main study, for example to ensure recruitment, randomisation, treatment, and follow-up assessments all run smoothly. It will therefore resemble the main study in many respects. In some cases this will be the first phase of the substantive study and data from the pilotterm phase may contribute to the final analysis; this can be referred to as an internal pilotterm. Alternatively at the end of the pilot studyterm the data may be analysed and set aside, a so-called external pilotterm.
Discussion In our repeat of LancasterpersName et al's study [ 3 ] we found that the reporting of pilot studiesterm was still poor. It is generally accepted that small, underpowered clinical trials are unethical [ 4 ]. Thus it is not an excuse to label such a study as a pilotterm and hope to make it ethical. We have shown that pilot studiesterm have different objectives to RCTsterm and these should be clearly described. Participants in such studies should be informed that they are in a pilot studyterm and that there may not be a further larger study.
It is helpful to make a more formal distinction between a 'pilotterm' and a 'feasibilityterm' study. We found that studies labeled 'feasibilityterm' were conducted with more flexible methodology compared to those labeled 'pilotterm'. For example the term 'feasibilityterm' has been used for large scale studies such as a screening programme applied at a population level to determine the initial feasibilityterm of the programme. On the other hand 'pilotterm' studies were reported with more rigorous methodological components like sample size estimation, randomization and control group selection than studies labeled 'feasibilityterm'. We found the NETSCC definition to be the most helpful since it distinguishes between these types of study.
In addition it was observed that most of the pilot studiesterm report their results as inconclusive, with the intention of conducting a further, larger study. In contrast, several of the feasibility studiesterm did not admit such an intention. On the basis of their intention one would have expected about 45 of the studies identified by LancasterpersName et al in 2000/1 to have been followed by a bigger study whereas we only found 8. This would reflect the opinion of most of the journal editors and experts who responded to our survey, who felt that pilot studiesterm rarely act as a precursor for a bigger study. The main reason given was that if the pilotterm shows significant results then researchers may not find it necessary to conduct the main trial. In addition if the results are unfavorable or the authors find an unfeasible procedure, the main study is less likely to be considered useful. Our limited reviewterm·mth of funding bodies was encouraging. Certainly when reviewing grant applications, we have found it helpful to have the results of a pilot studyterm included in the bid. We think that authors of pilotsterm studies should be explicit as to their purpose, e.g. to test a new procedure in preparation for a clinical trial. We also think that authors of proposals for pilot studiesterm should be more explicit as to the criteria which lead to further studies being abandoned, and that this should be an important part of the proposal.
In the Portfolio Database reviewterm·mth, only pilot studiesterm cited an intention to estimate sample size calculations for future studies and the majority of pilot studiesterm were full studies run with smaller sample sizes to test out a number of methodological components and clinical outcomes simultaneously. In comparison the feasibility studiesterm tended to focus on fewer methodological components within individual studies. For example, the 6 pilot studiesterm reported the intention to evaluate a total of 17 methodological components whereas in the 5 feasibility studiesterm a total of only 6 methodological components were specifically identified as being under investigation (Table 4 ). However, both pilotterm and feasibility studiesterm included trials run as complete studies, including randomization, but with sample sizes smaller than would be intended in the full study and the distinction between the two terms was not clear-cut.
Another reason for conducting a pilot studyterm is to provide information to enable a sample size calculation in a subsequent main study. However since pilot studiesterm tend to be small, the results should be interpreted with caution [ 5 ]. Only a small proportion of published pilot studiesterm reported pre-study sample size calculations. Most journal editors reported that a sample size calculation is not a mandatory criterion for publishing pilot studiesterm and suggested that it should not be done.
Some authors suggest that analysis of pilot studiesterm should mainly be descriptive,[ 3 , 6 ] as hypothesis testing requires a powered sample size which is usually not available in pilot studiesterm. In addition, inferential statistics and testing hypothesis for effectiveness require a control arm which may not be present in all pilot studiesterm. However most of the pilotterm interventional studies in this reviewterm·mth contained a control group and the authors performed and reported hypothesis testing for one or more variables. Some tested the effectiveness of an intervention and others just performed statistical testing to discover any important associations in the study variables. Observed practice is not necessarily good practice and we concur with Thabane et al [ 2 ] that any testing of an intervention needs to be reported cautiously.
The views of the journal editors, albeit from a small sample, were not particularly encouraging and reflected the experience of LancasterpersName et al [ 3 ]. Pilot studiesterm, by their nature, will not produce 'significant' (i.e P < 0.05) results. We believe that publishing the results of well conducted pilotterm or feasibility studiesterm is important for research, irrespective of outcome.. There is an increasing awareness that publishing only 'significant' results can lead to considerably error [ 7 ]. The journals we considered were all established, paper journals and perhaps the newer electronic journals will be more willing to consider the publication of the results from these types of studies.
We may expect that trials will increasingly be used to evaluate 'complex interventions'[ 8 , 9 ]. The MRC guidelines [ 8 ] explicitly suggest that preliminary studies, including pilotsterm, be used prior to any major trial which seeks to evaluate a package of interventions (such as an educational course), rather than a single intervention (such as a drug). Thus it is likely that reviewers will be increasingly asked to pronounce on these and will require guidance as to how to reviewterm·mth them.
Conclusions We conclude that pilot studiesterm are still poorly reported, with inappropriate emphasis on hypothesis-testing. We believe authors should be aware of the different requirements of pilot studiesterm and feasibility studiesterm and report them appropriately. We found that in practice the definitions of feasibilityterm and pilot studiesterm are not distinct and vary between health research funding bodies and we suggest use of the NETSCC definition to clarify terminology.
Competing interests The authors declare that they have no competing interests.
Authors' contributions MA reviewed the papers of 2000/1 and those of 2007/8 under the supervision of MC and helped to draft the manuscript. MC conceived of the study, and participated in its design and coordination and drafted the manuscript. CC conducted the portfolio database study and commented on the manuscript. GA conducted the original study, reviewed 20 papers and commented on the manuscript. All authors read and approved the final manuscript.
Pre-publication history The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1471-2288/10/67/prepub Contributor Information Mubashir ArainpersName, Email: mp08ma@sheffield.ac.uk.
Michael J CampbellpersName, Email: m.j.campbellpersName@sheffield.ac.uk.
Cindy L CooperpersName, Email: c.l.cooperpersName@sheffield.ac.uk.
Gillian A LancasterpersName, Email: g.lancasterpersName@lancasterpersName.ac.uk.