Lehana ThabanepersName, Jinhui MapersName, Rong ChupersName, Ji ChengpersName, Afisi IsmailapersName, Lorena P RiospersName, Reid RobsonpersName, Marroon ThabanepersName, Lora GiangregoriopersName, Charles H. GoldsmithpersName (2010date). “A tutorial on pilot studies: the what, why and how.” BMC Medical Research Methodologytitle·j.
Object of study —
Research question —
Contribution —
phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | A tutorial on pilot studies: the what, why and how — “The objective of this paper is to provide a detailed examination of the key aspects of pilot studies for phase III trials including: 1) the general reasons for conducting a pilot study; 2) the relationships between pilot studies, proof-of-concept studies, and adaptive designs; 3) the challenges of and misconceptions about pilot studies; 4) the criteria for evaluating the success of a pilot study; 5) frequently asked questions about pilot studies; 7) some ethical aspects related to pilot studies; and 8) some suggestions on how to report the results of pilot investigations using the CONSORT format.” | |
| claim-type | method — “The objective of this paper is to provide a detailed examination of the key aspects of pilot studies for phase III trials including: 1) the general reasons for conducting a pilot study; 2) the relationships between pilot studies, proof-of-concept studies, and adaptive designs; 3) the challenges of and misconceptions about pilot studies; 4) the criteria for evaluating the success of a pilot study; 5) frequently asked questions about pilot studies; 7) some ethical aspects related to pilot studies; and 8) some suggestions on how to report the results of pilot investigations using the CONSORT format.” | |
| research question — evidence | — “The objective of this paper is to provide a detailed examination of the key aspects of pilot studies for phase III trials including: 1) the general reasons for conducting a pilot study; 2) the relationships between pilot studies, proof-of-concept studies, and adaptive designs; 3) the challenges of and misconceptions about pilot studies; 4) the criteria for evaluating the success of a pilot study; 5) frequently asked questions about pilot studies; 7) some ethical aspects related to pilot studies; and 8) some suggestions on how to report the results of pilot investigations using the CONSORT format.” | |
| mechanism | The adaptations can be prospective (e.g. stopping a trial early due to safety or futility or efficacy at interim analysis); concurrent (e.g. changes in eligibility criteria, hypotheses or study endpoints) or retrospective (e.g. changes to statistical analysis plan prior to locking database or revealing treatment codes to trial investigators or patients). — “The adaptations can be prospective (e.g. stopping a trial early due to safety or futility or efficacy at interim analysis); concurrent (e.g. changes in eligibility criteria, hypotheses or study endpoints) or retrospective (e.g. changes to statistical analysis plan prior to locking database or revealing treatment codes to trial investigators or patients).” | mechanisms[0] |
| mechanism | Unlike other phase III pilots, an internal pilot investigation does not usually address any other feasibility aspects - because it is essentially part of the main study [ 10 , 19 , 20 ].. — “Unlike other phase III pilots, an internal pilot investigation does not usually address any other feasibility aspects - because it is essentially part of the main study [ 10 , 19 , 20 ]..” | mechanisms[1] |
| mechanism | However, many of them never get published, often because of the way the results are presented [ 13 ]. — “However, many of them never get published, often because of the way the results are presented [ 13 ].” | mechanisms[2] |
| mechanism | However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes. — “However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes.” | mechanisms[3] |
| builds-on | Lancaster et al. approach to quantitative pilot studies — “Adopting a similar approach to Lancaster et al . [ 10 ], we focus on quantitative pilot studies - particularly those done prior to full-scale phase III trials. Phase I trials are non-randomized studies designed to investigate the pharmacokinetics of a drug (i.e. how a drug is distributed and metabolized in the body) including finding a dose that can be tolerated with minimal toxicity. Phase II trials provide preliminary evidence on the clinical efficacy of a drug or intervention. They may or may not be randomized. Phase III trials are randomized studies comparing two or more drugs or intervention strategies to assess efficacy and safety. Phase IV trials, usually done after registration or marketing of a drug, are non-randomized surveillance studies to document experiences (e.g. side-effects, interactions with other drugs, etc) with using the drug in practice. For the purposes of this paper, our approach to utilizing pilot studies relies on the model for complex interventions advocated by the British Medical Research Council - which explicitly recommends the use of feasibility studies prior to Phase III clinical trials, but stresses the iterative nature of the processes of development, feasibility and piloting, evaluation and implementation [ 11 ].” | claim_relations[0] |
| builds-on | British Medical Research Council model for complex interventions — “Adopting a similar approach to Lancaster et al . [ 10 ], we focus on quantitative pilot studies - particularly those done prior to full-scale phase III trials. Phase I trials are non-randomized studies designed to investigate the pharmacokinetics of a drug (i.e. how a drug is distributed and metabolized in the body) including finding a dose that can be tolerated with minimal toxicity. Phase II trials provide preliminary evidence on the clinical efficacy of a drug or intervention. They may or may not be randomized. Phase III trials are randomized studies comparing two or more drugs or intervention strategies to assess efficacy and safety. Phase IV trials, usually done after registration or marketing of a drug, are non-randomized surveillance studies to document experiences (e.g. side-effects, interactions with other drugs, etc) with using the drug in practice. For the purposes of this paper, our approach to utilizing pilot studies relies on the model for complex interventions advocated by the British Medical Research Council - which explicitly recommends the use of feasibility studies prior to Phase III clinical trials, but stresses the iterative nature of the processes of development, feasibility and piloting, evaluation and implementation [ 11 ].” | claim_relations[1] |
| extends | CONSORT Statement — “Adopted from the CONSORT Statement [ 9 ], Table 3 provides a checklist of items to consider including in a report of a pilot study.” | claim_relations[2] |
| qualifies | use of pilot-study treatment-effect estimates for sample-size or power calculations — “Quite often the emphasis is wrongly placed on statistical significance, not on feasibility - which is the main focus of the pilot study. Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibility objectives; no clear analytic plans; and certainly no clear criteria for success of feasibility. In many cases, pilot studies are conducted to generate data for sample size calculations. This seems especially sensible in situations where there are no data from previous studies to inform this process. However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes. Therefore if not used cautiously, results of pilot studies can potentially mislead sample size or power calculations [ 21 ] -- particularly if the pilot study was done to see if there is likely to be a treatment effect in the main study.” | claim_relations[3] |
| supports | preplanned pooling of pilot and main-study data — “We recommend if any pooling of pilot and main study data is considered, this should be planned beforehand, described clearly in the protocol with clear discussion of the statistical consequences and methods. The goal is to avoid or minimize the potential bias that may occur due to multiple testing issues or any other opportunistic actions by investigators. In general, pooling when done appropriately can increase the efficiency of the main study [ 23 ].” | claim_relations[4] |
| sense | pilot study - a feasibility study intended to guide planning of a large-scale investigation — “A closer look at these definitions reveals that they are similar to the ones above in that a pilot study is synonymous with a feasibility study intended to guide the planning of a large-scale investigation.” | senses[0] |
| sense | proof-of-concept study - a clinical trial carried out to determine whether a treatment is biologically active or inactive — “Relationships between Pilot Studies, Proof-of-Concept Studies, and Adaptive Designs A proof-of-concept (PoC) study is defined as a clinical trial carried out to determine if a treatment (drug) is biologically active or inactive [ 14 ].” | senses[1] |
| note | Lancaster et al. approach to quantitative pilot studies — The paper explicitly adopts a similar quantitative-pilot approach. — “Adopting a similar approach to Lancaster et al . [ 10 ], we focus on quantitative pilot studies - particularly those done prior to full-scale phase III trials. Phase I trials are non-randomized studies designed to investigate the pharmacokinetics of a drug (i.e. how a drug is distributed and metabolized in the body) including finding a dose that can be tolerated with minimal toxicity. Phase II trials provide preliminary evidence on the clinical efficacy of a drug or intervention. They may or may not be randomized. Phase III trials are randomized studies comparing two or more drugs or intervention strategies to assess efficacy and safety. Phase IV trials, usually done after registration or marketing of a drug, are non-randomized surveillance studies to document experiences (e.g. side-effects, interactions with other drugs, etc) with using the drug in practice. For the purposes of this paper, our approach to utilizing pilot studies relies on the model for complex interventions advocated by the British Medical Research Council - which explicitly recommends the use of feasibility studies prior to Phase III clinical trials, but stresses the iterative nature of the processes of development, feasibility and piloting, evaluation and implementation [ 11 ].” | claim_relations[0] |
| weight | Lancaster et al. approach to quantitative pilot studies — 0.95 — “Adopting a similar approach to Lancaster et al . [ 10 ], we focus on quantitative pilot studies - particularly those done prior to full-scale phase III trials. Phase I trials are non-randomized studies designed to investigate the pharmacokinetics of a drug (i.e. how a drug is distributed and metabolized in the body) including finding a dose that can be tolerated with minimal toxicity. Phase II trials provide preliminary evidence on the clinical efficacy of a drug or intervention. They may or may not be randomized. Phase III trials are randomized studies comparing two or more drugs or intervention strategies to assess efficacy and safety. Phase IV trials, usually done after registration or marketing of a drug, are non-randomized surveillance studies to document experiences (e.g. side-effects, interactions with other drugs, etc) with using the drug in practice. For the purposes of this paper, our approach to utilizing pilot studies relies on the model for complex interventions advocated by the British Medical Research Council - which explicitly recommends the use of feasibility studies prior to Phase III clinical trials, but stresses the iterative nature of the processes of development, feasibility and piloting, evaluation and implementation [ 11 ].” | claim_relations[0] |
| note | British Medical Research Council model for complex interventions — Categorical reliance stated by the authors. — “Adopting a similar approach to Lancaster et al . [ 10 ], we focus on quantitative pilot studies - particularly those done prior to full-scale phase III trials. Phase I trials are non-randomized studies designed to investigate the pharmacokinetics of a drug (i.e. how a drug is distributed and metabolized in the body) including finding a dose that can be tolerated with minimal toxicity. Phase II trials provide preliminary evidence on the clinical efficacy of a drug or intervention. They may or may not be randomized. Phase III trials are randomized studies comparing two or more drugs or intervention strategies to assess efficacy and safety. Phase IV trials, usually done after registration or marketing of a drug, are non-randomized surveillance studies to document experiences (e.g. side-effects, interactions with other drugs, etc) with using the drug in practice. For the purposes of this paper, our approach to utilizing pilot studies relies on the model for complex interventions advocated by the British Medical Research Council - which explicitly recommends the use of feasibility studies prior to Phase III clinical trials, but stresses the iterative nature of the processes of development, feasibility and piloting, evaluation and implementation [ 11 ].” | claim_relations[1] |
| note | CONSORT Statement — The paper adapts CONSORT into a pilot-study reporting checklist. — “Adopted from the CONSORT Statement [ 9 ], Table 3 provides a checklist of items to consider including in a report of a pilot study.” | claim_relations[2] |
| weight | CONSORT Statement — 0.95 — “Adopted from the CONSORT Statement [ 9 ], Table 3 provides a checklist of items to consider including in a report of a pilot study.” | claim_relations[2] |
| note | use of pilot-study treatment-effect estimates for sample-size or power calculations — Limited sample sizes can make pilot treatment-effect estimates unrealistic or biased. — “Quite often the emphasis is wrongly placed on statistical significance, not on feasibility - which is the main focus of the pilot study. Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibility objectives; no clear analytic plans; and certainly no clear criteria for success of feasibility. In many cases, pilot studies are conducted to generate data for sample size calculations. This seems especially sensible in situations where there are no data from previous studies to inform this process. However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes. Therefore if not used cautiously, results of pilot studies can potentially mislead sample size or power calculations [ 21 ] -- particularly if the pilot study was done to see if there is likely to be a treatment effect in the main study.” | claim_relations[3] |
| weight | use of pilot-study treatment-effect estimates for sample-size or power calculations — 0.95 — “Quite often the emphasis is wrongly placed on statistical significance, not on feasibility - which is the main focus of the pilot study. Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibility objectives; no clear analytic plans; and certainly no clear criteria for success of feasibility. In many cases, pilot studies are conducted to generate data for sample size calculations. This seems especially sensible in situations where there are no data from previous studies to inform this process. However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes. Therefore if not used cautiously, results of pilot studies can potentially mislead sample size or power calculations [ 21 ] -- particularly if the pilot study was done to see if there is likely to be a treatment effect in the main study.” | claim_relations[3] |
| note | preplanned pooling of pilot and main-study data — The recommendation is conditional on advance planning and explicit statistical handling. — “We recommend if any pooling of pilot and main study data is considered, this should be planned beforehand, described clearly in the protocol with clear discussion of the statistical consequences and methods. The goal is to avoid or minimize the potential bias that may occur due to multiple testing issues or any other opportunistic actions by investigators. In general, pooling when done appropriately can increase the efficiency of the main study [ 23 ].” | claim_relations[4] |
| weight | preplanned pooling of pilot and main-study data — 0.95 — “We recommend if any pooling of pilot and main study data is considered, this should be planned beforehand, described clearly in the protocol with clear discussion of the statistical consequences and methods. The goal is to avoid or minimize the potential bias that may occur due to multiple testing issues or any other opportunistic actions by investigators. In general, pooling when done appropriately can increase the efficiency of the main study [ 23 ].” | claim_relations[4] |
| note | pilot study —same-as→ feasibility study — Categorical synonymy stated by the paper. — “A closer look at these definitions reveals that they are similar to the ones above in that a pilot study is synonymous with a feasibility study intended to guide the planning of a large-scale investigation.” | concept_edges[0] |
| relation | pilot study —same-as→ feasibility study — A pilot study is synonymous with a feasibility study used to plan a larger investigation. — “A closer look at these definitions reveals that they are similar to the ones above in that a pilot study is synonymous with a feasibility study intended to guide the planning of a large-scale investigation.” | concept_edges[0] |
| note | Process —part-of→ rationale for a pilot study — Categorical membership in the stated four-part classification. — “In general, the rationale for a pilot study can be grouped under several broad classifications - process, resources, management and scientific (see also http://www.childrens-mercy.org/stats/plan/pilot.asp for a different classification): • Process: This assesses the feasibility of the steps that need to take place as part of the main study. Examples include determining recruitment rates, retention rates, etc. • Resources: This deals with assessing time and budget problems that can occur during the main study. The idea is to collect some pilot data on such things as the length of time to mail or fill out all the survey forms. • Management: This covers potential human and data optimization problems such as personnel and data management issues at participating centres. • Scientific: This deals with the assessment of treatment safety, determination of dose levels and response, and estimation of treatment effect and its variance. Table 2 summarizes this classification with specific examples.” | concept_edges[1] |
| relation | Process —part-of→ rationale for a pilot study — Process feasibility is one of the four broad rationales for a pilot study. — “In general, the rationale for a pilot study can be grouped under several broad classifications - process, resources, management and scientific (see also http://www.childrens-mercy.org/stats/plan/pilot.asp for a different classification): • Process: This assesses the feasibility of the steps that need to take place as part of the main study. Examples include determining recruitment rates, retention rates, etc. • Resources: This deals with assessing time and budget problems that can occur during the main study. The idea is to collect some pilot data on such things as the length of time to mail or fill out all the survey forms. • Management: This covers potential human and data optimization problems such as personnel and data management issues at participating centres. • Scientific: This deals with the assessment of treatment safety, determination of dose levels and response, and estimation of treatment effect and its variance. Table 2 summarizes this classification with specific examples.” | concept_edges[1] |
| note | Resources —part-of→ rationale for a pilot study — Categorical membership in the stated four-part classification. — “In general, the rationale for a pilot study can be grouped under several broad classifications - process, resources, management and scientific (see also http://www.childrens-mercy.org/stats/plan/pilot.asp for a different classification): • Process: This assesses the feasibility of the steps that need to take place as part of the main study. Examples include determining recruitment rates, retention rates, etc. • Resources: This deals with assessing time and budget problems that can occur during the main study. The idea is to collect some pilot data on such things as the length of time to mail or fill out all the survey forms. • Management: This covers potential human and data optimization problems such as personnel and data management issues at participating centres. • Scientific: This deals with the assessment of treatment safety, determination of dose levels and response, and estimation of treatment effect and its variance. Table 2 summarizes this classification with specific examples.” | concept_edges[2] |
| relation | Resources —part-of→ rationale for a pilot study — Resource feasibility is one of the four broad rationales for a pilot study. — “In general, the rationale for a pilot study can be grouped under several broad classifications - process, resources, management and scientific (see also http://www.childrens-mercy.org/stats/plan/pilot.asp for a different classification): • Process: This assesses the feasibility of the steps that need to take place as part of the main study. Examples include determining recruitment rates, retention rates, etc. • Resources: This deals with assessing time and budget problems that can occur during the main study. The idea is to collect some pilot data on such things as the length of time to mail or fill out all the survey forms. • Management: This covers potential human and data optimization problems such as personnel and data management issues at participating centres. • Scientific: This deals with the assessment of treatment safety, determination of dose levels and response, and estimation of treatment effect and its variance. Table 2 summarizes this classification with specific examples.” | concept_edges[2] |
| note | Management —part-of→ rationale for a pilot study — Categorical membership in the stated four-part classification. — “In general, the rationale for a pilot study can be grouped under several broad classifications - process, resources, management and scientific (see also http://www.childrens-mercy.org/stats/plan/pilot.asp for a different classification): • Process: This assesses the feasibility of the steps that need to take place as part of the main study. Examples include determining recruitment rates, retention rates, etc. • Resources: This deals with assessing time and budget problems that can occur during the main study. The idea is to collect some pilot data on such things as the length of time to mail or fill out all the survey forms. • Management: This covers potential human and data optimization problems such as personnel and data management issues at participating centres. • Scientific: This deals with the assessment of treatment safety, determination of dose levels and response, and estimation of treatment effect and its variance. Table 2 summarizes this classification with specific examples.” | concept_edges[3] |
| relation | Management —part-of→ rationale for a pilot study — Management feasibility is one of the four broad rationales for a pilot study. — “In general, the rationale for a pilot study can be grouped under several broad classifications - process, resources, management and scientific (see also http://www.childrens-mercy.org/stats/plan/pilot.asp for a different classification): • Process: This assesses the feasibility of the steps that need to take place as part of the main study. Examples include determining recruitment rates, retention rates, etc. • Resources: This deals with assessing time and budget problems that can occur during the main study. The idea is to collect some pilot data on such things as the length of time to mail or fill out all the survey forms. • Management: This covers potential human and data optimization problems such as personnel and data management issues at participating centres. • Scientific: This deals with the assessment of treatment safety, determination of dose levels and response, and estimation of treatment effect and its variance. Table 2 summarizes this classification with specific examples.” | concept_edges[3] |
| note | Scientific —part-of→ rationale for a pilot study — Categorical membership in the stated four-part classification. — “In general, the rationale for a pilot study can be grouped under several broad classifications - process, resources, management and scientific (see also http://www.childrens-mercy.org/stats/plan/pilot.asp for a different classification): • Process: This assesses the feasibility of the steps that need to take place as part of the main study. Examples include determining recruitment rates, retention rates, etc. • Resources: This deals with assessing time and budget problems that can occur during the main study. The idea is to collect some pilot data on such things as the length of time to mail or fill out all the survey forms. • Management: This covers potential human and data optimization problems such as personnel and data management issues at participating centres. • Scientific: This deals with the assessment of treatment safety, determination of dose levels and response, and estimation of treatment effect and its variance. Table 2 summarizes this classification with specific examples.” | concept_edges[4] |
| relation | Scientific —part-of→ rationale for a pilot study — Scientific feasibility is one of the four broad rationales for a pilot study. — “In general, the rationale for a pilot study can be grouped under several broad classifications - process, resources, management and scientific (see also http://www.childrens-mercy.org/stats/plan/pilot.asp for a different classification): • Process: This assesses the feasibility of the steps that need to take place as part of the main study. Examples include determining recruitment rates, retention rates, etc. • Resources: This deals with assessing time and budget problems that can occur during the main study. The idea is to collect some pilot data on such things as the length of time to mail or fill out all the survey forms. • Management: This covers potential human and data optimization problems such as personnel and data management issues at participating centres. • Scientific: This deals with the assessment of treatment safety, determination of dose levels and response, and estimation of treatment effect and its variance. Table 2 summarizes this classification with specific examples.” | concept_edges[4] |
| note | proof-of-concept (PoC) study —distinct-from→ pilot study — Gradable distinction: the paper explicitly hedges with 'may not necessarily'. — “A proof-of-concept (PoC) study is defined as a clinical trial carried out to determine if a treatment (drug) is biologically active or inactive [ 14 ]. PoC studies usually use surrogate markers as endpoints. In general, they are phase I/II studies - which, as noted above, investigate the safety profile, dose level and response to new drugs [ 15 ]. Thus, although designed to inform the planning of phase III trials for registration or licensing of new drugs, PoC studies may not necessarily fit our restricted definition of pilot studies aimed at assessing feasibility of phase III trials as outlined in Section 2.” | concept_edges[5] |
| relation | proof-of-concept (PoC) study —distinct-from→ pilot study — A proof-of-concept study may fall outside the paper's restricted feasibility-based definition of a pilot study. — “A proof-of-concept (PoC) study is defined as a clinical trial carried out to determine if a treatment (drug) is biologically active or inactive [ 14 ]. PoC studies usually use surrogate markers as endpoints. In general, they are phase I/II studies - which, as noted above, investigate the safety profile, dose level and response to new drugs [ 15 ]. Thus, although designed to inform the planning of phase III trials for registration or licensing of new drugs, PoC studies may not necessarily fit our restricted definition of pilot studies aimed at assessing feasibility of phase III trials as outlined in Section 2.” | concept_edges[5] |
| weight | proof-of-concept (PoC) study —distinct-from→ pilot study — 0.5 — “A proof-of-concept (PoC) study is defined as a clinical trial carried out to determine if a treatment (drug) is biologically active or inactive [ 14 ]. PoC studies usually use surrogate markers as endpoints. In general, they are phase I/II studies - which, as noted above, investigate the safety profile, dose level and response to new drugs [ 15 ]. Thus, although designed to inform the planning of phase III trials for registration or licensing of new drugs, PoC studies may not necessarily fit our restricted definition of pilot studies aimed at assessing feasibility of phase III trials as outlined in Section 2.” | concept_edges[5] |
| note | statistical significance —contrasted-with→ feasibility — Strong normative contrast; the paper explicitly calls one focus wrong and the other primary. — “Quite often the emphasis is wrongly placed on statistical significance, not on feasibility - which is the main focus of the pilot study. Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibility objectives; no clear analytic plans; and certainly no clear criteria for success of feasibility. In many cases, pilot studies are conducted to generate data for sample size calculations. This seems especially sensible in situations where there are no data from previous studies to inform this process. However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes. Therefore if not used cautiously, results of pilot studies can potentially mislead sample size or power calculations [ 21 ] -- particularly if the pilot study was done to see if there is likely to be a treatment effect in the main study.” | concept_edges[6] |
| relation | statistical significance —contrasted-with→ feasibility — Statistical significance is contrasted with feasibility as the proper focus of a pilot study. — “Quite often the emphasis is wrongly placed on statistical significance, not on feasibility - which is the main focus of the pilot study. Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibility objectives; no clear analytic plans; and certainly no clear criteria for success of feasibility. In many cases, pilot studies are conducted to generate data for sample size calculations. This seems especially sensible in situations where there are no data from previous studies to inform this process. However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes. Therefore if not used cautiously, results of pilot studies can potentially mislead sample size or power calculations [ 21 ] -- particularly if the pilot study was done to see if there is likely to be a treatment effect in the main study.” | concept_edges[6] |
| weight | statistical significance —contrasted-with→ feasibility — 0.95 — “Quite often the emphasis is wrongly placed on statistical significance, not on feasibility - which is the main focus of the pilot study. Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibility objectives; no clear analytic plans; and certainly no clear criteria for success of feasibility. In many cases, pilot studies are conducted to generate data for sample size calculations. This seems especially sensible in situations where there are no data from previous studies to inform this process. However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes. Therefore if not used cautiously, results of pilot studies can potentially mislead sample size or power calculations [ 21 ] -- particularly if the pilot study was done to see if there is likely to be a treatment effect in the main study.” | concept_edges[6] |
| audit | treatment effects —causes→ sample size or power calculations — Audit 2026-09-28: DOUBTFUL - read against its span and kept, but pilot ESTIMATES of treatment effects can mislead power calculations; causes drops 'estimates' and the hedge. — “Quite often the emphasis is wrongly placed on statistical significance, not on feasibility - which is the main focus of the pilot study. Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibility objectives; no clear analytic plans; and certainly no clear criteria for success of feasibility. In many cases, pilot studies are conducted to generate data for sample size calculations. This seems especially sensible in situations where there are no data from previous studies to inform this process. However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes. Therefore if not used cautiously, results of pilot studies can potentially mislead sample size or power calculations [ 21 ] -- particularly if the pilot study was done to see if there is likely to be a treatment effect in the main study.” | concept_edges[7] |
| note | treatment effects —causes→ sample size or power calculations — Gradable conditional causal claim: 'if not used cautiously' and 'can potentially'. — “Quite often the emphasis is wrongly placed on statistical significance, not on feasibility - which is the main focus of the pilot study. Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibility objectives; no clear analytic plans; and certainly no clear criteria for success of feasibility. In many cases, pilot studies are conducted to generate data for sample size calculations. This seems especially sensible in situations where there are no data from previous studies to inform this process. However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes. Therefore if not used cautiously, results of pilot studies can potentially mislead sample size or power calculations [ 21 ] -- particularly if the pilot study was done to see if there is likely to be a treatment effect in the main study.” | concept_edges[7] |
| relation | treatment effects —causes→ sample size or power calculations — Pilot-derived treatment-effect estimates can mislead sample-size or power calculations when they are used incautiously. — “Quite often the emphasis is wrongly placed on statistical significance, not on feasibility - which is the main focus of the pilot study. Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibility objectives; no clear analytic plans; and certainly no clear criteria for success of feasibility. In many cases, pilot studies are conducted to generate data for sample size calculations. This seems especially sensible in situations where there are no data from previous studies to inform this process. However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes. Therefore if not used cautiously, results of pilot studies can potentially mislead sample size or power calculations [ 21 ] -- particularly if the pilot study was done to see if there is likely to be a treatment effect in the main study.” | concept_edges[7] |
| weight | treatment effects —causes→ sample size or power calculations — 0.5 — “Quite often the emphasis is wrongly placed on statistical significance, not on feasibility - which is the main focus of the pilot study. Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibility objectives; no clear analytic plans; and certainly no clear criteria for success of feasibility. In many cases, pilot studies are conducted to generate data for sample size calculations. This seems especially sensible in situations where there are no data from previous studies to inform this process. However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes. Therefore if not used cautiously, results of pilot studies can potentially mislead sample size or power calculations [ 21 ] -- particularly if the pilot study was done to see if there is likely to be a treatment effect in the main study.” | concept_edges[7] |
| audit | pooling of pilot and main study data —causes→ efficiency of the main study — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but attributed: the span says pooling CAN increase efficiency [23]; hedged. — “We recommend if any pooling of pilot and main study data is considered, this should be planned beforehand, described clearly in the protocol with clear discussion of the statistical consequences and methods. The goal is to avoid or minimize the potential bias that may occur due to multiple testing issues or any other opportunistic actions by investigators. In general, pooling when done appropriately can increase the efficiency of the main study [ 23 ].” | concept_edges[8] |
| effect direction | pooling of pilot and main study data —causes→ efficiency of the main study — increases — “We recommend if any pooling of pilot and main study data is considered, this should be planned beforehand, described clearly in the protocol with clear discussion of the statistical consequences and methods. The goal is to avoid or minimize the potential bias that may occur due to multiple testing issues or any other opportunistic actions by investigators. In general, pooling when done appropriately can increase the efficiency of the main study [ 23 ].” — “pooling when done appropriately can increase the efficiency of the main study” | concept_edges[8] |
| note | pooling of pilot and main study data —causes→ efficiency of the main study — Gradable conditional causal claim: pooling must be done appropriately and can increase efficiency. — “We recommend if any pooling of pilot and main study data is considered, this should be planned beforehand, described clearly in the protocol with clear discussion of the statistical consequences and methods. The goal is to avoid or minimize the potential bias that may occur due to multiple testing issues or any other opportunistic actions by investigators. In general, pooling when done appropriately can increase the efficiency of the main study [ 23 ].” | concept_edges[8] |
| relation | pooling of pilot and main study data —causes→ efficiency of the main study — Appropriate pooling can increase the efficiency of the main study. — “We recommend if any pooling of pilot and main study data is considered, this should be planned beforehand, described clearly in the protocol with clear discussion of the statistical consequences and methods. The goal is to avoid or minimize the potential bias that may occur due to multiple testing issues or any other opportunistic actions by investigators. In general, pooling when done appropriately can increase the efficiency of the main study [ 23 ].” | concept_edges[8] |
| weight | pooling of pilot and main study data —causes→ efficiency of the main study — 0.75 — “We recommend if any pooling of pilot and main study data is considered, this should be planned beforehand, described clearly in the protocol with clear discussion of the statistical consequences and methods. The goal is to avoid or minimize the potential bias that may occur due to multiple testing issues or any other opportunistic actions by investigators. In general, pooling when done appropriately can increase the efficiency of the main study [ 23 ].” | concept_edges[8] |
| audit | pilot study —causes→ likelihood of success of the main study — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says a pilot CAN enhance the likelihood of success; hedged. — “Pilot or vanguard studies provide a good opportunity to assess feasibility of large full-scale studies. Pilot studies are the best way to assess feasibility of a large expensive full-scale study, and in fact are an almost essential pre-requisite. Conducting a pilot prior to the main study can enhance the likelihood of success of the main study and potentially help to avoid doomed main studies. Pilot studies should be well designed with clear feasibility objectives, clear analytic plans, and explicit criteria for determining success of feasibility. They should be used cautiously for determining treatment effects and variance estimates for power or sample size calculations. Finally, they should be scrutinized the same way as full scale studies, and every attempt should be taken to publish the results in peer-reviewed journals.” | concept_edges[9] |
| effect direction | pilot study —causes→ likelihood of success of the main study — increases — “Pilot or vanguard studies provide a good opportunity to assess feasibility of large full-scale studies. Pilot studies are the best way to assess feasibility of a large expensive full-scale study, and in fact are an almost essential pre-requisite. Conducting a pilot prior to the main study can enhance the likelihood of success of the main study and potentially help to avoid doomed main studies. Pilot studies should be well designed with clear feasibility objectives, clear analytic plans, and explicit criteria for determining success of feasibility. They should be used cautiously for determining treatment effects and variance estimates for power or sample size calculations. Finally, they should be scrutinized the same way as full scale studies, and every attempt should be taken to publish the results in peer-reviewed journals.” — “can enhance the likelihood of success of the main study” | concept_edges[9] |
| note | pilot study —causes→ likelihood of success of the main study — Gradable causal claim because the paper says a pilot can enhance success and potentially avoid failure. — “Pilot or vanguard studies provide a good opportunity to assess feasibility of large full-scale studies. Pilot studies are the best way to assess feasibility of a large expensive full-scale study, and in fact are an almost essential pre-requisite. Conducting a pilot prior to the main study can enhance the likelihood of success of the main study and potentially help to avoid doomed main studies. Pilot studies should be well designed with clear feasibility objectives, clear analytic plans, and explicit criteria for determining success of feasibility. They should be used cautiously for determining treatment effects and variance estimates for power or sample size calculations. Finally, they should be scrutinized the same way as full scale studies, and every attempt should be taken to publish the results in peer-reviewed journals.” | concept_edges[9] |
| relation | pilot study —causes→ likelihood of success of the main study — Conducting a pilot can enhance the likelihood that the main study succeeds. — “Pilot or vanguard studies provide a good opportunity to assess feasibility of large full-scale studies. Pilot studies are the best way to assess feasibility of a large expensive full-scale study, and in fact are an almost essential pre-requisite. Conducting a pilot prior to the main study can enhance the likelihood of success of the main study and potentially help to avoid doomed main studies. Pilot studies should be well designed with clear feasibility objectives, clear analytic plans, and explicit criteria for determining success of feasibility. They should be used cautiously for determining treatment effects and variance estimates for power or sample size calculations. Finally, they should be scrutinized the same way as full scale studies, and every attempt should be taken to publish the results in peer-reviewed journals.” | concept_edges[9] |
| weight | pilot study —causes→ likelihood of success of the main study — 0.75 — “Pilot or vanguard studies provide a good opportunity to assess feasibility of large full-scale studies. Pilot studies are the best way to assess feasibility of a large expensive full-scale study, and in fact are an almost essential pre-requisite. Conducting a pilot prior to the main study can enhance the likelihood of success of the main study and potentially help to avoid doomed main studies. Pilot studies should be well designed with clear feasibility objectives, clear analytic plans, and explicit criteria for determining success of feasibility. They should be used cautiously for determining treatment effects and variance estimates for power or sample size calculations. Finally, they should be scrutinized the same way as full scale studies, and every attempt should be taken to publish the results in peer-reviewed journals.” | concept_edges[9] |
| relation | criteria for success → proceed to the main study — Success criteria govern the proceed/stop decision for the main study. — “It is always important to state the criteria for success of a pilot study. The criteria should be based on the primary feasibility objectives. These provide the basis for interpreting the results of the pilot study and determining whether it is feasible to proceed to the main study. In general, the outcome of a pilot study can be one of the following: (i) Stop - main study not feasible; (ii) Continue, but modify protocol - feasible with modifications; (iii) Continue without modifications, but monitor closely - feasible with close monitoring and (iv) Continue without modifications - feasible as is.” | unhoused_relations[0] |
| relation | surrogate markers → proof-of-concept (PoC) study — Surrogate markers are usually used as proof-of-concept study endpoints. — “A proof-of-concept (PoC) study is defined as a clinical trial carried out to determine if a treatment (drug) is biologically active or inactive [ 14 ]. PoC studies usually use surrogate markers as endpoints. In general, they are phase I/II studies - which, as noted above, investigate the safety profile, dose level and response to new drugs [ 15 ]. Thus, although designed to inform the planning of phase III trials for registration or licensing of new drugs, PoC studies may not necessarily fit our restricted definition of pilot studies aimed at assessing feasibility of phase III trials as outlined in Section 2.” | unhoused_relations[1] |
| weight | surrogate markers → proof-of-concept (PoC) study — 0.75 — “A proof-of-concept (PoC) study is defined as a clinical trial carried out to determine if a treatment (drug) is biologically active or inactive [ 14 ]. PoC studies usually use surrogate markers as endpoints. In general, they are phase I/II studies - which, as noted above, investigate the safety profile, dose level and response to new drugs [ 15 ]. Thus, although designed to inform the planning of phase III trials for registration or licensing of new drugs, PoC studies may not necessarily fit our restricted definition of pilot studies aimed at assessing feasibility of phase III trials as outlined in Section 2.” | unhoused_relations[1] |
| relation | adaptive trial design → a priori decision rules — A priori decision rules guide piloting adaptations in adaptive trial designs. — “An adaptive trial design refers to a design that allows modifications to be made to a trial's design or statistical procedures during its conduct, with the purpose of efficiently identifying clinical benefits/risks of new drugs or to increase the probability of success of clinical development [ 16 ]. The adaptations can be prospective (e.g. stopping a trial early due to safety or futility or efficacy at interim analysis); concurrent (e.g. changes in eligibility criteria, hypotheses or study endpoints) or retrospective (e.g. changes to statistical analysis plan prior to locking database or revealing treatment codes to trial investigators or patients). Piloting is normally built into adaptive trial designs by determining a priori decision rules to guide the adaptations based on cumulative data. For example, data from interim analyses could be used to refine sample size calculations [ 17 , 18 ]. This approach is routinely used in internal pilot studies - which are primarily designed to inform sample size calculation for the main study, with recalculation of the sample size as the key adaptation. Unlike other phase III pilots, an internal pilot investigation does not usually address any other feasibility aspects - because it is essentially part of the main study [ 10 , 19 , 20 ].. Nonetheless, we need to emphasize that whether or not a study is a pilot, depends on its objectives. An adaptive method is used as a strategy to reach that objective. Both a pilot and a non-pilot could be adaptive.” | unhoused_relations[2] |
| weight | adaptive trial design → a priori decision rules — 0.75 — “An adaptive trial design refers to a design that allows modifications to be made to a trial's design or statistical procedures during its conduct, with the purpose of efficiently identifying clinical benefits/risks of new drugs or to increase the probability of success of clinical development [ 16 ]. The adaptations can be prospective (e.g. stopping a trial early due to safety or futility or efficacy at interim analysis); concurrent (e.g. changes in eligibility criteria, hypotheses or study endpoints) or retrospective (e.g. changes to statistical analysis plan prior to locking database or revealing treatment codes to trial investigators or patients). Piloting is normally built into adaptive trial designs by determining a priori decision rules to guide the adaptations based on cumulative data. For example, data from interim analyses could be used to refine sample size calculations [ 17 , 18 ]. This approach is routinely used in internal pilot studies - which are primarily designed to inform sample size calculation for the main study, with recalculation of the sample size as the key adaptation. Unlike other phase III pilots, an internal pilot investigation does not usually address any other feasibility aspects - because it is essentially part of the main study [ 10 , 19 , 20 ].. Nonetheless, we need to emphasize that whether or not a study is a pilot, depends on its objectives. An adaptive method is used as a strategy to reach that objective. Both a pilot and a non-pilot could be adaptive.” | unhoused_relations[2] |
| relation | internal pilot studies → recalculation of the sample size — Sample-size recalculation is the key adaptation described for internal pilot studies. — “An adaptive trial design refers to a design that allows modifications to be made to a trial's design or statistical procedures during its conduct, with the purpose of efficiently identifying clinical benefits/risks of new drugs or to increase the probability of success of clinical development [ 16 ]. The adaptations can be prospective (e.g. stopping a trial early due to safety or futility or efficacy at interim analysis); concurrent (e.g. changes in eligibility criteria, hypotheses or study endpoints) or retrospective (e.g. changes to statistical analysis plan prior to locking database or revealing treatment codes to trial investigators or patients). Piloting is normally built into adaptive trial designs by determining a priori decision rules to guide the adaptations based on cumulative data. For example, data from interim analyses could be used to refine sample size calculations [ 17 , 18 ]. This approach is routinely used in internal pilot studies - which are primarily designed to inform sample size calculation for the main study, with recalculation of the sample size as the key adaptation. Unlike other phase III pilots, an internal pilot investigation does not usually address any other feasibility aspects - because it is essentially part of the main study [ 10 , 19 , 20 ].. Nonetheless, we need to emphasize that whether or not a study is a pilot, depends on its objectives. An adaptive method is used as a strategy to reach that objective. Both a pilot and a non-pilot could be adaptive.” | unhoused_relations[3] |
| weight | internal pilot studies → recalculation of the sample size — 0.95 — “An adaptive trial design refers to a design that allows modifications to be made to a trial's design or statistical procedures during its conduct, with the purpose of efficiently identifying clinical benefits/risks of new drugs or to increase the probability of success of clinical development [ 16 ]. The adaptations can be prospective (e.g. stopping a trial early due to safety or futility or efficacy at interim analysis); concurrent (e.g. changes in eligibility criteria, hypotheses or study endpoints) or retrospective (e.g. changes to statistical analysis plan prior to locking database or revealing treatment codes to trial investigators or patients). Piloting is normally built into adaptive trial designs by determining a priori decision rules to guide the adaptations based on cumulative data. For example, data from interim analyses could be used to refine sample size calculations [ 17 , 18 ]. This approach is routinely used in internal pilot studies - which are primarily designed to inform sample size calculation for the main study, with recalculation of the sample size as the key adaptation. Unlike other phase III pilots, an internal pilot investigation does not usually address any other feasibility aspects - because it is essentially part of the main study [ 10 , 19 , 20 ].. Nonetheless, we need to emphasize that whether or not a study is a pilot, depends on its objectives. An adaptive method is used as a strategy to reach that objective. Both a pilot and a non-pilot could be adaptive.” | unhoused_relations[3] |
| no type for this | criteria for success → proceed to the main study — Pilot-study success criteria provide the decision basis for whether the main study should proceed. — “It is always important to state the criteria for success of a pilot study. The criteria should be based on the primary feasibility objectives. These provide the basis for interpreting the results of the pilot study and determining whether it is feasible to proceed to the main study. In general, the outcome of a pilot study can be one of the following: (i) Stop - main study not feasible; (ii) Continue, but modify protocol - feasible with modifications; (iii) Continue without modifications, but monitor closely - feasible with close monitoring and (iv) Continue without modifications - feasible as is.” | unhoused_relations[0] |
| no type for this | surrogate markers → proof-of-concept (PoC) study — Proof-of-concept studies usually use surrogate markers as endpoints. — “A proof-of-concept (PoC) study is defined as a clinical trial carried out to determine if a treatment (drug) is biologically active or inactive [ 14 ]. PoC studies usually use surrogate markers as endpoints. In general, they are phase I/II studies - which, as noted above, investigate the safety profile, dose level and response to new drugs [ 15 ]. Thus, although designed to inform the planning of phase III trials for registration or licensing of new drugs, PoC studies may not necessarily fit our restricted definition of pilot studies aimed at assessing feasibility of phase III trials as outlined in Section 2.” | unhoused_relations[1] |
| no type for this | adaptive trial design → a priori decision rules — Piloting is normally built into adaptive designs through a priori decision rules applied to cumulative data. — “An adaptive trial design refers to a design that allows modifications to be made to a trial's design or statistical procedures during its conduct, with the purpose of efficiently identifying clinical benefits/risks of new drugs or to increase the probability of success of clinical development [ 16 ]. The adaptations can be prospective (e.g. stopping a trial early due to safety or futility or efficacy at interim analysis); concurrent (e.g. changes in eligibility criteria, hypotheses or study endpoints) or retrospective (e.g. changes to statistical analysis plan prior to locking database or revealing treatment codes to trial investigators or patients). Piloting is normally built into adaptive trial designs by determining a priori decision rules to guide the adaptations based on cumulative data. For example, data from interim analyses could be used to refine sample size calculations [ 17 , 18 ]. This approach is routinely used in internal pilot studies - which are primarily designed to inform sample size calculation for the main study, with recalculation of the sample size as the key adaptation. Unlike other phase III pilots, an internal pilot investigation does not usually address any other feasibility aspects - because it is essentially part of the main study [ 10 , 19 , 20 ].. Nonetheless, we need to emphasize that whether or not a study is a pilot, depends on its objectives. An adaptive method is used as a strategy to reach that objective. Both a pilot and a non-pilot could be adaptive.” | unhoused_relations[2] |
| no type for this | internal pilot studies → recalculation of the sample size — Internal pilot studies are primarily designed around sample-size recalculation as their key adaptation. — “An adaptive trial design refers to a design that allows modifications to be made to a trial's design or statistical procedures during its conduct, with the purpose of efficiently identifying clinical benefits/risks of new drugs or to increase the probability of success of clinical development [ 16 ]. The adaptations can be prospective (e.g. stopping a trial early due to safety or futility or efficacy at interim analysis); concurrent (e.g. changes in eligibility criteria, hypotheses or study endpoints) or retrospective (e.g. changes to statistical analysis plan prior to locking database or revealing treatment codes to trial investigators or patients). Piloting is normally built into adaptive trial designs by determining a priori decision rules to guide the adaptations based on cumulative data. For example, data from interim analyses could be used to refine sample size calculations [ 17 , 18 ]. This approach is routinely used in internal pilot studies - which are primarily designed to inform sample size calculation for the main study, with recalculation of the sample size as the key adaptation. Unlike other phase III pilots, an internal pilot investigation does not usually address any other feasibility aspects - because it is essentially part of the main study [ 10 , 19 , 20 ].. Nonetheless, we need to emphasize that whether or not a study is a pilot, depends on its objectives. An adaptive method is used as a strategy to reach that objective. Both a pilot and a non-pilot could be adaptive.” | unhoused_relations[3] |
finding · quantity · variable · definition · key-term — grounded in the full text| finding | This can increase the efficiency of the main study - see Section 5. • Can I combine the results of a pilot with the results of another study or in a meta-analysis? - Yes, provided the sampling frame and methodologies are the same. - No, if the main study is reported and it includes the pilot study. • Can the results of the pilot study be valid on their own, without existence of the main study - Yes, if the results show that it is not feasible to proceed to the main study or there is insufficient funding. • Can I apply for funding for a pilot study? - Yes. — “This can increase the efficiency of the main study - see Section 5. • Can I combine the results of a pilot with the results of another study or in a meta-analysis? - Yes, provided the sampling frame and methodologies are the same. - No, if the main study is reported and it includes the pilot study. • Can the results of the pilot study be valid on their own, without existence of the main study - Yes, if the results show that it is not feasible to proceed to the main study or there is insufficient funding. • Can I apply for funding for a pilot study? - Yes.” | content_tags.findings[0] |
| finding | Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibility objectives; no clear analytic plans; and certainly no clear criteria for success of feasibility. — “Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibility objectives; no clear analytic plans; and certainly no clear criteria for success of feasibility.” | content_tags.findings[1] |
| finding | Such studies typically do not show statistically significant results - remember that underpowered studies (with no statistically significant results) are inconclusive, not negative since "no evidence of effect" is not "evidence of no effect" [ 27 ]. • Can I combine data from a pilot with data from the main study? - Yes, provided the sampling frame and methodologies are the same. — “Such studies typically do not show statistically significant results - remember that underpowered studies (with no statistically significant results) are inconclusive, not negative since "no evidence of effect" is not "evidence of no effect" [ 27 ]. • Can I combine data from a pilot with data from the main study? - Yes, provided the sampling frame and methodologies are the same.” | content_tags.findings[2] |
| finding | A review of the commonly cited research ethics guidelines - the Nuremburg Code [ 29 ], Helsinki Declaration [ 30 ], the Belmont Report [ 31 ], ICH Good Clinical Practice [ 32 ], and the International Ethical Guidelines for Biomedical Research Involving Human Subjects [ 33 ] - shows that pilot studies are not addressed in any of these guidelines. — “A review of the commonly cited research ethics guidelines - the Nuremburg Code [ 29 ], Helsinki Declaration [ 30 ], the Belmont Report [ 31 ], ICH Good Clinical Practice [ 32 ], and the International Ethical Guidelines for Biomedical Research Involving Human Subjects [ 33 ] - shows that pilot studies are not addressed in any of these guidelines.” | content_tags.findings[3] |
| finding | State clearly what modifications in the design of the main study (if any) would be necessary to make it feasible Overall evidence of feasibility 17 General interpretation of the results in the context of current evidence of feasibility • Focus should be on feasibility — “State clearly what modifications in the design of the main study (if any) would be necessary to make it feasible Overall evidence of feasibility 17 General interpretation of the results in the context of current evidence of feasibility • Focus should be on feasibility” | content_tags.findings[4] |
| finding | Discussion Item # 15: Interpretation Interpretation of the results should focus on feasibility, taking into account the stated criteria for success of feasibility, study hypotheses, sources of potential bias or imprecision (given the feasibility nature of the study) and the dangers associated with multiplicity - repeated testing on multiple outcomes. — “Interpretation of the results should focus on feasibility, taking into account the stated criteria for success of feasibility, study hypotheses, sources of potential bias or imprecision (given the feasibility nature of the study) and the dangers associated with multiplicity - repeated testing on multiple outcomes.” | content_tags.findings[5] |
| finding | Abstract Pilot studies for phase III trials - which are comparative randomized trials designed to provide preliminary evidence on the clinical efficacy of a drug or intervention - are routinely performed in many clinical areas. — “Pilot studies for phase III trials - which are comparative randomized trials designed to provide preliminary evidence on the clinical efficacy of a drug or intervention - are routinely performed in many clinical areas.” | content_tags.findings[6] |
| finding | Conducting a pilot prior to the main study can enhance the likelihood of success of the main study and potentially help to avoid doomed main studies. — “Conducting a pilot prior to the main study can enhance the likelihood of success of the main study and potentially help to avoid doomed main studies.” | content_tags.findings[7] |
| finding | Pilot studies are sometimes referred to as "vanguard trials" (i.e. pre-studies) intended to assess the safety of treatment or interventions; to assess recruitment potential; to assess the feasibility of international collaboration or coordination for multicentre trials; to evaluate surrogate marker data in diverse patient cohorts; to increase clinical experience with the study medication or intervention, and identify the optimal dose of treatments for the phase III trials [ 4 ]. — “Pilot studies are sometimes referred to as "vanguard trials" (i.e. pre-studies) intended to assess the safety of treatment or interventions; to assess recruitment potential; to assess the feasibility of international collaboration or coordination for multicentre trials; to evaluate surrogate marker data in diverse patient cohorts; to increase clinical experience with the study medication or intervention, and identify the optimal dose of treatments for the phase III trials [ 4 ].” | content_tags.findings[8] |
| finding | As suggested by an African proverb from the Ashanti people in Ghana " You never test the depth of a river with both feet ", the main goal of pilot studies is to assess feasibility so as to avoid potentially disastrous consequences of embarking on a large study - which could potentially "drown" the whole research effort. — “As suggested by an African proverb from the Ashanti people in Ghana " You never test the depth of a river with both feet ", the main goal of pilot studies is to assess feasibility so as to avoid potentially disastrous consequences of embarking on a large study - which could potentially "drown" the whole research effort.” | content_tags.findings[9] |
| finding | It is fair to say that every major clinical trial had to start with some piloting or a small scale investigation to assess the feasibility of conducting a larger scale study: critical care [ 5 ], diabetes management intervention trials [ 6 ], cardiovascular trials [ 7 ], primary healthcare [ 8 ], to mention a few. — “It is fair to say that every major clinical trial had to start with some piloting or a small scale investigation to assess the feasibility of conducting a larger scale study: critical care [ 5 ], diabetes management intervention trials [ 6 ], cardiovascular trials [ 7 ], primary healthcare [ 8 ], to mention a few.” | content_tags.findings[10] |
| finding | Despite their noted importance, the reality is that pilot studies receive little or no attention in scientific research training. — “Despite their noted importance, the reality is that pilot studies receive little or no attention in scientific research training.” | content_tags.findings[11] |
| finding | In fact, we are not aware of any textbook that dedicates a chapter on this issue - many just mention it in passing or provide a cursory coverage of the topic. — “In fact, we are not aware of any textbook that dedicates a chapter on this issue - many just mention it in passing or provide a cursory coverage of the topic.” | content_tags.findings[12] |
| finding | The objective of this paper is to provide a detailed examination of the key aspects of pilot studies. — “The objective of this paper is to provide a detailed examination of the key aspects of pilot studies.” | content_tags.findings[13] |
| finding | Evaluation of a pilot study (i.e. how to determine if a pilot study was successful) is covered in Section 6. — “Evaluation of a pilot study (i.e. how to determine if a pilot study was successful) is covered in Section 6.” | content_tags.findings[14] |
| quantity | 6 patients (a similar study) against 12 patients, quoted as a mistaken sentiment — reported sample, observation, or event count — “In reviewing submissions to a research ethics board, we have come across sentiments such as • So-and-so did a similar study with 6 patients and got statistical significance - ours uses 12 patients (double the size)! • We did a similar pilot before (and it was published!) The third most common reason is that a pilot is a small study done by a student or an intern - which can be completed quickly and does not require funding.” | content_tags.quantities[0] |
| quantity | 95% CI; margin of error 0.05; lower bound 0.70; expected completion rate 75%; at least 75 patients — reported sample, observation, or event count — “Using a 95% CI for the proportion of eligible patients who complete the assessment form, a margin of error (ME) of 0.05, a lower bound of this CI of 0.70, and an expected completion rate of 75% based on an educated guess, the required sample for the pilot study would be at least 75 patients.” | content_tags.quantities[1] |
| quantity | 10 — reported sample, observation, or event count — “Specific arguments include • I have funding for 10 patients only; • I have limited seed (start-up) funding; • This is just a student project! • My supervisor (boss) told me to do it as a pilot .” | content_tags.quantities[2] |
| quantity | PROTECT feasibility criteria: 98.5% receive study drug within 12 hours of randomization; 91.7% receive every scheduled dose; 90% or more have lower limb compression ultrasounds on schedule; > 90% of necessary dose adjustments made appropriately — reported sample, observation, or event count — “For example, the Prophylaxis of Thromboembolism in Critical Care Trial (PROTECT) was designed to assess the feasibility of a large-scale trial with the following criteria for determining success [ 25 ]: • 98.5% of patients had to receive study drug within 12 hours of randomization; • 91.7% of patients had to receive every scheduled dose of the study drug in a blinded manner; • 90% or more of patients had to have lower limb compression ultrasounds performed at the specified times; and • > 90% of necessary dose adjustments had to have been made appropriately in response to pre-defined laboratory criteria .” | content_tags.quantities[3] |
| quantity | POET feasibility criteria: one subject per centre per week (200 subjects from four centres over 50 weeks); at least 70% of eligible patients recruited; no more than 5% of recruited subjects crossed over; complete follow-up in at least 95% — reported sample, observation, or event count — “In a second example, the PeriOperative Epidural Trial (POET) Pilot Study was designed to assess the feasibility of a large, multicentre trial with the following criteria for determining success [ 26 ]: • one subject per centre per week (i.e., 200 subjects from four centres over 50 weeks) can be recruited ; • at least 70% of all eligible patients can be recruited ; • no more than 5% of all recruited subjects crossed over from one modality to the other; and • complete follow-up in at least 95% of all recruited subjects .” | content_tags.quantities[4] |
| definition | and Adaptive Designs A proof-of-concept (PoC) study — “Relationships between Pilot Studies, Proof-of-Concept Studies, and Adaptive Designs A proof-of-concept (PoC) study is defined as a clinical trial carried out to determine if a treatment (drug) is biologically active or inactive [ 14 ].” | content_tags.definitions[0] |
| concept | pilot study — not defined in this paper's own words; a feasibility study intended to guide planning of a large-scale investigation — “A closer look at these definitions reveals that they are similar to the ones above in that a pilot study is synonymous with a feasibility study intended to guide the planning of a large-scale investigation.” | concepts[0] |
| concept | feasibility study — not defined in this paper's own words; a pilot investigation used to guide planning of a larger study — “A closer look at these definitions reveals that they are similar to the ones above in that a pilot study is synonymous with a feasibility study intended to guide the planning of a large-scale investigation.” | concepts[1] |
| concept | rationale for a pilot study — not defined in this paper's own words; the process, resources, management, and scientific reasons for piloting — “In general, the rationale for a pilot study can be grouped under several broad classifications - process, resources, management and scientific (see also http://www.childrens-mercy.org/stats/plan/pilot.asp for a different classification): • Process: This assesses the feasibility of the steps that need to take place as part of the main study. Examples include determining recruitment rates, retention rates, etc. • Resources: This deals with assessing time and budget problems that can occur during the main study. The idea is to collect some pilot data on such things as the length of time to mail or fill out all the survey forms. • Management: This covers potential human and data optimization problems such as personnel and data management issues at participating centres. • Scientific: This deals with the assessment of treatment safety, determination of dose levels and response, and estimation of treatment effect and its variance. Table 2 summarizes this classification with specific examples.” | concepts[2] |
| concept | Process — not defined in this paper's own words; feasibility of the steps required by the main study — “In general, the rationale for a pilot study can be grouped under several broad classifications - process, resources, management and scientific (see also http://www.childrens-mercy.org/stats/plan/pilot.asp for a different classification): • Process: This assesses the feasibility of the steps that need to take place as part of the main study. Examples include determining recruitment rates, retention rates, etc. • Resources: This deals with assessing time and budget problems that can occur during the main study. The idea is to collect some pilot data on such things as the length of time to mail or fill out all the survey forms. • Management: This covers potential human and data optimization problems such as personnel and data management issues at participating centres. • Scientific: This deals with the assessment of treatment safety, determination of dose levels and response, and estimation of treatment effect and its variance. Table 2 summarizes this classification with specific examples.” | concepts[3] |
| concept | Resources — not defined in this paper's own words; time and budget feasibility for the main study — “In general, the rationale for a pilot study can be grouped under several broad classifications - process, resources, management and scientific (see also http://www.childrens-mercy.org/stats/plan/pilot.asp for a different classification): • Process: This assesses the feasibility of the steps that need to take place as part of the main study. Examples include determining recruitment rates, retention rates, etc. • Resources: This deals with assessing time and budget problems that can occur during the main study. The idea is to collect some pilot data on such things as the length of time to mail or fill out all the survey forms. • Management: This covers potential human and data optimization problems such as personnel and data management issues at participating centres. • Scientific: This deals with the assessment of treatment safety, determination of dose levels and response, and estimation of treatment effect and its variance. Table 2 summarizes this classification with specific examples.” | concepts[4] |
| concept | Management — not defined in this paper's own words; human and data-management feasibility at participating centres — “In general, the rationale for a pilot study can be grouped under several broad classifications - process, resources, management and scientific (see also http://www.childrens-mercy.org/stats/plan/pilot.asp for a different classification): • Process: This assesses the feasibility of the steps that need to take place as part of the main study. Examples include determining recruitment rates, retention rates, etc. • Resources: This deals with assessing time and budget problems that can occur during the main study. The idea is to collect some pilot data on such things as the length of time to mail or fill out all the survey forms. • Management: This covers potential human and data optimization problems such as personnel and data management issues at participating centres. • Scientific: This deals with the assessment of treatment safety, determination of dose levels and response, and estimation of treatment effect and its variance. Table 2 summarizes this classification with specific examples.” | concepts[5] |
| concept | Scientific — not defined in this paper's own words; feasibility concerning safety, dose, response, effect, and variance — “In general, the rationale for a pilot study can be grouped under several broad classifications - process, resources, management and scientific (see also http://www.childrens-mercy.org/stats/plan/pilot.asp for a different classification): • Process: This assesses the feasibility of the steps that need to take place as part of the main study. Examples include determining recruitment rates, retention rates, etc. • Resources: This deals with assessing time and budget problems that can occur during the main study. The idea is to collect some pilot data on such things as the length of time to mail or fill out all the survey forms. • Management: This covers potential human and data optimization problems such as personnel and data management issues at participating centres. • Scientific: This deals with the assessment of treatment safety, determination of dose levels and response, and estimation of treatment effect and its variance. Table 2 summarizes this classification with specific examples.” | concepts[6] |
| concept | proof-of-concept (PoC) study — not defined in this paper's own words; a clinical trial that determines whether a treatment is biologically active or inactive — “A proof-of-concept (PoC) study is defined as a clinical trial carried out to determine if a treatment (drug) is biologically active or inactive [ 14 ]. PoC studies usually use surrogate markers as endpoints. In general, they are phase I/II studies - which, as noted above, investigate the safety profile, dose level and response to new drugs [ 15 ]. Thus, although designed to inform the planning of phase III trials for registration or licensing of new drugs, PoC studies may not necessarily fit our restricted definition of pilot studies aimed at assessing feasibility of phase III trials as outlined in Section 2.” | concepts[7] |
| concept | surrogate markers — not defined in this paper's own words; endpoints usually used in proof-of-concept studies — “A proof-of-concept (PoC) study is defined as a clinical trial carried out to determine if a treatment (drug) is biologically active or inactive [ 14 ]. PoC studies usually use surrogate markers as endpoints. In general, they are phase I/II studies - which, as noted above, investigate the safety profile, dose level and response to new drugs [ 15 ]. Thus, although designed to inform the planning of phase III trials for registration or licensing of new drugs, PoC studies may not necessarily fit our restricted definition of pilot studies aimed at assessing feasibility of phase III trials as outlined in Section 2.” | concepts[8] |
| concept | adaptive trial design — not defined in this paper's own words; a design permitting modifications to design or statistical procedures during trial conduct — “An adaptive trial design refers to a design that allows modifications to be made to a trial's design or statistical procedures during its conduct, with the purpose of efficiently identifying clinical benefits/risks of new drugs or to increase the probability of success of clinical development [ 16 ]. The adaptations can be prospective (e.g. stopping a trial early due to safety or futility or efficacy at interim analysis); concurrent (e.g. changes in eligibility criteria, hypotheses or study endpoints) or retrospective (e.g. changes to statistical analysis plan prior to locking database or revealing treatment codes to trial investigators or patients). Piloting is normally built into adaptive trial designs by determining a priori decision rules to guide the adaptations based on cumulative data. For example, data from interim analyses could be used to refine sample size calculations [ 17 , 18 ]. This approach is routinely used in internal pilot studies - which are primarily designed to inform sample size calculation for the main study, with recalculation of the sample size as the key adaptation. Unlike other phase III pilots, an internal pilot investigation does not usually address any other feasibility aspects - because it is essentially part of the main study [ 10 , 19 , 20 ].. Nonetheless, we need to emphasize that whether or not a study is a pilot, depends on its objectives. An adaptive method is used as a strategy to reach that objective. Both a pilot and a non-pilot could be adaptive.” | concepts[9] |
| concept | a priori decision rules — not defined in this paper's own words; predefined rules that guide adaptations using cumulative data — “An adaptive trial design refers to a design that allows modifications to be made to a trial's design or statistical procedures during its conduct, with the purpose of efficiently identifying clinical benefits/risks of new drugs or to increase the probability of success of clinical development [ 16 ]. The adaptations can be prospective (e.g. stopping a trial early due to safety or futility or efficacy at interim analysis); concurrent (e.g. changes in eligibility criteria, hypotheses or study endpoints) or retrospective (e.g. changes to statistical analysis plan prior to locking database or revealing treatment codes to trial investigators or patients). Piloting is normally built into adaptive trial designs by determining a priori decision rules to guide the adaptations based on cumulative data. For example, data from interim analyses could be used to refine sample size calculations [ 17 , 18 ]. This approach is routinely used in internal pilot studies - which are primarily designed to inform sample size calculation for the main study, with recalculation of the sample size as the key adaptation. Unlike other phase III pilots, an internal pilot investigation does not usually address any other feasibility aspects - because it is essentially part of the main study [ 10 , 19 , 20 ].. Nonetheless, we need to emphasize that whether or not a study is a pilot, depends on its objectives. An adaptive method is used as a strategy to reach that objective. Both a pilot and a non-pilot could be adaptive.” | concepts[10] |
| concept | internal pilot studies — not defined in this paper's own words; pilots within the main study whose key adaptation is sample-size recalculation — “An adaptive trial design refers to a design that allows modifications to be made to a trial's design or statistical procedures during its conduct, with the purpose of efficiently identifying clinical benefits/risks of new drugs or to increase the probability of success of clinical development [ 16 ]. The adaptations can be prospective (e.g. stopping a trial early due to safety or futility or efficacy at interim analysis); concurrent (e.g. changes in eligibility criteria, hypotheses or study endpoints) or retrospective (e.g. changes to statistical analysis plan prior to locking database or revealing treatment codes to trial investigators or patients). Piloting is normally built into adaptive trial designs by determining a priori decision rules to guide the adaptations based on cumulative data. For example, data from interim analyses could be used to refine sample size calculations [ 17 , 18 ]. This approach is routinely used in internal pilot studies - which are primarily designed to inform sample size calculation for the main study, with recalculation of the sample size as the key adaptation. Unlike other phase III pilots, an internal pilot investigation does not usually address any other feasibility aspects - because it is essentially part of the main study [ 10 , 19 , 20 ].. Nonetheless, we need to emphasize that whether or not a study is a pilot, depends on its objectives. An adaptive method is used as a strategy to reach that objective. Both a pilot and a non-pilot could be adaptive.” | concepts[11] |
| concept | recalculation of the sample size — not defined in this paper's own words; the key adaptation in the described internal-pilot approach — “An adaptive trial design refers to a design that allows modifications to be made to a trial's design or statistical procedures during its conduct, with the purpose of efficiently identifying clinical benefits/risks of new drugs or to increase the probability of success of clinical development [ 16 ]. The adaptations can be prospective (e.g. stopping a trial early due to safety or futility or efficacy at interim analysis); concurrent (e.g. changes in eligibility criteria, hypotheses or study endpoints) or retrospective (e.g. changes to statistical analysis plan prior to locking database or revealing treatment codes to trial investigators or patients). Piloting is normally built into adaptive trial designs by determining a priori decision rules to guide the adaptations based on cumulative data. For example, data from interim analyses could be used to refine sample size calculations [ 17 , 18 ]. This approach is routinely used in internal pilot studies - which are primarily designed to inform sample size calculation for the main study, with recalculation of the sample size as the key adaptation. Unlike other phase III pilots, an internal pilot investigation does not usually address any other feasibility aspects - because it is essentially part of the main study [ 10 , 19 , 20 ].. Nonetheless, we need to emphasize that whether or not a study is a pilot, depends on its objectives. An adaptive method is used as a strategy to reach that objective. Both a pilot and a non-pilot could be adaptive.” | concepts[12] |
| concept | feasibility — not defined in this paper's own words; the primary focus of a pilot study rather than statistical significance — “Quite often the emphasis is wrongly placed on statistical significance, not on feasibility - which is the main focus of the pilot study. Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibility objectives; no clear analytic plans; and certainly no clear criteria for success of feasibility. In many cases, pilot studies are conducted to generate data for sample size calculations. This seems especially sensible in situations where there are no data from previous studies to inform this process. However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes. Therefore if not used cautiously, results of pilot studies can potentially mislead sample size or power calculations [ 21 ] -- particularly if the pilot study was done to see if there is likely to be a treatment effect in the main study.” | concepts[13] |
| concept | statistical significance — not defined in this paper's own words; an often misplaced emphasis in pilot-study interpretation — “Quite often the emphasis is wrongly placed on statistical significance, not on feasibility - which is the main focus of the pilot study. Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibility objectives; no clear analytic plans; and certainly no clear criteria for success of feasibility. In many cases, pilot studies are conducted to generate data for sample size calculations. This seems especially sensible in situations where there are no data from previous studies to inform this process. However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes. Therefore if not used cautiously, results of pilot studies can potentially mislead sample size or power calculations [ 21 ] -- particularly if the pilot study was done to see if there is likely to be a treatment effect in the main study.” | concepts[14] |
| concept | treatment effects — not defined in this paper's own words; pilot-derived estimates that may be biased by limited sample sizes — “Quite often the emphasis is wrongly placed on statistical significance, not on feasibility - which is the main focus of the pilot study. Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibility objectives; no clear analytic plans; and certainly no clear criteria for success of feasibility. In many cases, pilot studies are conducted to generate data for sample size calculations. This seems especially sensible in situations where there are no data from previous studies to inform this process. However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes. Therefore if not used cautiously, results of pilot studies can potentially mislead sample size or power calculations [ 21 ] -- particularly if the pilot study was done to see if there is likely to be a treatment effect in the main study.” | concepts[15] |
| concept | sample size or power calculations — not defined in this paper's own words; calculations that pilot-derived treatment-effect estimates may mislead — “Quite often the emphasis is wrongly placed on statistical significance, not on feasibility - which is the main focus of the pilot study. Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibility objectives; no clear analytic plans; and certainly no clear criteria for success of feasibility. In many cases, pilot studies are conducted to generate data for sample size calculations. This seems especially sensible in situations where there are no data from previous studies to inform this process. However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes. Therefore if not used cautiously, results of pilot studies can potentially mislead sample size or power calculations [ 21 ] -- particularly if the pilot study was done to see if there is likely to be a treatment effect in the main study.” | concepts[16] |
| concept | criteria for success — not defined in this paper's own words; criteria based on primary feasibility objectives that guide whether a main study proceeds — “It is always important to state the criteria for success of a pilot study. The criteria should be based on the primary feasibility objectives. These provide the basis for interpreting the results of the pilot study and determining whether it is feasible to proceed to the main study. In general, the outcome of a pilot study can be one of the following: (i) Stop - main study not feasible; (ii) Continue, but modify protocol - feasible with modifications; (iii) Continue without modifications, but monitor closely - feasible with close monitoring and (iv) Continue without modifications - feasible as is.” | concepts[17] |
| concept | primary feasibility objectives — not defined in this paper's own words; the objectives on which pilot-study success criteria should be based — “It is always important to state the criteria for success of a pilot study. The criteria should be based on the primary feasibility objectives. These provide the basis for interpreting the results of the pilot study and determining whether it is feasible to proceed to the main study. In general, the outcome of a pilot study can be one of the following: (i) Stop - main study not feasible; (ii) Continue, but modify protocol - feasible with modifications; (iii) Continue without modifications, but monitor closely - feasible with close monitoring and (iv) Continue without modifications - feasible as is.” | concepts[18] |
| concept | pooling of pilot and main study data — not defined in this paper's own words; preplanned combination of pilot and main-study data with explicit statistical methods — “We recommend if any pooling of pilot and main study data is considered, this should be planned beforehand, described clearly in the protocol with clear discussion of the statistical consequences and methods. The goal is to avoid or minimize the potential bias that may occur due to multiple testing issues or any other opportunistic actions by investigators. In general, pooling when done appropriately can increase the efficiency of the main study [ 23 ].” | concepts[19] |
| concept | efficiency of the main study — not defined in this paper's own words; a potential benefit of appropriately pooling pilot and main-study data — “We recommend if any pooling of pilot and main study data is considered, this should be planned beforehand, described clearly in the protocol with clear discussion of the statistical consequences and methods. The goal is to avoid or minimize the potential bias that may occur due to multiple testing issues or any other opportunistic actions by investigators. In general, pooling when done appropriately can increase the efficiency of the main study [ 23 ].” | concepts[20] |
| concept | CONSORT Statement — not defined in this paper's own words; the reporting framework adapted for the paper's pilot-study checklist — “Adopted from the CONSORT Statement [ 9 ], Table 3 provides a checklist of items to consider including in a report of a pilot study.” | concepts[21] |
| concept | checklist of items — not defined in this paper's own words; items to consider when reporting a pilot study — “Adopted from the CONSORT Statement [ 9 ], Table 3 provides a checklist of items to consider including in a report of a pilot study.” | concepts[22] |
| concept | likelihood of success of the main study — not defined in this paper's own words; an outcome that conducting a pilot can enhance — “Pilot or vanguard studies provide a good opportunity to assess feasibility of large full-scale studies. Pilot studies are the best way to assess feasibility of a large expensive full-scale study, and in fact are an almost essential pre-requisite. Conducting a pilot prior to the main study can enhance the likelihood of success of the main study and potentially help to avoid doomed main studies. Pilot studies should be well designed with clear feasibility objectives, clear analytic plans, and explicit criteria for determining success of feasibility. They should be used cautiously for determining treatment effects and variance estimates for power or sample size calculations. Finally, they should be scrutinized the same way as full scale studies, and every attempt should be taken to publish the results in peer-reviewed journals.” | concepts[23] |
| effect size | — “For example, consider having some discussions with clinicians using the approach suggested by Lenth [ 24 ] to illicit additional information on possible effect size and variance estimates.” | content_tags.statistics[0] |
| confidence interval | — “In some cases, one could use a confidence interval [CI] approach to estimate the sample size required to establish feasibility.” | content_tags.statistics[1] |
| confidence interval | — “However, confidence interval approach may be used to calculate and justify the sample size based on key feasibility objective(s).” | content_tags.statistics[2] |
| confidence interval | — “However, the confidence interval approach may be used to calculate and justify the sample size based on key feasibility objective(s).” | content_tags.statistics[3] |
Full text — tagged
The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.
Abstract. Abstract Pilot studies for phase III trials - which are comparative randomized trials designed to provide preliminary evidence on the clinical efficacy of a drug or intervention - are routinely performed in many clinical areas. Also commonly know as "feasibilityterm" or "vanguard" studies, they are designed to assess the safety of treatment or interventions; to assess recruitment potential; to assess the feasibilityterm of international collaboration or coordination for multicentre trials; to increase clinical experience with the study medication or intervention for the phase III trials. They are the best way to assess feasibilityterm of a large, expensive full-scale study, and in fact are an almost essential pre-requisite. Conducting a pilot prior to the main study can enhance the likelihood of success of the main studyterm and potentially help to avoid doomed main studies. The objective of this paper is to provide a detailed examination of the key aspects of pilot studies for phase III trials including: 1) the general reasons for conducting a pilot studyterm·mth; 2) the relationships between pilot studies, proof-of-concept studies, and adaptive designs; 3) the challenges of and misconceptions about pilot studies; 4) the criteria for evaluating the success of a pilot studyterm·mth; 5) frequently asked questions about pilot studies; 7) some ethical aspects related to pilot studies; and 8) some suggestions on how to report the results of pilot investigations using the CONSORT format. 1. Introduction The Concise Oxford Thesaurus [ 1 ] defines a pilot project or study as an experimental, exploratory, test, preliminary, trial or try out investigation. Epidemiology and statistics dictionaries provide similar definitions of a pilot studyterm·mth as a small scale • " ... test of the methods and procedures to be used on a larger scale if the pilot studyterm·mth demonstrates that the methods and procedures can work" [ 2 ]; • "...investigation designed to test the feasibilityterm of methods and procedures for later use on a large scale or to search for possible effects and associations that may be worth following up in a subsequent larger study" [ 3 ]. Table 1 provides a summary of definitions found on the Internet. A closer look at these definitions reveals that they are similar to the ones above in that a pilot studyterm·mth is synonymous with a feasibility studyterm·mth intended to guide the planning of a large-scale investigation. Pilot studies are sometimes referred to as "vanguard trials" (i.e. pre-studies) intended to assess the safety of treatment or interventions; to assess recruitment potential; to assess the feasibilityterm of international collaboration or coordination for multicentre trials; to evaluate surrogate markerterm data in diverse patient cohorts; to increase clinical experience with the study medication or intervention, and identify the optimal dose of treatments for the phase III trials [ 4 ]. As suggested by an African proverb from the Ashanti people in Ghana " You never test the depth of a river with both feet ", the main goal of pilot studies is to assess feasibilityterm so as to avoid potentially disastrous consequences of embarking on a large study - which could potentially "drown" the whole research effort. Table 1 Some Adapted Definitions of Pilot Studies on the Web (Date of last access: December 22, 2009) Definition* Source A trial study carried out before a research design is finalised to assist in defining the research question or to test the feasibilityterm, reliability and validity of the proposed study design http://www.cirem.org.uk/definitions.html A smaller version of a study is carried out before the actual investigation is done. Researchers use information gathered in pilot studies to refine or modify the research methodology for a study and to develop large-scale studies http://www.mh.state.oh.us/what-we-do/promote/research-and-evaluation/learning-lab/research-glossary.shtml A small scale study conducted to test the plan and method of a research study. http://www.umm.edu/nursing/docs/glossary_research_terms.pdf A small study carried out before a large-scale study to try out a procedure or to test a principle http://www.psych-sci.manchester.ac.uk/actnow/glossary/ An experimental use of a treatment in a small group of patients to learn if it will be effective and safe on a broad scale http://www.lungcanceralliance.org/news/glossary.html The initial study examining a new method or treatment http://www.cdc.gov/des/consumers/resourcesterm/glossary.html#P A small study often done to assist the preparation of a larger, more comprehensive investigation . http://www.informedesign.umn.edu/Glossary.aspx?id=1952# Small, preliminary test or trial run of an intervention, or of an evaluation activity such as an instrument or sampling procedure. The results of the pilot are used to improve the program or evaluation procedure being piloted before it is used on a larger scale . http://www.nsf.gov/pubs/2005/nsf0531/nsf0531_6.pdf *Emphasis is ours Feasibilityterm studies are routinely performed in many clinical areas. It is fair to say that every major clinical trial had to start with some piloting or a small scale investigation to assess the feasibilityterm of conducting a larger scale study: critical care [ 5 ], diabetes managementterm intervention trials [ 6 ], cardiovascular trials [ 7 ], primary healthcare [ 8 ], to mention a few. Despite their noted importance, the reality is that pilot studies receive little or no attention in scientificterm research training. Few epidemiology or research textbooks cover the topic with the necessary detail. In fact, we are not aware of any textbook that dedicates a chapter on this issue - many just mention it in passing or provide a cursory coverage of the topic. The objective of this paper is to provide a detailed examination of the key aspects of pilot studies. In the next section, we narrow the focus of our definition of a pilot to phase III trials. Section 3 covers the general reasons for conducting a pilot studyterm·mth. Section 4 deals with the relationships between pilot studies, proof-of-concept studies, and adaptive designs, while section 5 addresses the challenges of pilot studies. Evaluation of a pilot studyterm·mth (i.e. how to determine if a pilot studyterm·mth was successful) is covered in Section 6. We deal with several frequently asked questions about pilot studies in Section 7 using a "question-and-answer" approach. Section 8 covers some ethical aspects related to pilot studies; and in Section 9, we follow the CONSORT format [ 9 ] to offer some suggestions on how to report the results of pilot investigations. 2. Narrowing the focus: Pilot studies for randomized studies Pilot studies can be conducted in both quantitative and qualitative studies. Adopting a similar approach to Lancaster et al . [ 10 ], we focus on quantitative pilot studies - particularly those done prior to full-scale phase III trials. Phase I trials are non-randomized studies designed to investigate the pharmacokinetics of a drug (i.e. how a drug is distributed and metabolized in the body) including finding a dose that can be tolerated with minimal toxicity. Phase II trials provide preliminary evidence on the clinical efficacy of a drug or intervention. They may or may not be randomized. Phase III trials are randomized studies comparing two or more drugs or intervention strategies to assess efficacy and safety. Phase IV trials, usually done after registration or marketing of a drug, are non-randomized surveillance studies to document experiences (e.g. side-effects, interactions with other drugs, etc) with using the drug in practice. For the purposes of this paper, our approach to utilizing pilot studies relies on the model for complex interventions advocated by the British Medical Research Council - which explicitly recommends the use of feasibilityterm studies prior to Phase III clinical trials, but stresses the iterative nature of the processesterm of development, feasibilityterm and piloting, evaluation and implementation [ 11 ]. 3. Reasons for Conducting Pilot Studies Van Teijlingen et al . [ 12 ] and van Teijlingen and Hundley [ 13 ] provide a summary of the reasons for performing a pilot studyterm·mth. In general, the rationale for a pilot studyterm can be grouped under several broad classifications - processterm, resourcesterm, managementterm and scientificterm (see also http://www.childrens-mercy.org/stats/plan/pilot.asp for a different classification): • Processterm: This assesses the feasibilityterm of the steps that need to take place as part of the main study. Examples include determining recruitment rates, retention rates, etc. • Resourcesterm: This deals with assessing time and budget problems that can occur during the main study. The idea is to collect some pilot data on such things as the length of time to mail or fill out all the survey forms. • Managementterm: This covers potential human and data optimization problems such as personnel and data managementterm issues at participating centres. • Scientificterm: This deals with the assessment of treatment safety, determination of dose levels and response, and estimation of treatment effectterm and its variance. Table 2 summarizes this classification with specific examples. Table 2 Reasons for conducting pilot studies Main Reason Examples Processterm: This assesses the feasibilityterm of the processesterm that are key to the success of the main study • Recruitment rates • Retention rates • Refusal rates • Failure/success rates • (Non)compliance or adherence rates • eligibility criteria - Is it obvious who meets and who does not meet the eligibility requirements? - Are the eligibility criteria sufficient or too restrictive? • Understanding of study questionnaires or data collection tools: - Do subjects provide no answer, multiple answers, qualified answers, or unanticipated answers to study questions? Resourcesterm: This deals with assessing time and resourceterm problems that can occur during the main study • Length of time to fill out all the study forms • Determining capacity: - Will the study participants overload your phone lines or overflow your waiting room? • Determining processterm time - How much time does it take to mail out a thousand surveys? • Is the equipment readily available when and where it is needed? • What happens when it breaks down or gets stolen? • Can the software used for capturing data read and understand the data? • Determining centre willingness and capacity - Do the centres do what they committed to doing? - Do investigators have the time to Perform the tasks they committed to doing? - Are there any capacity issues at each participating centre? Managementterm: This covers potential human and data managementterm problems • What are the challenges that participating centres have with managing the study? • What challenges do study personnel have? • Is there enough room on the data collection form for all of the data you receive? • Are there any problems entering data into the computer? • Can data coming from different sources be matched? • Were any important data values forgotten about? • Do data show too much or too little variability? Scientificterm: This deals with the assessment of treatment safety, dose, response, effect and variance of the effect • Is it safe to use the study drug/intervention? • What is the safe dose level? • Do patients respond to the drug? • What is the estimate of the treatment effectterm? • What is the estimate of the variance of the treatment effectterm? 4. Relationships between Pilot Studies, Proof-of-Concept Studies, and Adaptive Designs A proof-of-concept (PoC) studyterm·def is defined as a clinical trial carried out to determine if a treatment (drug) is biologically active or inactive [ 14 ]. PoC studies usually use surrogate markersterm as endpoints. In general, they are phase I/II studies - which, as noted above, investigate the safety profile, dose level and response to new drugs [ 15 ]. Thus, although designed to inform the planning of phase III trials for registration or licensing of new drugs, PoC studies may not necessarily fit our restricted definition of pilot studies aimed at assessing feasibilityterm of phase III trials as outlined in Section 2. An adaptive trial designterm refers to a design that allows modifications to be made to a trial's design or statistical procedures during its conduct, with the purpose of efficiently identifying clinical benefits/risks of new drugs or to increase the probability of success of clinical development [ 16 ]. The adaptations can be prospective (e.g. stopping a trial early due to safety or futility or efficacy at interim analysis); concurrent (e.g. changes in eligibility criteria, hypotheses or study endpoints) or retrospective (e.g. changes to statistical analysis plan prior to locking database or revealing treatment codes to trial investigators or patients). Piloting is normally built into adaptive trial designsterm by determining a priori decision rulesterm to guide the adaptations based on cumulative data. For example, data from interim analyses could be used to refine sample size calculations [ 17 , 18 ]. This approach is routinely used in internal pilot studiesterm·mth - which are primarily designed to inform sample size calculation for the main study, with recalculation of the sample sizeterm as the key adaptation. Unlike other phase III pilots, an internal pilot investigation does not usually address any other feasibilityterm aspects - because it is essentially part of the main study [ 10 , 19 , 20 ].. Nonetheless, we need to emphasize that whether or not a study is a pilot, depends on its objectives. An adaptive method is used as a strategy to reach that objective. Both a pilot and a non-pilot could be adaptive. 5. Challenges of and Common Misconceptions about Pilot Studies Pilot studies can be very informative, not only to the researchers conducting them but also to others doing similar work. However, many of them never get published, often because of the way the results are presented [ 13 ]. Quite often the emphasis is wrongly placed on statistical significanceterm, not on feasibilityterm - which is the main focus of the pilot studyterm·mth. Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibilityterm objectives; no clear analytic plans; and certainly no clear criteria for successterm of feasibilityterm. In many cases, pilot studies are conducted to generate data for sample size calculations. This seems especially sensible in situations where there are no data from previous studies to inform this processterm. However, it can be dangerous to use pilot studies to estimate treatment effectsterm, as such estimates may be unrealistic/biased because of the limited sample sizes. Therefore if not used cautiously, results of pilot studies can potentially mislead sample size or power calculationsterm [ 21 ] -- particularly if the pilot studyterm·mth was done to see if there is likely to be a treatment effectterm in the main study. In section 6, we provide guidance on how to proceed with caution in this regard. There are also several misconceptions about pilot studies. Below are some of the common reasons that researchers have put forth for calling their study a pilot. The first common reason is that a pilot studyterm·mth is a small single-centre study. For example, researchers often state lack of resourcesterm for a large multi-centre study as a reason for doing a pilot. The second common reason is that a pilot investigation is a small study that is similar in size to someone else's published study. In reviewing submissions to a research ethics board, we have come across sentiments such as • So-and-so did a similar study with 6 patients and got statistical significanceterm - ours uses 12 patients (double the size)! • We did a similar pilot before (and it was published!) The third most common reason is that a pilot is a small study done by a student or an intern - which can be completed quickly and does not require funding. Specific arguments include • I have funding for 10 patients only; • I have limited seed (start-up) funding; • This is just a student project! • My supervisor (boss) told me to do it as a pilot . None of the above arguments qualifies as sound reasons for calling a study a pilot. A study should only be conducted if the results will be informative; studies conducted for the reasons above may result in findings of limited utility, which would be a waste of the researchers' and participants' efforts. The focus of a pilot studyterm·mth should be on assessment of feasibilityterm, unless it was powered appropriately to assess statistical significanceterm. Further, there is a vast number of poorly designed and reported studies. Assessment of the quality of a published report may be helpful to guide decisions of whether the report should be used to guide planning or designing of new studies. Finally, if a trainee or researcher is assigned a project as a pilot it is important to discuss how the results will inform the planning of the main study. In addition, clearly defined feasibilityterm objectives and rationale to justify piloting should be provided. Sample Size for Pilot Studies In general, sample size calculations may not be required for some pilot studies. It is important that the sample for a pilot be representative of the target study population. It should also be based on the same inclusion/exclusion criteria as the main study. As a rule of thumb, a pilot studyterm·mth should be large enough to provide useful information about the aspects that are being assessed for feasibilityterm. Note that PoC studies require sample size estimation based on surrogate markersterm [ 22 ], but they are usually not powered to detect meaningful differences in clinically important endpoints. The sample used in the pilot may be included in the main study, but caution is needed to ensure the key features of the main study are preserved in the pilot (e.g. blinding in randomized controlled trials). We recommend if any pooling of pilot and main study dataterm is considered, this should be planned beforehand, described clearly in the protocol with clear discussion of the statistical consequences and methods. The goal is to avoid or minimize the potential bias that may occur due to multiple testing issues or any other opportunistic actions by investigators. In general, pooling when done appropriately can increase the efficiency of the main studyterm [ 23 ]. As noted earlier, a carefully designed pilot studyterm·mth may be used to generate information for sample size calculations. Two approaches may be helpful to optimize information from a pilot studyterm·mth in this context: First, consider eliciting qualitative data to supplement the quantitative information obtained in the pilot. For example, consider having some discussions with clinicians using the approach suggested by Lenth [ 24 ] to illicit additional information on possible effect sizeterm·stat and variance estimates. Second, consider creating a sample size table for various values of the effect or variance estimates to acknowledge the uncertainty surrounding the pilot estimates. In some cases, one could use a confidence intervalterm·stat [CI] approach to estimate the sample size required to establish feasibilityterm. For example, suppose we had a pilot trial designed primarily to determine adherence rates to the standardized risk assessment form to enhance venous thromboprophylaxis in hospitalized patients. Suppose it was also decided a priori that the criterion for success would be: the main trial would be ' feasibl e' if the risk assessment form is completed for ≥ 70% of eligible hospitalized patients. Using a 95% CI for the proportion of eligible patients who complete the assessment form, a margin of error (ME) of 0.05, a lower bound of this CI of 0.70, and an expected completion rate of 75% based on an educated guess, the required sample for the pilot studyterm·mth would be at least 75 patients. This calculation is based on a common formula for obtaining a 95% CI for a single proportion: p ± 1.96 where " p " is the prior estimate of the proportion of interest and " n " is the sample size. 6. How to Interpret the Results of a Pilot Studyterm·mth: Criteria for Successterm It is always important to state the criteria for successterm of a pilot studyterm·mth. The criteria should be based on the primary feasibility objectivesterm. These provide the basis for interpreting the results of the pilot studyterm·mth and determining whether it is feasible to proceed to the main study. In general, the outcome of a pilot studyterm·mth can be one of the following: (i) Stop - main study not feasible; (ii) Continue, but modify protocol - feasible with modifications; (iii) Continue without modifications, but monitor closely - feasible with close monitoring and (iv) Continue without modifications - feasible as is. For example, the Prophylaxis of Thromboembolism in Critical Care Trial (PROTECT) was designed to assess the feasibilityterm of a large-scale trial with the following criteria for determining success [ 25 ]: • 98.5% of patients had to receive study drug within 12 hours of randomization; • 91.7% of patients had to receive every scheduled dose of the study drug in a blinded manner; • 90% or more of patients had to have lower limb compression ultrasounds performed at the specified times; and • > 90% of necessary dose adjustments had to have been made appropriately in response to pre-defined laboratory criteria . In a second example, the PeriOperative Epidural Trial (POET) Pilot Studyterm·mth was designed to assess the feasibilityterm of a large, multicentre trial with the following criteria for determining success [ 26 ]: • one subject per centre per week (i.e., 200 subjects from four centres over 50 weeks) can be recruited ; • at least 70% of all eligible patients can be recruited ; • no more than 5% of all recruited subjects crossed over from one modality to the other; and • complete follow-up in at least 95% of all recruited subjects . 7. Frequently asked questions about pilot studies In this Section, we offer our thoughts on some of the frequently asked questions about pilot studies. These could be helpful to not only clinicians and trainees, but to anyone who is interested in health research. • Can I publish the results of a pilot studyterm·mth? - Yes, every attempt should be made to publish. • Why is it important to publish the results of pilot studies? - To provide information about feasibilityterm to the research community to save resourcesterm being unnecessarily spent on studies that may not be feasible. Further, having such information can help researchers to avoid duplication of efforts in assessing feasibilityterm. - Finally, researchers have an ethical and scientificterm obligation to attempt publishing the results of every research endeavor. However, our focus should be on feasibilityterm goals. Emphasis should not be placed on statistical significanceterm when pilot studies are not powered to detect minimal clinically important differences. Such studies typically do not show statistically significant results - remember that underpowered studies (with no statistically significant results) are inconclusive, not negative since "no evidence of effect" is not "evidence of no effect" [ 27 ]. • Can I combine data from a pilot with data from the main study? - Yes, provided the sampling frame and methodologies are the same. This can increase the efficiency of the main studyterm - see Section 5. • Can I combine the results of a pilot with the results of another study or in a meta-analysis? - Yes, provided the sampling frame and methodologies are the same. - No, if the main study is reported and it includes the pilot studyterm·mth. • Can the results of the pilot studyterm·mth be valid on their own, without existence of the main study - Yes, if the results show that it is not feasible to proceed to the main study or there is insufficient funding. • Can I apply for funding for a pilot studyterm·mth? - Yes. Like any grant, it is important to justify the need for piloting. - The pilot has to be placed in the context of the main study. • Can I randomize patients in a pilot studyterm·mth? - Yes. For a phase III pilot studyterm·mth, one of the goals could be to assess how a randomization procedure might work in the main study or whether the idea of randomization might be acceptable to patients [ 10 ]. In general, it is always best for a pilot to maintain the same design as the main study. • How can I use the information from a pilot to estimate the sample size? - Use with caution, as results from pilot studies can potentially mislead sample size calculations. - Consider supplementing the information with qualitative discussions with clinicians - see section 5; and - Create a sample size table to acknowledge the uncertainty of the pilot information - see section 5. • Can I use the results of a pilot studyterm·mth to treat my patients? - Not a good idea! - Pilot studies are primarily for assessing feasibilityterm. • What can I do with a failed or bad pilot studyterm·mth? - No study is a complete failure; it can always be used as bad example! However, it is worth making clear that a pilot studyterm·mth that shows the main study is not likely to be feasible is not a failed (pilot) study. In fact, it is a success - because you avoided wasting scarce resourcesterm on a study destined for failure! 8. Ethical Aspects of Pilot Studies Halpern et al . [ 28 ] stated that conducting underpowered trials is unethical. However, they proposed that underpowered trials are ethical in two situations: (i) small trials of interventions for rare diseases -- which require documenting explicit plans for including results with those of similar trials in a prospective meta-analysis; (ii) early-phase trials in the development of drugs or devices - provided they are adequately powered for defined purposes other than randomized treatment comparisons. Pilot studies of phase III trials (dealing with common diseases) are not addressed in their proposal. It is therefore prudent to ask: Is it ethical to conduct a study whose feasibilityterm can not be guaranteed (i.e. with a high probability of success)? It seems unethical to consider running a phase III study without having sufficient data or information about the feasibilityterm. In fact, most granting agencies often require data on feasibilityterm as part of their assessment of the scientificterm validity for funding decisions. There is however one important ethical aspect about pilot studies that has received little or no attention from researchers, research ethics boards and ethicists alike. This pertains to the issue of the obligation that researchers have to patients or participants in a trial to disclose the feasibilityterm nature of pilot studies. This is essential given that some pilot studies may not lead to further studies. A review of the commonly cited research ethics guidelines - the Nuremburg Code [ 29 ], Helsinki Declaration [ 30 ], the Belmont Report [ 31 ], ICH Good Clinical Practice [ 32 ], and the International Ethical Guidelines for Biomedical Research Involving Human Subjects [ 33 ] - shows that pilot studies are not addressed in any of these guidelines. Canadian researchers are also encouraged to follow the Tri-Council Policy Statement (TCPS) [ 34 ] - it too does not address how pilot studies need to be approached. It seems to us that given the special nature of feasibilityterm or pilot studies, the disclosure of their purpose to study participants requires special wording - that informs them of the definition of a pilot studyterm·mth, the feasibilityterm objectives of the study, and also clearly defines the criteria for successterm of feasibilityterm. To fully inform participants, we suggest using the following wording in the consent form: " The overall purpose of this pilot studyterm·mth is to assess the feasibilityterm of conducting a large study to [state primary objective of the main study]. A feasibilityterm or pilot studyterm·mth is a study that... [state a general definition of a feasibility studyterm·mth]. The specific feasibilityterm objectives of this study are ... [state the specific feasibilityterm objectives of the pilot studyterm·mth]. We will determine that it is feasible to carry on the main study if ... [state the criteria for successterm of feasibilityterm] ." 9. Recommendation for Reporting the Results of Pilot Studies Adopted from the CONSORT Statementterm·mth [ 9 ], Table 3 provides a checklist of itemsterm to consider including in a report of a pilot studyterm·mth. Table 3 Pilot Studyterm·mth - Checklist: Items to include when reporting a pilot studyterm·mth PAPER SECTION Item Descriptor Reported on Page # TITLE and ABSTRACT 1 Does the title or abstract indicate that the study is a "pilot"? INTRODUCTION Background 2 Scientificterm background for the main study and explanation of rationale for assessing feasibilityterm through piloting METHODS Participants and setting 3 • Eligibility criteria for participants in the pilot studyterm·mth (these should be the same as in the main study -- if different, state the differences) • The settings and locations where the data were collected Interventions 4 Provide precise details of the interventions intended for each group and how and when they were actually administered (if applicable) -- state clearly if any aspects of the intervention are assessed for feasibilityterm Objectives 5 • Specific scientificterm objectives and hypotheses for the main study • Specific feasibilityterm objectives Outcomes 6 • Clearly defined primary and secondary outcome measures for the main study • Clearly define the feasibilityterm outcomes and how they were operationalized -- these should include key elements such as recruitment rates, consent rates, completion rates, variance estimates, etc Sample size 7 Describe how sample size was determined • In general for a pilot of a phase III trial, there is no need for a formal sample size calculation. However, confidence intervalterm·stat approach may be used to calculate and justify the sample size based on key feasibilityterm objective(s). Feasibilityterm Criteria 8 Clearly describe the criteria for assessing success of feasibilityterm -- these should be based on the feasibilityterm objectives Statistical Methods 9 Describe the statistical methods for the analysis of primary and secondary feasibilityterm outcomes Ethical Aspects 10 • State whether the study received research ethics approval • State how informed consent was handled -- given the feasibilityterm nature of the study RESULTS Participant flow 11 Flow of participants through each stage (a flow-chart is strongly recommended). • Describe protocol deviations from pilot studyterm·mth as planned, together with reasons • State the number of exclusions at each stage and reasons for exclusions Recruitment 12 Report the dates defining the periods of recruitment and follow-up Baseline data 13 Report the baseline demographic and clinical characteristics of the participants Outcomes and estimation 14 For each primary and secondary feasibilityterm outcome, report the point estimate of effect and its precision ( e.g ., 95% confidence intervalterm·stat [CI]) -- if applicable DISCUSSION Interpretation 15 Interpretation of the results should focus on feasibilityterm, taking into account • the stated criteria for successterm of feasibilityterm; • study hypotheses, sources of potential bias or imprecision -- given the feasibilityterm nature of the study • the dangers associated with multiplicity of analyses and outcomes Generalizability 16 Generalizability (external validity) of the feasibilityterm. State clearly what modifications in the design of the main study (if any) would be necessary to make it feasible Overall evidence of feasibilityterm 17 General interpretation of the results in the context of current evidence of feasibilityterm • Focus should be on feasibilityterm Title and abstract Item #1: The title or abstract should indicate that the study is a "pilot" or "feasibilityterm" As a number one summary of the contents of any report, it is important for the title to clearly indicate that the report is for a pilot or feasibility studyterm·mth. This would also be helpful to other researchers during electronic information search about feasibilityterm issues. Our quick search of PUBMED [on July 13, 2009], using the terms "pilot" OR "feasibilityterm" OR "proof-of-concept" for revealed 24423 (16%) hits of studies that had these terms in the title or abstract compared with 149365 hits that had these terms anywhere in the text. Background Item #2: Scientificterm background for the main study and explanation of rationale for assessing feasibilityterm through piloting The rationale for initiating a pilot should be based on the need to assess feasibilityterm for the main study. Thus, the background of the main study should clearly describe what is known or not known about important feasibilityterm aspects to provide context for piloting. Methods Item #3: Participants and setting of the study The description of the inclusion-exclusion or eligibility criteria for participants should be the same as in the main study. The settings and locations where the data were collected should also be clearly described. Item #4: Interventions Precise details of the interventions intended for each group and how and when they were actually administered (if applicable) - state clearly if any aspects of the intervention are assessed for feasibilityterm. Item #5: Objectives State the specific scientificterm primary and secondary objectives and hypotheses for the main study and the specific feasibilityterm objectives. It is important to clearly indicate the feasibilityterm objectives as the primary focus for the pilot. Item #6: Outcomes Clearly define primary and secondary outcome measures for the main study. Then, clearly define the feasibilityterm outcomes and how they were operationalized - these should include key elements such as recruitment rates, consent rates, completion rates, variance estimates, etc. In some cases, a pilot studyterm·mth may be conducted with the aim to determine a suitable (clinical or surrogate) endpoint for the main study. In such a case, one may not be able to define the primary outcome of the main study until the pilot is finished. However, it is important that determining the primary outcome of the main study be clearly stated as part of feasibilityterm outcomes. Item #7: Sample Size Describe how sample size was determined. If the pilot is a proof-of-concept study, is the sample size calculated based on primary/key surrogate markerterm(s)? In general if the pilot is for a phase III study, there may be no need for a formal sample size calculation. However, the confidence intervalterm·stat approach may be used to calculate and justify the sample size based on key feasibilityterm objective(s). Item #8: Feasibilityterm criteria Clearly describe the criteria for assessing success of feasibilityterm - these should be based on the feasibilityterm objectives. Item #9: Statistical Analysis Describe the statistical methods for the analysis of primary and secondary feasibilityterm outcomes. Item #10: Ethical Aspects State whether the study received research ethics approval. Describe how informed consent was handled - given the feasibilityterm nature of the study. Results Item #11: Participant Flow Describe the flow of participants through each stage of the study (use of a flow-diagram is strongly recommended -- see CONSORT [ 9 ] for a template). Describe protocol deviations from pilot studyterm·mth as planned with reasons for deviations. State the number of exclusions at each stage and corresponding reasons for exclusions. Item #12: Recruitment Report the dates defining the periods of recruitment and follow-up. Item #13: Baseline Data Report the baseline demographic and clinical characteristics of the participants. Item #14: Outcomes and Estimation For each primary and secondary feasibilityterm outcomes, report the point estimate of effect and its precision ( e.g ., 95% CI) - if applicable. Discussion Item # 15: Interpretation Interpretation of the results should focus on feasibilityterm, taking into account the stated criteria for successterm of feasibilityterm, study hypotheses, sources of potential bias or imprecision (given the feasibilityterm nature of the study) and the dangers associated with multiplicity - repeated testing on multiple outcomes. Item #16: Generalizability Discuss the generalizability (external validity) of the feasibilityterm aspects observed in the study. State clearly what modifications in the design of the main study (if any) would be necessary to make it feasible. Item #17: Overall evidence of feasibilityterm Discuss the general results in the context of overall evidence of feasibilityterm. It is important that the focus be on feasibilityterm. 9. Conclusions Pilot or vanguard studies provide a good opportunity to assess feasibilityterm of large full-scale studies. Pilot studies are the best way to assess feasibilityterm of a large expensive full-scale study, and in fact are an almost essential pre-requisite. Conducting a pilot prior to the main study can enhance the likelihood of success of the main studyterm and potentially help to avoid doomed main studies. Pilot studies should be well designed with clear feasibilityterm objectives, clear analytic plans, and explicit criteria for determining success of feasibilityterm. They should be used cautiously for determining treatment effectsterm and variance estimates for power or sample size calculations. Finally, they should be scrutinized the same way as full scale studies, and every attempt should be taken to publish the results in peer-reviewed journals. Competing interests The authors declare that they have no competing interests. Authors' contributions LT drafted the manuscript. All authors reviewed several versions of the manuscript, read and approved the final version. Pre-publication history The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1471-2288/10/1/prepub Contributor Information Lehana ThabanepersName, Email: thabanl@mcmaster.ca. JinhuipersName Ma, Email: maj26@mcmaster.ca. Rong ChupersName, Email: chur@mcmaster.ca. Ji ChengpersName, Email: chengj2@mcmaster.ca. Afisi IsmailapersName, Email: ismailaspersName@mcmaster.ca. Lorena P RiospersName, Email: lprios@yahoo.com. Reid RobsonpersName, Email: Reid.C.RobsonpersName@gsk.com. Marroon ThabanepersName, Email: mthaban@mcmaster.ca. Lora GiangregoriopersName, Email: lmgiangr@healthy.uwaterloo.ca. Charles H GoldsmithpersName, Email: goldsmit@mcmaster.ca. Acknowledgements Dr Lehana ThabanepersName is clinical trials mentor for the Canadian Institutes of Health Research. We thank the reviewers for insightful comments and suggestions which led to improvements in the manuscript.