Page 1

Executive Summary

ABSTRACT

Scientific research has a long history of using well-established, well documented, and validated methods for the design, conduct, and analysis of clinical trials. A study design that is considered appropriate includes sufficient sample size (n) and statistical power and proper control of bias to allow a meaningful interpretation of the results. Whenever feasible, clinical trials should be designed and performed so that they have adequate statistical power. However, when the clinical context does not provide a sufficient number of research participants for a trial with adequate statistical power but the research question has great clinical significance, research can still proceed under certain conditions. Small clinical trials might be warranted for the study of rare diseases, unique study populations (e.g., astronauts), individually tailored therapies, in environments that are isolated, in emergency situations, and in instances of public health urgency. Properly designed trials with small sample sizes may provide substantial evidence of efficacy and are especially appropriate in particular situations. However, the conclusions derived from such studies may require careful consideration of the assumptions and inferences, given the small number of paticipants.

Bearing in mind the statistical power, precision, and validity limitations of trials with small sample sizes, there are innovative design and analysis approaches that can improve the quality of such trials. A number of trial designs



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Page 1 Executive Summary ABSTRACT Scientific research has a long history of using well-established, well documented, and validated methods for the design, conduct, and analysis of clinical trials. A study design that is considered appropriate includes sufficient sample size (n) and statistical power and proper control of bias to allow a meaningful interpretation of the results. Whenever feasible, clinical trials should be designed and performed so that they have adequate statistical power. However, when the clinical context does not provide a sufficient number of research participants for a trial with adequate statistical power but the research question has great clinical significance, research can still proceed under certain conditions. Small clinical trials might be warranted for the study of rare diseases, unique study populations (e.g., astronauts), individually tailored therapies, in environments that are isolated, in emergency situations, and in instances of public health urgency. Properly designed trials with small sample sizes may provide substantial evidence of efficacy and are especially appropriate in particular situations. However, the conclusions derived from such studies may require careful consideration of the assumptions and inferences, given the small number of paticipants. Bearing in mind the statistical power, precision, and validity limitations of trials with small sample sizes, there are innovative design and analysis approaches that can improve the quality of such trials. A number of trial designs

OCR for page 1
Page 2 especially lend themselves to use in studies with small sample sizes, including one subject (n-of-1) designs, sequential designs, “within-subject” designs, decision analysis-based designs, ranking and selection designs, adaptive designs, and risk-based allocation designs. Data analysis for trials with small numbers of participants in particular must be focused. In general, certain types of analyses are more amenable to studies with small numbers of participants, including sequential analysis, hierarchical analysis, Bayesian analysis, decision analysis, statistical prediction, meta-analysis, and risk-based allocation. Because of the constraints of conducting research with small sample sizes, the committee makes recommendations in several areas: defining the research question, tailoring the study design by giving careful consideration to alternative methods, clarifying sample characteristics and methods for the reporting of results of clinical trials with small sample sizes, performing corroborative analyses to evaluate the consistency and robustness of the results of clinical trials with small sample sizes, and exercising caution in the interpretation of the results before attempting to extrapolate or generalize the findings of clinical trials with small sample sizes. The committee also recommends that more research be conducted on the development and evaluation of alternative experimental designs and analysis methods for trials with small sample sizes. INTRODUCTION Clinical trials are used to elucidate the most appropriate preventive, diagnostic, or treatment options for individuals with a given medical condition. Perhaps the most essential feature of a clinical trial is that it aims to use results based on a limited sample of research participants to see if the intervention is safe and effective or if it is comparable to a comparison treatment. Sample size is a crucial component of any clinical trial. A trial with a small number of research participants is more prone to variability and carries a considerable risk of failing to demonstrate the effectiveness of a given intervention when one really is present. This may occur in phase I (safety and pharmacologic profiles), II (pilot efficacy evaluation), and III (extensive assessment of safety and efficacy) trials. Although phase I and II studies may have smaller sample sizes, they usually have adequate statistical power, which is the committee's definition of a “large” trial. Sometimes a trial with eight participants may have adequate statistical power, statistical power being the probability of rejecting the null hypothesis when the hypothesis is false. Thus, a critical aspect of clinical trial design is determination of the sample size needed to establish the feasibility of the study (i.e., sufficient statistical power). The number of participants in a clinical trial should al-

OCR for page 1
Page 3 ways be large enough to provide a sufficiently precise answer to the research question posed, but it should also be the minimum necessary to achieve this aim. A proposed study that cannot answer the question being asked because the necessary sample size cannot be attained should not be conducted on ethical grounds. That is, it is unacceptable to expose patients or research participants to harms even inconveniences if there is no prospect that useful and potentially generalizable information will result from the study. Adequately powered randomized clinical trials and double-blind, randomized clinical trials are generally regarded as the most authoritative research methods for establishment of the efficacies of therapeutic interventions. By allocating sufficient numbers of individuals to groups (e.g., experimental or control groups), investigators can estimate or determine with some degree of certainty the effect of a given intervention. Nevertheless, even though the size of the available research population does not allow a randomized clinical trial with adequate statistical power to be conducted, there might still be a need to design and perform the research (e.g., because treatments are unavailable for a rare disorder or a unique patient population or because studies require the participation of patients with terminal or severely debilitating or incapacitating disorders). In addition, some distinctive research populations—such as astronauts or members of a small, isolated community—may consist of less than five individuals. This research situation, in which large numbers of study participants cannot be obtained, is defined as a “small n clinical trial,” where n refers to the sample size. The sample size in small clinical trials might be very small, for example, a group of astronauts during a space mission, or could range upward to more than 100 individuals. This is in contrast to the sample sizes of some large clinical trials, where the number of participants is in the thousands. This report focuses on the issues and challenges presented by clinical trials with very small sample sizes. Because of the design and analysis constraints of small-sample-size trials and because of their inherent uncertainties, they require at least as much— and probably more—thought and planning than traditional large clinical trials. Small-sample-size studies may also require additional methods for evaluation of the effectiveness of a therapeutic intervention. In addition, inferences should consider the size of the population relative to the size of the sample. For example, in some trials with small sample sizes, the size of the potential population might be large (e.g., phase II studies of treatments for cancer). In other cases, the sample size is necessarily small by virtue of the limited available population (e.g., astronauts). Designs focused on individual effects, such as n-of-1 studies seem more appropriate when the avail-

OCR for page 1
Page 4 able population is limited than when the size of the potential population is large. Sampling of small populations is a problem in its own right and is distinct from the problem of making inferences (extrapolations) from the results of studies with small sample sizes. A threshold question, however, is whether the scientific bases of alternative and emerging methods, such as decision analysis or statistical prediction, alone or in combination, are sufficiently developed to demonstrate the efficacy or effectiveness of a therapeutic intervention in a small clinical trial. New approaches to protocol design are needed for studies with small sample sizes that can assess the potential therapeutic efficacies of drugs, biologics, devices, and other medical interventions. The rapid progress that is being made in a variety of areas (e.g., biotechnology, organ transplantation, gene therapy, cell and tissue engineering and therapies, biologically based artificial organs, designer drugs, and space travel) highlights the need to evaluate the effects of experimental interventions so that the benefits that arise from these advances can be made available safely and expeditiously. CHARGE TO THE COMMITTEE AND PLAN OF ACTION The Institute of Medicine, at the request of the National Aeronautics and Space Administration, asked a committee of experts to assess the current methodologies and the appropriate situations for the conduct of clinical trials with small sample sizes. The charge included a request to assess the published literature on various strategies such as (1) meta-analysis to combine disparate information from several studies including Bayesian techniques as in the confidence profile method and (2) other alternatives such as assessing therapeutic results in a single treated population (e.g., astronauts) by sequentially measuring whether the intervention is falling above or below a preestablished probability outcome range and meeting predesigned specifications as opposed to incremental improvement. A committee of nine members comprising experts with knowledge in biostatistics, clinical pharmacology, clinical research, ethics, and research design methods reviewed the scientific literature relevant to clinical trials with small sample sizes and held three meetings, including an invitational conference on future directions in clinical trials with small sample sizes. Conference participants consisted of individuals from federal research and regulatory agencies, industry, academia, and other areas of clinical research and practice. They were asked to provide information and perspective on the progress in developing strategies for the design, conduct, and evaluation of clinical trials with small sample sizes.

OCR for page 1
Page 5 FINDINGS Scientific research has a long history of using well-established, documented, and validated methods for the design, conduct, and analysis of clinical trials ( Box 1). A study design that is considered appropriate includes one with a sufficient sample size and statistical power and proper control of bias to allow a meaningful interpretation of the results. The committee strongly reaffirms that, whenever feasible, clinical trials should be designed and performed so that they have adequate statistical power. However, when the clinical context does not provide a sufficient number of research participants for a trial with an adequate statistical power but the research question has great clinical significance, research can still proceed under certain conditions. Properly designed trials with small sample sizes can contribute to substantial evidence of efficacy and are especially appropriate in particular situations ( Box 2). However, the conclusions derived from such studies may require careful consideration of the assumptions and inferences, given the BOX 1 Important Concepts in Clinical Trial Design Does the trial measure efficacy or effectiveness? A method of reducing bias (randomization and masking [blinding]) Inclusion of control groups      Placebo concurrent controls      Active treatment concurrent controls (superiority versus equivalence trial)      No-treatment concurrent controls      Dose-comparison concurrent controls      External controls (historical or retrospective controls) Use of masking (blinding) or an open-label trial      Double-blind trial      Single-blind trial Randomization      Use of randomized versus nonrandomized controls Outcomes (endpoints) to be measured: credible, validated, and responsive to change Sample size and statistical power Significance tests to be used

OCR for page 1
Page 6 BOX 2 Situations That Might Warrant a Small Clinical Trial Rare diseases Unique study populations (e.g., astronauts) Individually tailored therapies Environments that are isolated Emergency situations Public health urgency Restricted resources coupled with an important need paucity of data. There is nothing very different about small clinical trials relative to larger clinical trials other than greater uncertainty about the inferences made from the results of the trials. Design and Analysis of Small Clinical Trials Bearing in mind the statistical power, precision, and validity limitations of trials with small sample sizes, the committee notes that there are innovative design and analysis approaches that can improve the quality of such trials. A number of trial designs especially lend themselves to use in studies with small sample sizes, including n-of-1 designs, sequential designs, decision analysis-based designs, ranking and selection designs, adaptive designs, and risk-based allocation designs ( Box 3). A necessary companion to a well-designed clinical trial is an appropriate statistical analysis of the data from that trial. Assuming that a clinical trial will produce data that could reveal differences in effect between two or more interventions, statistical analyses are used to determine whether such differences are real or due to chance. Analysis of data from trials with small sample sizes in particular must be focused. In general, certain types of analyses are more amenable to trials with small sample sizes ( Box 4). Analysis should include confidence intervals when appropriate, although in trials with small sample sizes the confidence intervals will often be uninformative because they will be too wide. Although Bayesian methods require the use of subjective prior distributions, in small trials it will often be possible to use data from other sources to define the prior distributions. For example, for the problem of loss of bone mineral density during spaceflight, data from earlier spaceflights and studies

OCR for page 1
Page 7 BOX 3 Design Methods for Clinical Trials Traditional Designs for Clinical Trials Parallel group design Cross-over design Factorial design Add-on design Randomized withdrawal design Early escape design Special Designs for Small Clinical Trials n-of-1 design Sequential design Decision analysis-based design Ranking and selection design Adaptive design Risk-based allocation design of osteoporosis in immobilized individuals could provide a strong basis for development of prior distributions. These prior distributions could be used in a sequential trial setting that uses Bayesian methods, which would possibly add considerably to the power of the study. RECOMMENDATIONS Because of the constraints of trials with small sample sizes, for example, trials with participants with unique or rare diseases or health conditions, it is BOX 4 Statistical Approaches to Analysis of Data from Small Clinical Trials Sequential analysis Hierarchical models Bayesian analysis Decision analysis Statistical prediction Meta-analysis Risk-based allocation

OCR for page 1
Page 8 particularly important to define the research questions and select outcome measures that will make the best possible use of the available research participants while minimizing the risks to those participants. The limitations of small trials make it especially important that intermediate and surrogate outcomes be considered for measurement. It will not always be possible to measure directly the effect of an intervention on a given condition. RECOMMENDATION 1: Define the research question. Before undertaking a small clinical trial it is particularly important that the research question be well defined and that the outcomes and conditions to be evaluated be selected in a manner that will most likely help clinicians make therapeutic decisions. A multidisciplinary team of experts should be assembled to plan the research effort prospectively. Planning of clinical trials is a multistep process, and alternative methods should be considered to identify the most meaningful answer. To ensure that all approaches and limitations are considered, individuals experienced in trial design, statistics, and medicine are needed during the development of the research plan. In general, a small clinical trial is conducted because of external constraints, not necessarily by choice. Nonetheless, the common requirements for these trials should be no different from those for larger trials; that is, they must be soundly designed and appropriately analyzed to provide a reasonable measure of the effect of an intervention. They should be designed to have an outcome measure for determination of success, a baseline for the measurement of change, and a means to follow up the study participants to assess change. RECOMMENDATION 2: Tailor the design. Careful consideration of alternative statistical design and analysis methods should occur at all stages in the multistep process of planning a clinical trial. When designing a small clinical trial, it is particularly important that the statistical design and analysis methods be customized to address the clinical research question and study population. Because of the limitations of small clinical trials, it is especially important that the results be reported with accompanying details about the sample size, sample characteristics, and study design. The details necessary to combine evidence from several related studies, for example, measurement methods, main outcomes, and predictors for individual participants should be published. There are two reasons for this: first, it allows the clinician to

OCR for page 1
Page 9 interpret appropriately the data within the clinical context; second, it paves the way for future analyses of the study, for example, as part of a sequential design or a meta-analysis. In the clinical setting, the consequences might be greater if one misinterprets the results. In the research setting, insufficiently described design strategies and methods diminish the study's value for future analyses. RECOMMENDATION 3: Clarify methods of reporting of results of clinical trials. In reporting the results of a small clinical trial, with its inherent limitations, it is particularly important to carefully describe all sample characteristics and methods of data collection and analysis for synthesis of the data from the research. Since analysis of the data from small clinical trials will inevitably involve a number of assumptions, the use of several different statistical analyses is likely to enhance the acceptance (or rejection) of various assumptions. For example, if several different analyses give consistent results under different assumptions, one can be more confident that the results are not due to unwarranted assumptions. Conversely, if the analyses produce different results, depending on which sets of assumptions are used, one might be less certain about the original assumptions than might have been the case before the trial was conducted. In sum, the use of alternative statistical analyses might help identify the more sensitive variables and the key interactions in applying heterogeneous results across trials or in trying to draw generalizations from a number of trials. RECOMMENDATION 4: Perform corroborative statistical analyses. Given the greater uncertainties inherent in small clinical trials, several alternative statistical analyses should be performed to evaluate the consistency and robustness of the results of a small clinical trial. In small clinical trials, more so than in large clinical trials, one must be particularly cautious about recognizing individual variability among participants in terms of their biology and health care preferences and administrative variability in terms of what can be done from one setting to another. The diminished power of studies with small sample sizes might mean that the generalizability of the findings might not be a possibility in the short term, if at all. Thus, caution should be exercised in the interpretation of the results of small clinical trials.

OCR for page 1
Page 10 RECOMMENDATION 5: Exercise caution in interpretation. One should exercise caution in the interpretation of the results of small clinical trials before attempting to extrapolate or generalize those results. Researchers who participate in clinical trials have proposed alternative clinical trial designs, some of which have been applied to small clinical trials. The committee believes that the research base in this area requires further development. Alternative designs have been proposed in a variety of contexts; however, they have not been adequately examined in the context of small clinical trials. Studies of the use and effectiveness of various designs should be conducted and new methods should be developed. Evaluations of the utilities of individual and combined statistical analyses in a variety of small clinical trial designs will be necessary. RECOMMENDATION 6: More research on alternative designs is needed. Appropriate federal agencies should increase support for expanded theoretical and empirical research on the performances of alternative study designs and analysis methods that can be applied to small studies. Areas worthy of more study may include theory development, simulated and actual testing including comparison of existing and newly developed or modified alternative designs and methods of analysis, simulation models, study of limitations of trials with different sample sizes, and modification of a trial during its conduct. CONCLUDING REMARKS It should be noted that the various strategies and methodologies presented in this report are by no means an exhaustive list of those methods that are applicable to small clinical trials. Indeed, other strategies may be useful on a case-by-case basis. Moreover, the committee believes that all of the strategies described here have potential utility in specific settings and in studies with particular research characteristics and challenges. As a result, no single approach is advocated above all others. In addition, this is a developing area of research. New approaches to this problem will surely arise in the future and may well be in progress. The importance of conducting small clinical trials only when there are no alternatives cannot be overemphasized. The committee is not encouraging the use of small clinical trials, but, rather provides advice on strategies that should be considered in the design and analysis of small clinical trials

OCR for page 1
Page 11 when the opportunity to perform a randomized clinical trial with adequate statistical power is not possible. In doing so, it recognizes that small clinical trials frequently need to be viewed as part of a continuing process of data collection. Thus, for some trials it might be impossible to definitively answer a research question with a high degree of confidence. In those cases, perhaps the best that one can do is assess the next set of questions to be asked. SUMMARY OF THE COMMITTEE'S RECOMMENDATIONS RECOMMENDATION 1: Define the research question. Before undertaking a small clinical trial it is particularly important that the research question be well defined and that the outcomes and conditions to be evaluated be selected in a manner that will most likely help clinicians make therapeutic decisions. RECOMMENDATION 2: Tailor the design. Careful consideration of alternative statistical design and analysis methods should occur at all stages in the multistep process of planning a clinical trial. When designing a small clinical trial, it is particularly important that the statistical design and analysis methods be customized to address the clinical research question and study population. RECOMMENDATION 3: Clarify methods of reporting the results of clinical trials. In reporting the results of a small clinical trial, with its inherent limitations, it is particularly important to carefully describe all sample characteristics and methods of data collection and analysis for synthesis of the data from the research. RECOMMENDATION 4: Perform corroborative statistical analyses. Given the greater uncertainties inherent in small clinical trials, several alternative statistical analyses should be performed to evaluate the consistency and robustness of the results of a small clinical trial. RECOMMENDATION 5: Exercise caution in interpretation. One should exercise caution in the interpretation of the results of small clinical trials before attempting to extrapolate or generalize those results. RECOMMENDATION 6: More research on alternative designs is needed. Appropriate federal agencies should increase support for expanded theoretical and empirical research on the performances of alternative study designs and analysis methods that can be applied to small studies. Areas worthy of more study may include theory development, simulated and actual testing including comparison of existing and newly developed or modified alternative designs and methods of analysis, simulation models, study of limitations of trials with different sample sizes, and modification of a trial during its conduct.