National Academies Press: OpenBook

The Prevention and Treatment of Missing Data in Clinical Trials (2010)

Chapter: 6 Conclusions and Recommendations

« Previous: 5 Principles and Methods of Sensitivity Analyses
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2010. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press. doi: 10.17226/12955.
×

6
Conclusions and Recommendations

Missing data in clinical trials can seriously undermine the benefits provided by randomization into control and treatment groups. Two approaches to the problem are to reduce the frequency of missing data in the first place and to use appropriate statistical techniques that account for the missing data. The former approach is preferred, since the choice of statistical method requires unverifiable assumptions concerning the mechanism that causes the missing data, and so always involves some degree of subjectivity.

In Chapters 2 and 3, we detail some of the causes of missing data in clinical trials and discuss how to reduce the amount of missing data. However, because it is impossible to eliminate all occurrences of missing data, in Chapters 4 and 5 we discuss analysis methods that properly account for the missing outcome values.

In this concluding chapter, we bring all our recommendations together from the preceding chapters and offer three additional broad recommendations, two addressed to the U.S. Food and Drug Administration (FDA) and the companies that sponsor clinical trials. One is for the FDA and the National Institutes of Health to use their extensive database to develop a better understanding of the various causes of dropout from clinical trials, the typical extent of missing data in different types of trials, and the reductions in the rates of missing data that can be anticipated from the application of various alternative trial designs and techniques for trial conduct. A second recommendation is for the training of analysts in the latest techniques for the treatment of missing data in clinical trials. Finally, a third recommendation is for various research problems to be pursued.

Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2010. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press. doi: 10.17226/12955.
×

TRIAL OBJECTIVES

Questions such as whether to continue to collect trial outcome data after a participant has discontinued use of the study treatment, whether to use a single or composite outcome measure, how long to measure outcome data, all depend on the estimation goal of the trial. This estimation includes not only the outcome of interest, but also whether the focus is on short- or long-term effects of the intervention and the target population of interest. Possibilities for the latter include the “intent-to-treat” population, or the population of treatment compliers. Before selecting a trial design, it is important to decide on the primary parameter and population of interest, the “causal estimand.” Once the estimand is decided, the clinical trial design can be optimized for the measurement of that estimand.

Recommendation 1: The trial protocol should explicitly define (a) the objective(s) of the trial; (b) the associated primary outcome or outcomes; (c) how, when, and on whom the outcome or outcomes will be measured; and (d) the measures of intervention effects, that is, the causal estimands of primary interest. These measures should be meaningful for all study participants, and estimable with minimal assumptions. Concerning the latter, the protocol should address the potential impact and treatment of missing data.

REDUCING DROPOUTS THROUGH TRIAL DESIGN

The interpretation of the trial findings is more difficult when participants discontinue their assigned interventions before the end of the study. Therefore, the trial design should be selected to maximize the number of participants who are maintained on the study intervention throughout the duration of the trial.

Recommendation 2: Investigators, sponsors, and regulators should design clinical trials consistent with the goal of maximizing the number of participants who are maintained on the protocol-specified intervention until the outcome data are collected.

There is a key distinction between treatment dropout and analysis dropout, and although there are trials in which treatment dropout will understandably be substantial, there is very little reason for substantial amount of missing data, that is, analysis dropouts. Furthermore, for many trial estimands, the benefits of retaining participants in the study can be substantial, including to support an analysis of effectiveness (comparison

Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2010. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press. doi: 10.17226/12955.
×

of treatment policies) and to be able to monitor side effects that occur after discontinuation of treatment.

Recommendation 3: Trial sponsors should continue to collect information on key outcomes on participants who discontinue their protocol-specified intervention in the course of the study, except in those cases for which a compelling cost-benefit analysis argues otherwise, and this information should be recorded and used in the analysis.


Recommendation 4: The trial design team should consider whether participants who discontinue the protocol intervention should have access to and be encouraged to use specific alternative treatments. Such treatments should be specified in the study protocol.


Recommendation 5: Data collection and information about all relevant treatments and key covariates should be recorded for all initial study participants, whether or not participants received the intervention specified in the protocol.

REDUCING DROPOUTS THROUGH TRIAL CONDUCT

In addition to trial design, aspects of trial conduct can also substantially reduce the amount of missing data. Chapter 3 outlines specific trial conduct techniques that should be considered. Given the importance of reducing the frequency of missing data, the monitoring of missing data from the design stage throughout the conduct of a trial needs to be accounted for in the trial protocol.

Recommendation 6: Study sponsors should explicitly anticipate potential problems of missing data. In particular, the trial protocol should contain a section that addresses missing data issues, including the anticipated amount of missing data, and steps taken in trial design and trial conduct to monitor and limit the impact of missing data.


Recommendation 7: Informed consent documents should emphasize the importance of collecting outcome data from individuals who choose to discontinue treatment during the study, and they should encourage participants to provide this information whether or not they complete the anticipated course of study treatment.


Recommendation 8: All trial protocols should recognize the importance of minimizing the amount of missing data, and, in particular, they

Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2010. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press. doi: 10.17226/12955.
×

should set a minimum rate of completeness for the primary outcome(s), based on what has been achievable in similar past trials.

TREATING MISSING DATA

Missing data are often unavoidable, despite best efforts to reduce their occurrence in trial design and conduct. The validity of assumptions concerning the source of missing data can only be assessed jointly by both data analysts and clinicians. Therefore, it is important that the assumptions underlying any selected analysis technique be clearly articulated so that they can be evaluated by clinicians as well as by statistical analysts.

Recommendation 9: Statistical methods for handling missing data should be specified by clinical trial sponsors in study protocols, and their associated assumptions stated in a way that can be understood by clinicians.

Methods like last observation carried forward (LOCF) and baseline observation carried forward (BOCF) are commonly applied in situations in which their underlying assumptions are unrealistic. The analysis methods used should yield confidence intervals for the treatment effect that have the claimed coverage properties and tests should have their nominal size when data are missing.

Recommendation 10: Single imputation methods like last observation carried forward and baseline observation carried forward should not be used as the primary approach to the treatment of missing data unless the assumptions that underlie them are scientifically justified.


Recommendation 11: Parametric models in general, and random effects models in particular, should be used with caution, with all their assumptions clearly spelled out and justified. Models relying on parametric assumptions should be accompanied by goodness-of-fit procedures.


Recommendation 12: It is important that the primary analysis of the data from a clinical trial should account for the uncertainty attributable to missing data, so that under the stated missing data assumptions the associated significance tests have valid type I error rates and the confidence intervals have the nominal coverage properties. For inverse probability weighting and maximum likelihood methods, this analysis can be accomplished by appropriate computation of standard errors, using either asymptotic results or the bootstrap. For imputation, it

Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2010. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press. doi: 10.17226/12955.
×

is necessary to use appropriate rules for multiply imputing missing responses and combining results across imputed datasets because single imputation does not account for all sources of variability.


Recommendation 13: Weighted generalized estimating equations methods should be more widely used in settings when missing at random can be well justified and a stable weight model can be determined, as a possibly useful alternative to parametric modeling.

One very useful source of information that appears to have been rarely used is the follow-up of a sample of participants who withdrew from a study. Such data could be very useful in determining reasons for withdrawal and their missing outcome measurements.

Recommendation 14: When substantial missing data are anticipated, auxiliary information should be collected that is believed to be associated with reasons for missing values and with the outcomes of interest. This could improve the primary analysis through use of a more appropriate missing at random model or help to carry out sensitivity analyses to assess the impact of missing data on estimates of treatment differences. In addition, investigators should seriously consider following up all or a random sample of trial dropouts, who have not withdrawn consent, to ask them to indicate why they dropped out of the study, and, if they are willing, to collect outcome measurements from them.

Given that the assumptions for the missing data mechanism cannot be validated, the sensitivity of inferences for treatment effects in clinical trials to those assumptions needs to be assessed.

Recommendation 15: Sensitivity analyses should be part of the primary reporting of findings from clinical trials. Examining sensitivity to the assumptions about the missing data mechanism should be a mandatory component of reporting.

UNDERSTANDING THE CAUSES AND DEGREE OF DROPOUTS IN CLINICAL TRIALS

A crucial issue that sponsors must wrestle with in planning a clinical trial is how much missing data they are likely to experience, how much could be reduced through the use of various techniques (such as those outlined in this report), and consequently if they implement these various techniques, what degree of missingness is likely to remain. The answers to

Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2010. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press. doi: 10.17226/12955.
×

these questions will help trial sponsors decide on how to plan for missing data when determining sample size, whether steps are needed to reduce the amount of missing data (some of which may be resource intensive), and the potential for lack of robustness of final estimates of intervention effects to assumptions about missing data. In addition, analysts need to know what assumptions about the missing data mechanisms are scientifically defensible.

Information from previously collected clinical studies would help in answering these questions. Although FDA retains data from all clinical trials over which it has oversight, the data are confidential to the company that sponsored the trial and are therefore not shared. And although there is some research on why participants drop out of different kinds of clinical trials, empirical evidence is lacking for many types of trials. There is a need for more standardized data collection, documentation, and analysis of the reasons for and the frequency of missing data. Systematic investigations of factors related to treatment dropout and withdrawal and to missing data more generally are needed.

A pharmaceutical company that has been researching interventions in a particular area for a long time may have internal data that can provide some of this information. However, if a company is small or has limited prior experience, having access to information from prior clinical trials would be extremely useful in trial design.

Recommendation 16: The U.S. Food and Drug Administration and the National Institutes of Health should make use of their extensive clinical trial databases to carry out a program of research, both internal and external, to identify common rates and causes of missing data in different domains and how different models perform in different settings. The results of such research can be used to inform future study designs and protocols.

While it is difficult to be specific, characteristics of a trial that has failed because of missing data concerns include (a) rates of missing data that are two to three times as large as the difference in rate of successful outcome between the groups; (b) differential rates of missing data across treatment arms, indicating a high likelihood that biases would not cancel out in the treatment comparison; (c) lack of a record about the reasons for missing data, making it unclear whether the mechanisms are MAR; (d) lack of auxiliary data that would be the basis for plausible missing data adjustments; (e) marginal treatment effects that might be easily overturned by uncertainty from missing data; and (f) inadequate analysis methods that do not reflect uncertainty from missing data.

Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2010. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press. doi: 10.17226/12955.
×

Many of the analysis techniques described in Chapters 4 and 5 have been explored both theoretically and in applications in the research literature over the past 20 years. However, their applications to clinical trials have been limited. There seems to be a reticence on the part of biostatisticians, both at the drug and device companies and at FDA, to embrace these various techniques. We conjecture that this reticence may be a by-product of the regulatory environment, a result of the limited development of supporting software for newer methods, or a result of a lack of training and education. We believe that once these techniques are implemented in a few major clinical trials, any conservatism from the regulatory environment would be offset by the more effective use of the information provided by these techniques. Also, as pointed out in Chapters 4 and 5, software now exists for most of the techniques as part of the most commonly used statistical packages. The remaining possibility, training and education, can be addressed by making education at FDA a higher priority. It is also important that FDA’s clinical reviewers have some understanding of modern analysis methods so that they can assist in the judgment as to the reasonableness of assumed missing data mechanisms.

Recommendation 17: The U.S. Food and Drug Administration (FDA) and drug, device, and biologic companies that sponsor clinical trials should carry out continued training of their analysts to keep abreast of up-to-date techniques for missing data analysis. FDA should also encourage continued training of their clinical reviewers to make them broadly familiar with missing data terminology and missing data methods.

Throughout this report, we have advocated that further research be carried out in a number of important areas. We have decided to bring together those calls for additional research here in this final chapter. Areas in need of further research include

  1. designs for the follow-up of participants in clinical trials who have dropped out of the study (e.g., referred to here as analysis dropouts) and who have not withdrawn their consent,

  2. collecting the typical rates and likely causes of missing data in various kinds of clinical trials,

  3. the effect of missing data on the power of clinical trials,

  4. how to set useful target rates and acceptable rates of missing data in clinical trials,

  5. the robustness of missing data methods such as inverse probability weighting methods and multiple imputation methods to assumptions,

Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2010. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press. doi: 10.17226/12955.
×
  1. the assessment of goodness-of-fit for the parametric models used to analyze data from clinical trials (when there is missing data),

  2. the performance of double-robust procedures in comparison to more commonly used procedures,

  3. the impact of missingness in auxiliary variables on the various current methods, and ways of reducing the associated bias,

  4. methods of sensitivity analysis in clinical trials, particularly for nonmonotone patterns in longitudinal data,

  5. methods for assessing and limiting the impact of informative censoring for time-to-event outcomes, and

  6. how to develop effective decision rules based on the input from sensitivity analyses.

We have collected the highest priority of these calls for additional research in a final recommendation, adding to that a call for the development of the associated software tools.

Recommendation 18: The treatment of missing data in clinical trials, being a crucial issue, should have a higher priority for sponsors of statistical research, such as the National Institutes of Health and the National Science Foundation. There remain several important areas where progress is particularly needed, namely: (1) methods for sensitivity analysis and principled decision making based on the results from sensitivity analyses, (2) analysis of data where the missingness pattern is non-monotone, (3) sample size calculations in the presence of missing data, and (4) design of clinical trials, in particular plans for follow-up after treatment discontinuation (degree of sampling, how many attempts are made, etc.), and (5) doable robust methods, to more clearly understand their strengths and vulnerabilities in practical settings. The development of software that supports coherent missing data analyses is also a high priority.

Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2010. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press. doi: 10.17226/12955.
×
Page 107
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2010. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press. doi: 10.17226/12955.
×
Page 108
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2010. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press. doi: 10.17226/12955.
×
Page 109
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2010. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press. doi: 10.17226/12955.
×
Page 110
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2010. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press. doi: 10.17226/12955.
×
Page 111
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2010. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press. doi: 10.17226/12955.
×
Page 112
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2010. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press. doi: 10.17226/12955.
×
Page 113
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2010. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press. doi: 10.17226/12955.
×
Page 114
Next: References »
The Prevention and Treatment of Missing Data in Clinical Trials Get This Book
×
Buy Paperback | $44.00 Buy Ebook | $35.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Randomized clinical trials are the primary tool for evaluating new medical interventions. Randomization provides for a fair comparison between treatment and control groups, balancing out, on average, distributions of known and unknown factors among the participants. Unfortunately, these studies often lack a substantial percentage of data. This missing data reduces the benefit provided by the randomization and introduces potential biases in the comparison of the treatment groups.

Missing data can arise for a variety of reasons, including the inability or unwillingness of participants to meet appointments for evaluation. And in some studies, some or all of data collection ceases when participants discontinue study treatment. Existing guidelines for the design and conduct of clinical trials, and the analysis of the resulting data, provide only limited advice on how to handle missing data. Thus, approaches to the analysis of data with an appreciable amount of missing values tend to be ad hoc and variable.

The Prevention and Treatment of Missing Data in Clinical Trials concludes that a more principled approach to design and analysis in the presence of missing data is both needed and possible. Such an approach needs to focus on two critical elements: (1) careful design and conduct to limit the amount and impact of missing data and (2) analysis that makes full use of information on all randomized participants and is based on careful attention to the assumptions about the nature of the missing data underlying estimates of treatment effects. In addition to the highest priority recommendations, the book offers more detailed recommendations on the conduct of clinical trials and techniques for analysis of trial data.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!