Trial Strategies to Reduce the Frequency of Missing Data
This chapter discusses a number of strategies that can be applied to reduce the amount of missing data during trial implementation and conduct. The approaches in this chapter deal with the practical aspects of trial conduct, rather than the more fundamental design aspects covered in Chapter 2. We classify trial strategies into two types: (1) actions for design and management teams and (2) actions for investigators and site personnel. Before turning to those strategies, we briefly comment on the research literature on clinical trial dropouts, and we end the chapter with a look at setting and meeting targets for missing data.
REASONS FOR DROPOUTS
The literature on the factors associated with and the effectiveness of various measures to reduce the occurrence of missing outcome values is relatively diffuse, possibly dependent on the medical condition, the intervention under study, and the population of interest. Thus, the literature is difficult to summarize and often not a great deal is known for particular situations. However, several statements can be supported: there is a lack of consensus regarding how to measure dropout rates; dropout can often be very substantial, sometimes more than 30 percent; and the rate of missing outcome data can sometimes be substantially reduced by applying some of the ideas suggested in Chapter 2 and below (see, e.g., Sprague et al., 2003; Oleske et al., 2007; Robinson et al., 2007; Snow et al., 2007; Warden et al., 2007; Williams et al., 2008).
ACTIONS FOR DESIGN AND MANAGEMENT TEAMS
This section discusses some techniques that trial design and management teams can use to reduce the frequency of dropouts. First, designers and managers can limit participants’ burden and inconvenience in the data collection stage. This can be done in at least five ways: (1) minimizing the number of visits and assessments, (2) collecting only the information that is needed at each visit, (3) using user-friendly case report forms, (4) using direct data capture that does not require a clinic visit whenever feasible, and (5) allowing a relatively large time window for each follow-up assessment. Examples of information not needed at each visit include aspects of the participant’s medical history and contact information that were provided at earlier visits and information available from medical records. The overall aim is to balance the competing goals of reducing response burden and collecting sufficient information to fully support the analytic goals and to guide the next steps in treatment. (Regarding use of direct data capture to minimize the response burden, it would also be useful to attempt to collect whatever information is available from administrative records.)
Second, design and management teams can increase the incentives for participation and completion by the provision of effective treatments to participants after the trial. Such incentives might include continued access to effective study treatments on extension protocols until the treatment is licensed.
Third, designer and managers can select investigators with a good track record of both enrolling and following participants and collecting complete data in previous trials, and provide good training. The training (and onstudy reinforcement) needs to emphasize the importance of complete data collection and the difference between discontinuing the study treatment and discontinuing data collection. Training should stress the value of collecting data after a participant discontinues the study (or the control) treatment.
As discussed in Chapter 2, many trial sponsors and investigators mistakenly assume that there is little reason for additional data collection when participants discontinue study treatment. But as we emphasize, the continued collection of data is important in many trials. The trial objectives and estimands need to be considered.
Training can also emphasize the importance of the informed consent process as a mechanism for ensuring that participants understand the commitment they are making, including their intent to complete the trial regardless of the treatment they are receiving. Training of investigators and research staff should also emphasize how to work with participants to minimize the extent of missing data (see later in this chapter). Finally, trainers need to know and explain to participants that any decision to withdraw consent is a participant’s decision, not the investigator’s decision. However,
when participants are dissatisfied with the conduct of the trial but have not yet withdrawn, the investigator should make an effort to address their concerns and retain them in the trial, rather than simply indicating that the participants withdrew consent. In doing so, investigators must be careful that their efforts do not cross over into coercion.
Fourth, designers and managers need to consider how investigators are paid. Paying investigators solely by the number of participants enrolled should be avoided because it places too much emphasis on enrollment and not enough on follow-up; payments should also reflect follow-up work (e.g., payment per visit or procedure). In addition, linking some additional compensation to the completeness of the data collection should be considered. It is acceptable and generally advisable to link a final payment to completion of forms at a study closeout visit (i.e., a final visit at the end of follow-up to assess the participant’s status). However, care must be taken on this point. Providing extra compensation to investigators for encouraging participants to complete a study when the participants are thereby exposed to significant additional risks could create a conflict of interest on the part of the investigator and would therefore be unethical. But if there are minimal risks associated with data collection to the participant, it may be acceptable to provide financial incentives to the investigator to continue to collect data, whether or not the participant continues treatment.
Fifth, designers and managers can ensure that data collection is monitored and reported during the trial. Missing data and missed visits that could affect important outcomes need to be assessed in real time by site personnel during a clinical trial. The information from these assessments should be available and shown to investigators at regular meetings and on study websites, creating a climate to encourage other investigators to collect more complete data. Also, identification of poorly performing sites can help identify the need for some sort of remediation, including additional training, site visits, or even site closure. Site visits should be targeted on the basis of assessments of the amount of missing data, with the goal of helping to correct the problem.
ACTIONS FOR INVESTIGATORS AND SITE PERSONNEL
Investigators and site personnel can also act in several ways to reduce the amount of missing data. First, in the informed consent process, they can emphasize to participants the importance of continued participation for the full duration of the trial. Similarly, they can ensure that the trial procedures allow for an informed withdrawal of consent so that participants recognize the importance of continued follow-up for data collection if they discontinue study treatment: see Box 3-1 for an example of language for withdrawal. Second, investigators and site personnel can provide incentives
Example of Language for Withdrawal of Informed Consent
for participants. In general, paying for voluntary participation in a clinical trial is regarded as ethical (see, e.g., Emmanuel, 2005). When compensation is to be provided, the Code of Federal Regulations requires that the responsible Institutional Review Board (IRB) ensure that the compensation is neither coercive nor at the level that would present undue influence (21 CFR 50.20). Providing cash is generally not viewed as being coercive, as it is a benefit. Most IRBs allow cash payments to be slightly backloaded (retaining a small proportion as an incentive for completion), but, generally, payments accrue as a study progresses in payment for participation activities that are completed. Compensating people for taking risks is not uncommon, and as noted it is generally acceptable if not judged as coercive. Payments for return visits of participants who have stopped taking medication are virtually always considered ethical, since the risk to the participant is zero or minimal. Study-branded gifts are also ethical and may have the added effect of increasing the participant’s engagement with the trial.
Third, investigators and site personnel can collect information on which participants are at risk for dropping out and why: formal “intent-to-attend” questioning may help to identify reasons for dropout (see, e.g., Leon et al., 2007) and may yield useful covariates in missing data models. Factors influencing decisions to participate include: (i) time and duration of visits, (ii) need for assistance with transportation or child care, (iii) need for reminders, (iv) problems in relations with the staff, (v) problems with blood drawing or other procedures, (vi) side effects, and (vii) perceptions of intervention efficacy.
Fourth, investigators and site personnel can educate participants on the importance of continued engagement in the trial in order to help contribute
to important scientific knowledge. Mechanisms for such education include the production of a study newsletter, maintenance of a regularly updated website for trial participants, and providing access to interim papers and presentations on study progress and findings. (We note that IRBs may require approval for some communications with study participants.)
Fifth, investigators and site personnel can increase participants’ engagement and retention in the study by such mechanisms as study-branded gifts; regular expressions of thanks, both verbal and written; social networking; and solicitation of input regarding relevant issues of study conduct. Other ways to encourage participation and involvement include reminders before a visit and after missed visits help to encourage participation.
Sixth, investigators and site personnel can make participation enjoyable in many ways, including: (i) development of a welcoming environment, (ii) hiring of friendly staff, (iii) operational practices that are respectful of participants’ time and schedules, (iv) availability of on-site diversions for small children, and (v) valued education at the site.
Seventh, investigators and site personnel can ensure that participants’ contact information is updated at each visit, recognizing that in some studies, home visits may be needed to keep all contact information current. The provision of transportation and child care costs can also improve retention.
For participants who want to discontinue treatment, it is important for site personnel to determine the reasons and to make sure that the participants understand the importance of continuing on the study for the purpose of data collection. If participants switch to alternative treatments due to intolerance, it is important for investigators and site personnel to document the changes because they may be useful in summarizing the study results.
Recommendation 6: Study sponsors should explicitly anticipate potential problems of missing data. In particular, the trial protocol should contain a section that addresses missing data issues, including the anticipated amount of missing data, and steps taken in trial design and trial conduct to monitor and limit the impact of missing data.
Recommendation 7: Informed consent documents should emphasize the importance of collecting outcome data from individuals who choose to discontinue treatment during the study, and they should encourage participants to provide this information whether or not they complete the anticipated course of study treatment.
TARGETS FOR ACCEPTABLE RATES OF MISSING DATA
Although some missing data should be anticipated for every clinical trial, levels that are unacceptable given the design should be considered in
writing the protocol. One way to set target rates and maximally acceptable rates for missing data would be to use the results from similar trials to help determine what is reasonably achievable and did not excessively impact study conclusions and to determine how missing data can be further minimized. For example, using the findings from completed trials, some percentile could be used for the target and some higher percentile for the maximally acceptable value. Another possibility, which is likely not currently feasible but would be after a sufficient number of effective sensitivity analyses have been carried out (depending on the characteristics of the trials) is to observe what rates of missing values, in trials in which the primary analysis demonstrated a significant benefit, resulted in alternative analyses in which the treatment effect was no longer significant. In this way, one could try to limit the amount of missing data to ensure that a sensitivity analysis would not contradict the findings of the primary analysis.
Once goals are established, performance against these goals can be monitored, and the goals can be used to motivate investigators. Comparison of targets and current rates of missing data could also be used by a data monitoring committee to halt a trial for underperformance.
Establishing reasonable goals and adhering to them may not be an easy task for several reasons: (1) there may not be many similar trials to use as a basis for acceptable levels of missing data; (2) it may be difficult to determine the steps that trial investigators took to reduce missing data; and (3) it may be difficult to determine what, if any, sensitivity analyses were carried out for trials conducted by other sponsors. Nevertheless, it is important to set high standards for the participating investigators and monitor the amount of missing data for key outcomes in real time.
We note that one cannot be specific as to how to set target and maximally acceptable rates for missing data in all clinical trials. The amount of acceptable missing data will depend on many characteristics of the trial, including whether the assumption that the missing data are missing at random is reasonable, the size of the anticipated effect of the intervention under study, and the likelihood that a sensitivity analysis would render the results of the trial inconclusive. Applied research is needed on this topic, and techniques will need to evolve on the basis of that research.
Recommendation 8: All trial protocols should recognize the importance of minimizing the amount of missing data, and, in particular, they should set a minimum rate of completeness for the primary outcome(s), based on what has been achievable in similar past trials.
Finally, to monitor the degree of missing outcome data, the data and safety monitoring board (DSMB) for a trial should be aware of the trial’s target for missing data, and the investigators should report to the DSMB