The Vaccine Safety Datalink (VSD) is a large linked database that was created in 1991 to fill a void in the ability of the United States to study vaccine safety issues. The VSD was developed through the collaborative efforts of the National Immunization Program (NIP) at the Centers for Disease Control and Prevention (CDC) and several private managed care organizations (MCOs) (Chen et al., 1997). The VSD is a unique national resource for evaluating vaccine safety. It includes data from administrative records for more than 7 million members of eight MCOs (Davis, 2004). The VSD database links data on patient characteristics, health outcomes (according to data resulting from inpatient, outpatient, and emergency-room records), and vaccination history (vaccine type, date of vaccination, manufacturer, lot number, and injection site) (Davis, 2004). The VSD can be a valuable tool for the retrospective assessment of vaccine safety because the number of people included is large, they generally receive most of their health services at the MCOs, and demographic, health outcome, and vaccination data are maintained electronically.
The opportunities offered by the VSD for thorough investigations of vaccine safety concerns and well-designed, planned, retrospective vaccine studies have led to heightened interest in the results of VSD studies and sometimes in the VSD data themselves. A few researchers interested in particular vaccine safety hypotheses also have shown interest in accessing and analyzing VSD data. The interest in the VSD shown by researchers, advocacy groups, members of Congress, and others has brought in-
creasing attention to its use, its limitations, and the implications of studies that were conducted through the analysis of its data.
Throughout the first decade of the VSD’s existence, researchers from the NIP and the MCOs participating in the VSD collaborated on studies that used VSD data. During that time, there was no way for an independent external researcher who did not pursue a collaborative relationship with a NIP-affiliated or MCO-affiliated researcher to use the VSD. In 2002, after requests by independent external researchers that VSD data be made available, the NIP announced the creation of the VSD data sharing program (CDC, 2004d). The VSD data sharing program guidelines (CDC, 2002, 2003a, 2004a,b,c) have been revised multiple times since the inception of the program.
Concerns about data sharing were stimulated in part by public concern over a study initiated in 1999 using VSD data. In fall 1999, researchers at the NIP began a screening study using VSD data to investigate whether exposure to thimerosal in vaccines (to which it was added as a preservative) was associated with neurodevelopmental disorders (DeStefano, 2004; Verstraeten et al., 2003a). Some members of the general public have criticized the thimerosal screening study for changes in the original study protocol, changes in eligibility criteria, the selective official release of preliminary findings, and the inclusion of vaccine-manufacturer representatives in a meeting intended to provide external expert review of the study (Bernard, 2004).
CHARGE TO THE COMMITTEE
The Institute of Medicine Committee on the Review of the National Immunization Program’s Research Procedures and Data Sharing Program was asked to address the following charge:1
(1a) review the design and the implementation to date of the new Vaccine Safety Datalink Data Sharing Program to assess compliance with the current standards of practice for data sharing in the scientific community and (1b) make recommendations to the National Immunization Program and the National Center for Health Statistics for any needed modifications that would facilitate use, ensure appropriate utilization, and protect confidentiality; and (2a) review the iterative approaches to conducting analysis that are characteristics of studies using the complex, automated
Vaccine Safety Datalink system. Examples of recent studies to be examined are a completed screening study on thimerosal and vaccines (Verstraeten et al.) and cohort studies on asthma; (2b) review whether, when, and how preliminary findings about potential vaccine-related risks obtained from the Vaccine Safety Datalink system should be shared with other scientists, communicated to the public, and used to make policy or recommendations to CDC; and (2c) make recommendations to the National Immunization Program on the release of such preliminary findings in the future.
The charge was expanded to include the National Center for Health Statistics (NCHS) in part 1 because organizational responsibility for the VSD data sharing program was transferred from the NIP to NCHS in March 2004.
In the course of its deliberations, the committee found that several overarching principles emerged. The principles can be described as common themes inherent in the committee’s recommendations and thus principles that should be considered in any modifications of the VSD data sharing program or any determinations about whether, when, and how to release VSD preliminary findings. The four overarching principles that emerged from the committee’s recommendations are these:
Independence. Ensure that potential biases and potential conflicts of interest are minimized, balanced, or otherwise managed in the design and implementation of all processes, practices, and policies related to the VSD.
Transparency. Ensure that all processes, practices, and policies related to the VSD are developed in the spirit of openness, clearly articulated, and easily available to interested persons or entities, and that any deviations from them are documented and justified.
Fairness. Ensure that all processes, practices, and policies related to the VSD are designed and implemented in a fair manner.
Protection of confidentiality. Ensure that the design and implementation of the VSD protect the confidentiality of individually identifiable information.
Concerns about trust and how the public perceives the reliability of findings based on VSD data have spilled over from the NIP’s analyses to the development and implementation of the VSD data sharing program. Taking steps to improve the independence, transparency, and fairness of VSD procedures while continuing to protect confidentiality will help to
enhance trust in the processes and procedures used for VSD research and the VSD data sharing program. Trust can be enhanced only if the public has confidence in the independence and fairness of the decision-making process for VSD research priorities and approval of VSD data sharing proposals.
The VSD is a public resource that is designed to inform important public health policy decisions. Though it is a resource supported by public funds, there are restrictions on access because the data are provided by and remain the property of private MCOs. By the very nature of its potential to influence policy, the public demands and deserves access to the data used to influence those decisions and transparency in the processes that permit or restrict access. If the VSD is intended to be used as a foundation of policy decisions, there is a public need to share data fairly and to be as transparent as possible while protecting the confidentiality of individually identifiable information in the VSD. Confidentiality protections must not be jeopardized; a single breach of confidentiality, no matter how minor, could undermine the contractual arrangements between the MCOs and the NIP and lead to the termination of cooperation and the loss of a unique resource of potentially great national value.
The committee determined that it could not adequately address issues of independence, transparency, fairness, and protection of confidentiality without examining how the VSD research process supports or hinders the application of those principles. When concerns arise about the independence and transparency, in particular, of the general VSD research process, those concerns spill over to people’s perception of the independence and transparency of the VSD data sharing program and of the determinations about whether, when, and how to release VSD preliminary findings. To provide the most appropriate and useful recommendations requested in its charge, the committee believed that it had to consider how the VSD research plan, the priority-setting of VSD studies, and the VSD peer-review process affect the VSD data sharing program and the release of preliminary findings.
The committee’s recommendations are related to four main topics: the VSD data sharing program; the release of preliminary findings based on VSD data; independent review of VSD activities; and the applicability of the Shelby Amendment and Information Quality Act to VSD data and VSD preliminary findings.
THE VACCINE SAFETY DATALINK DATA SHARING PROGRAM
Reflecting on all the information gathered throughout its study, the committee finds that the VSD data sharing program has three short-term goals:
To facilitate access to and use of the VSD;
To protect the confidentiality of individually identifiable data in the VSD; and
To enhance public trust in the VSD as a tool to address specific concerns about vaccine safety.
On the basis of those three goals, the committee developed its recommendations related to the VSD data sharing program in the following framework:
The NIP should support the broadest feasible use of the VSD for vaccine safety research within the constraints of law, protection of confidentiality, and VSD contract provisions;
Bureaucratic and technical barriers to accessing the VSD should be minimized, although some types of studies may require collaboration with or facilitation by data custodians;
Guidelines for proposals from independent external researchers should be developed and publicized to facilitate access;
Responses to proposals should be timely;
Criteria for the independent review of proposals should be publicly accessible;
Costs to researchers should approximate the incremental costs of access;
Descriptions of the objectives and methods of current and published studies should be made publicly available;
All VSD users should provide a timely and detailed public report of their results to the NIP; and
All completed VSD studies should be subjected to scientific peer review before any public release.
LIMITATIONS OF THE VACCINE SAFETY DATALINK DATA SHARING PROGRAM AND THE NEED FOR COLLABORATION
The VSD and the VSD data sharing program have a number of limitations. Because VSD data come from administrative databases of the MCOs, additional data collection (for example, medical-chart reviews) or data cleaning must be done before the automated data are suitable for many specific research studies. For new vaccine safety studies, the VSD data sharing program allows external researchers access only to automated data and only to data from before 2001 (CDC, 2004a). Because the quality of automated data cannot be guaranteed, the inclusion of chart-review-verified data in new vaccine safety studies improves the quality of such studies. Chart-review-verified data can be obtained only by collaborating
with NIP-affiliated or MCO-affiliated researchers. Thus, it is important for independent external researchers to try to collaborate with a NIP-affiliated or MCO-affiliated researcher to produce a new, high-quality vaccine safety study with VSD data from 2001 and later.
Because of the limitations of the data available through the program and the differing levels of access to VSD data that depend on the type of researcher requesting access, the VSD data sharing program does not meet the traditional definition of data sharing. For the VSD data sharing program to be considered a true data sharing program, changes must be made.
All the committee’s recommendations related to the VSD data sharing program can be found in Box ES-1 at the end of this executive summary.
RELEASE OF PRELIMINARY FINDINGS BASED ON VACCINE SAFETY DATALINK DATA
Solely internal peer-review processes may be needed when preliminary findings could have a substantial impact on public health. The need to release preliminary findings rapidly may force a decision to limit peer review to peers inside the federal government, but if so, the internal peer review should be as extensive as possible. In such situations, purely internal review should be followed by external review on an expedited schedule. In the case of the VSD, however, the committee finds that because the data are incorporated into the VSD data files annually rather than continually, there will rarely be situations in which preliminary findings are so urgent that they cannot undergo independent external peer review.
Numerous concerns arise in the release of preliminary findings, but in some situations it is appropriate to release preliminary findings about potential vaccine-related risks. Conditions governing whether, when, and how to share preliminary VSD findings with other scientists, the public, and policy-makers should be defined a priori. Release of preliminary findings shared with others, used to make policy decisions, or superseded by later findings are special situations that require special considerations. Any preliminary findings that are released under such conditions need to be communicated in an appropriate context.
All the committee’s recommendations related to the release of preliminary findings based on VSD data can be found in Box ES-1 at the end of this executive summary.
INDEPENDENT REVIEW OF VACCINE SAFETY DATALINK ACTIVITIES
There are legitimate concerns about the independence and fairness of the implementation of review procedures applied to VSD data sharing
proposals and of determinations about the release of preliminary findings from VSD analyses. The lack of transparency of some of the processes also affects the trust relationship between the NIP and the general public. To address some of those concerns, the committee recommends that two independent groups be used.
To give the full array of stakeholders an opportunity to provide input into the VSD research plan priority-setting process and to ensure that the process is as transparent as possible, an independent group should be used to review and provide advice on the VSD research plan. The committee recommends that a subcommittee of the National Vaccine Advisory Committee that includes representatives of a wide variety of stakeholders (such as advocacy groups, vaccine manufacturers, FDA, and CDC) review and provide advice to the NIP on the VSD research plan annually. The subcommittee charged with this role could be the existing Subcommittee on Safety and Communications or a subcommittee created specifically for the purpose.
In addition, to enhance trust in the fairness of the VSD research process, an independent review committee (advisory to the director of CDC) with minimal and balanced biases and conflicts of interest should be established to review various aspects of VSD research activities and of the VSD data sharing program. The committee recommends that an independent review committee with minimal and balanced biases and conflicts of interest be created to:
Review independent external researchers’ proposals to use VSD data through the data sharing program;
Review research proposals from internal researchers and provide oversight of changes in or deviations from research protocols for internal VSD studies; and
Provide advice on when and how preliminary findings based on VSD data should be made public.
The committee makes other recommendations throughout its report that focus on specific issues related to the VSD data sharing program and the determination of whether, when, and how to release preliminary findings. However, the committee’s recommendations regarding independent oversight of VSD-related activities are the primary means of improving transparency and ensuring fair implementation of processes, practices, and policies related to VSD access, and thereby enhancing public trust in the use of the VSD to answer vaccine safety questions.
The VSD database has many strengths, but it also has limitations. The value of the VSD data sharing program will be enhanced by easy access to
the data, so that a variety of researchers can conduct a range of studies and have their findings reviewed by peers and discussed in ways conducive to the advancement of knowledge about vaccine safety. The VSD is a valuable resource for the nation. Efforts should be made to facilitate access to VSD data and their appropriate utilization, while protecting the confidentiality of information contained therein. Ensuring the independence, transparency, and fairness of VSD research activities is important for ensuring public trust in the VSD as a tool for addressing critical vaccine safety questions.
DESCRIPTION OF THE VACCINE SAFETY DATALINK
Recommendation 2.1: The committee recommends that the NIP and NCHS seek legal advice to clarify the applicability of the Shelby Amendment and the Information Quality Act to VSD data and VSD preliminary findings.
THE VACCINE SAFETY DATALINK DATA SHARING PROGRAM
Recommendation 3.1: The committee recommends that future revisions of the VSD data sharing guidelines clearly and explicitly describe the VSD data that are and are not available to independent external researchers for new vaccine studies through the VSD data sharing program.
Recommendation 3.2: The committee recommends that the distinction between the annual automated VSD data (whose quality cannot always be guaranteed) available to independent external researchers through the data sharing program and the study-specific data potentially available to researchers affiliated with the NIP or the participating MCOs be explained more clearly in the data sharing guidelines so that potential users are informed about the limitations of the data that are available through the data sharing program.
Recommendation 3.3: Because of the limitations in the data available to independent external researchers through the VSD data sharing program, the committee recommends that the NIP require the designation of a facilitator for collaboration at each MCO as a condition of the VSD contract.
Recommendation 3.4: To formulate alternative hypotheses or to conduct alternative analyses, researchers need to have access to information or variables that would allow the use of different inclusion and exclusion criteria, different variables for inclusion in models, and, in general, earlier versions of a dataset that would support such restructuring. The committee believes that it is appropriate to allow independent external researchers access to such datasets and recommends that such datasets be made available through the VSD data sharing program.
Recommendation 3.5: The committee recommends that the VSD data sharing guidelines reflect a more specific categorization of the types of studies that can be done with VSD data to conceptualize the full range of studies that independent external researchers may wish to conduct with the data: an audit, a broader reanalysis, a corroboration study, and an investigation of a new hypothesis.
Recommendation 3.6: The committee recommends that there be specific evaluation criteria for VSD proposals and that interested persons have an opportunity to comment on the draft evaluation criteria before they are finalized; the evaluation criteria should be identified clearly in the VSD data sharing guidelines.
Recommendation 3.7: The committee recommends that the technical feasibility of a proposed VSD study be the primary evaluation criterion in the review of proposals submitted to the VSD data sharing program.
Recommendation 3.8: To assist independent external researchers who want to use VSD data through the data sharing program, the committee recommends that the NIP and NCHS add to the VSD data sharing program guidelines a list of recommended competencies for VSD data analysis.
Recommendation 3.9: To facilitate use of the VSD data sharing program, the committee recommends that the NIP work with the VSD-participating MCOs to determine the feasibility of using IRB authorization agreements for VSD research proposals.
Recommendation 3.10: The committee recommends that the NIP work with the MCOs participating in the VSD and America’s Health Insurance Plans (the VSD contractor) to evaluate the feasibility of streamlining the
IRB review process for audits or broader reanalyses in accordance with appropriate regulations.
Recommendation 3.11: Because the confidentiality concerns are integral to the continuation of the VSD, the committee recommends that NCHS in conjunction with the MCOs develop policies and procedures to address confidentiality violations of VSD data and that they be clearly described in the VSD data sharing program guidelines and the agreements that external researchers must sign before using the RDC.
Recommendation 3.12: The committee concludes that it is reasonable to expect researchers who request access to VSD data to have their own funding and it therefore recommends that RDC costs not be waived for independent external researchers.
Recommendation 3.13: The committee recommends that, as a condition of accessing VSD data, all independent external researchers that use the VSD data sharing program be required to submit a report to the NIP (with a copy to NCHS) within a reasonable time (to be determined by the NIP) on the status of their study, the type of study conducted (an audit, a broader reanalysis, a corroboration study, or an investigation of a new hypothesis), the results obtained, and their planned further activities. The reports should be made public by the NIP and should be easily accessible.
Recommendation 3.14: The committee recommends that, as a condition of accessing VSD data, all independent external researchers that use the VSD data sharing program be required to submit to the NIP (with a copy to NCHS) a copy of a manuscript intended for publication at least 30 days before submission to a journal or other print or electronic media. Copies of presentations to be delivered at conferences or meetings that are open to the public or that have media coverage should also be submitted to the NIP and NCHS at least 15 days before presentation.
THE VACCINE SAFETY DATALINK RESEARCH PROCESS AND THE RELEASE OF PRELIMINARY FINDINGS
Recommendation 4.1: To enhance the value of the VSD, to improve the credibility of results derived from it, and to support CDC’s role in assessing vaccine safety, the committee recommends that the NIP develop an annual VSD research plan. The plan should define the priorities for new studies and support of current studies. The annual VSD research plan should be made public. Material deviations from the plan should be identified and be publicly available.
Recommendation 4.2: To support greater use of the VSD and to promote opportunities for collaborative work outside the existing community of VSD researchers, the committee recommends that the annual VSD research plan include provisions for allocating some existing funds, on a competitive basis, to external researchers interested in conducting collaborative work with VSD data.
Recommendation 4.3: The committee recommends that detailed research protocols for each study conducted by an internal VSD researcher be developed, peer-reviewed, and archived. Each protocol should include well-specified definitions of the study population, exposures, and cases; detailed analytic plans; sample size requirements; and study timelines. Data collection forms, procedures, data and analysis files, programming code, and database versions should be documented, cataloged, and archived for a period of at least 7 years after completion of a study.
Recommendation 4.4: To promote collaboration and information-sharing, the committee recommends that the NIP update and improve its list of publications and presentations by establishing a VSD research clearinghouse that provides on a timely basis status reports, study findings, and conclusions for current and completed VSD studies.
Recommendation 4.5: The committee recommends that the NIP and NCHS release publicly the procedures that will be used for record-keeping of VSD data sharing program documents and update the status of the program regularly.
Recommendation 4.6: The committee recommends that in nearly all situations preliminary findings from the VSD be subject to independent external peer review before being communicated to the public or used as the basis of a policy decision. When CDC determines that purely internal peer review is necessary before release, external peer review should be undertaken as soon as possible.
Recommendation 4.7: The committee recommends that preliminary findings from VSD data be shared with the public whenever the findings are presented to anyone other than collaborators in the research, federal employees responsible for research activities, MCO-affiliated VSD researchers, scientific journals, peer reviewers for scientific journals, and people responsible for oversight of the research.
Recommendation 4.8: The committee recommends that preliminary findings from VSD data be shared with the public whenever these findings contribute to the basis of a policy decision or are used to change guidelines on vaccine administration.
Recommendation 4.9: The committee recommends that when final results from VSD analyses or studies are released through publication or through presentation at a meeting, preliminary findings be shared only rarely, but that the dataset from which the final results were obtained be available to other researchers who may verify and extend the results through an audit or broader reanalysis.
Recommendation 4.10: The committee recommends that any preliminary findings based on VSD data that are shared with the public be put into appropriate statistical and scientific context with clear characterization of the uncertainties in the findings, of the strengths and limitations of the data, and of the possibility that new data or new analyses could change interpretations.
INDEPENDENT REVIEW OF VACCINE SAFETY DATALINK ACTIVITIES
Recommendation 5.1: The committee recommends that a subcommittee of the National Vaccine Advisory Committee that includes representatives of a wide variety of stakeholders (such as advocacy groups, vaccine manufacturers, FDA, and CDC) review and provide advice to the NIP on the VSD research plan annually. The subcommittee charged with this role could be the existing Subcommittee on Safety and Communications or a subcommittee created specifically for the purpose.
Recommendation 5.2: The committee recommends that the NIP propose to the National Vaccine Program that additional liaison representatives be appointed to ensure that all perspectives are heard by adequately representing advocacy groups and other members of the public at subcommittee meetings addressing the VSD research plan.
Recommendation 5.3: The committee recommends that an independent review committee with minimal and balanced biases and conflicts of interest be created to: