The Vaccine Safety Datalink Research Process and the Release of Preliminary Findings
REVIEW OF ITERATIVE ANALYSIS APPROACHES USED FOR VACCINE SAFETY DATALINK STUDIES
In its charge, the committee was asked to “review the iterative approaches to conducting analysis that are characteristics of studies using the complex, automated Vaccine Safety Datalink (VSD) system. Examples of recent studies to be examined are a completed screening study on thimerosal and vaccines (Verstraeten et al., 2003a) and cohort studies on asthma.” The studies by DeStefano et al. (2002) and by Verstraeten et al. (2003a) are reviewed below.
Childhood Vaccinations and Risk of Asthma
The 2002 study by DeStefano and colleagues evaluated a suggested association between the risk of asthma and childhood vaccines by studying a retrospective cohort. The researchers found no association between asthma and diptheria-tetanus-pertussis vaccine, oral polio vaccine, or measles-mumps-rubella vaccine. They found weak associations of asthma with Haemophilus influenzae type b vaccine and Hepatitis B vaccine, which the authors attributed partially to a bias in health care utilization or information bias (for example, they could not verify that a child who according to the medical record was unvaccinated was not accessing health care elsewhere) (DeStefano et al., 2002).
Review of Timeline
In January 1997, a proposal on asthma studies was put forward by a researcher at the Centers for Disease Control and Prevention (CDC), and a working group consisting of interested investigators from the VSD, managed care organizations (MCOs), and CDC was established to conduct such studies.1 In fall 1997, a paper by Kemp and colleagues (Kemp et al., 1997) suggested an association between whole-cell pertussis vaccine and asthma. Around that time, several articles were published on the hygiene hypothesis (Mawson, 2001), which posits that some infections in infancy or early childhood may protect against asthma or other allergic conditions. That raised the idea that vaccines, by preventing the infections, may increase the risk of asthma. The National Immunization Program (NIP) decided that this topic should be addressed by using VSD data, so NIP-affiliated VSD researchers conducted a pilot study in 1998 (DeStefano, 2004).
The study team was able to conduct data-quality and data-assurance checks of the electronic data and perform chart review on a sample of about 5% of the VSD population. The study team used that 5% sample to do the analyses. The analyses were presented at a meeting of VSD investigators in June 1998 and at the Interscience Conference on Antimicrobial Agents and Chemotherapy (ICAAC) in September 1998. The researchers determined that even though they had reviewed the patient charts, there was uncertainty about the MCO enrollment status of children; some were enrolled from birth, and others were enrolled when they were 2 or 3 years old. The researchers could not determine with certainty the patients’ vaccine histories before they were enrolled in the MCO. From the pilot study, the researchers learned that the data were not complete enough to be sure that the children who did not have vaccinations in their records were not vaccinated (DeStefano, 2004).
The researchers revised their study on the basis of what they had learned and restricted the data analysis to children born as MCO members to obtain a complete vaccination history. The initial findings from the new analysis were presented at the May 1999 meeting of VSD investigators. There was still some concern about the validity of vaccination status, and to account for it, the researchers conducted a subanalysis of children who had at least some indication of using the MCO for health care. The findings of the analysis were presented at the International Conference on Pharmacoepidemiology in August 1999 and at the September 1999
ICAAC. The findings were the same as the main results in the final paper (DeStefano, 2004).
The publication of the manuscript took over 3 years (January 1999-June 2002). The first draft was completed in January 1999, and it was cleared by CDC and ready for submission to a scientific journal by October 2000 (the paper was not accepted by the first journal and was submitted to a second journal in May 2001) (DeStefano, 2004). The second journal also did not accept the submission, so it was sent to a third journal (Journal of Pediatric Infectious Disease) in August 2001, was accepted in February 2002, and published in June 2002 (DeStefano, 2004; DeStefano et al., 2002).
Safety of Thimerosal-Containing Vaccines: A Two-Phase Study of Computerized MCO Databases
The 2003 study by Verstraeten and colleagues explored possible associations between thimerosal (a preservative that contains ethylmercury) in vaccines and neurodevelopmental disorders. The study was intended to be an initial screen of possible associations; a detailed study would be planned if any associations were identified (DeStefano, 2004). The final results based on one MCO’s data indicated that cumulative exposure to thimerosal at the age of 3 months resulted in a significant association with tics (Verstraeten et al., 2003a). In data from a second MCO, an increased risk of language delay was associated with cumulative exposure at the ages of 3 months and 7 months (DeStefano, 2004). Those findings could not be replicated with data from a third, comparable MCO. None of the analyses showed a significant increase in risk of attention deficit hyperactivity disorder (ADHD) or autism (DeStefano, 2004).
Review of Timeline
The concern about thimerosal-containing vaccines and possible health effects received a lot of attention in summer 1999 when a joint Public Health Service (PHS) and American Academy of Pediatrics statement recommended reduction or elimination of thimerosal in vaccines as a precautionary measure (CDC, 1999b). That was suggested because it had been determined that with some vaccination schedules at that time, a child could have exceeded the Environmental Protection Agency’s guidelines for exposure to methylmercury. In August 1999, a National Vaccine Advisory Committee meeting was convened at the National Institutes of Health (NIH) to review the research then available on the question and to recommend additional research (Egan, 2000). After the meeting, a working group of the PHS, including some external experts, was established to
develop ideas for research projects that could begin to evaluate if there may have been adverse effects from thimerosal-containing vaccines.2 One of the proposed projects was to use the VSD to explore possible associations between thimerosal and neurodevelopmental disorders. The neurologic conditions that potentially could be caused by thimerosal were not clear, so the initial analysis was used as a screening tool (DeStefano, 2004).
In fall 1999, the thimerosal working group deemed the VSD project to have high priority. The protocol for the study was developed in collaboration with the working group and the VSD principal investigators. In late fall and early winter 1999, preliminary analyses were conducted. From late February to April 2000, the analyses and preliminary findings began to be discussed among the VSD investigators. Suggestions were made on how to revise the analysis. At the April 2000 meeting of VSD investigators, the analysis had progressed to a point where there were indications of possible associations with speech or language delay and some possible associations with ADHD. The VSD investigators alerted NIP leadership of their results and sought input on how to proceed (DeStefano, 2004).
In April 2000, the lead study author presented preliminary findings at the annual Epidemic Intelligence Service conference. On April 27, 2000, the researchers briefed the CDC associate director of science on the results. It was suggested that the researchers convene a review panel of CDC scientists outside the NIP. The meeting took place in May 2000; it was determined that the evidence of an association was weak but that the results should be explored further (DeStefano, 2004). At a different May 2000 meeting with the CDC scientific review panel, a recommendation was made to try to replicate the findings in an independent dataset. The researchers conducted the replication with data from Harvard Pilgrim MCO (DeStefano, 2004).
In early June 2000, an external expert review group was convened at Simpsonwood Conference and Retreat Center (Simpsonwood Transcript, 2000). That group also determined that the evidence was weak but should be explored further along several lines of inquiry, including attempted replication with an independent database. In June 2000, the results were presented to the Advisory Committee on Immunization Practices, including findings from the third MCO (the independent database). In that third database, the researchers did not replicate findings of any of the associations seen in data from the two VSD MCOs (DeStefano, 2004). In August 2000, the findings were presented to the World Health Organization Global Vaccine Advisory Committee (GVAC). The GVAC recommended that
a similar study be done in the United Kingdom with the General Practice Research Database. In July 2001, the researchers presented updated preliminary findings to the Institute of Medicine Committee on Immunization Safety Review (Verstraeten, 2001). The updated results were fairly similar to those of earlier analyses; there were still some indications of association with speech or language delay and perhaps with attention problems (DeStefano, 2004).
The principal researcher on the thimerosal study left CDC in July 2001, but in December 2000, before he left, he wrote a first draft of a manuscript on the study and submitted it for CDC clearance (DeStefano, 2004). Because many of the researchers had moved on to other studies, it took over a year for the manuscript to be completed. Additional follow-up data became available, and improved ideas for addressing concerns about healthcare-seeking bias emerged (DeStefano, 2004).
In October 2002, the revised manuscript was submitted for CDC clearance. It was cleared by December 2002.3 The manuscript was also submitted to the first journal for publication in that month. In May 2003, it was accepted by the second journal it was submitted to (Pediatrics), pending revision. The researchers revised it and resubmitted it in June. It was accepted in July, and published in November 2003 (DeStefano, 2004; Verstraeten et al., 2003a).
VACCINE SAFETY DATALINK RESEARCH PLAN
The VSD is the only population-based resource in the nation that has sufficient sample size to address possible concerns about rare adverse effects of vaccines. The VSD is an important national resource. As a resource, however, rather than a study, its value depends primarily on the nature and extent of its use. The investigators who have conducted studies with it have been almost exclusively those who are also responsible for its funding, creation, development, or maintenance. Within that community of researchers, opportunities to propose and lead studies have been created and prioritized, systems for conducting studies have been developed, and funding has been allocated.
For researchers outside the VSD research network, the opportunities and support for use of this resource have been, at best, unclear and narrow. Similarly, there appear to be few opportunities for individuals or parties outside of the NIP or the VSD MCOs to have direct input into research priorities and the allocation of resources. Finally, the public has not been routinely informed about the status and ultimate findings of research efforts undertaken with the VSD.
Those apparent shortcomings arise in part from the changing public expectations about data sharing, policy-making, and public access to information. Because the NIP is responsible for all those roles as they are related to vaccine safety data, the confluence of increasing expectations brings considerable pressure to bear on the NIP. Because the mission of the NIP is perceived to be contradictory (promoting vaccines and assessing their safety), it is all the more incumbent on the VSD to define a process whereby interested parties may give voice to their concerns, the priorities and allocations of resources are determined in open discussion, and accountability is established.
Recommendation 4.1: To enhance the value of the VSD, to improve the credibility of results derived from it, and to support CDC’s role in assessing vaccine safety, the committee recommends that the NIP develop an annual VSD research plan. The plan should define the priorities for new studies and support of current studies. The annual VSD research plan should be made public. Material deviations from the plan should be identified and be publicly available.
Public Input on VSD Research Plan
The annual VSD research plan should be developed with broad input from interested parties. Many individuals and agencies have a stake in VSD activities and findings, some with conflicting needs and agendas. It is appropriate for all of them to understand and have input into the identification and ranking of new research initiatives that will use the VSD.
Scientific organizations have recognized the importance of public input in establishing priorities for research. To establish a process for gathering public input on VSD research priorities, NIP could learn from the activities of those organizations and the models they have used to involve the public in the research priority-setting process. The NIH, for example, has a Council of Public Representatives (COPR), which regularly provides advice to the director of NIH on issues related to public participation in NIH activities, outreach efforts, and other matters of public interest (NIH, 2005a). A recent report by COPR identified principles and recommendations to improve public input and transparency in the NIH research priority-setting process (COPR and NIH, 2004). Several of the principles identified in that report (for example, fostering two-way communication on an individual and community level, ensuring that senior decision-makers receive and fully consider public input, and partnering with local communities, grassroots organizations, and local leaders) could inform future VSD research activities.
Set-Aside of VSD Funding for Collaborative Projects
The principal investigators at the VSD MCOs and key senior investigators at the MCOs and CDC make decisions about which studies to undertake with VSD funds.4 Only researchers affiliated with the NIP or the MCOs have an opportunity to propose VSD studies. Substantial federal funds are used to support the VSD infrastructure and specific VSD studies. The committee finds it reasonable for the public to expect that some portion of the contract funding will be made available to support meritorious research proposals by researchers not affiliated with the NIP or the VSD MCOs.
Recommendation 4.2: To support greater use of the VSD and to promote opportunities for collaborative work outside the existing community of VSD researchers, the committee recommends that the annual VSD research plan include provisions for allocating some existing funds, on a competitive basis, to external researchers interested in conducting collaborative work with VSD data.
Proposals should be reviewed by an independent committee that has expertise in vaccines, immunology, epidemiology, statistics, and research with administrative databases. Review should consider scientific merit, feasibility, innovative use of the VSD, and the potential to interest others in its use.
If one of the identified high-priority research subjects interests external researchers, they should have the opportunity to compete for VSD research funds. The committee believes that all public funds for VSD research should be allocated according to the priorities established through the public process. However, the committee’s recommendations do not preclude submission by independent external researchers of a VSD research proposal on any hypothesis that is technically feasible; if the hypothesis is not considered to have high-priority for VSD research funds, the researchers would have to obtain other funding to conduct the study.
Detailed Documentation of Research Protocols by Internal VSD Researchers
Research proposals are generated before a study starts to describe the proposed research hypothesis, outline the research methods that will be used, identify the staff that will work on the study, and estimate a budget for the study. Detailed research protocols generally are developed after a research proposal is approved but before the study commences and in-
clude specific information about how each step in the study process will be accomplished (such as all the variables that will be included in the analyses, a detailed description of the approach to data analysis, the specific analyses and statistical tests that will be done, and the confounders that will be considered). In contrast with the more general description of research methods included in research proposals, research protocols include detailed specifications of research methods for all phases of the study.
Detailed documentation of research protocols, analysis decisions, and deviations from protocols is important for ensuring the integrity of the scientific process and specifically for ensuring public confidence in the integrity of VSD studies conducted by NIP-affiliated and MCO-affiliated VSD researchers. Thorough documentation and archiving of all study methods and analysis decisions can be considered the equivalent of keeping good laboratory notebooks. If an audit or a reanalysis of a study by an internal VSD researcher ever is conducted, thorough documentation of methods will insulate the original investigator from unwarranted criticism of the research methods used and aid the later researcher in conducting an audit or reanalysis. Having detailed research protocols for all VSD studies conducted by NIP-affiliated and MCO-affiliated researchers will support transparency of the VSD research program. Increasing the rigor with which such studies are documented and conducted will enhance public trust in findings from the VSD.
Recommendation 4.3: The committee recommends that detailed research protocols for each study conducted by an internal VSD researcher be developed, peer-reviewed, and archived. Each protocol should include well-specified definitions of the study population, exposures, and cases; detailed analytic plans; sample size requirements; and study timelines. Data collection forms, procedures, data and analysis files, programming code, and database versions should be documented, cataloged, and archived for a period of at least 7 years after completion of a study.
SHARING VACCINE SAFETY DATALINK PROGRAM INFORMATION
At its meetings, the committee heard requests for more information about VSD studies and the VSD data sharing program (Bernard, 2004; Fisher, 2004a). Transparency through the provision of more information can help to ease concerns about the implementation of the VSD program. The VSD is supported by public funds, and the committee finds it reasonable to expect that as much study-specific and programmatic information
as possible be shared with the public without jeopardizing confidentiality or researchers’ ability to publish their work.
Sharing Information About Current and Completed Studies
Sharing information about current and completed studies is an important part of promoting transparency. The NIP has shared information about published VSD studies, but the committee concludes that it is just as important to share information about current and completed studies that are not yet published. The list of VSD publications and presentations on the NIP Web site has not been updated since December 2000 (CDC, 2000). The committee encourages the NIP to update the list often and on a fixed schedule so that the public can be assured that all studies that have been done with VSD data are disclosed quickly. Only when information about how VSD resources are being allocated is openly shared can there be public accountability for the VSD research studies that are being pursued.
A VSD research clearinghouse like that established for the California Health Interview Survey (CHIS) (CHIS, 2003) could help to promote collaboration and information-sharing among new and experienced VSD researchers. The CHIS Research Clearinghouse provides information about studies that have been completed or are in progress (CHIS, 2003), and a mailing list is used to distribute the latest news and information about CHIS to interested persons—an example of another activity that the NIP could emulate to promote transparency of VSD activities. Providing information about studies that are in progress is an important way to promote credibility, trust, and transparency between the NIP and members of the public who are concerned about vaccine safety. Such a research clearinghouse would also constitute a mechanism for promoting collaboration in that external researchers who are interested in conducting studies with VSD data could more easily identify experienced VSD researchers with whom they might collaborate.
Recommendation 4.4: To promote collaboration and information-sharing, the committee recommends that the NIP update and improve its list of publications and presentations by establishing a VSD research clearinghouse that provides on a timely basis status reports, study findings, and conclusions for current and completed VSD studies.
Sharing Information About Utilization of the VSD Data Sharing Program
Transparency is an important part of ensuring public trust and confidence in the VSD. Considering that the VSD data sharing program demands could increase in the future, releasing information to the public on the use of the program (such as number and types of proposals submitted, disposition of each proposal, and timelines of notifications to researchers) could promote transparency and help to foster public trust. If the public is confident that there is a transparent, standardized process for documenting the status of proposals and that information about use of the VSD data sharing program is made known on a regular schedule, there may be less concern and suspicion about the processes that the NIP and NCHS use to implement the data sharing program.
Recommendation 4.5: The committee recommends that the NIP and NCHS release publicly the procedures that will be used for recordkeeping of VSD data sharing program documents and update the status of the program regularly.
Such information could be made available electronically (for example, on the NIP or NCHS Web site) to balance the public’s need for information with the administrative burden on NIP and NCHS staff. The NIP Web site may be an effective place to provide such information, especially considering that the site recently has been recognized by the World Health Organization as a vaccine safety Web site that meets essential information practices criteria (WHO, 2005).
THE ROLE OF PEER REVIEW
Peer review of completed manuscripts is an important component of the scientific process. Regardless of the type of researcher, the data being used, or the sponsor of the research, all research findings should go through peer review before being considered scientifically valid. Whereas external peer-review processes are normally used before findings are released in a peer-reviewed journal, solely internal peer-review processes may be used when questions arise about the advisability of releasing preliminary findings. Independent external peer-review processes offer the greatest confidence in the accuracy and validity of study findings. When external peer review is not possible (for example, in the case of release of preliminary findings with an urgent public health impact or through a mechanism other than a peer-reviewed journal), all findings that will be released should at least go through an extensive internal peer-review process.
The Peer-Review Process for Intramural Research at CDC
Research conducted by federal scientists goes through a slightly different peer-review process from that of research conducted by academic scientists. Because intramural research conducted by federal employees is conducted as part of the employees’ regular duties, the employer (the federal government) requires that there be additional layers of review.
The NIP has described two processes that are used for review of intramural research conducted by NIP employees. The choice depends on whether the public health concern is considered routine or nonroutine. For routine public health concerns, the process that the NIP uses consists of (Bernier, 2004b):
Presentations at scientific meetings (considered work in progress)
Internal peer review and clearance
External peer review prior to publication (confidential)
Publication in journal
Sharing of final dataset for published study
For nonroutine public health concerns, the process that NIP uses consists of (Bernier, 2004b):
Consultations with internal scientists
Consultations with external scientists, including those from the vaccine industry
Presentations to standing advisory committees for policy recommendations
Communication with the public
Release of presented findings in summary form
Presentations at scientific meetings
Internal peer review and clearance
External peer review (confidential)
Publication in journal
Sharing of final dataset in published study
The delineation of the two processes leads to several questions: At what point in the research process is a determination made about whether the research topic is routine or nonroutine, and who makes the decision? What are the criteria for deciding whether a topic being investigated is routine or nonroutine? Because it may be difficult to distinguish between a routine and a nonroutine public health concern, should there be a single, standardized peer-review process? Could the validity of findings from work treated as a routine public health concern be questioned if it eventu-
ally has an important public health impact, since the findings did not have the intense internal peer review of the abbreviated process? Such questions could affect the perceived reliability and credibility of findings based on VSD data that are released by NIP-affiliated VSD researchers.
RELEASE OF PRELIMINARY FINDINGS
Standards of Practice for Preliminary Findings
To discuss whether, when, and how to release and share preliminary findings with others, it is necessary first to have a common understanding of what is meant by the term preliminary findings. For purposes of this report, preliminary findings refers to results or summaries and associated conclusions that are based on incomplete data or incomplete analyses. Because data are incorporated into the VSD database annually, concerns about preliminary findings affecting continual data accrual are not applicable here.
The question of the release of preliminary findings based on VSD data has an additional layer of complexity related to the multistage process that is used to test a hypothesis. In the VSD setting, preliminary findings may refer to results of the first-level analyses (based on automated data) or to results of incomplete analyses or incomplete data in second-level (case-control) studies.
Preliminary Findings Because of Incomplete Data
Data may be incomplete because the expected data have not all been collected, validated, or otherwise processed for analysis. In some literature, particularly that related to clinical trials, interim findings or preliminary findings refers to analyses of incomplete data that are expected to be repeated as data accumulate. In such circumstances, the statistical issues of multiple analyses of interim data have been examined, and methods for controlling false-positive error rates have been developed (DeMets, 2004). Repeated analyses of findings do not appear to be germane to the part of the committee’s charge on releasing preliminary findings, because these studies are not typically designed to incorporate additional data over time as events accumulate.
Preliminary Findings Because of Incomplete Analyses
Even with complete data, analyses may be incomplete because some protocol-specified analyses have not been conducted. Analyses of observational data usually progress along the following lines. Early in the pro-
cess, univariate distributions and simple correlations or other estimates of bivariate associations between some exposure and the outcomes of interest are calculated. If there is evidence of a relationship, increasingly sophisticated analytic approaches are used in an attempt to determine whether the associations could be attributed to other factors. Because of the inherent limitations of observational studies, the later analyses are designed to eliminate alternative explanations for the association, and to increase confidence in the strength of any remaining relationship. Only when an association persists despite every attempt to explain it away can a conclusion be drawn that a cause-effect relationship exists. The simple correlations or odds ratios first calculated would be considered preliminary findings. Normally, such first-level findings are provided in publications only in the context of the complete analyses on which conclusions are based.
Preliminary Data Compared with Preliminary Findings
The difference between preliminary data and preliminary findings must also be understood. At the committee’s October 2004 meeting, representatives of the NIP described how the NIP conceptualizes preliminary data compared with preliminary findings. In the context of the VSD, the NIP defines data as “the underlying elements of information that lead to findings but are not the findings per se. They permit analyses or reanalyses” (Bernier, 2004b). It describes preliminary data as “the underlying elements of information of a study or investigation which are still incomplete or subject to change” (Bernier, 2004b). In contrast, the NIP defines preliminary findings as “initial results obtained from investigations or studies often expressed in summary statements or summary-like form such as tables or graphs. These results are incomplete and subject to change prior to peer-reviewed publication” (Bernier, 2004b). As those terms are related to the VSD, the committee uses the NIP’s distinction between preliminary data and preliminary findings. However, the committee argues that the term preliminary should be used slightly differently.
Should There Ever Be Preliminary Findings?
When researchers examine study data for accuracy, perform initial analyses of them, and analyze the effect of confounding factors, their findings should be described as preliminary. When findings are considered valid enough to share with external groups (even if by invitation and limited) or to be the basis of a policy decision, the findings should not be characterized as preliminary, even if other findings or reports will be released later. Only expectations of additional data or additional revela-
tions from further analyses should cause findings to be designated as preliminary. The committee believes that preliminary should be used only when there is an appreciable likelihood of a material change.
Although the committee argues for limited use of preliminary findings in the future, for the sake of clarity and adherence to the committee’s charge it uses the NIP’s conceptualization of preliminary findings in discussing findings, conclusions, and recommendations in this report.
The Role of Peer Review in the Release of Preliminary Findings
Conducting scientific research can be a long and slow process. From the time that a research study is proposed, it can take many months or years to conduct the research, analyze the results, draft a manuscript, have the manuscript reviewed by peers, and finally have the research results published. Data analysis typically begins with simple data summaries and simple analyses that progress to more sophisticated analyses and refinements. Associations and relative risks often appear, diminish, or vanish when additional causal factors are considered in the increasingly detailed analyses. Repeated looks at the data are often required to make sure that results are robust and reliable.
Advice and feedback from colleagues during study analyses are normal parts of the scientific process. When findings are deemed important enough to be the basis of a policy decision or to be communicated to the public, extensive and independent peer review is necessary.
Solely internal peer-review processes may be needed when preliminary findings could have a substantial impact on public health. The need to release preliminary findings rapidly may force a decision to limit peer review to peers inside the federal government; if so, the internal peer review should be as extensive as possible. The committee recognizes that an extensive, independent, external peer review may be even more necessary in such a situation, because potentially influential preliminary findings may be based on a small number of cases or on incomplete analyses. In such situations, purely internal review should be followed by external review on an expedited schedule.
In the case of the VSD, however, the committee finds that because the data are incorporated into the VSD data files annually rather than continually, there will rarely be situations in which preliminary findings are so urgent that they cannot undergo independent external peer review. Preliminary findings from true surveillance systems may yield quickly emerging findings, but this will rarely be the case with findings from the VSD. If the NIP determines that preliminary findings about potential vaccine-related risks should be communicated to the public, the peer-review process may be abbreviated, but it should not be less rigorous.
Recommendation 4.6: The committee recommends that in nearly all situations preliminary findings from the VSD be subject to independent external peer review before being communicated to the public or used as the basis of a policy decision. When CDC determines that purely internal peer review is necessary before release, external peer review should be undertaken as soon as possible.
Peer review of findings that are the basis of a vaccine safety-related policy decision should always be linked to the policy role of the NIP. The committee believes that there should be external peer review of any preliminary findings that the NIP uses as the basis of or to support a policy decision, although the external review need not be done in the traditional way. Novel approaches, such as standing committees of external reviewers or convening of reviewers by conference call, could be explored as mechanisms for obtaining independent external review.
Preliminary findings that could have an impact on public health may need to be released quickly, but the committee believes that the possibility of releasing findings through an expedited process at a peer-reviewed journal should not be dismissed outright. Many journals are able to obtain rapid peer review and release articles in electronic format on an expedited basis. The committee believes that releasing preliminary findings through an expedited process at a peer-reviewed journal should always be seen as preferable to releasing preliminary findings through a solely internal peer-review process.
Concerns About Releasing Preliminary Findings
Using the NIP’s conceptualization of preliminary findings, the committee finds that various concerns arise in the release of preliminary findings. Most important is the need for careful balance of the needs and rights of the public to be kept informed, the added costs of satisfying those needs and rights, and the risk of unnecessary alarm or unwarranted complacency if interpretations change. The committee’s deliberations about the release of preliminary findings have been guided by the need for balance of costs, risk, and benefits.
Need for Scientific Exchange
The scientific process is characterized by exchange of information, response, and revision among colleagues. That process helps to ensure that scientific results are valid and account for all possible confounding factors. During the process, preliminary findings may be shared among colleagues.
Some of the process of scientific exchange occurs outside the public context. Preliminary findings should rarely be exchanged publicly, because early interpretations are often wrong and there is scientific value in having peers review methods and analyses for rigor and accuracy. Routinely releasing preliminary findings can have an adverse effect on the broader peer-review process by seeming to condone the release of preliminary findings before they have been reviewed by peers.
Distinction Between Signals and Noise
Evidence of relationships or of a lack of relationships—signals—in scientific data is always subject to the possibility of bias and randomness—noise. The smaller the signal in relation to the noise, the greater the difficulty of interpretation and the greater the likelihood of erroneous conclusions. In scientific research, especially when data are examined iteratively, it can be difficult to determine when the results of data analysis indicate a true signal of a risk as opposed to noise. The likelihood that a conclusion is inaccurate is greater—sometimes much greater—when it is based on preliminary or interim data than when it is based on a full dataset. In summary, preliminary findings, as the committee defines them, are meant to push the envelope and are expected to be found wrong in many situations; otherwise, there would be no reason to try to improve the analysis and interpretation of the data.
Consequences When Published Findings Differ from Preliminary Findings
When final interpretations of a scientific study differ substantially from preliminary findings that are already in the public domain, there is an obvious need to explain and justify the changes. That can be a healthy part of the scientific process, but it can undermine the credibility of a study among persons who do not have a full understanding of how the scientific process works. When a policy decision is based on preliminary findings and later published findings indicate a risk that is different from the risk that influenced a policy decision, the changes may create a particularly serious problem. Such a situation can undermine not only the particular study but also the policy as a whole if the public begins to question the credibility of all the data that served as the basis of the policy. Ultimately, routine and widespread dissemination of preliminary results can sometimes undermine public trust if preliminary conclusions fail verification and are changed (Weijer, 2004).
Public Health Impact of Releasing Preliminary Findings
The decision to release preliminary findings should mean that someone deemed the findings to be important for the health of the public and that urgent notice to the public is needed. Accordingly, it can be expected—and indeed intended—that knowledge of the findings will change policy or the behavior of some segments of the public (such as consumers, health care practitioners, and researchers studying similar issues). The acknowledgment that findings can cause behavior change in some way emphasizes the importance of ensuring that preliminary findings represent a true signal and not just data noise. It can be difficult to change established health behaviors (McCall, 2003), and this underscores the importance of minimizing the likelihood that final results will support a different conclusion from preliminary results. In a larger scientific context (beyond the VSD), releasing preliminary findings can affect future data collection. That can confound final results and undermine the legitimacy of a study’s conclusions.
Whenever a decision to release preliminary findings is contemplated, the risks, costs, and benefits related to early disclosure of findings compared with delayed disclosure of findings must be examined and weighed (Ball et al., 1998; Dittmann, 2001; Slovic, 1987). When releasing preliminary findings, it is important to communicate the risk properly and in a way that protects the credibility of the source so that the public can trust the findings and base appropriate health decisions on them (McComas and Trumbo, 2001). When contemplating the release of preliminary findings about vaccines, it is important to consider the effect of interrupting immunization programs, the probable size of true and perceived risks and the likelihood of unknown risks, and the risk of vaccine-preventable diseases compared with the risk reduction that could be achieved through immunization (Dittmann, 2001). Such an approach would have added benefits the next time results are released.
When to Release Preliminary Findings
Despite those concerns, in various situations it is appropriate to release preliminary findings about potential vaccine-related risks based on VSD data. In the case of the thimerosal screening analysis (Verstraeten et al., 2003a), some people retrospectively questioned the decision process used by the NIP to determine whether, when, and how to share preliminary findings with others (Bernard, 2004). The NIP did not use a formal decision mechanism to guide the release of VSD preliminary findings to others.
Retrospective assessments of the appropriateness of such decisions
can offer lessons for similar situations in the future, but they do not directly provide guidance for making such decisions. That underscores the need to define a priori the conditions governing whether, when, and how to share preliminary VSD findings about potential vaccine-related risks with other scientists, the public, and policy-makers.
Release of Preliminary Findings When Shared with Others
The NIP should expect, of course, that a public release to one group will soon, and appropriately, spread to others, and the risks and costs related to selective release are powerful arguments against selective release. However, a public release should not be confused with consultations with particularly knowledgeable scientists providing peer review or the equivalent.
One of the conditions that should determine when preliminary findings are shared with the public is the process used to share the findings with other scientists. It is reasonable to expect that preliminary findings will be shared among researchers affiliated with the NIP and the MCOs and among federal employees without having to be shared with the public; this is part of the normal peer-review process for ensuring the rigor and validity of research. However, with respect to sharing preliminary findings in broader venues (for example, in presentations at scientific meetings and in meetings that involve people who do not have a role in scientific peer review), the committee believes that no members of the public have a greater right to knowledge than others. That all members of the general public should have equal access to information from the federal government is a vital component of the Federal Advisory Committee Act, the Government in the Sunshine Act, and the Freedom of Information Act.
Recommendation 4.7: The committee recommends that preliminary findings from VSD data be shared with the public whenever the findings are presented to anyone other than collaborators in the research, federal employees responsible for research activities, MCO-affiliated VSD researchers, scientific journals, peer reviewers for scientific journals, and people responsible for oversight of the research.
Release of Preliminary Findings When Used to Make Policy Decisions
Policy decisions or recommendations from the federal government can have a large and direct impact on the lives of Americans. The public’s trust in decisions or recommendations that are based on scientific infor-
mation can be affected by their trust in the scientific findings that were the basis for those decisions. The committee believes that if preliminary findings have a direct impact on a policy decision or vaccine administration guidance, the data and the findings that were the basis of that decision or guidance should be made available to the public.
Recommendation 4.8: The committee recommends that preliminary findings from VSD data be shared with the public whenever these findings contribute to the basis of a policy decision or are used to change guidelines on vaccine administration.
Although “preliminary analyses” are not considered to be research data under the provisions of the Shelby Amendment, research findings may be subject to the law if they support “an agency action that has the force and effect of law” (Pub. L. No. 105-277 ). The committee encourages the NIP to determine the applicability of the Shelby Amendment in situations where preliminary findings contribute to the basis of a policy decision or a change in guidelines.
Release of Preliminary Findings Superseded by Later Findings
Different considerations affect the release of preliminary findings that have already been superseded by later findings and the release of preliminary findings that represent the most recent analysis of data. In the normal scientific process, preliminary results often differ from the final results obtained through the rigorous, comprehensive analysis of a full dataset. The peer-review process is meant to ensure that final results represent the most rigorous analysis and account for confounding factors and data anomalies that were present in the preliminary results. When the peer-review process works appropriately, final results should always be considered as representing the most accurate and valid analysis of the data. However, there may be reasons to review earlier stages in the research process; for example, to determine whether and how something may have gone wrong, to allocate proper credit for scientific advances, or to simply understand scientific processes.
Because the peer-review process is designed to ensure the validity and scientific quality of the final results, preliminary findings should not be relied on as the most valid interpretations of the data. For VSD studies, the committee believes that preliminary findings superseded by later findings should not normally be released.
Recommendation 4.9: The committee recommends that when final results from VSD analyses or studies are released through publication or through presentation at a meeting, preliminary findings be
shared only rarely, but that the dataset from which the final results were obtained be available to other researchers who may verify and extend the results through an audit or broader reanalysis.
Putting Preliminary Findings into Appropriate Context
Any preliminary findings that are released under the conditions specified above need to be communicated in an appropriate context. The type and degree of risk suggested by the preliminary findings will influence the public’s reactions. To help the public to determine what, if any, actions to take, an indication of the public health importance of the findings should accompany their release. The public health importance of the risk and the behavior change resulting from it will be modulated by the statistical and scientific context used to communicate the risk. To portray accurately the risk conveyed by any preliminary findings from VSD data, communication about preliminary findings needs to include an assessment of the quality and integrity of the data used as the basis of the findings and possibly should include sensitivity analyses.
Recommendation 4.10: The committee recommends that any preliminary findings based on VSD data that are shared with the public be put into appropriate statistical and scientific context with clear characterization of the uncertainties in the findings, of the strengths and limitations of the data, and of the possibility that new data or new analyses could change interpretations.