Vaccine Safety Research, Data Access, and Public Trust (2005)

One of the key goals of the Vaccine Safety Datalink (VSD) data sharing program should be maintenance of public trust in the use of the VSD to draw scientific conclusions about vaccine safety. Because of the contentious nature of some of the issues surrounding the VSD and the strained relationship between the Centers for Disease Control and Prevention (CDC) and some people who have been critical of CDC’s vaccine safety activities, the committee recognizes that there may be public concerns about the role of CDC in reviewing proposals to use VSD data and in setting the VSD research agenda. A perception of bias in the VSD proposal-review process and in the priorities established for the VSD research plan could jeopardize public confidence in VSD activities.

There are legitimate reasons for public concern about the independence and fairness of the review of VSD data sharing proposals and of determinations about when and how to release preliminary findings of VSD analyses. The lack of transparency of some of those processes affects the trust relationship between the National Immunization Program (NIP) and some members of the general public. To address some of those concerns, the committee believes that two independent groups should advise on different aspects of the VSD program:

1. A subcommittee of the National Vaccine Advisory Committee (NVAC) that includes representatives of a wide variety of stakeholders—such as advocacy groups, vaccine manufacturers, the Food and Drug Administration (FDA), and CDC—to review and provide advice on the VSD research plan annually.

2. An independent review committee with minimal and balanced biases and conflicts of interest to:

• Review independent external researchers’ proposals to use VSD data through the data sharing program;

• Review research proposals from internal researchers and provide oversight of changes in or deviations from research protocols for internal VSD studies; and

• Provide advice on when and how preliminary findings from VSD data should be shared with the public.

The key characteristic of each of the committees is scientific independence. Independent review of the VSD research plan and of various aspects of specific VSD studies is integral to public trust in the use of the VSD to answer questions about vaccine safety.

Every year, each VSD managed care organization (MCO) is provided an annual budget allocation. Each MCO conducts or participates in VSD studies given their available resources determined from their yearly budget. For high-priority VSD studies that require additional resources, the NIP sometimes will supplement the budget.¹ Decisions about which VSD studies should be pursued with the available resources are reached by consensus among the VSD investigators at the MCOs and the NIP.²

It is somewhat unclear how the priorities for the VSD research plan are set and how much input is sought from stakeholders outside the VSD steering committee. Presentations during the open sessions of the committee’s meetings showed that the public also does not understand how research priorities are set.

The limitations of the VSD data sharing program and the limited ability of independent external researchers to conduct high-quality corroboration studies or studies of new hypotheses create a special need to involve the public in the priority-setting process for the VSD research plan. Only NIP-affiliated or MCO-affiliated researchers have access to VSD data for events before and after January 1, 2001, for corroboration studies and studies of new hypotheses, so independent external researchers may not be able to conduct studies that members of the public consider to have high priority. Novel hypotheses or approaches for studying previously

investigated hypotheses essentially cannot be pursued independently by researchers who do not wish to collaborate with other researchers or cannot find willing collaborators. In view of the limited ability of independent researchers to conduct high-quality VSD studies of new hypotheses and the limited ability of the public to provide input on which VSD studies should be pursued with federal tax dollars, there needs to be greater opportunity for input into the setting of priorities in the VSD research plan and greater transparency of the priority-setting process.

To give the full array of stakeholders an opportunity to provide input into the VSD research priority-setting process and to ensure that the process is as transparent as possible, an independent group should be used to review and provide advice on the plan. It is important that such a group represent all relevant stakeholders so that the priorities of each can be heard. Such a group should represent a broad cross-section of stakeholders and be charged with thinking strategically about VSD research priorities. The group should meet publicly and allow interested persons to observe the process and provide input through established mechanisms.

It is expected that such a group would have conflicts of interest, inasmuch as the goal would be to hear from knowledgeable persons, most of whom will be stakeholders. But the group’s deliberations are also expected to occur through an open, public process. Conflicts of interest should not be avoided in a research priority-setting process meant to include all stakeholders, but the process should be transparent so that anyone can observe the deliberations that are influencing it.

The committee believes that a subcommittee of the NVAC would be the most appropriate group to review and provide advice on the VSD research plan because: (1) one of its functions is to recommend research priorities and other measures to enhance the safety and efficacy of vaccines (HHS, 2003b); (2) it includes federal officials as nonvoting ex officio members, and this allows input into the VSD research plan from other federal partners; and (3) it reports to the director of the National Vaccine Program (NVP) and is managed and supported by the NVP, which is organizationally in the Office of the Secretary of the Department of Health and Human Services. The NVAC is governed by the provisions of the Federal Advisory Committee Act (FACA). Because the NVAC is a FACA committee, the membership of the NVAC is subject to the NVAC Charter, and provisions are in place to ensure that there is public notice of meetings and that meetings are open to the public. By utilizing a subcommittee of the NVAC to review the VSD research plan annually, regular voting members of the NVAC, nonvoting ex officio members (such as CDC, FDA, and other federal agencies), and nonvoting liaison representatives (such as a representative of America’s Health Insurance Plans) (HHS, 2003b) can provide input on priorities for VSD research.

The committee considered the appropriateness of having a subcommittee of the Advisory Committee on Immunization Practices (ACIP) serve this role, but it found that use of an ACIP subcommittee would not achieve the desired level of independence, because the NIP has programmatic responsibility for managing and supporting the ACIP. Use of a subcommittee of the NVAC would achieve an additional level of independence.

The committee heard about public concerns regarding the review of VSD research proposals and the procedures that independent external researchers must follow to use VSD data (Geier and Geier, 2004). Many of the specific concerns and the committee’s related recommendations have been described earlier in this report. The concerns have direct implications for the release of preliminary findings.

Independence, transparency, and fairness must characterize VSD research activities if the public is to trust findings and conclusions based on VSD data. If the public questions whether the rules for access to VSD data are applied equitably to all researchers who request use of VSD data, confidence in the legitimacy of the VSD data sharing program and, ultimately, the findings from any VSD studies could be jeopardized.

Because only one group of researchers has accessed VSD data through the data sharing program, the committee could not determine whether the VSD data sharing guidelines have been applied equitably to independent external researchers. However, it has been asked to provide recommendations on any needed modifications of the data sharing program

that would ensure its appropriate utilization in the future. One way to ensure fairness in utilization of the program is the creation of a committee independent of the NIP and the National Center for Health Statistics (NCHS) to review research proposals submitted by independent external researchers who request use of VSD data. Only when there is confidence that all research proposals are reviewed fairly will there be confidence in the legitimacy of all findings based on VSD data. Both the NIP and the public need a system to ensure the fair review of research proposals.

The adequacy and appropriateness of research proposals for VSD studies carried out by independent external researchers receive much scrutiny by both NCHS staff responsible for the data sharing program and the MCO Institutional Review Boards. However, it is unclear how research proposals from NIP-affiliated and MCO-affiliated researchers are evaluated. The extent to which there is oversight of NIP-affiliated and MCO-affiliated researchers’ adherence to their research protocols (outlining the specification of the study population, the detailed analytic plan, sample size requirements, and study timelines) is also unclear to the public (Bernard, 2004). Lack of public confidence in adherence to detailed research protocols by researchers from the NIP or the MCOs affiliated with the VSD could jeopardize public confidence in the entire program. Especially in light of the substantial federal resources spent on the VSD program, the committee believes that the public would be served by greater transparency of VSD research activities and by assurances that all VSD research proposals are evaluated independently and that there is oversight of changes in or deviations from research protocols for internal VSD studies.

Whether, when, and how to release preliminary findings is an aspect of the VSD that needs to be characterized by independence and transparency. As the committee has outlined earlier, various conditions should govern decisions about the release of preliminary findings. The committee recognizes that not all decisions about the release of preliminary findings may fall neatly into one of these categories. The advice of an independent committee can be critical in that regard. An independent committee can evaluate the public’s right to know compared with the risk of alarming the public about a risk that might not exist. The public should feel confident that the risks and benefits related to releasing such findings will always be evaluated and that an independent committee will offer advice on the most appropriate course of action.

For all those reasons, the committee believes that establishment of an independent review committee (advisory to the director of CDC) will enhance trust in the transparency and fairness of the VSD research process. It is important that an NVAC subcommittee that represents all stakeholders think strategically and provide advice on priorities for the VSD

research plan, but it is also important that an operational, independent committee that is free of serious conflicts of interest review VSD research proposals and provide advice on the release of preliminary findings.

• Review independent external researchers’ proposals to use VSD data through the data sharing program;

• Review research proposals from internal researchers and provide oversight of changes in or deviations from research protocols for internal VSD studies; and

• Provide advice on when and how preliminary findings based on VSD data should be made public.

It is possible that an established committee could appropriately serve the role described here, but the committee could not identify any established committee that would be suitable. Thus, the committee concluded that an independent review committee charged with carrying out those functions should be created de novo. Whether a new committee is created or an established committee is found or reconfigured, the following criteria should guide its creation and operation:

• The committee’s organization, operation, and deliberations are characterized by independence;

• Members’ biases and conflicts of interest are minimal and balanced; and

• Members are chosen on the basis of scientific and technical expertise.

As long as an independent review committee meets those criteria, the committee believes that the NIP and NCHS should have flexibility in determining the structure and operating procedures of the committee. The NIP should consider having trained members of the public serve on the independent review committee if they meet the criteria of scientific and technical expertise. Matters that could be considered in the formation of such a committee include:

• Membership of the committee (number and types of members);

• Frequency of meetings;

• Staff support;

• Criteria that the committee will use to evaluate VSD research proposals;

• Process for allowing VSD program staff to provide comments on proposals;

• Which decisions or deliberations should be made public and when;

• Whether researchers or the public or both should be allowed to make presentations to the committee; and

• Types of reports or feedback provided by the committee.

The committee recognizes that the workload may be very small, at least at the beginning, and that a less weighty approach may be needed. Regardless, the three criteria listed above should still be applied.

When there is proper detailed documentation of research protocols, any deviations from the protocols should be clear, explicit, and adequately justified; otherwise, problems may arise. Some of the public criticisms (Bernard, 2004) of the VSD thimerosal screening study (Verstraeten et al., 2003a) were related to its alleged deviations from the original research protocol.

Good science and public accountability are enhanced when researchers adhere to original, peer-reviewed research protocols and thoroughly document and justify substantive deviations from original protocols. Transparency and public trust in the VSD would be served best by allowing an independent review committee to oversee VSD researchers’ adherence to research protocols and provide advice on the best course of action if protocol deviations are not sufficiently documented and justified.

The committee encourages adherence to research protocols and documentation and justification of deviations from protocols, but it also recognizes the great benefits that may come from unstructured, unplanned research. There can be great value in informal examinations of data for unexpected signals. Such unplanned, unstructured research should not be inhibited but should be viewed as exploratory.

The committee recognizes that some people may dispute decisions made by the independent review committee, and an appeals process may need to be established. The entity considering the appeal should be separate from the independent review committee and should have the authority to overrule a decision of the independent review committee. The committee encourages the NIP and NCHS to establish an appeals process and

specify who will be responsible for appeals decisions; the deputy director for science and public health at CDC may be an appropriate entity for this role.

Although an appeals process is needed, it is hoped that researchers and the public will trust the decisions made by the independent review committee. That will limit the number of appeals.