workshop on December 7, 1993, in Washington, D.C. The intent of theworkshop was not to address questions about specific vaccines oradverse events but to consider generally how questions about suchtopics should be addressed in future research. The workshop participantsincluded academic researchers, members of vaccine safety advocacygroups, vaccine manufacturers, health care providers, representativesof federal research and regulatory agencies, and IOM staff. A listof participants is included at the end of this summary.
This document represents a summary of the workshop discussion. Althoughthe workshop was held under the auspices of the Committee to StudyNew Research on Vaccines, this summary contains no committee recommendations,because committee consensus was not sought. The views stated in thissummary are those of the individual workshop participants.
The discussion at the workshop was organized into three major areas:case reports and passive surveillance systems, large-linked databases(LLDBs), and clinical trials. For each topic, one or more individualsoffered introductory comments, which were followed by a group discussionof the topic.
CASE REPORTS AND PASSIVE SURVEILLANCE
As discussed extensively in the IOM reports on adverse events aftervaccines and elsewhere (Institute of Medicine, 1991, 1994a,b; Kramerand Lane, 1992), three different questions can be asked about causality:
Did a particular vaccine cause a particular adverse event in a givenindividual? This question is asked, for example, in the assessmentof individual cases for the purpose of compensation.
Can a particular vaccine cause a particular adverse event? For the populationas a whole, this question is best answered by well-designed epidemiologicstudies and clinical trials. In the absence of such data, however,it can be concluded that if a vaccine did cause a particular adverse event, even if only in one documentedcase, then it can cause that event. Thus, case reports can provide pivotal informationin a causality assessment.
Will (or how often will) a vaccine cause an adverse event? This question is answerable onlyby epidemiologic studies or clinical trials.
Well-established general principles for evaluating case reports ofadverse drug reactions (Kramer, 1981) are generally applicable tothe study of vaccines. Elements to be considered in evaluating causationin case reports include the following: (1) previous clinical experiencewith the vaccine, (2) alternative etiologic candidates, (3) priorhistory of the recipient, (4) timing of receipt of the
vaccine with respect to the onset of the event, (5) characteristicsof the adverse event, both clinical and pathologic, (6) dechallenge(removing or ceasing the exposure; this element is less relevantto evaluation of vaccines because vaccine exposure occurs at a singlepoint exposure, rather than regularly for a period of time as a drugmight be), and (7) rechallenge (again, this occurs less commonlywith vaccines, but this element can sometimes be helpful in the evaluationif a vaccine is readministered). The features of case reports thatare most useful in helping investigators reach conclusions aboutthe relation between a vaccine and an adverse event include the absenceof a good alternative etiologic candidate (this feature is necessarybut not sufficient for inferring causality), timing of receipt ofthe vaccine with respect to the onset of the adverse event (particularlyfor well-defined events such as anaphylaxis), characteristics ofthe adverse event (for specific clinical and/or pathologic features[e.g., anaphylaxis or vaccine strain-disseminated viral infection])and rechallenge.
Passive surveillance systems refer to systems that collect and analyzeindividual case reports of adverse reactions that are sent to thesystem voluntarily by physicians and patients. Pharmaceutical companiesas well as government agencies maintain passive surveillance systems.
The Vaccine Adverse Event Reporting System (VAERS) of the PHS isa fairly new passive surveillance system. Under the aegis of theCenters for Disease Control and Prevention (CDC) and the U.S. Foodand Drug Administration (FDA), VAERS was established by Public Law99-660. Anyone who knows of a possible adverse reaction—the patient,a parent, or even a non relative who witnessed or merely heard ofa reaction—can submit a report to VAERS. Letters are sent to about240,000 health care providers every 6 to 9 months explaining whatthey are required to report and emphasizing that a temporal relationto vaccination is the criterion for reporting an adverse event. Thereare plans to increase educational efforts aimed at physicians. ThePHS closely examines the reports to determine whether a particularpattern is emerging; weekly reports are reviewed to discover whatnew serious events or fatalities have been reported. Events relatedto newly-marketed vaccines are especially carefully scrutinized.
Passive surveillance systems can identify a problem, serving as analerting system. Once there is an alert, other surveillance methodsthat are more sensitive and specific for determining causation canbe used. For example, in the United Kingdom, reports to a passivesurveillance system indicated a problem with meningitis after theadministration of the Urabe strain mumps vaccine. (This strain isNOT used in the United States.) Stimulated reporting, that is, activesurveillance (letters were sent to every pediatrician), was thenemployed. A rate of one case of meningitis (diagnosis confirmed byvaccine-strain virus isolated from the cerebrospinal fluid) per 250,000doses of vaccine distributed was found by that method. By using LLDBsto link records of lumbar punctures with immunization histories,the rate of meningitis after administration of that vaccine was foundto be 1 in 10,000 immunized children.
Some workshop participants felt that, if case reports are to servea useful function in assessing the safety of vaccines, there is agreat need to improve the analysis of the case reports received inpassive surveillance systems by improving the quality of the informationcontained within an individual case report, by harmonizing the systemsthat collect and analyze the data, and by increasing the proportionof serious events that are reported. The enthusiasm for better utilizingcase reports was not, however, universal. Those with reservationsworried that increasing reporting will merely increase the noiseinherent in the systems and that the resources needed to improvepassive surveillance systems sufficiently would be better spent onother research tools.
Improvements at the level of the individual case report have bothquantitative (more reports and more information in each report) andqualitative (complete, relevant, and necessary information) aspects.A significant factor in the success of passive surveillance systemsis whether the information about the reaction is complete and accurate.This is probably best achieved if a health care professional reportsthe reaction or verifies information received from a nonhealth careprofessional. The problem of underreporting of adverse events associatedwith vaccines is discussed in a subsequent section. Complete informationon the vaccinee and the reaction to a vaccination are important.In addition to information on the patient's history and completedetails of the medical problems with which the patient is presenting,it is important to know about other risk factors (for example, thepossibility that an infectious disease is circulating within thecommunity) that might account for the reaction. As mentioned previously,the absence of an alternative etiologic candidate is one factor insupport of causality. Results of a comprehensive examination areuseful for diagnostic and treatment purposes (What is wrong withthe patient and what treatment should the patient be given?) andfor research purposes (Did the vaccine cause the adverse event? Ifso, what is the mechanism?). A complete workup could be expensive,and it is not clear who should pay for a comprehensive medical examinationof someone experiencing an adverse event after vaccination.
The analysis of adverse events could be greatly improved by harmonizingof the adverse reaction reporting forms, the coding terminologies,and the search strategies used to summarize the information containedin the multiple databases managed by different manufacturers anddifferent countries. In September 1993 FDA sponsored a workshop onharmonization efforts, and the participants were hopeful that theworkshop marked the beginning of more long-term efforts in that direction.Harmonization is an issue for manufacturers (How much effort is madeto obtain complete data? How is the documentation organized? Whatcoding
system and coding practices are used?); for regulatory agencies (agenciesmay not use the same coding systems as the manufacturers); and fordata analysis and report generation (e.g., How many cases of adverseevent x have been reported with vaccine y?). The number of reports of a given adverse event identified withina system depends on the completeness of the search strategies andthe categories used to classify the adverse event. Many errors canbe made if search strategies are not standardized and validated.If the main purpose of passive surveillance systems is a signal detectionsystem (particularly for the detection of previously unrecognizedadverse events but also for the detection of increased numbers ofreports of a recognized adverse event), participants suggested thatsearch strategies should err in the direction of false positivesrather than false negatives.
The use of summary statistics (e.g., how many reports of reactionx after vaccine y) from passive surveillance systems is a first step in assessingthe safety of vaccines, but limiting the analysis to those statisticscan lead to erroneous conclusions. The reported events occur in temporalrelation to vaccination, but not all events that occur within anappropriate interval from the time of vaccination are necessarilycaused by the vaccination. Although this might seem obvious, summarystatistics of reported events can be misinterpreted as evidence that the vaccine trulycauses the adverse event.
As a passive surveillance system, VAERS suffers problems of bothunderreporting and overreporting. Although health care professionalsare required to report some adverse events, specifically, those thatare covered by the no-fault component of the Vaccine Injury CompensationProgram, there are no provisions for enforcement. It is likely thatmany events that occur after the receipt of vaccines, like thosethat occur after the receipt of medications, go unreported. Peoplemay fail to report an adverse event because they are unaware of thereporting system, because they do not believe that the event is relatedto the vaccine, or because they know that the adverse event is anaccepted reaction (e.g., fever or sore arm) to that vaccine and considerit unnecessary to report what is already established. Because anyonecan report an adverse event to VAERS, it is possible that the patient,the manufacturer, and the health care provider could report the sameevent. If this duplication is not recognized and accounted for bythe system, the number of adverse events in the system would be spuriouslyhigh. Duplication is not the major source of overreporting. Seriousadverse events that occur after a vaccination may be reported evenif the reporting physician knows or strongly suspects another cause.
In addition to over- and underreporting, it is difficult to assessthe meaning of absolute numbers of reports for several reasons. Considerthe following examples:
Many people who experience an adverse event have more than one symptomand each symptom is given a separate code. A child who experiencesa fever, a sore arm, and vomiting is listed as having three distinctadverse events. Thus, the number of adverse events recorded by thesystem is far greater than the number of people who have experiencedan adverse event.
One would expect more reports of adverse events following receiptof diphtheria and tetanus toxoids and pertussis vaccine (DPT) vaccinethan following receipt of measles, mumps, and rubella vaccine simplybecause each child receives more doses of DPT than measles, mumps,and rubella vaccine.
One would expect to find more reports of sudden infant death syndrome(SIDS) associated with a lot of vaccine in use in the winter monthsthan with lots in use during summer because the background rate (andtherefore the expected number of coincidental reports) of SIDS ishigher in winter than in summer.
Childhood vaccines are frequently given in combination, and eachadverse event occurring following receipt of each vaccine given onthe same day is given a separate code. Summary statistics from apassive surveillance system such as VAERS will show a great manyreports of a sore arm following receipt of the oral polio vaccine(OPV). What that summary statistic does not indicate is that thechild concurrently received other vaccines that were responsiblefor the sore arm. In fact, there is no causality assessment madefor the reports in VAERS, because the system was not designed tosupport such an assessment.
These examples describe the expected complications of passive surveillancesystems. Reporting and analysis of summary statistics require carefulconsideration of and explanations for these factors.
Signal Detection and Hypothesis Generation
Passive surveillance systems can be used to establish patterns ofadverse event reporting and to monitor changes in those patternsas vaccines change (different lots, different formulations, differentantigens), or as the populations that receive particular vaccineschange. Although case reports can provide important information regardingcausality under specific circumstances, the participants generallyagreed that passive surveillance systems best serve to detect signalsor warnings that there might be a problem rather than to answer questionsabout causation. These signals can lead to hypotheses about causality,which can then be tested by other methods, such as epidemiologicor laboratory studies.
A participant noted that because VAERS is intended to detect newsignals, it would be inadvisable to impede that process by providingcase definitions to those who are doing the reporting. He suggested,instead, the creation of standardized protocols, so that when a reportfalling into a particular category is received, a follow-up protocolis sent to that reporter for completion. The question that then arisesis how to finance the extra workup that may be required, especiallyif it includes costly medical tests.
Participants expressed a desire for a more clearly defined mechanismby which passive surveillance systems are evaluated and hypothesesare generated. FDA and CDC routinely review VAERS reports in a searchfor signals, and they are still trying to determine the best methodfor doing this. Internal review of current reports looks for patternsof events as well as the overall numbers of reports. Summary datafrom the VAERS system have not routinely been published, but actionis under way to determine the best format for doing so. It was suggestedthat the publication of well-documented case reports should be encouragedand facilitated.
More rigorous, well-established epidemiologic and statistical methodscould be applied to passive reporting systems, and PHS representativesindicated that some such methods are under development. For example,in the fall of 1993, a joint meeting of FDA and CDC investigatorslooked at clusters of adverse events, particularly deaths associatedwith specific vaccine lots. Although analysis of such clusters canbe useful, it usually does not permit inferences about causality.Again, it serves to sound an alert that there might be a problem,that then needs to be analyzed in some other way. It was suggestedthat VAERS data are amenable to time series analysis; outcomes canbe examined across time (by month, by 6-month period, etc.) and canbe used as systematic signal-generating mechanisms.
A participant expressed caution in interpreting the information frompassive surveillance systems, noting that “one person's signal is another person's noise.” If all the signals are preserved, there is so much attendant noisethat undue fears about the safety of vaccines may be raised. Thatparticipant felt that various signals need to be preserved and madereadily accessible to the people who are generating and testing hypotheses,but urged that care be taken to avoid misinterpretation of what commonscientific wisdom would suggest is uninterpretable noise as a signalof a causal relation. Such spurious associations could unnecessarilyalarm the public and could lead to decreases in immunization coverage.
There was discussion of how much money and effort should be devotedto passive surveillance systems given their inherent limitations.One participant expressed the opinion that financial resources shouldbe allocated on the basis of the potential benefit of a particularresearch method. That participant preferred an emphasis on allocatingtight monetary resources to rigorous scientific studies once thealert has been sounded. However, scientific understanding of the
mechanisms of adverse reactions to vaccines has not yet advancedto the point that it is always evident what studies should be undertaken.
Use of Case Reports in Causality Assessment
One participant stressed that individual case reports from passivesystems may in some specific cases, be able to answer causation questionsthat population-based studies never could. For example, the questionof whether the Urabe-strain mumps vaccine can cause meningitis orwhether OPV can cause polio can be answered by a single persuasivecase in which virus is isolated and is typed by molecular biologictechniques as the vaccine strain, rather than the wild-type strain,and there is no other etiologic factor that could cause the disease.The absence of an alternative etiologic factor makes the isolationof the vaccine strain of the virus more compelling, because the presenceof vaccine strain virus does not always prove that the virus causeddisease; it is not known how often vaccine-strain virus might bepresent in asymptomatic individuals.
Several participants agreed that evaluation of case reports of fullyevaluated events should not be minimized as an important method ofexamining very rarely occurring adverse events. Even LLDBs do nothave access to data on populations in the millions, such as mightbe necessary to have enough power to make conclusions about rarelyoccurring adverse events in a population. When a suspected rarelyoccurring adverse event is reported following vaccination, intenseand targeted clinical evaluations and research might help in assessingcausality. As discussed above, this could be costly, and currentlythere is no mechanism to fund these investigations.
Some shortcomings of passive surveillance systems are common to otherstrategies for evaluating adverse events; for example, no data-gatheringsystem, including LLDBs, can easily assess causation when multiplevaccines are administered simultaneously. As another example, becauseof the emphasis on events temporally associated with vaccination,events that occur long after vaccination may not be detected.
Because causation may not be the most fruitful focus for a discussionof VAERS and other passive surveillance systems, one participantsuggested that two additional questions be asked.
Do preexisting conditions play a role in the occurrence or reportingof adverse events?
Are particular lots associated with higher reporting rates of particularadverse events?
Passive surveillance systems such as VAERS could be helpful in addressingboth of these questions. However, as with questions of causality,without complete information in each report, these questions areunanswerable by a passive surveillance system.
Education of Health Care Providers
A participant noted a problem in persuading doctors and other healthcare providers to report adverse events. Health care providers needto be educated that VAERS is an adverse “event” reporting system, not anadverse “effect” reporting system. That is, physicians or other health careproviders sometimes fail to report an event after vaccination becausethey do not believe that the event is related to vaccination. Thistacit decision to report only well-established adverse consequencesof vaccines means that the previously unrecognized effects of vaccineswould continue to go unnoticed. If a passive surveillance systemis to function effectively as a signal detection system, then allserious adverse events that occur following vaccination should bereported.
A participant noted that, with the advent of the new presidentialadministration, immunization policy in general is being reevaluated.The education of practitioners, including information about adverseevents and the provision of instructions on how to report them, isexpected to be an integral part of a new immunization program.
Vaccine Injury Compensation Program
A participant noted that the Vaccine Injury Compensation Program(VICP) also serves as a passive reporting system that might supplementthe information in VAERS. There is potentially more useful informationin the records from VICP than in those from VAERS, because vaccineesor their parents are required to submit complete medical informationto be considered for compensation. In addition, there could be casesin the VICP that are not in VAERS. For example, the participant noteda case submitted to VICP (but not to VAERS) of two episodes of Guillain-Barré syndrome (GBS) in the same child after administration of a tetanustoxoid-containing preparation. The possibility of linking the VAERSand VICP databases is under active discussion within PHS. Anotherparticipant noted that databases from Japan might be informative,for example, for providing information on that country's experienceswhen it changed from recommending DPT vaccination in infancy to age2 years and older.
Evaluation of Passive Surveillance Systems
An evaluation of VAERS itself and how it has functioned thus farwas suggested. It should be determined, for example, whether VAERSactually has served well as the signal detector that a passive surveillancesystem is intended to be. Research to compare surveillance systemswas suggested—to find out, for example, what proportion of casesof various sorts are reported to VAERS. This