Methodological Considerations in Evaluating the Evidence
QUESTIONS TO BE ADDRESSED
The committee was charged with the task of summarizing the strength of the scientific evidence concerning the association between herbicide exposure during Vietnam service and each of a set of diseases or conditions suspected to be associated with such exposure. Public Law 102–4 (codified as 38 USC Sec. 1116) specifies three scientific determinations concerning diseases that must be made. It charges the committee to determine (to the extent that available scientific data permit meaningful determinations) the following:
whether a statistical association with herbicide exposure exists, taking into account the strength of the scientific evidence and the appropriateness of the statistical and epidemiologic methods used to detect the association;
the increased risk of each disease among those exposed to herbicides during service in the Republic of Vietnam during the Vietnam era; and
whether there exists a plausible biologic mechanism or other evidence of a causal relationship between herbicide exposure and the disease.
P.L. 102–4 did not provide a specific list of diseases and conditions suspected to be associated with herbicide exposure. The committee staff and members responsible for the 1994 report Veterans and Agent Orange: Health Effects of Herbicides Used in Vietnam (hereafter referred to as VAO) (IOM, 1994) developed such a list based on diseases and conditions that had been mentioned in the scientific literature or in other documents that came to their attention through
extensive literature searches. The VAO list has been supplemented over time in response to developments in the literature.
The information used by the committee was developed through a comprehensive search of relevant data bases. Public and commercial data bases covering biological, medical, toxicological, chemical, historical, and regulatory information were examined. The majority of these data bases were bibliographic, providing citations to scientific literature. Committee staff examined the reference lists of major review articles, books, and reports for relevant citations. Reference lists of individual articles were also scanned for pertinent citations. Search engines were used to scan for information posted on the Internet. Literature identification continued through September 30, 2000. The input received both in written and oral form from veterans and other interested persons at public hearings and in written submissions served as a valuable source of additional information.
This third biennial update concentrates on evaluating the evidence published following the completion of work on Veterans and Agent Orange: Health Effects of Herbicides Used in Vietnam (IOM, 1994), Veterans and Agent Orange: Update 1996 (IOM, 1996), Veterans and Agent Orange: Update 1998 (IOM, 1999), and Veterans and Agent Orange: Herbicide/Dioxin Exposure and Type 2 Diabetes (IOM, 2000). For each health outcome, the new evidence is reviewed in detail. Conclusions are based on the totality of accumulated evidence, not just on recently published studies. In other words, new evidence is not interpreted alone but is put in the context of evidence addressed in previous reports.
The committee’s judgments have both quantitative and qualitative aspects; they reflect both the evidence examined and the approach taken to evaluate it. In VAO, the committee delineated how it approached its task so that readers would be able to assess and interpret the committee’s findings. By offering this information, the committee wished to make the report useful to those seeking to update its conclusions as new information was obtained. The committees responsible for subsequent reports have adopted the original committee’s approach.
The remainder of this chapter delineates the primary considerations underlying the evaluation process. A more complete description of methodological issues may be found in Chapter 5 of VAO and in Chapter 4 of Update 1996 and Update 1998.
Is Herbicide Exposure Statistically Associated with the Health Outcome?
The committee necessarily focused on a pragmatic question: What is the nature of the relevant evidence for or against a statistical association between exposure and the health outcome? The evidentiary base that the committee found to be most helpful derived from epidemiologic studies of populations—that is, investigations in which large groups of people are studied to determine the association between the occurrence of particular diseases and exposure to the substances at issue. To determine whether an association exists, epidemiologists
estimate the magnitude of an appropriate quantitative measure (such as the relative risk or the odds ratio) that describes the relationship between exposure and disease in defined populations or groups. However, the use of terms such as “relative risk,” “odds ratio,” or “estimate of relative risk” is not consistent in the literature. In this report, the committee intends relative risk to refer to the results of cohort studies and odds ratio (an estimate of relative risk) to refer to the results of case-control studies. Values of relative risk greater than 1 may indicate a positive or direct association—that is, a harmful association—whereas values between 0 and 1 may indicate a negative or inverse association—that is, a protective association. A “statistically significant” difference is one that, under the assumptions made in the study and the laws of probability, would be unlikely to occur if there were no true difference and no biases.
Determining whether an observed statistical association between exposure and a health outcome is “real” requires additional scrutiny because there may be alternative explanations for the observed association. These include: error in the design, conduct, or analysis of the investigation; bias, or a systematic tendency to distort the measure of association so that it may not represent the true relation between exposure and outcome; confounding, or distortion of the measure of association because another factor related to both exposure and outcome has not been recognized or taken into account in the analysis; and chance, the effect of random variation, which produces spurious associations that can, with a known probability, sometimes depart widely from the true relation.
Therefore, in deciding whether an association between herbicide exposure and a particular outcome exists, the committee examined the quantitative estimates of risk and evaluated whether these estimates might be due to error, bias, confounding, or chance or were likely to represent a true association.
In pursuing the question of statistical association, the committee recognized that an absolute conclusion about the absence of association may never be attained. As in science generally, studies of health outcomes following herbicide exposure are not capable of demonstrating that the purported effect is impossible or could never occur. Any instrument of observation, including epidemiologic studies, has a limit to its resolving power. Hence, in a strict technical sense, the committee could not prove the absolute absence of a health outcome associated with herbicide or dioxin exposure.
What Is the Increased Risk of the Outcome in Question Among Those Exposed to Herbicides in Vietnam?
This question, which is pertinent principally (but not exclusively) if there is evidence for a positive association between exposure and a health outcome, concerns the likely magnitude of the association in Vietnam veterans exposed to herbicides. The most desirable evidence in answering this type of question involves knowledge of the rate of occurrence of the disease in those Vietnam
veterans who were actually exposed to herbicides, the rate in those who were not exposed (the “background” rate of the disease in the population of Vietnam veterans), and the degree to which any other differences between the exposed and unexposed groups of veterans influence the difference in rates. When exposure levels among Vietnam veterans have not been adequately determined, which has been the case in most studies, this question is very difficult to answer. The committees have found the available evidence sufficient for drawing conclusions about the association between herbicide exposure and a number of health outcomes. However, the lack of good data on Vietnam veterans per se, especially with regard to herbicide exposure, complicates the assessment of the increased risk of disease among individuals exposed to herbicides during service in Vietnam. Indeed, most of the evidence on which the findings in this and other reports are based comes from studies of people exposed to dioxin or herbicides in occupational and environmental settings rather than from studies of Vietnam veterans.
Is There a Plausible Biologic Mechanism?
Chapter 3 details the cellular and animal experimental evidence that provides the basis for the assessment of biologic plausibility, that is, the extent to which a statistical association is consistent with existing biological or medical knowledge. As with the epidemiologic evidence, the chapter concentrates on studies published during 1999–2000 but considers all relevant studies in drawing conclusions. The issue of whether a given chemical exposure-health outcome relationship reflects a true association in humans is addressed in the context of research regarding the mechanism of interaction between the chemical and biological systems, evidence in animal studies, evidence of an association between exposure and the occurrence of a health outcome in humans, and/or evidence that a given outcome is associated with occupational or environmental chemical exposures. It must be recognized, however, that lack of data in support of a plausible biologic mechanism does not rule out the possibility that a causal relationship does exist.
ISSUES IN EVALUATING THE EVIDENCE
A valid surrogate animal model for the study of a human disease must reproduce with some degree of fidelity the manifestations of the disease in humans. Whole-animal studies or animal-based experimental systems continue to be used to study herbicide toxicity because they allow for rigid control of chemical exposures and close monitoring of health outcomes. Because many of the chemical exposures presently associated with certain diseases in humans have been confirmed in experimental studies, data derived from such studies are generally accepted as a valuable guide in the assessment of biologic plausibility.
As discussed in Chapter 3, many of the toxic effects of the herbicides used in Vietnam have been ascribed to 2,3,7,8-tetrachlorodibenzo-p-dioxin (TCDD), a contaminant of some of these herbicides. This has not, however, simplified the risk assessment process because the toxicologic profile of TCDD is rather complex. There is consensus that most of the toxic effects of TCDD involve interaction with the aryl hydrocarbon receptor (AhR), a protein that binds TCDD and other aromatic hydrocarbons with high affinity. Attempts to establish correlations between the effects of TCDD in experimental systems and in humans are particularly problematic, because species-, gender-, and endpoint-specific differences in susceptibility to TCDD have been documented. Although studies in which transformed human cell lines are employed to study AhR biology minimize the inherent error associated with species extrapolations, caution must be exercised because the extent to which transformation itself influences toxicity outcomes has yet to be fully defined. In addition, while it is generally accepted that genetics plays a key role in determining the adverse effects of environmental chemicals, the impact of different genetic backgrounds on AhR function is not yet well understood.
Environmental and/or occupational exposures to herbicides or TCDD provide data on human responses that can be compared directly to data obtained in experimental studies. Higher-than-background body burdens of dioxin have been documented in many of these groups, and details describing the major findings from these studies are reviewed in Chapters 7–10 of this report. In general, the elevated risks of cancers at various sites reported in epidemiologic studies are consistent with the known biological actions of the agents present in herbicide formulations. Although its full potential has yet to be realized, the application of molecular and cellular measurements to epidemiologic research promises to increase our understanding of the association between herbicide exposure and disease occurrence. This may provide a significant advantage in the assessment of biologic plausibility, because biologically based epidemiologic data allow more accurate identification and quantification of exposures. For instance, the analytical data available from individuals known to have been exposed to herbicides during the Vietnam War constitute a valuable resource for the study of TCDD-related disease, with documented TCDD body burdens providing a quantitative bridge between experimental studies and human epidemiology. Taken together, experimental studies and epidemiologic investigations provide complementary perspectives from which to view human health effects of exposure to herbicides. However, it must be recognized that the ultimate test of associations between exposure and disease occurrence lies in data obtained from human populations.
To obtain additional information pertinent to evaluation of the potential effects of herbicide exposure of veterans, this and previous committees decided
to review studies of other groups potentially exposed to the herbicides contained in Agent Orange, to other herbicides, and to dioxin, the contaminant believed to cause many of the purported adverse effects of Agent Orange. These study populations include industrial and agricultural workers, Vietnamese citizens, and people exposed environmentally as a result of residing near the site of an accident or a toxic waste dump. The committee felt that reviewing the studies of such groups would help in determining (1) whether these compounds could be associated with particular health outcomes in veterans and (2) the nature of any dose-response relationships, although the committee acknowledged that such findings may have only an indirect bearing on the association in veterans themselves. It is also important to note that the categories of association described below relate to the association between exposure to chemicals and health outcomes in human populations, not to the likelihood that any individual’s health problem is associated with or caused by the herbicides in question.
With the exception of acute and subacute transient peripheral neuropathy, the committee did not specifically consider case studies or other published studies lacking a control or comparison group. The committee elected to consider case histories when evaluating the association between exposure and these conditions because their transient nature precluded using case-control and other types of studies with comparison populations.
It has been well documented (Song et al., 2000) in biomedical research that studies with a statistically significant finding are more likely to be published than studies with nonsignificant results. Thus, evaluations of disease-exposure associations that are based solely on the published literature could be biased in favor of a positive association. In general, however, for reports of overall associations with exposure, the committee did not consider the risk of publication bias to be high among studies of herbicide exposure and health risks. The committee took this position because there are numerous published studies showing no positive association; because it examined a substantial amount of unpublished material; and because the committee felt that publicity surrounding the issue of exposure to herbicides, particularly regarding Vietnam veterans, has been so intense that any studies showing no association would be unlikely to be viewed as unimportant by the investigators. In short, the pressure to publish such “negative” findings would be considerable.
The Role of Judgment
The evaluation of evidence to reach conclusions about statistical associations goes beyond quantitative procedures at several stages: assessing the relevance and validity of individual reports; deciding on the possible influence of error,
bias, confounding, or chance on the reported results; integrating the overall evidence within and across diverse areas of research; and formulating the conclusions themselves. These aspects of the committee’s review required thoughtful consideration of alternative approaches at several points. They could not be accomplished by adherence to a narrowly prescribed formula.
Rather, the approach described here evolved throughout the process of review and was determined in important respects by the nature of the evidence, exposures, and health outcomes at issue. Both the quantitative and the qualitative aspects of the process that could be made explicit were important to the overall review. Ultimately, the conclusions about association expressed in this report are based on the committee’s collective judgment. The committee has endeavored to express its judgments as clearly and precisely as the data allowed.
IOM (Institute of Medicine). 1994. Veterans and Agent Orange: Health Effects of Herbicides Used in Vietnam. Washington, DC: National Academy Press.
IOM. 1996. Veterans and Agent Orange: Update 1996. Washington, DC: National Academy Press.
IOM. 1999. Veterans and Agent Orange: Update 1998. Washington, DC: National Academy Press.
IOM. 2000. Veterans and Agent Orange: Herbicide/Dioxin Exposure and Type 2 Diabetes. Washington, DC: National Academy Press.
Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ. 2000. Publication and related biases. Health Technology Assessment 4(10):1–115.