This chapter outlines the approach used by the Committee to Review the Health Effects in Vietnam Veterans of Exposure to Herbicides (Eighth Biennial Update) and its predecessors to evaluate the available scientific evidence. A more complete description is found in Chapter 5 of Veterans and Agent Orange: Health Effects of Herbicides Used in Vietnam, hereafter referred to as VAO (IOM, 1994).
As discussed in Chapter 1, the committee was charged with summarizing the strength of the scientific evidence of associations between exposure to various herbicides and contaminants during service in the Vietnam War and individual diseases or other health outcomes. Public Law 102-4, which mandated the committee’s work, however, did not specify particular health outcomes suspected of being associated with herbicide exposure. Such a list of outcomes was developed on the basis of diseases and conditions addressed in the scientific literature identified through the original VAO’s extensive literature searches. The list has been amended in the VAO updates in response to new publications, to requests from the Department of Veterans Affairs (VA) and various veterans’ service organizations, and to concerns of Vietnam veterans and their families. Comments received at public hearings and in written submissions from veterans and other interested persons have been valuable in identifying issues to be pursued in greater depth in the scientific literature.
The VAO committees began their evaluation by presuming neither the presence nor the absence of an association between exposure and any particular health outcome. Over the series of reviews, evidence of various degrees of association,
lack of association, or persistent indeterminacy with respect to a wide array of disease states has accrued. For many conditions, however, particularly ones that are very uncommon, associations with the chemicals of interest have remained unaddressed in the medical research literature; for these, the committee remains neutral on the basis of the understanding that “absence of evidence is not evidence of absence.”
IDENTIFICATION OF RELEVANT LITERATURE
Mixtures of 2,4-dichlorophenoxyacetic acid (2,4-D), 2,4,5-trichlorophen-oxyacetic acid (2,4,5-T), picloram, and cacodylic acid made up the bulk of the herbicides sprayed in Vietnam. At the time of the spraying, 2,3,7,8-tetrachlorodibenzo-p-dioxin (TCDD, one form of dioxin) was an unintended contaminant from the production of 2,4,5-T and so was present in Agent Pink, Agent Green, Agent Purple, Agent Orange, and Agent Orange II, which all contained 2,4,5-T. It is important to note that TCDD and Agent Orange are not the same. Databases have been searched for the names of those compounds, their synonyms and abbreviations, and their Chemical Abstracts Service (CAS) numbers. The evidence indicates that a single protein, the aryl hydrocarbon receptor (AHR), mediates essentially all the toxicity of TCDD, so aryl hydrocarbon receptor also was used as a keyword, as were dioxin, Agent Orange, and Vietnam veteran.
One of the herbicides used in Vietnam was cacodylic acid, or dimethylar-sinic acid of valency 5 (DMAV), an organic form of arsenic. In addition to being synthesized as a herbicide, DMAV is a metabolite of inorganic arsenic exposure in humans. DMAV was long thought to be a biologically inactive metabolite, but evidence suggests that methylated forms such as MMAIII (Aposhian et al., 2000), and perhaps DMAIII and DMAV (Cohen et al., 2006), might be responsible for some of the adverse effects of inorganic arsenic. The committee carefully reconsidered that evidence but again determined that it does not support a conclusion that exposure to cacodylic acid (DMAV) would be expected to result in the same adverse health effects as would exposure to toxic concentrations of inorganic arsenic. Therefore, as in prior VAO reports, the literature on the health effects of inorganic arsenic was not considered here. Further details on the effects of inorganic arsenic can be found in Arsenic in Drinking Water (NRC, 1999) and Arsenic in Drinking Water: 2001 Update (NRC, 2001). For cacodylic acid and picloram, the search terms were the chemical names, synonyms, and CAS numbers of the herbicides.
This report concentrates on the evidence published after the completion of work on Veterans and Agent Orange: Update 2008 (IOM, 2009). Relevant new contributions to the literature made during the period October 1, 2008–September 30, 2010, were sought. The information that the committee used was compiled from a comprehensive electronic search of public and commercial databases— biologic, medical, toxicologic, chemical, historical, and regulatory—that pro-
vide citations of the scientific literature. In addition, the reference lists of some review and research articles, books, and reports were examined for potentially relevant articles. As noted above, the terms used in the search strategy included the chemical names, synonyms, and CAS numbers of the specific chemicals of interest—2,4-D, 2,4,5-T, TCDD, cacocylic acid, and picloram (see Figure 2-1 for chemical structures and CAS numbers)—and the more generic terms involved with this project: Vietnam veteran, Agent Orange, aryl hydrocarbon receptor, dioxin, herbicide, and phenoxy. Results on other specific phenoxy herbicides are also of interest: 2-methyl-4-chlorophenoxyacetic acid (MCPA) and 2-(2-methyl-4 -chlorophenoxy) propionic acid (MCPP or Mecoprop) for 2,4-D and 2-(2,4,5-trichlorophenoxy) propionic acid (2,4,5-TP or Silvex) for 2,4,5-T (see Figure 2-1); although the benzoate herbicide dicamba (2-methoxy-3,6-dichlorobenzoic acid) is not always categorized with the phenoxy herbicides, it has structural similarities with this class, and measures of its association with various adverse health outcomes have been factored into the evidence. Because some polychlorinated biphenyls (PCBs) and polychlorodibenzofurans (PCDFs) have dioxin-like biologic activity, studies of populations exposed to PCBs or PCDFs were reviewed when results were presented in terms of TOTSL toxic equivalents (TEQs). Findings related only to exposure to the diverse chemical families of pesticides were considered too nonspecific for inclusion in the evidence database used to draw conclusions about associations. (An ancillary analysis conducted during preparation of Update 2008 determined that the term pesticide did not identify any relevant citation that was not picked up by more specific terms, and so it was eliminated from the searches conducted for the current update, and this reduced the number of extraneous hits to be culled.)
(With the structural representation at hand in Figure 2-1, the committee will respond to an assertion it has heard repeatedly from individual Vietnam veterans that “benzene is contained in TCDD.” Indeed, the two rings at the ends of the three-ring structure constituting the basic structure of dioxin compounds, to which chlorine molecules or other chemical radicals can be attached, do have the molecular structure of a single benzene molecule and “dibenzo-dioxin” in TCDD’s chemical name does mean the molecule is indeed a benzene-substituted dioxane. The benzene ring structure is a basic building block of a vast number of organic compounds, both industrial [such as polyaromatic hydrocarbons, the phenoxy herbicides, picloram, and PCBs] and natural [such as estradiol, a hormone present in both men and women]. However, the biologically active compound benzene does not emerge from dioxin, whose three-ring structure is extremely stable and very resistant to metabolism.)
Because they are the target population of the charge to the VAO committees, studies of Vietnam veterans (serving in any of the armed forces, American or otherwise) have always been accorded considerable weight in the committees’ deliberations, whether or not estimation of exposure to herbicide-related substances has been attempted. Characterization of exposure in studies of the
FIGURE 2-1 Chemical structures and CAS numbers for specific chemicals of interest.
veterans was extremely uncommon at the time of the original VAO report, and the Vietnam veterans’ own ages were still below the ages at which many chronic illnesses are manifested. Consequently, the original committee made extensive efforts to consider several groups known or thought to have potentially higher and better-characterized exposure to TCDD or phenoxy herbicides than Vietnam veterans themselves—both occupational exposure (as of chemical-production, paper and pulp, sawmill, tannery, waste-incinerator, railroad, agricultural, and forestry workers) and environmental exposure (as of residents of Seveso, Times Beach, Quail Run, and Vietnam).
Successive committees have been able to concentrate more on studies that explicitly addressed the exposures specified in the charge. Some occupational and environmental cohorts that received exceptionally high exposures (such as the International Agency for Research on Cancer and Seveso cohorts) are now well characterized and producing a stream of informative results. The Agricultural Health Study, a continuing prospective cohort study of agricultural populations with specific information on the chemicals of interest, is also now contributing a steady stream of information to the database. Most important, the Vietnam veterans themselves are advancing in age and when studied are capable of directly providing substantial information on chronic health conditions and, in some study populations, information related to serum TCDD concentrations. The committee for Update 2006 decided that exhaustive searches on job titles, occupations, or industries to identify additional study populations with possible, but not specifically characterized, exposure to the chemicals of interest were no longer an efficient means of augmenting the evidence database, in that they are more likely to retrieve citations with information about a health outcome at the expense of considerable uncertainty about exposure.
The previous and current committees followed the Update 2006 committee’s practice of more circumscribed searching. As the information in the database on populations that had established exposure to the chemicals of interest has grown, VAO committees have become less dependent on data from studies with nonspecific exposure information and have been able to focus more on findings of studies with refined exposure specificity. In recognition of the more pivotal role that findings drawn directly from Vietnam veterans were able to play in its decisions, the committee for Update 2008 reordered its consideration of populations. For each health outcome, studies of Vietnam veterans, the target population of the VAO series, are addressed first and then occupational and environmental studies.
It is well accepted that any TCDD or herbicide effect may be diluted somewhat in studies of Vietnam veterans because some of the veterans may not have been exposed or may have been exposed only at low concentrations. The problem is exacerbated in studies in which exposure is defined in terms of occupation (even on the basis of a full job history). Exploratory studies based on linking to a one-time statement of occupation (for example, on a death certificate or in a census) are thought to be of little use even when a job–exposure matrix is
used to “convert” standardized job codes to specific exposures. Not only is there uncertainty about whether all members of the sample have been exposed to one of the chemicals of interest unless detailed personal monitoring and industrial-hygiene work have been performed but for most occupational categories there is considerable certainty that the workers were exposed to many other potentially toxic agents. Thus, such studies may well minimize the effects of exposure to TCDD or the herbicides of interest while yielding misleading indications of health problems resulting from other exposures.
The search strategy was devised to ensure that abstracts of all potentially relevant articles were subjected to closer screening, but it also resulted in the identification of a large number of nonrelevant studies. The searches produced in excess of 6,600 “hits,” including some studies that were identified more than once. It was evident from the abstracts of most of the cited articles that they did not address health effects in association with exposure to the chemicals of interest; for example, many of the cited studies investigated the efficacy of herbicides in killing weeds. All studies that discussed health effects were considered if the search-related information (title, abstract, and keywords) indicated that any of the herbicides of interest (or any of their components) may have been investigated. For each of the more than 1,300 potentially relevant citations ultimately identified, a copy of the entire article was obtained online and reviewed more thoroughly by the committee for inclusion in its report. For the present update, very few documents of interest had to be retrieved as hard copies from library sources.
In large part, included reports are peer-reviewed journal articles, but generally available and formally published government studies (particularly those investigating health effects in Vietnam veterans) are also included under the presumption that they have been carefully reviewed. In practice, the articles are generally in English, but the committee obtained translations for crucial ones that were not in English, as in the case of reports of a study of Korean veterans of the Vietnam War (Kim HA et al., 2003; Kim JS et al., 2003) when Update 2004 was produced. For the present update, in an effort to determine whether the results of epidemiologic studies of the Vietnamese population had been published in foreign-language publications, a search of non-English journals was conducted on this topic; no additional literature of this sort was identified.
TCDD, the 2,3,7,8-chlorinated congener of dioxin, is the most potent of the polychlorinated dibenzo-p-dioxins (PCDDs), dibenzofurans, and biphenyls, so it is presumed to be most problematic of the dioxin-like chemicals contaminating the phenoxy herbicides used in Vietnam. However, our concern is not limited to that congener. In nonlaboratory settings—for example, epidemiologic studies— exposures occur not only to TCDD but to mixtures of dioxins, dibenzofurans, and PCBs, which vary in their degree of chlorination. The concept of toxic equivalency has been developed primarily to permit overarching estimation of oral exposure and risk from certain environmentally persistent chemicals with structural similarities to PCDDs and PCDFs that bind the AHR, induce the same spectrum
of effects, and bioacumulate in the food chain (van den Berg et al., 2006). A toxicity equivalency factor (TEF) is an estimate of the dioxin-like potency of an individual congener relative to the toxicity of TCDD. TEQs are often used to estimate the cumulative toxic potency of mixtures as the sum of TEFs weighted by the concentrations of the corresponding congeners in the mixture; this total is denoted as the mixture’s TEQ in terms of dioxin-like activity. That approach is often taken in epidemiologic studies that focus on PCBs. Many epidemiologic studies of PCBs were recovered in the literature search although they were not specifically sought. Because dioxin-like and non–dioxin-like PCB congeners are found together in environmental mixtures and are known to mediate toxicity by various mechanisms, the relative contribution of dioxin-like PCBs to an individual health outcome can be difficult to determine. Therefore, evidence from epidemiologic studies of PCB exposure was retained only for results reported for specific dioxin-like congeners or in terms of TEQs.
The committee for Update 2008 investigated what pesticides are used in greenhouses and determined that greenhouse workers are not likely to be exposed to herbicides, particularly those of interest for VAO committee deliberations (Czarnota, 2004; Neal, 2006; University of Connecticut, 2006). Results on such populations (Abell et al., 2000, on fertility; Hansen et al., 1992, on cancer in female workers) were retroactively excluded from the evidence database considered in Update 2008, and no new citations for studies of such workers have been retained.
More than 60 articles on epidemiologic studies and several score of toxicology studies contributed new information to the present update. New evidence on each health outcome was reviewed in detail. The committee’s conclusions, however, are based on the accumulated evidence, not just on recently published studies. In a considerable number of instances over the course of the VAO reports, single study populations have generated multiple entries for a given health outcome. The past procedure has been to enter new results into the summary results tables in groups corresponding to successive updates, so it has been difficult to recognize which findings are based on the experience of the same set of people.
For the present update, the committee adopted a major change in the formatting of the tables of cumulative results for the health outcomes in an effort to make the interrelationships more evident for its own deliberations and for the reader. The practice followed in previous VAO updates of inserting the findings from new publications at the beginning of the sections on veteran, occupational, and environmental studies, thereby creating chronologic bands of studies reviewed in an individual update, has been replaced with a presentation of results by study population. The reported findings on a given condition from a particular study population have been gathered and presented in reverse chronologic order so that the most mature set of statistics appears first. In many instances, this will represent the most informative set of data, the set that has the greatest power to demonstrate an adverse effect in this population. For some health problems,
particularly those common in old age, the toxic effect associated with an external factor may be to make a disease manifest itself sooner. In such situations, the evidence of an association with an exposure may consist of a wave of diagnoses in younger people, and the prevalence will equalize with that of the control group as the populations age. The committee therefore decided that it could not retain only the most recent findings when considering the experience of a given study population.
The cohorts themselves have been ordered in the tables to reflect the overarching cohorts of which they are a subgroup. And, the exposure of interest for each cohort is explicitly noted in the tables to facilitate judgments about when consistency might be expected among populations that experience the same exposure. That should minimize misapprehensions that there are inconsistencies if two excellent studies of groups exposed to different chemicals of interest have incongruent findings.
Primary findings are the components of the evidence that the committee endeavors to integrate in drawing its conclusions; reanalyses (without the incorporation of additional information), pooled analyses, reviews, and so on, may be discussed in conjunction with primary results or in synthesis sections for a given health outcome, but they are not themselves part of the evidence dataset.
The committee’s general approach to the evaluation of scientific evidence corresponds closely with the approach developed by the original VAO committee as delineated in detail in Chapter 5 of VAO. The committee had three specific tasks: to determine whether there is a statistical association between exposure to the herbicides used in Vietnam and health outcomes, to determine the increase in risk of effects among Vietnam veterans, and to determine whether plausible biologic mechanisms provide support for a causal relationship with a given health outcome.
Statistical Association
The issues in determining whether a statistical association exists are detailed in Chapter 5 of VAO. The committee found that the most relevant evidence came from epidemiologic studies—investigations in which large groups of people are studied to identify an association between exposure to a chemical of interest and the occurrence of particular health outcomes.
Epidemiologists estimate associations between exposure and outcome in a specific population or group by using such measures as relative risk, standardized mortality ratio, and odds ratio. Those measures indicate the magnitude of a difference in the rate of an outcome between two populations. For example, if the rate in an exposed population is twice the rate in a nonexposed population, the
relative risk, or rate ratio, is 2. Similarly, if the odds of a health outcome are 1:20 in an exposed population but 1:100 in a nonexposed population, the odds ratio is 5. In this report, both relative risk (also called risk ratio) and odds ratio are used to represent the association between exposure and adverse outcome. Both measures are often reported in prospective cohort studies. Case–control studies usually report odds ratios, and cannot report relative risk because the base rate in the control group is usually not available in these studies. However, it is possible for case–control studies to provide estimates for relative risk if ancillary information on the base rate is available (Hsieh et al., 1985; Langholz, 2010). For rare diseases with low rates for both the exposed group and the control group, odds is approximately identical to risk, therefore an odds ratio is approximately identical to a relative risk (that is,
odds = risk / [1 - risk],
so that when risk is close to zero, [1 - risk] is close to one, and therefore odds will be close to risk). An estimated relative risk or odds ratio greater than 1 indicates a positive association (that is, it is more likely that the outcome will be seen in exposed people than in nonexposed people), whereas a relative risk or odds ratio between 0 and 1 indicates a negative or inverse association (that is, the outcome is less likely in exposed people). A relative risk or odds ratio of 1 suggests the absence of association, which is usually the null hypothesis to be tested. A statistically significant association is one that would be unlikely to occur by chance (that is, if the null hypothesis is true). (Chapters 6–11 contain tables of results abstracted from the studies providing evidence about individual health outcomes. Because the distinction between risk and odds is of little consequence in the deliberations of VAO committees, the column labeled “Estimated Risk” presents these findings without specifying the precise nature of the reported statistic.)
Determining whether an estimated association between an exposure and an outcome represents a real relationship requires careful scrutiny because there can be more than one explanation for an estimate. Bias is a distortion of the measure of association that results from flawed selection in the assembly of the study population or from error in measurement of studied characteristics. Confounding is a distortion of the measure of association that results from failure to recognize or account for some factor related both to exposure and to outcome. Chance is the degree to which an estimated association might vary randomly among different samples of the population studied. The width of a confidence interval is used to quantify the likely statistical variability of an exposure–disease association, but it does not incorporate quantification of distortions that may arise from the systematic problems mentioned above. Even when a relative risk or standardized mortality ratio substantially exceeds 1, a conclusion regarding increased risk must be qualified when the confidence interval is wide. In drawing its conclusions, the committee examined the quantitative estimates of association and evaluated the
potential influences of bias, confounding, and chance. In integrating the findings of various studies, the committee considered the degree of statistical significance associated with every estimated risk (a reflection of the magnitude of the observed effect and the power of the study designs) rather than simply tallying the “significant” and “nonsignificant” outcomes as dichotomous items of evidence. The committee also considered whether controlled laboratory investigations provide information consistent with the chemicals’ of interest being associated with a given effect and perhaps causally linked to it.
In pursuing the question of statistical association, the committee recognized that an absolute conclusion about the absence of association is unattainable. As in science generally, studies of health effects associated with herbicide exposure cannot demonstrate that a purported effect is impossible or could never occur, only that it is statistically improbable. Any instrument of observation, even an excellent epidemiologic study, is limited in its resolving power. In a strict technical sense, therefore, the absence of an association between even one chemical and a health outcome cannot be proved. Convincingly demonstrating the lack of a particular effect of all five of the compounds of interest simultaneously would be a daunting effort, especially in light of the paucity of information concerning picloram and cacodylic acid. The present committee therefore endorses the decision by the committee for Update 2006 to reclassify several types of cancer that had been classified since VAO (1994) as having “suggestive evidence of no association” with “exposure to herbicides.”
Interaction or synergism among the chemicals of interest or with other agents is another theoretical concern. The committee was not charged with attributing effects to specific chemicals of interest, and joint effects among them should be adequately identified by the committee’s approach. The number of combinations of the chemicals with other agents that might be problematic is virtually infinite. Real-life experience, as investigated with epidemiologic studies, effectively integrates any results of exposure to a target substance over all other possibly detrimental or mitigating exposures that a population might have. It may not be possible to partition contributions of the chemicals of interest from those of all other factors quantitatively, but, to the extent that the possibility of confounding influences can be appraised, the committee will have achieved its objective.
Increased Risk in Vietnam Veterans
When all the available epidemiologic evidence has been evaluated, it is presumed that Vietnam veterans are at increased risk for a specific health outcome if there is evidence of a positive association between one or more of the chemicals of interest and the outcome. The best measure of potency for the quantification of risk to veterans would be the rate of the outcome in exposed Vietnam veterans compared with the rate in nonexposed veterans, adjusted for the degree to which any other factors that differ between exposed and nonexposed veterans might
influence those rates. A dose–response relationship established in another human population suitably adjusted for such factors would be similarly suitable.
It is difficult to quantify risk when exposures of a population have not been measured accurately. Recent serum TCDD concentrations are available only on subgroups enrolled in the Air Force Health Study (AFHS) (the Ranch Hand and Southeast Asia comparison subjects) and from VA’s study of deployed and nondeployed members of the Army Chemical Corps. Pharmacokinetic models, with their own set of assumptions, must be applied to extrapolate from contemporary readings to obtain presumably accurate estimates of original exposure of Vietnam-era veterans. The absence of reliable measures of exposure of Vietnam veterans to the chemicals of interest limits the committee’s ability to quantify risks of specific diseases in this population.
Although serum TCDD measurements in only a small portion of Vietnam-era veterans are available, the observed distributions of these most reliable measures of exposure make it clear that they cannot be used as a standard to partition veterans into discrete exposure groups, such as service on Vietnamese soil, service in the Blue Water Navy, and service elsewhere in Southeast Asia. For example, many TCDD values observed in the comparison group from the AFHS exceeded US background levels and overlapped considerably with those of the Ranch Hand subjects.
As explained in Chapter 1, the committee for Update 2006 decided to make a general statement about its continuing inability to address that aspect of its charge quantitatively rather than to reiterate a disclaimer in the concluding section for every health outcome, and the present committee has retained that approach.
Plausible Biologic Mechanisms
Chapter 4, “Information Related to Biologic Plausibility” and previously denoted as “Toxicology,” details the experimental basis of assessment of biologic plausibility or the extent to which an observed statistical association in epidemiologic studies is consistent with other biologic or medical knowledge. Does the observation of a particular health effect make sense on the basis of what is known about how the chemicals in question act at the tissue, cellular, or molecular level? The relationship between a particular exposure and a specific human health outcome is addressed in the context of research on the effects of the chemicals on biologic systems and of evidence from animal studies.
Chapter 4 presents an integrated toxicity profile of each of the chemicals of interest without providing detailed commentary on each possibly relevant toxicology article published in the update period. Experimental information pertinent to a particular health outcome is now presented immediately after the epidemiologic evidence on that outcome in the “Biologic Plausibility” sections of the individual health outcomes (Chapters 6–11).
A positive statistical association between an exposure and an outcome does not necessarily mean that the exposure is the cause of the outcome. Data from toxicology studies may support or conflict with a hypothesis that a specific chemical can contribute to the occurrence of a particular disease. Many toxicology studies are conducted with laboratory animals so that variables, including the amount and duration of exposure, can be controlled precisely. Studies that use isolated cells in culture also can elucidate how a chemical alters cellular processes. The objectives of those toxicology studies are to determine what toxic effects are observed at different exposure concentrations and to identify the mechanisms by which the effects are produced. Ultimately, the results of the toxicology studies should be consistent with what is known about the human disease process to support a conclusion that the development of the disease was influenced by an exposure.
Animal studies and in vitro studies with human cells and cell lines do provide important links to understanding underlying biochemical mechanisms associated with toxicity induced by xenobiotics (exogenous chemicals). In some cases, however, toxic effects are observed in animal studies that are not detected in humans. There are many potential factors that may contribute to differences between controlled animal studies and effects observed in humans. The following are among the most important:
- Physiologic differences. Laboratory animals are not miniature humans. Depending on the biologic process under investigation, a particular test species may match the human system more closely and so be a better experimental model.
- Magnitude of exposures. In general, the TCDD exposure used for animal studies has been many orders of magnitude higher than Vietnam veterans are likely to have received during military service.
- Duration of exposure. Although TCDD is a persistent organic pollutant, animal studies seldom examine chronic low-level exposure that occurs over a period of many months or even years.
- Timing of exposure. It is well known that many organ systems are highly susceptible to xenobiotic exposure during critical stages of development, such as during gestation; the response of some systems (such as the immune system) may also depend on the timing of exposure to antigens relative to the timing of exposure to xenobiotics such as TCDD.
- Other genetic susceptibilities. The etiologies of most diseases in humans and in animals are likely to be under the influence of numerous genes and to involve complex gene-by-environment interactions, and preliminary evidence suggests that TCDD can induce epigenetic modifications to an organism’s DNA that may alter future expression of the genome.
- Sex differences. There are well-known differences in susceptibility to
-
xenobiotic exposures between male and female animals, some of which are modified by sex steroids; there are probably other reasons for sex differences.
- Prior and recurring exposures to multiple sources. Humans are exposed to xenobiotics from multiple sources throughout their lifetime.
- Complex mixtures. Most xenobiotic exposures occur in complex mixtures; the makeup of these mixtures can greatly influence the ultimate toxic effects; in addition to dietary modulation of response to other exposures in both humans and animals, human metabolism is further perturbed by dietary supplements, prescription and over-the-counter pharmaceuticals, and other factors (such as cigarette-smoking or ambient pollution).
- Stress. Stress of known or unknown origin is a well-known modifier of human disease responses (such as immune responses); stress is an ever-present variable that is difficult to assess or control for in epidemiologic studies because there is substantial individual variation in response to stress (Cohen et al., 2007).
The absence of evidence of biologic plausibility from toxicology studies, however, does not rule out the possibility of a biologic relationship. In fact, cases in which the epidemiologic evidence is strong but toxicologic support is lacking often drive new toxicology research.
As noted in VAO, not only is information on biologic plausibility one of the primary elements in the oft-cited list of factors that has rather imprecisely become known as the Bradford Hill (1965) “criteria” for causality (discussed in more detail at the end of this chapter) but insights about biologic processes inform whether an observed pattern of statistical association might be interpreted as the product of more than error, bias, confounding, and chance. The committee used toxicologic information in that fashion and placed the information before its synthesis and conclusion to provide readers with a more coherent argument for its ultimate conclusion about the adequacy of the available evidence to support the existence of a particular association.
Associations between exposures to the chemicals of interest and specific health outcomes are determined through an analysis of available epidemiologic studies that is informed by an understanding of the toxicology of the chemicals and their exposure pathways. In reaching conclusions, VAO committees consider the nature of the exposures, the nature of the health outcomes, the populations exposed, and the quality of the evidence examined. Some specific issues that this and prior committees have considered are addressed below.
Human Studies
The committee reviewed studies of Vietnam veterans and of other populations that might have been exposed to the chemicals of interest. The other populations factored into the committee’s evaluation included cohorts of workers in chemical production and agriculture and populations residing near sites of environmental contamination. The committee believes that studies of such non-veteran subjects can help in the assessment of whether the chemicals of interest are associated with particular health outcomes. As noted above in describing the literature search, studies of nonveteran subjects were identified because one of the chemicals of interest was specified by the original researchers as presenting a possible toxic exposure rather than on the basis of occupational definitions. Some of the studies provide stronger evidence about health outcomes than do studies of veterans because exposures were measured sooner after occurrence and were more thoroughly characterized than has been the case in most studies of veterans. Furthermore, in the studies of workers in chemical-production plants, the magnitude and duration of exposure to the chemicals were generally greater, so the likelihood that any possible health consequence would be manifested was greater. The studies were often large enough to examine health risks among groups of people with different levels of exposure, so dose–response relationships could be investigated. The general practice of VAO committees has been to evaluate all studies, whether or not their subjects were Vietnam veterans, according to the same criteria in determining the strength and validity of findings. Because the subjects of studies of Vietnam veterans are the concern of the legislation that mandated the present review, however, demonstrations of increased incidence of particular health outcomes among them are of unquestionable pertinence in drawing conclusions.
The committee has concluded that it would be inappropriate to use quantitative techniques, such as meta-analysis, to combine individual study results into a single summary measure of statistical association. The committee reached that conclusion because of the many differences among studies in definitions of exposure, health outcomes considered, criteria for defining study populations, correction for confounding factors, and degree of detail in reporting results. The appropriate use of meta-analysis requires more methodologic consistency among studies, especially in the definition of exposure, than is present in the literature reviewed by the committee (Egger et al., 2002; Petitti, 2000). A detailed discussion of the results of individual studies in appropriate categories (Vietnam-veteran, occupational, or environmental exposure; and exposure to Agent Orange or equivalent dioxin-contaminated phenoxy herbicides, to dioxin, to phenoxy herbicides without dioxin contamination, to cacodylic acid, or to picloram) with a thorough examination of each study’s strengths and weaknesses is fully informative without making unfounded assumptions of homogeneity.
In general, VAO committees have not considered case reports, case series, or
other published studies that lacked control or comparison groups. An exception was made, however, for early-onset peripheral neuropathy. Individual case reports were reviewed because the rapid appearance and frequently transient nature of the condition impose methodologic constraints that might have precluded the application of standard epidemiologic techniques.
Because any effect of Agent Orange in individuals or groups of veterans is evaluated in terms of disease or medical outcome, attention to disease classification was important to the committee in assembling pertinent data from various investigations related to a particular outcome before integrating the information. The researchers who conducted the studies reviewed by the committee faced the same challenge in interpreting the available documentation when assigning diagnostic labels to given subjects and then grouping the labels for analysis.
Pathologists, clinicians, and epidemiologists use several classification systems, including the International Classification of Diseases (ICD); the International Classification of Diseases, 9th Revision (ICD-9), Clinical Modification (ICD-9-CM), and the International Classification of Diseases for Oncology. The 10th revision of ICD (ICD-10) is currently used to classify mortality information. Most of the subjects investigated in the studies cited in this update were diagnosed under earlier systems, and most of the articles report results in accordance with ICD-9 if they use ICD codes at all, so the committee has also used ICD-9. ICD codes are a hierarchic system for indicating type of disease and site. For example, ICD-9 162 specifies cancers of the lung, trachea, or bronchus; 162.2, cancer of a main bronchus; 162.3, cancer of an upper lobe; 162.4, cancer of a middle lobe, and 162.5, cancer of a lower lobe.
For a patient to receive a correct cancer diagnosis, careful staging of the extent of disease is necessary, and a biopsy of the tissue must be analyzed with microscopy, often with special immunohistochemical stains, to confirm a clinical impression. Many of the epidemiologic studies reviewed by this committee did not use the ICD approach to classification of disease and relied instead on clinical impression alone. Death-certificate diagnoses are notoriously inaccurate if the certificates are completed by medical officers who are not familiar with the decedents’ medical history (Smith Sehdev and Hutchins, 2001). Self-reported diagnoses, which are obtained from survey questionnaires, often are partially or completely inaccurate; for instance, a patient may report having been treated for stomach cancer although the correct diagnosis was gastric adenocarcinoma, gastric lymphoma, pancreatic cancer, large bowel cancer, or peritoneal cancer.
Many epidemiologic studies report disease outcome by organ system. For instance, the term digestive system may be used for conditions that are benign or malignant and that affect the esophagus, stomach, liver, pancreas, small bowel, large bowel, or rectum. Therefore, if a report indicated that a cohort has an increased incidence of digestive system cancer, it would be unclear whether the association was attributable to excess cases of esophageal, gastric, hepatic, pancreatic, or intestinal cancers or to some combination thereof. Such generalization
is complicated by the fact that the cause of cancer may differ between anatomic sites. For instance, there are strong associations between gastric cancer and Helicobacter pylori infection, between smoking and squamous cell carcinoma of the esophagus, and between chronic hepatitis B infection and hepatic cancer. Furthermore, a single site may experience a carcinogenic response to multiple agents.
The committee recognizes that outcome misclassification is a possibility when recording of a diagnosis with a specific ICD code is used as the means of entering an observation into an analysis, but this system has been refined over many decades and is virtually universally used and understood, in addition to being exhaustive and explicit. Therefore, this and previous VAO committees have opted to use the ICD system as an organizing tool. Although the groupings of cancer sites for which conclusions about association have been presented may correspond more closely to National Institute for Occupational Safety and Health or National Cancer Institute Surveillance Epidemiology and End Results categories (see Appendix B), the underlying ICD codes provide the most exactitude. In this report, ICD codes appear almost exclusively in the introductory sections of health-outcome discussions (particularly for cancers) to specify precisely what outcome the committee is addressing and, when possible, in the results table to indicate exactly what the primary researchers believed that they were investigating. (See Appendix B for cancer groupings with corresponding ICD-9 and ICD-10 codes.)
Rare diseases, such as hairy cell leukemia and tonsil cancer, are difficult to study because it is hard to accumulate enough cases to permit analysis. Often, the result is that observed cases are included in a broader, less specific category. Thus, epidemiologic data may not be available for assessing whether a particular rare disease is associated with Agent Orange exposure. In some instances, such as chronic lymphocytic leukemia and AL amyloidosis, VAO committees have reached conclusions on the basis of the data available and the etiology of the disease. Through systematic application of the hierarchic nature of the ICD coding system, committees intend to draw, for every type of cancer, an explicit conclusion about the adequacy of available evidence to support an association between herbicide exposure and that type of cancer. For nonmalignant conditions, however, the diversity of disease processes involved makes the use of broad ICD ranges less useful, but, because VAO committees could not possibly address every rare nonmalignant disease, they do not draw explicit conclusions about diseases that are not discussed. Thus, the category of “inadequate or insufficient evidence to determine an association” is the default or starting point for any health outcome; if a condition or outcome is not addressed specifically, it will be in this category.
The committee is aware of the concerns of some veterans about the role of herbicide exposure in the occurrence of multiple health outcomes, such as multiple cancers, in a given person. Little research has been done to address whether the rate of concurrence is greater than would be expected by chance.
Simultaneous analysis of multiple health outcomes could potentially provide more insight into whether the chemicals of interest cause multiple health effects, into competing risks between various health outcomes, and into the interactive effects of health outcomes; but addressing health conditions individually has remained challenging.
VAO committees wanted to be clear in indicating what evidence is factored into their conclusions. The practice in the VAO reports has been to augment the results table for a given health outcome with any additional publications considered in the current update in the categories of Vietnam-veteran, occupational, or environmental studies. Inclusion of sequential sets of results from follow-ups of a study population has the potential to create the appearance of a greater weight of evidence than is warranted, so Update 2006 and Update 2008 used italicized citations in results tables to indicate that results had been superseded. The current committee did not want to convey the notion that earlier findings were of no importance. In an effort to get a comprehensive and comprehensible picture of the history of each study population, the current committee decided to abandon the sequential entries by update that had been the format for the result tables since Update 1996. The new format adopted for the results tables is a refinement of the cohort-based approach introduced in Update 2006 for cardiovascular diseases. To facilitate the reader’s locating the discussion of the characteristics of particular study populations and the attributes of the publications based on them, the order of studies in the results tables corresponds to their presentation in Chapter 5. The main categorization of veteran, occupational, and environmental studies has been retained in both instances.
An issue related to evidence evaluation that was of concern for the Update 2006 committee was the evidence category of “no association.” That committee determined that a conclusion of no association would require substantive evidence of such a lack of effect of each of the chemicals of interest. Given the paucity of information that exists for cacodylic acid and picloram, that conclusion would seem suspect even if substantial evidence uniformly supported a finding of no association both with exposure to the phenoxy herbicides and with exposure to TCDD. The Update 2008 and current committees concurred in that determination and adopted a similar approach to the placement of health outcomes in this category.
Exposure Assessment
Much of the evidence that VAO committees have considered has been drawn from studies of populations that were not in Vietnam during the period when Agent Orange and other herbicides were used as defoliants. The most informative of those studies were well-documented investigations of occupational exposures to TCDD or specific herbicides, such as 2,4-D and 2,4,5-T. In many other studies, TCDD exposure was combined with exposures to an array of “dioxin-like”
compounds, and the herbicides were often analyzed as members of a functional class; this is less informative for the committee’s purpose than individual results on a specific compound. In the real-world situations investigated in epidemiologic studies, exposure to multiple possibly toxic chemicals is the rule rather than the exception; for example, farmers and other agricultural populations are likely to be exposed to insecticides, fungicides, and herbicides. In such studies, the committee looked for evidence of health effects that are associated with the specific compounds in the defoliants used in Vietnam and sought consideration of and adjustment for other possibly confounding exposures.
The quality of exposure information in the scientific literature reviewed by this and previous VAO committees spans a broad range. Some studies relied on interviews or questionnaires to determine the extent and frequency of exposure. Such self-reported information generally carries less weight than would more objective measures of exposure. The strength of questionnaire-based information as evidence of exposure is enhanced to the extent that the information can be corroborated or validated by other sources. Written records of chemical purchase or production can provide one type of objective information. Even more useful are scientific measurements of exposure. In some occupational studies, for example, workers wear air-sampling instruments that measure the concentration of a contaminant in each worker’s breathing zone. Measurement of chemicals or their products in biologic specimens, such as blood and urine, can provide reliable indications of exposure for specific periods. Studies that categorize exposure from well-documented environmental sources of contaminants can be useful in the identification of exposed populations, but their results may be inaccurate if people with different magnitudes of exposure are assigned to the same general category of exposure. Studies that explore environmental exposure and disease frequency in regional populations (such as states and counties) are known as ecologic studies. Most ecologic studies are considered preliminary or “hypothesis-generating” studies because they lack information on exposure and disease on an individual basis and are unable to address potential confounding factors.
Chapter 3 of this update addresses issues of exposure estimation in more detail. The agent of interest may be assessed with various degrees of specificity. For instance, any of the four herbicides in question could be individually measured, and phenoxy herbicides would be a useful broader category for 2,4,5-T and 2,4-D; but a report of findings in terms simply of “herbicides” is only on the margin of being informative, and results stated in terms of “pesticides” are too vague to be useful. For a given chemical of interest, the measure of exposure may be increasingly imprecise—for example, concentrations in target tissue, serum concentrations, cumulative exposure, possible exposure, and so on down to merely a report of service in a job or industry category. Those approaches can address complexities in specificity, duration, and intensity of exposure with various degrees of success. All may provide some information about association with a chemical of interest, but this committee has determined that investigation
of associations between an exposure of concern and most health outcomes has reached the stage where some characterizations of exposure are too nonspecific to promote insight. For health outcomes with very little evidence, a somewhat looser criterion would apply so that no possible signal of an association would be overlooked.
Animal and Mechanistic Studies
Animal models used as surrogates for the study of a human disease must reproduce, with some degree of fidelity, the manifestations of the disease in humans. However, a given effect of herbicide exposure in an animal species does not necessarily establish its occurrence in humans. In addition to possible species differences, many factors affect the ability to extrapolate from results of animal studies to health effects in humans. Animals used in experimental studies are most often exposed to purified chemicals, not to mixtures. Even if herbicide formulations or mixtures are used, the conditions of exposure might not realistically reproduce human exposures that occur in the field. Furthermore, Vietnam veterans were exposed to other agents—such as tobacco smoke, insecticides, therapeutics, drugs, diesel fumes, and alcohol—that may increase or decrease the ability of chemicals in herbicides to produce a particular adverse health outcome. Few, if any, studies either in humans or in experimental animals have examined those interactions.
As discussed in Chapter 4, TCDD is thought to be responsible for many of the toxic effects of the herbicides used in Vietnam. Attempts to establish correlations in the effects of TCDD between experimental systems and humans are particularly problematic because of known species-, sex-, and outcome-specific differences in susceptibility to TCDD toxicity. Some data indicate that humans might be more resistant than are other species to TCDD’s toxic effects (Ema et al., 1994; Moriguchi et al., 2003); other data suggest that, for some outcomes, human sensitivity could be the same as or greater than that of some experimental animals (DeVito et al., 1995). Differences in vulnerability may also be affected by variations in the rate at which TCDD is eliminated from the body (see Chapter 4 for details on the toxicokinetics of TCDD), although susceptibility is generally thought be an inherent biological response.
It is important to account for TCDD’s mode of action in considering species and strain differences. There is a consensus that most of or all the toxic effects of TCDD involve interaction with the AHR, a protein that binds TCDD and certain other aromatic hydrocarbons with high affinity. Formation of an active complex involving the intracellular receptor, the ligand (the TCDD molecule), and other proteins is followed by interaction of the activated complex with specific sites on DNA. That interaction can alter the expression of genes involved in the regulation of cellular processes. The development of mice that lack the AHR has helped to establish a definitive association between the AHR and TCDD-mediated toxicity.
The affinity of TCDD for the AHR is species- and strain-specific, and responses to binding of the receptor vary among cell types and developmental stages. In addition, genetic differences in the properties of the AHR are known in human populations, as they are in laboratory animals, so some people are at intrinsically greater or less risk for the toxic effects of TCDD.
Although studying AHR biology in transformed human cell lines minimizes the inherent error associated with species extrapolations, caution must be exercised because it is still not clear to what extent toxicity is affected by the transformation itself or by the conditions under which cell lines are cultured in vitro. Furthermore, humans have AHR with differing affinities for dioxin, and thus a single transformed human cell line will not accurately reflect the responses observed across the entire human population.
Publication Bias
Some studies are more likely to be published than others. That is the concept of publication bias, which has been documented in biomedical research (Song et al., 2000; Stern and Simes, 1997). Most commonly, bias can be introduced when studies whose hypotheses are supported by statistically significant results or that are otherwise deemed favorable by their authors are selectively submitted for publication. In addition, papers with “interesting findings” may be of more interest to journal editors and reviewers and thus be more likely to be accepted for publication after submission. Conversely, “negative” studies, in which the hypotheses being tested are not supported by the study findings, often go unpublished. Investigators employed by industry may be inhibited from submitting findings that have potential legal or economic ramifications.
Thus, conclusions about associations between exposure and outcome that are based solely on published results could be subject to bias. Despite that, the committee does not believe that its conclusions have been unduly affected by publication bias, for two reasons: the extensive publicity surrounding the possibility of health effects associated with the herbicides used in Vietnam has created considerable pressure to publish all findings on the subject, and the many published studies assembled and reviewed contain among their results the full range of possible statistical associations, from convincingly negative through indeterminate to strongly positive.
Role of Judgment
This committee’s process of reaching conclusions about statistical associations involved more than a formulaic application of quantitative procedures to the assembled evidence. First, the committee had to assess the relevance and validity of individual reports. Then, it had to evaluate the possible influences of measurement error, selection bias, confounding, and chance on the reported results.
Next, the committee integrated all the evidence within and among diverse fields of research. Finally, the conclusions drawn were based on consensus within the committee. Those aspects of the committee’s review required thoughtful consideration of alternative approaches at several points and could not be accomplished by adherence to a narrowly prescribed formula.
The realized approach, as described here, has been determined to a large extent by the nature of the exposures, of the health outcomes, and of the resulting evidence available for examination; therefore, it has evolved in the course of the work of this and previous VAO committees. The quantitative and qualitative procedures underlying this review have been made as explicit as possible, but ultimately the conclusions about association expressed in this report are based on the committee’s collective judgment. The committee has endeavored to express its judgments as clearly and precisely as the data allowed.
In delivering the committee its charge for this update, VA’s representative requested that the committee delineate, for health outcomes found to have some evidence supporting statistical association, how well each of the factors that have rather imprecisely become known as the Hill (1965) “criteria” for causality have been satisfied. Having a scientific perspective on the extent to which these factors, in addition to biologic plausibility, were met would help facilitate the Secretary in making a policy decision concerning a presumptive relationship of any new health outcome to exposure to the herbicides used by the military in Vietnam.
The committee is uniformly and strongly of the opinion that execution of a checklist of the Hill criteria would not be an appropriate approach for fulfilling its charge. The list of issues that Hill discussed are not a definitive set of factors to be addressed in evaluating whether a collection of evidence supports causality. The nine aspects of a statistical association noted by Hill (1965)—strength, consistency, specificity, temporality, biologic gradient, plausibility, coherence, experiment, and analogy—as contributing to a finding of causality vary in the importance that might be assigned to them, but none is sufficient, and only temporality (that the cause precede the effect) is necessary. Philosophers of science have established that a set of sufficient criteria for causality does not exist (Rothman and Greenland, 1998). Citing Weed and Gorelick (1996) and Holman et al. (2001), Rothman et al. (2008) noted that “epidemiologists have not agreed on a set of causal criteria or on how to apply them [emphasis in orginal]…. The typical use of causal criteria is to make a case for a position for or against causality that has been arrived at by other, unstated means.” The establishment of causality is not an absolute or discrete (or necessarily permanent) state. The Hill criteria have often been used as a point of reference in addressing the subject of causation in evaluating possible environmental harms, but even in theoretical and optimal circumstances scientists have not derived a definitive algorithm for recognizing causality. The extent to which a relationship is judged to be causal entails many subjective elements involving the universe of information considered and the weight accorded to each evidentiary component considered.
Furthermore, with regard to chronic diseases, causality is rarely limited to a single factor.
For those reasons, the present VAO committee has not adopted the suggestion to perform what would be in effect a checklist approach to distilling evidence concerning underlying causality for any observed statistical association between a human health effect and exposure to the components of the herbicides sprayed in Vietnam. The committee interprets its charge to be to summarize the scientific evidence for consideration by the Secretary, whose role is to make the policy decision of whether a contribution by herbicide exposure to the occurrence of an adverse health effect is likely enough to merit recognition as a presumptive condition.
Abell A, Juul S, Bonde JP. 2000. Time to pregnancy among female greenhouse workers. Scandinavian Journal of Work, Environment and Health 26(2):131–136.
Aposhian HV, Gurzau ES, Le XC, Gurzau A, Healy SM, Lu X, Ma M, Yip L, Zakharyan RA, Maiorino RM, Dart RC, Tircus MG, Gonzalez-Ramirez D, Morgan DL, Avram D, Aposhian MM. 2000. Occurrence of monomethylarsonous acid in urine of humans exposed to inorganic arsenic. Chemical Research in Toxicology 13(8):693–697.
Cohen S, Janicki-Deverts D, Miller GE. 2007. Psychological stress and disease. Journal of the American Medical Association 298(14):1685–1687.
Cohen SM, Arnold LL, Eldan M, Lewis AS, Beck BD. 2006. Methylated arsenicals: The implications of metabolism and carcinogenicity studies in rodents to human risk assessment. Critical Reviews in Toxicology 36(2):99–133.
Czarnota MA. 2004. Weed Control in Greenhouses: Bulletin 1246. University of Georgia, College of Agricultural and Environmental Sciences, Cooperative Extension.
DeVito MJ, Birnbaum LS, Farland WH, Gasiewicz TA. 1995. Comparisons of estimated human body burdens of dioxin-like chemicals and TCDD body burdens in experimentally exposed animals. Environmental Health Perspectives 103(9):820–831.
Egger M, Ebrahim S, Smith GD. 2002. Where now for meta-analysis? International Journal of Epidemiology 31(1):1–5.
Ema M, Ohe N, Suzuki M, Mimura J, Sogawa K, Ikawa S, Fujii-Kuriyama Y. 1994. Dioxin binding activities of polymorphic forms of mouse and human arylhydrocarbon receptors. Journal of Biological Chemistry 269(44):27337–27343.
Hansen ES, Hasle H, Lander F. 1992. A cohort study on cancer incidence among Danish gardeners. American Journal of Industrial Medicine 21(5):651–660.
Hill AB. 1965. The environment and disease: Association or causation? Proceedings of the Royal Society of Medicine 58:295–300.
Holman DJ, Arnold-Reed DE, de Klerk N, McComb C, English D. 2001. A psychometric experiment in casual inference to estimate evidential weights used by epidemiologists. Epidemiology 12(2):246–255.
_____________________
1 Throughout this report, the same alphabetic indicator after year of publication is used consistently for a given reference when there are multiple citations by the same first author in a given year. The convention of assigning the alphabetic indicators in order of citation in a given chapter is not followed.
Hsieh DA, Manski CF, McFadden D. 1985. Estimation of response probabilities from augmented retrospective observations. Journal of the American Statistical Association 80(391):651–662.
IOM (Institute of Medicine). 1994. Veterans and Agent Orange: Health Effects of Herbicides Used in Vietnam. Washington, DC: National Academy Press.
IOM. 2009. Veterans and Agent Orange: Update 2008. Washington, DC: The National Academies Press.
Kim HA, Kim EM, Park YC, Yu JY, Hong SK, Jeon SH, Park KL, Hur SJ, Heo Y. 2003. Immunotoxicological effects of Agent Orange exposure to the Vietnam War Korean veterans. Industrial Health 41(3):158-166.
Kim JS, Lim HS, Cho SI, Cheong HK, Lim MK. 2003. Impact of Agent Orange exposure among Korean Vietnam veterans. Industrial Health 41(3):149-157.
Langholz B. 2010. Case–control studies = odds ratios: Blame the retrospective model. Epidemiology 21(1):10–12.
Moriguchi T, Motohashi H, Hosoya T, Nakajima O, Takahashi S, Ohsako S, Aoki Y, Nishimura N, Tohyama C, Fujii-Kuriyama Y, Yamamoto M. 2003. Distinct response to dioxin in an arylhydrocarbon receptor (AHR)-humanized mouse. Proceedings of the National Academy of Sciences of the United States of America 100(10):5652–5657.
Neal NC. 2006. Greenhouse Weed Control: Horticultural Information Leaflet 570. North Carolina State University.
NRC (National Research Council). 1999. Arsenic in Drinking Water. Washington, DC: National Academy Press.
NRC. 2001. Arsenic in Drinking Water: 2001 Update. Washington, DC: National Academy Press.
Petitti DB. 2000. Meta-Analysis, Decision Analysis, and Cost-Effectiveness Analysis: Methods for Quantitative Synthesis in Medicine. New York: Oxford Press.
Rothman KJ, Greenland S. 1998. Modern Epidemiology (2nd edition). Philadelphia, PA: Lippincott Williams & Wilkins.
Rothman KJ, Greenland S, Lash TL. 2008. Modern Epidemiology (3rd edition). Philadelphia, PA: Lippincott Williams & Wilkins.
Smith Sehdev AE, Hutchins GM. 2001. Problems with proper completion and accuracy of the cause-of-death statement. Archives of Internal Medicine 161(2):277-284.
Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ. 2000. Publication and related biases. Health Technology Assessment 4(10):1–115.
Stern JM, Simes RJ. 1997. Publication bias: Evidence of delayed publication in a cohort study of clinical research projects. British Medical Journal 315(7109):640–645.
University of Connecticut. 2006. Greenhouse Weed Control. University of Connecticut, Storrs, Cooperative Extension System.
van den Berg M, Birnbaum L, Denison MS, De Vito M, Farland W, Feeley M, Fiedler H, Hakansson H, Hanberg A, Haws L, Rose M, Safe S, Schrenk D, Tohyama C, Tritscher A, Tuomisto J, Tysklind M, Walker N, Peterson R. 2006. The 2005 World Health Organization re-evaluation of human and mammalian toxic equivalency factors for dioxins and dioxin-like compounds. Toxicological Sciences 93(2):223–241.
Weed DL, Gorelick LS. 1996. The practice of casual inference in cancer epidemiology. Cancer Epidemiology, Biomarkers and Prevention 5:303–311.
