REVIEW OF ATSDR’S GREAT LAKES REPORT DRAFTS (LETTER REPORT)
ORGANIZATION OF THIS LETTER REPORT
A cover letter to Dr. James Stephens providing context for this report and presenting the conclusions of this report began on page 1. In this letter report, the committee presents the details of its evaluation of the Agency for Toxic Substances and Disease Registry (ATSDR) report drafts, of ATSDR’s response to review, and of the concerns of the Office of the Director of ATSDR and the Office of the Director of the Coordinating Center for Environmental Health and Injury Prevention (CCEHIP). Through that evaluation, the committee responds to the seven tasks presented in Box 1. This letter report begins with a brief discussion of the background of the drafts prepared by ATSDR. The committee then presents its overarching comments on those drafts. The committee next specifically addresses Tasks 1 and 4, evaluating the use of datasets, presentation of information, and conclusions of the drafts. That is followed sequentially by discussions of ATSDR’s responsiveness to reviewers’ comments on the 2004 draft (Task 2), the validity of ATSDR’s concerns regarding the 2007 draft (Task 3), ATSDR’s responsiveness to reviewers’ comments on the 2007 draft (Task 5), the scientific soundness of the 2008 draft (Task 6), and areas for improvement and other committee concerns (Task 7). Finally, the committee’s conclusions are summarized by task.
Statement of Task
The IOM committee will conduct a review of the “Public Health Implications of Hazardous Substances in the Twenty-Six U.S. Great Lakes AOCs” with a specific focus on the following:
BACKGROUND ON ATSDR DRAFTS EVALUATED
In 1972, the US-Canada Great Lakes Water Quality Agreement (GLWQA, 1972; amended 1978) committed the United States and Canada to restoring and maintaining the chemical, physical, and biological integrity of the Great Lakes Basin Ecosystem. The agreement reaffirmed the rights and obligations of Canada and the United States under the Boundary Waters Treaty of 1909. The International Joint Commission (IJC)1 monitors and assesses the progress under the agreement, including the protection of human health. The committee believes that investigating whether there is a relationship between contaminants and health outcomes is important for areas around the Great Lakes.
In a 2001 letter (IJC, 2001; see Appendix B), the IJC requested ATSDR’s2 assistance in “evaluating the public health implications of the presence of hazardous materials in Great Lakes Areas of Concern (AOC’s) by providing information on ATSDR's public health assessments of hazardous waste sites within these AOC’s.” An AOC is “a geographic area that fails to meet the General or Specific Objectives of the Agreement where such failure has caused or is likely to cause impairment of beneficial use or of the area’s ability to support aquatic life” (Great Lakes Water Quality Agreement, Annex 2; 1978). The letter further specified that it “would be most helpful if ATSDR could identify evaluated sites within each AOC, the Hazard Category assigned to each site, any relevant demographic information available to ATSDR concerning the populations at risk, completed exposure pathways identified, and the priority substances following these pathways.” The IJC had previously worked with Health Canada to produce a report for each of the AOCs in Canada (including the Canadian side of binational AOCs) outlining particular health outcomes in the AOCs. Those reports were summarized in a journal article (Elliott et al., 2001) and were available to the committee.
In April 2004, ATSDR prepared a draft report related to the IJC request (ATSDR, 2004).3 In that report draft, information from ATSDR’s health-evaluation documents, including the documents termed Public Health Assessments (PHAs),4 on sites both within and close to the AOCs, data from 2001 on contaminants from a national database (the Toxic Release Inventory, TRI), and summary county-level health data from the Community Health Status Reports (CHSRs) produced in 2000 by the Health Resources and Services Administration (HRSA) were compiled for each of the AOCs. The contaminant data focused on the pollutants that have been
The IJC is an independent binational organization established by the Boundary Waters Treaty of 1909. Its purpose is to help prevent and resolve disputes related to the use and quality of boundary waters and to advise Canada and the United States on related questions (IJC, 2008).
ATSDR’s mission is to serve the public by using the best science, taking responsive public health actions, and providing trusted health information to prevent harmful exposures and disease related exposures to toxic substances (http://www.atsdr.cdc.gov/).
The title of the 2004 draft was Public Health Implications of Hazardous Substances in the Twenty-Six U.S. Great Lakes Areas of Concern.
Since 1986, ATSDR has been required by law to conduct a PHA at each of the sites on the Environmental Protection Agency’s National Priorities List. The aim of these evaluations is to find out whether people are being exposed to hazardous substances and, if so, whether the exposure is harmful and should be stopped or reduced. If it is appropriate, ATSDR also conducts PHAs when petitioned by concerned individuals. PHAs are carried out by environmental and health scientists from ATSDR and from the states with which ATSDR has cooperative agreements.
designated as critical by the IJC.5 That draft was reviewed internally by ATSDR, and by the IJC, the Environmental Protection Agency (EPA), four external reviewers, and state public-health officials. Interim drafts were prepared and underwent further review. A revised draft was prepared before July 2007 (ATSDR, 2007).6 It differed from the 2004 draft in that information on some of the hazardous-waste sites discussed was updated, additional health information from individual studies cited by reviewers was added, and information from the National Pollutant Discharge Elimination System (NPDES) on the permissible release of chemicals were included.
The Office of the Director of ATSDR and the Office of the Director of CCEHIP, however, had concerns about the methods used and the conclusions drawn in the 2007 draft report, “held up the release of the report” (ATSDR, 2008a), and requested further review. In February 2008, an unofficial copy of the July 2007 draft appeared on a nongovernmental organization’s Web site. ATSDR later outlined its concerns about the report draft (ATSDR, 2008c), posted those concerns on its Web site and prepared a revised draft (ATSDR, 2008a).7 The 2008 draft is substantially different from the 2007 draft. The changes include a change in the title of the report, from Public Health Implications of Great Lakes Areas of Concern (AOC), to Selected Information on Chemical Releases within Great Lakes Counties Containing Areas of Concern (AOCs); reorganization of the subsections in the sections for each AOC county; provision of more context for and description of the methods in the report; and removal of the CHSR data. As indicated in the title of the 2008 draft, the focus is explicitly on AOC counties rather than on AOCs.
All three of those drafts, as well as reviewers’ comments and a discussion of the concerns of the Office of the Director of ATSDR and the Office of the Director of CCEHIP are now posted on ATSDR’s Great Lakes Web site (http://www.atsdr.cdc.gov/grtlakes/) under “Reports and Publications.” The timeline of events is summarized in Box 2.
OVERARCHING COMMENTS ON THE DRAFT REPORTS
In this section, the committee presents an overarching evaluation of the ATSDR drafts, beginning with comments on the purpose of the drafts, followed by a discussion of the scope of the drafts, including the scope of the literature reviewed by ATSDR in the drafts. The discussion below in part addresses the task to the committee related to the scientific quality of the documents (see Tasks 1 and 2) and the scientific soundness of the effort (Task 6) in that it
provides recommendations for developing clear statements of purpose and a sufficient rationale for the selection of data sources. Those steps are necessary to conduct research of good scientific quality.
Purpose of the Report Drafts
There is no well-defined, consistent statement of the objectives or purpose of the drafts. The apparent purpose also changed from one draft to another. The executive summary of the 2007 draft states that “it should be viewed as an assessment to identify the co-occurrence of elevated patterns of morbidity and mortality and environmental contamination that may merit further hypothesis-based epidemiologic study.” The introduction is less specific with respect to the stated aims: “in addition to evaluating information on the ATSDR public health assessments and other assessments for hazardous waste sites within the 26 U.S. AOCs, this document evaluates data on industrial sources of chemical emissions and on county-wide health outcomes, in order to provide a fuller perspective of potential impacts on environmental burdens and public health.” Throughout the draft the health data are discussed in the context of the chemical data; this leads the reader to infer that the purpose of the report is to present associations between the chemical and health data.
In the 2008 draft, the CHSR data are removed, and this changes the draft’s potential focus and purpose relative to the 2007 draft, inasmuch as health outcomes are no longer examined. Even so, the overall purpose remains unclear in the 2008 draft. The statement that most closely indicates a purpose in the executive summary of the 2008 draft is that “in response to the IJC request, this report summarizes previously-published health evaluation documents and chemical release information for the 26 U.S. AOCs and 54 counties that are in close geographic proximity to those AOCs.” The executive summary also states that “this report emphasizes the critical pollutants (within the constraints imposed by using existing data) but also presents information on other pollutants, when such information is available and relevant.” Although those statements delineate what is included, they do not indicate why those chemicals and datasets were selected, nor do they indicate the overall goal or purpose of the draft.
When considering the possible purposes of the 2008 draft in the context of whether it is responsive to the request by the IJC, the committee judged that the initial request from the IJC could be interpreted in two ways. One interpretation could focus on the more general request of the IJC (see Appendix B), which states, “The purpose of this letter is to request ATSDR’s assistance in evaluating the public health implications of the presence of hazardous materials in Great Lakes Areas of Concern (AOC’s) by providing information on ATSDR’s public health assessments of hazardous waste sites within these AOCs.” The second interpretation of the task appears to be more specific stating “that ATSDR provide to the Commission information on public health assessments that it has conducted on hazardous waste sites located within any of the 33 United States AOC’s. It would be most helpful if ATSDR could identify evaluated sites within each AOC, the Hazard Category assigned to each site, any relevant demographic information available to ATSDR concerning the populations at risk, completed exposure pathways identified, and the priority substances following these pathways.” According to the second interpretation, the agency could simply have provided the IJC with a summary of specific facts associated with the PHAs, or even just a compilation of the PHAs themselves, to allow the IJC to consider health implications. For that interpretation, the 2008 draft is reasonably adequate, but that straightforward approach arguably might not have been very helpful to the IJC. ATSDR could have responded that the PHAs in general do not and cannot provide information on the “hazards posed by the continuing presence of hazardous materials in the AOCs,” in that the PHAs aim to evaluate the hazards posed by specific waste sites, which generally do not coincide
with the AOCs.8 Even when the geographic region covered by a PHA is the same as an AOC, the information included in the PHA might not capture all potential environmental concerns or potential health effects in the AOC. A PHA could identify potential or observed direct effects of contaminants on human populations (through direct or indirect exposures) but miss the indirect effects of contaminants in AOCs, such as a reduction in species diversity. Such indirect effects may have implications for human health that are much larger than the direct effects (see Box 3 for an example). A discussion of the use and limitations of the PHAs in the context of the objectives of the drafts could have clarified why they were chosen.
Example of an Indirect Effect
A major effect of human activities in and around Saginaw Bay was the collapse of the extensive walleye fishery. That resulted in the removal of a whole industry from the Saginaw Bay, which probably had a larger, albeit indirect, human health effect than did direct contamination of the remaining fish. There is circumstantial evidence that the collapse was due primarily to increased turbidity that smothered eggs on the spawning reefs (Schneider, 1977).
Alternatively, the IJC request could be interpreted as having a larger scope of assisting the IJC in considering health implications. For that purpose, a larger, phased effort would be needed. The effort would have to be more involved than what was conducted for the draft reports, moving beyond such constraints such as using only the IJC critical pollutants, considering only a specific set of hazardous-waste sites (such as those for which there are PHAs), and incorporating only TRI releases and NPDES discharges for contaminant-release data. Even if the goal were to tease out which health effects were attributable specifically to the critical pollutants, it would be difficult or impossible to assign significance to those specific environmental pollutants, even if they were fully characterized, without considering other chemical exposures and other potential confounding influences.
Scope of Project and Literature Review
The scope of the project—including discussion of ATSDR’s interpretation of the IJC request, the datasets that were considered to address the request, the criteria for selecting particular datasets for the report, and any limitations imposed in the context of the request—is not presented in the 2007 draft. In the 2008 draft, improvements include better characterization of the data sources used (e.g., what is and is not included in the TRI), acknowledgment of fundamental data limitations (such as gaps in whether, when, and how people might have been exposed), and definition of technical terms (such as Beneficial Use Impairments, completed exposure pathways, and critical pollutants). Some problems still exist in the description of the scope of work, however. The 2008 draft’s preface states that “in assembling this report, ATSDR scientists surveyed many sources of data on environmental exposures and human health. Ultimately, four kinds of environmental data were included in the report.” However, the criteria for the inclusion or exclusion of datasets are not specified, and no datasets other than those used in the 2008 draft are discussed. A discussion of the breadth of data considered, and reasons for
inclusion or exclusion, would not only guide such decisions in the study, but add to the value of the draft as a compendium.
The rationale for inclusion of some hazardous-waste sites and exclusion of others is not presented in the 2007 draft, and although it is better explained in the 2008 draft, some data modifications made between the 2007 and 2008 drafts raise questions about the methods used in both drafts (e.g., for selecting hazardous-waste sites). For example, the criteria for adding and removing hazardous-waste sites between the two drafts are not evident or not clearly followed (see Box 4 for an example).
Example of the Addition or Removal of Hazardous-Waste Sites Without Following the Criteria
Appendix 2 states that the Lower Ecorse Creek Dump hazardous waste site is excluded from the 2008 draft despite its appearing to satisfy the criteria for inclusion in Section 1.4.
Neither the 2007 nor 2008 draft describes literature searches or the criteria or rationale for determining which articles are relevant and included. Although the drafts contain a reference section (2007 draft) or a bibliography (2008 draft) that refers to the data sources and lists papers related to Great Lakes research on contaminants and health, the papers listed are not explicitly cited in the text. Few public-health outcome studies are mentioned in the text (for example, the New York State Department of Health 1986 cancer-incidence investigation at Pollution Abatement Services), and the reasons for the selection of studies to discuss are not presented. Nowhere are the parameters of a literature search outlined. Furthermore, the present committee is aware of several studies conducted on populations living around the Great Lakes that are neither cited in the text nor listed in the reference or bibliography section (e.g., Weisskopf et al., 2005; McElroy et al., 2004; Clayton et al., 2002). There is also a large body of literature related to the potential health effects of the critical pollutants (for example, from occupational and population-based studies outside of the Great Lakes), but there is no evidence that those types of studies were considered by ATSDR despite their utility in determining the biological plausibility of effects. These drafts would have been improved by the inclusion of a clearly referenced background section describing previous work around the Great Lakes and support for the concerns about specific pollutants and their potential health effects with documentation of the search and selection strategies used.
APPROPRIATENESS OF DATASETS, SCIENTIFIC QUALITY OF DATA ANALYSIS AND PRESENTATION, AND CONCLUSIONS
In this section, the committee evaluates the 2007 and 2008 drafts in relation to Tasks 1 and 4 (see Box 1), including the appropriateness of the datasets used, the scientific quality of the data analysis and presentation, and the validity of the conclusions drawn.
Both the 2007 and 2008 drafts include contaminant-related data from a variety of sources, including 2001 data from the EPA’s TRI and 2004 data from the NPDES. The drafts also cite a variety of public health evaluation documents, including exposure investigation reports, health consultations and PHAs. The 2007 draft cites the ATSDR’s Hazardous Substance Release and Health Effects Database (HazDat), but HazDat does not appear to be cited in the text of the 2008
draft. The committee had concerns about the lack of delineated criteria for choosing the datasets, the relevance of the contaminant data to exposures (i.e., the lack of or uncertainty regarding completed exposure pathways, and lack of exposure-relevant environmental contaminant concentrations), temporal issues, and other issues related to the data. Those concerns are discussed below.
Selection of Contaminant-Related Datasets
The reasons for including or excluding contaminant data are not delineated in the 2007 or 2008 draft. For example, no rationale is given for the use of only 2001 data from the TRI and only 2004 data from the NPDES.
There is no evidence that available environmental data other than from the TRI or NPDES were considered. Other geographically specific sources of data on environmental contamination that may affect the AOCs or expose humans and that could be explored for use include the National Health and Nutrition Examination Surveys, fish and fishing surveys, EPA Air Quality System Data, data collected by or available to state agencies (which would have been particularly relevant to the assessment), information from the Emergency Planning and Community Right-to-Know Act9, the National Response Center Web-based query tool (http://www.nrc.uscg.mil/foia.html) and, presumably, Great Lakes monitoring efforts conducted by EPA’s Great Lakes National Program Office, which have yielded information on fish and water contaminants (http://www.epa.gov/glnpo/monitor.html). If ATSDR undertakes a similar activity in the future, it should catalog the data available, and justify the inclusion and exclusion of data.
Nonpoint pollution sources10 are not considered in the drafts. Such sources could be larger than the NPDES discharges, particularly for the IJC-critical pollutants, and the relative contribution of the different sources to exposures in AOCs, AOC counties, or the Great Lakes Basin as a whole (depending on the area of study) might be important.
The 2007 and 2008 drafts also exclude discussion of TRI information from offsite releases in the text (it is, however, included in the tables) and draw no distinction between the various types of disposal methods reported in the TRI database (for example, to Resource Conservation and Recovery Act C landfill versus land treatment versus other land disposal), despite the fact that the different methods have different implications for human exposure and risk.
Finally, although the drafts focus on IJC-critical pollutants, some sections, such as the concluding section, include discussion of other contaminants. ATSDR should discuss the criteria and rationale for inclusion and exclusion of other pollutants and the relation of such criteria to the overall objectives of the drafts, and follow the criteria consistently. As discussed below (in “Health Data”) non-IJC-critical chemicals that are known to be associated with a given disease must also be considered in looking at health effects.
Section 304 of the act, “Emergency Notifications,” captures information on accidental spills and releases that could result in human exposure.
A nonpoint pollution source is one whose releases cannot be traced back to a single origin or source. For example, a smokestack is a point pollution source, whereas septic systems and farm runoff are nonpoint pollution sources.
Completed Exposure Pathways
Despite criticisms by reviewers of the 2004 draft, the 2007 and 2008 drafts contain no general framework that would link environmental releases with resulting environmental concentrations, and with human exposures leading to risk of disease. In the 2008 draft ATSDR provides more clarity about the requirements for a completed exposure pathway.11 Under most circumstances, however, the overall implication of those requirements is the difficulty in ruling out or in the possibility of completed exposure pathways once a chemical release has occurred. Furthermore, without quantitative estimates of exposure it is impossible to assess human health implications. This underscores the importance of environmental concentration data that are relevant to human exposures (according to chemical, location, time, environmental medium), if human health risks are to be assessed.
For most of the site-specific discussions, “completed exposure pathways” are not specified, nor are populations at risk for exposure to AOC-related contamination identified. Instead, the drafts are focused on theoretical pathways and do not address the more relevant question of whether anyone is actually exposed or exposed in any important way. Although the lack of information regarding whether completed exposure pathways exist is acknowledged in Section 7.1 of the 2008 draft—,
The available data do not indicate whether people are actually exposed. For exposure to occur there needs to be a completed pathway from a source to people’s bodies. Discharge of a pollutant into a stream (as indicated in NPDES) does not mean that people are exposed to that pollutant, or if so, how much. Use of a chemical in a factory, as reported in TRI, does not mean that people are exposed to the chemical, or if so, to what extent.
—the implications of this limitation are not taken into account in the report as a whole.
The nature of the contaminant data relative to the AOCs is such that the contaminant data used in the drafts do not necessarily indicate potential human exposures within or from the AOCs themselves, within only the adjacent counties, or within the Great Lakes Basin as a whole. For chemical discharges in or from the AOCs to result in exposures, the released materials must be transported to potentially exposed populations. The TRI and NPDES data are included in both the 2007 and 2008 drafts without consideration of transport to (or out of) the AOCs or AOC counties. Such transport might not occur to a great extent if the release is downwind or downstream of the AOC (or in another watershed, as occurs for some of the waste sites for which PHAs are included in the drafts), or if the release would not be transported over long distances. Conversely, there may be releases in counties that are not adjacent to an AOC that could have a
In the 2008 draft, ATSDR outlines the requirements for a completed exposure pathway. Conversely, on the basis of those requirements, for there not to be a completed pathway, any one of the following characteristics would need to be demonstrated: (1) no possibility of environmental movement, or complete and rapid environmental degradation, such that the “released” contaminant is effectively immobilized; (2) lack of an exposure point (no possibility of public access to the contamination); (3) lack of an exposure route (no scenario in which the public might inhale, ingest, absorb through the skin, or otherwise experience exposure); and (4) lack of a potentially exposed population (there is no likelihood that people could get to or be at exposure points). Demonstrating those characteristics requires environmental sampling and access barriers (100 percent compliance could not be assumed), and unlikely exposure scenarios would need to be considered.
greater effect than releases within an AOC county (for example, if they are upwind or in the watershed of the AOC, and the released materials can be transported over sufficient distances).
Assessment of environmental transport and potential exposures as part of establishing a completed exposure pathway requires environmental concentrations of chemical agents (e.g., in air, water, and fish) near populations, whereas release information is typically in terms of total mass or mass per unit time and independent of populations. The geographic information system maps in the 2007 draft provide some information on subsets of the population within the AOC counties, but the 2008 draft explicitly states that “we did not provide general demographic data for AOC counties because existing data do not allow us to determine which populations might have some risk of exposure to the toxic substances discussed in this report.”12
The use of ATSDR’s PHAs for information on the geographic regions discussed in the 2007 and 2008 drafts also creates difficulties in having completed exposure pathways. ATSDR’s PHAs are developed with the purpose of addressing potential concerns for people who live close to specific waste sites. In contrast, evaluations relevant to the AOCs, AOC counties or the Great Lakes Basin would require consideration of all those who could be affected by the polluted areas under consideration, which are not limited to the waste sites evaluated in PHAs.
Potential temporal issues are also related to the contaminant data used. The TRI and NPDES databases are restricted to data on controlled or permitted releases of contaminants, and the TRI and NPDES data included in the ATSDR drafts are for recent activity (e.g., 2001 for the TRI and 2004 for the NPDES). In contrast, many of the hazardous-waste sites discussed in the drafts are the result of past uncontrolled spills or leaks or are due to historical waste-disposal practices that resulted in much higher levels of environmental contamination than would occur from current disposal practices. Current allowable releases might not be representative of the actual quantities of chemicals present, let alone the potential exposures from the sites. For example, it would be important to know where there are local areas with elevated chemical levels because of residual contamination or continuing releases that have not been sufficiently remediated. Moreover, given that historical levels of contamination may have been greater than current contamination levels, it would be helpful to discuss time trends, such as whether contamination is decreasing (as is the case, for example, with polychlorinated biphenyls).
The temporal issues are of even greater concern when the contaminant data are presented with the health data in the 2007 draft. The relevance of the 2001 TRI and 2004 NPDES data to the chosen health data (2000 CHSR) in the 2007 draft is questionable in that the health-outcome information preceded the releases. If the more recent contaminant releases are representative of releases that occurred before manifestation of the health outcomes, they might be relevant, but no evidence to support such a conclusion is presented. For example, if modeling techniques or information on half-lives indicates that current levels adequately estimate past levels (sufficiently earlier than disease to account for latency), the current levels might be relevant to disease outcomes. The selection criteria for contaminant data should be discussed in that context; and the same context should be considered in selection of contaminant data even for the 2008 draft
which omits health effects. A suitable, clearly delineated approach to the task would have highlighted the problem.
Other Contaminant-Related Data Issues
In discussing potential health effects, reporting the TRI and NPDES data in total pounds released without considering the potency of the various chemicals to cause health effects is not useful; for example, a pound of benzene is not the same as a pound of ethylbenzene in terms of toxicity. It would be much more helpful to report potency-weighted releases, for example using as a basis the reference doses and cancer potency factors reported in EPA’s Integrated Risk Information System database, which provide an estimate of the relationship between the concentration of a chemical and the likelihood of disease.
In the 2007 draft, ATSDR presents HazDat information that includes the number of records exceeding health-based screening values in specified media (e.g., air, soil, water). Those data, however, are not presented clearly,13 and the 2008 draft does not include them. The information in the tables (e.g., Table 2.2-B) that list site contaminants exceeding health-based screening levels in particular environmental media could be useful, if presented clearly, for sites where complete exposure pathways for contaminants have been established.
Criteria for inclusion or exclusion of demographic data are not included in the drafts. As discussed above, the reasons for inclusion or exclusion of data would not only guide the decisions but add to the value of any such report as a compendium.
When considering demographic information, ATSDR does not assess the time period when the data were collected or explain the relevance of the data. For example, the demographic information for the Newstead Site includes the statement that “two adults and two children under 5 years of age formerly resided on the site. The area is relatively rural, but there are some neighbors.” Who formerly resided on the site is of questionable relevance for evaluating current potential exposures. Although ethnicity is often important to consider in exploring health outcomes, the 2008 draft which does not focus on health outcomes, does not outline its importance. For example, it states that “according to 2000 U. S. Census Bureau data, approximately 16,500 persons live in the Village of Depew. Of those 16,500, 98.7% are Caucasian, with less than 1 percent each African-American, Native American, Asian, multiracial, Hispanic, and classified as other.” Such demographic data may be useful in discussing health effects in susceptible populations (if those are ethnically related), but appear to be largely irrelevant to the 2008 draft.
The two drafts also refer to vulnerable populations without explaining why they are considered vulnerable. “Vulnerable” populations are defined in the 2007 draft as children under 6 years of age, women of reproductive age, and older adults living within 1 mile of a hazardous-waste site. The draft states that those populations “have been shown to be vulnerable populations for health effects associated with exposure to contaminants found in contaminated Great Lakes fish.” The 2008 draft simply refers to those populations as “clearly vulnerable.” The committee
acknowledges that these are generally accepted vulnerable populations; however, the statements are not referenced, and the basis on which these populations are considered vulnerable is not stated (e.g., whether they are susceptible because of greater exposure, such as that due to food-consumption patterns of fish, or because of personal characteristics, such as pregnancy or age). Different populations could be vulnerable to effects of different chemicals, depending on the chemical characteristics, and the specific health end points being considered. For example, some populations might be vulnerable to the effects of some chemicals because of fish consumption but not to other chemicals, such as polycyclic aromatic hydrocarbons, that do not result in exposure through fish consumption.
The ATSDR 2007 draft uses the HRSA’s CHSR as indicators of health outcomes related to the AOCs. Those reports were derived largely from vital-statistics data, specifically, births and deaths. Thus, the health outcomes in the report are compiled at the county level and are limited to morbidity associated with birth outcomes (e.g., low birth weight) and mortality rates from infant deaths and from such diseases as cancer, and information from birth records such as access to prenatal care and age of mother at birth. The data for a given county can be compared with those for peer counties and national statistics. Peer counties are other counties (and county-like geographic areas; usually 20 or more) that are similar to the counties of interest in population size and density, poverty and age structure (ATSDR, 2007).
The 2007 draft lists the selected community-health indicators and states the data source, but the dates, sources, geographic boundaries, and relevance of the data are not explored further. Information about the methods used by HRSA to collect and analyze the CHSR data, including how the peer counties were selected and their appropriateness for the 2007 ATSDR draft is lacking. For example, the basis of selection of the peer counties does not control for many of the potential confounders relevant to the draft, such as migration, smoking, and the presence of environmental exposures in the peer counties.
For each AOC county in the 2007 draft, health-outcome measures are reported if they exceed the median for the peer-county group and highlighted (with boldface type) if they exceed the 90th percentile of the peer-county group, that is, the “upper limit of the peer county range.” The appropriate HRSA terminology, however, is not consistently used or defined in the draft, and this creates confusion.14 Regardless of the confusion, the utility of identifying indicators above the median as indicative of “elevated” rates is questionable in the context of the 2007 draft, even for hypothesis generation, because half the peer counties (which would be considered “unexposed” counties for use in the 2007 draft) would be expected to fall above the median for any given health endpoint.
The term upper limit of the peer county range sometimes appears in the ATSDR draft as upper end of the peer county range or simple peer county range (as in “none exceeded the peer county range”). Although peer county range was taken directly from the HRSA source documents, it is used differently in the ATSDR draft and the HRSA documents. In the HRSA source documents the phrase is defined by a footnote that states that “eighty percent of the peer group values fall within this range.” Chapter 7 of the 2007 ATSDR report states that values above the “upper limit of the peer county group” were supposed to “exceed the upper limit of the 90% confidence limit of the median for the peer county range.” The committee was unsure how to interpret that statement despite its examination of the HRSA source documents.
As with the contaminant data, the rationale and criteria for choosing the CHSR data are not provided. The 2007 draft states that data “were utilized because of their availability” but does not state what alternatives were considered and does not discuss other potentially relevant datasets. The 2008 draft also makes strong statements regarding the available health data without a detailed description of what sources were explored: “Except as noted in the context of ATSDR health evaluation documents, no currently available health data meet these needs; thus this report does not include other health data.” As outlined above in the section “Overarching Comments on the Draft Reports,” the draft should indicate which datasets were considered, and reasons for their inclusion and the exclusion of others.
The CHSR data have many useful applications but for the reasons outlined previously and below, the committee considers that they are not used appropriately by ATSDR in its 2007 draft. The Canadian study (Elliott et al., 2001) cited previously relied on morbidity data derived from hospital-discharge abstracts. The United States lacks national or state-wide hospital data systems like the province-wide systems in Canada, but some states, such as New York, have comparable data that are available from a state-wide hospital discharge abstract system,15 and others (Ohio,16 Pennsylvania,17 Illinois,18 and Michigan19) appear to have some capacity to generate such data. Moreover, Medicare is a national system that has hospital-discharge abstract data on people over 65 years old that could have been considered.
In addition, there is poor correspondence between the AOCs and county boundaries. AOCs may be small parts of counties (e.g., Ashtabula River AOC, see Appendix Figure A-2) or span county boundaries (e.g., Clinton River AOC, see Appendix Figure A-3), so the overall county data may bear little relation to the population being exposed to contaminants associated with the AOC. Thus, any estimate of the effect of exposure may be diluted by the larger numbers of those who do not live closer to or in an AOC. Furthermore, the county data are cross-sectional and fail to account for duration of residence in a county, so that the relative risk of exposure and the ability to detect outcomes in short-term residents versus longer-term residents are unknown. Use of the CHSR data is further complicated by the fact that many of the health outcomes of interest require years to develop after exposure to a risk factor. A closer fit between the AOCs and the health and other related data might have been achieved if the data were compiled on the level of census tracts, rather than the counties. More specific data, however, would not offset the fact that the data are cross-sectional and ecologic, and resulting limitations would have to be discussed.
As important as the dataset used are the biologic specificity and plausibility of health effects reported, given the exposures of interest and other possible confounding exposures. With respect to biologic specificity, many chemicals have been shown to be associated with the same health outcomes. For example, lung cancer is associated with exposure to chromium and arsenic, as well as with smoking; therefore, in examining rates of lung cancer it is important to acknowledge the potential for other confounders or underlying contributors, not just a subset of chemical pollutants, such as the pollutants considered critical by the IJC.
With respect to biologic plausibility, health indicators with no plausible direct relationship to contaminant exposure (such as being a teen mother, an older mother, or an unmarried mother; homicide; and lack of first-trimester prenatal care) have no clear role in the assessment. Those indicators have not been shown to be, nor would they be expected to be, directly linked to specific chemical exposures and they should not be taken into consideration.20
The CHSRs data were removed in preparation of the 2008 draft presumably because of concerns regarding health data expressed in the letter of scientific concerns (ATSDR, 2008c) and the executive summary of the 2008 draft, including lack of a biologically plausible connection between release and some health indicators, asynchrony between the release data and the health indicator databases, and dilution of health-indicator data by the inclusion of large unexposed populations. The committee agrees that the use of the health data in the 2007 draft raised many issues, but removal of any reference to those data dramatically changes the focus and potential uses of the draft.
The only health data remaining in the 2008 draft are derived from community health assessments and health studies, including PHAs previously completed by ATSDR, or in some cases from equivalent studies completed by state health departments. These data reflect the exposures and outcomes at the studied sites but cannot be easily compared with the experiences of other communities and in many cases are outdated. With respect to the health evaluations, the draft noted that “if conditions at a site differ from those described in the most recent ATSDR site assessment included here, this report’s listed hazard category may not reflect current conditions” but continued to include and discuss the conclusions of the evaluations in the present tense. Some past environmental contamination could be relevant or of interest, such as those associated with health outcomes of concern with a long latency period or affected by contaminants with a long half-life, and therefore it is reasonable that those documents be included in the report. The text, however, should clearly indicate that the data on environmental contamination refer to prior conditions. It is also important to summarize the current remediation status because it is typically more relevant than whether a site had posed an urgent public-health risk at some time in the past.
No statistical analysis of associations between exposures and health outcomes was conducted for the 2007 draft. Rather, the draft contains a descriptive analysis with contaminant and health data presented together. Despite the caveats in the draft, such a reporting of co-occurrence of “elevated” morbidity and mortality with environmental contamination is likely to be misinterpreted as evidence of an association, or even as evidence of a causal link. For example, although the 2007 draft does state in some sections that no causal inferences or associations are made, causal association is inappropriately implied in such text as the following:
According to analysis of ATSDR’s HazDat database for 2003, there were over 15,000 instances where contaminants of concern were found at levels above health–based screening values in a variety of media (i.e., water, air, and soil). While no causal inferences or associations are made in this report of the 26 AOCs, elevated rates were observed for infant mortality in 21 AOCs, low birth weight in 6 AOCs, and premature births in 4 AOCs.
Elevated cancer mortality was also seen for breast cancer in 17 AOCs, colon cancer in 16 AOCs, and lung cancer in 12 AOCS (see Table 7.2).
Increased disease rates alone cannot be considered evidence that living near a waste site is the cause of the occurrence of a specific disease and should be used only in an initial step in determining the nature, extent, and implications of disease in the community around a site. Issues important to the question of association or causation that are not adequately addressed in the 2007 draft report include population-relevant exposure levels, the relationship between exposure and observed effect, and possible confounders.
The committee agrees that investigating whether there is a relationship between contaminants and health outcomes is important for areas around the Great Lakes. A number of methods and tools are available for such an evaluation, and the report draft does not explain why only the descriptive analysis was used. For example, exposure-modeling methods are available to estimate exposures, and risk-assessment methods to estimate potential health effects. The results of drinking-water sampling, air sampling, river sampling, or biomonitoring could have been explored to assess human exposures better. An epidemiologic study design could also inform whether there is enough power to detect plausible associations between exposure to contaminants and adverse health effects in the AOC or nearby populations.
The 2007 draft and to a lesser extent the 2008 draft report also suffer from the differences between the geographic units captured by the different datasets and the areas studied. As stated previously, the AOCs are typically relatively small regions in the Great Lakes Basin that are defined by criteria of ecologic degradation or impairment of beneficial use. The AOCs appear to be primary units of study in the 2007 draft, although there is no clear statement of this and all the compiled data are on the adjacent counties. The draft does not explain the selection of AOC-adjacent counties as the units of study (that is, how the county-level data would allow evaluation of AOCs, as requested by the IJC). The data in the 2007 and 2008 drafts offer no means of identifying the pollutants now present in the AOCs. Given both the geography of the regions of interest and the misaligned nature of existing health and contaminant datasets in the United States, it is possible that there are insufficient data with enough specificity to the AOCs to respond to the IJC’s specific request. Early identification of such limitations might have led to a re-formulation of the problem.
Presentation of the Information
The committee was asked to assess the presentation of the information in the 2007 and 2008 drafts.
In the 2007 draft, the tone, especially of the executive summary, is inappropriately alarmist, as was noted by the Offices of the Director of ATSDR and CCEHIP and by reviewers. Examples of potentially alarmist sections in the 2007 draft report include the following:
The executive summary states that there are “over 15,000 instances where contaminants of concern were found at levels above health-based screening values,” and that “there are 25 AOCs which have not been remediated and that over 9 million residents are living within these AOC counties.” Such statements and claims could be read to imply that 9 million residents are at risk for exposure to contaminants that could have adverse health outcomes.
Chapter 7 (Section 7.4) states that “no causal inference or associations are made in this report,” but this statement is followed by lists of putatively increased rates.
The large number of AOC counties inappropriately labeled as having public-health problems could cause inappropriate public concern. For example, Table 7.2 contains a list of “elevated” morbidity and mortality in the counties corresponding to the 26 AOCs, including infant mortality which was “elevated” essentially everywhere. However, “elevated” is not defined or footnoted in the table except for a statement that the “limitations of the report are in the text”. Furthermore, “elevated” cancer mortality (from breast, colon, and lung cancer) is noted for twelve to seventeen AOCs without indicating which data were used to come to this conclusion. A footnote also states that “blank cells indicate that the morbidity or mortality condition was not reported,” which could be interpreted to mean that every reported entry corresponds to an increased rate.
Although many of the limitations in presentation are improved in the 2008 draft, some problems remain. Overall, the organization of the 2008 draft is such that determining potential contaminant issues for any particular site is difficult, as is determining how many hazardous-waste sites represent urgent public-health hazards or how many sites have been remediated. In future similar projects, a summary table highlighting sites where a potential for exposure still exists (accounting for all remediation activities, not just obtaining information from ATSDR’s most recent public health evaluation) would be helpful.
As mentioned in some of the previous sections, the committee identified inconsistencies in the individual drafts, such as inconsistency in which data are included according to the introduction and which data are actually included in later sections of the draft; or, in the case of the 2008 draft, inclusion of demographic data on some sites but not others.
Finally, the reasons for and context of the preparation of both the 2007 and 2008 drafts are not well described. That could be accomplished in a preface, but it is important for the tone of a preface to be consistent with the nature and certainty of the conclusions reached.
Conclusions of Report Drafts
This section summarizes the committee’s views on the conclusions of the 2007 and 2008 report drafts, in response to Tasks 1 and 4 of the charge to the committee. As detailed below, many of the conclusions in the 2007 draft and 2008 draft reports were either not clearly stated or, as in the executive summary, overstated without adequate support in the body of the report, especially given the uncertainties and limitations of the way the data were used. In the 2008 draft, the conclusions presented in the executive summary differ from those presented in the conclusions section. In addition, some conclusions that could have been drawn on the basis of the evidence in the draft and would have been useful to the reader were not presented.
Although many of the uncertainties (such as, a lack of information about the degree of contamination, human exposure pathways, populations at risk, and public-health outcomes) that limit the value of ATSDR’s health evaluations in addressing population health in AOC communities are acknowledged in the 2007 draft report, there is no discussion of potential strategies to overcome those uncertainties, and those uncertainties are not taken into account when conclusions are drawn. For example, the 2007 draft lists substantial uncertainties (exposure was not well characterized, demographic data were not available, health-outcome data were not available, vulnerable populations were not discussed, and no public-health data were reported)
regarding many of the hazardous-waste sites discussed, but in the conclusions the sites are considered to be of concern without any accounting of the uncertainty of those assessments.
Problems previously discussed regarding health-outcome data are amplified in the conclusions section of the 2007 draft. For example, the description of the health-outcome data implies that exceeding the 90 percent confidence limit for median health-indicator rates was generally used as a threshold but, in fact, the median was generally used. Having conclusions that report the number of AOCs with “elevated rates” of health outcomes (such as infant mortality, low birth weight, and cause-specific cancer) is not appropriate.
In addition, in the conclusions of the 2007 draft, the discussion of limitations focuses on the potential for underestimating human health risk due to the data sources and the expanded geographic boundaries. Because the report does not estimate risks, and given the other limitations of the report, any discussion of risk estimates is inappropriate. However, if there is such a discussion it should also include the potential to overestimate risks. For example, failure to consider the potential for over-estimating risks by including pollution sources that may not have an impact in the AOC or by reporting health indicators that are not strongly linked to pollutants in the AOC would be an important flaw.
The executive summary of the 2008 draft lists five conclusions regarding the current status of the Great Lakes region, the limitation of the available data, and the need for better data analysis. These conclusions do not appear elsewhere in the report and are not tied to supporting evidence. Thus, although those conclusions might be true (the present committee was not charged to assess that), the 2008 draft does not provide a basis for determining their validity in that it does not adequately address the current status of the region, document the data that are available or conduct any data analysis.
For example, although the first conclusion that there is evidence of pollution in the Great Lakes region is well documented elsewhere, the report does not cite evidence on the extent of pollution in the Great Lakes region, and its conclusions do not take into account remediation actions that have been undertaken. The second conclusion is that the available information on environmental pollution in the Great Lakes region is “limited and incomplete.” That might be considered always true for any area, but the systematic survey of existing data that would support such a statement is not present in the 2008 draft. Similarly, a systematic search of available data and analytical techniques would be needed to support the third conclusion, that available information on environmental pollution provides little insight on people’s exposure to pollutants; the fourth conclusion, that available health data are not well matched to the exposure data; and the fifth conclusion, that it is currently impossible to define “the threat to human health from critical pollutants” found in the Great Lakes Basin. Since the report does not include human health data or exposure data, these three conclusions are beyond the scope of the 2008 report.
Other conclusions in the report state that the 2008 draft highlights or supports the need for better data or data analysis (see Box 5 for an example), but the text does not justify such statements because it does not survey the data or data analyses that are available. Similarly, statements are made in the recommendations section that are not justified in the report (see Box 6 for an example).
Example of Conclusion That Was Not Justified by Report Contents
The executive summary states "this report serves to highlight the pressing need for better data, properly collected, organized, and analyzed, to help define threats to human health and optimal strategies for protecting health.”
Example of Recommendation That Was Not Justified by Report Contents
The Recommendations section contains the further statements that “ATSDR strongly supports the need … to help elucidate the links between chemical contamination and health effects. Existing efforts at ATSDR and CDC … represent important steps toward those goals. This report suggests that further such efforts are well justified.” However, the report neither analyzes those “existing efforts” nor identifies any “links between chemical contamination and health effects.”
ATSDR missed the opportunity to draw conclusions from its assessment that were documented in the body of the draft but not presented as conclusions. For instance, one conclusion is that fish contaminant levels in many AOCs continue to exceed health-based values on the basis of documented restrictions on fish consumption (although some of the restrictions are due to lake-wide restrictions, rather than being specific to AOCs). In addition, ATSDR could have documented the number of sites that were delisted from the National Priority List or whose remediation had been completed or is still needed as part of its conclusions.
RESPONSIVENESS TO REVIEWERS’ COMMENTS ON 2004 REPORT DRAFT
This section provides the committee’s evaluation of ATSDR’s response to reviewers’ comments on the 2004 draft, and specifically addresses the committee’s Task 2. That is, are the peer-review comments on the 2004 draft adequately addressed in the 2007 draft? The reviewers’ comments and ATSDR’s responses are discussed in general below. A comment-by-comment documentation of ATSDR’s response to comments is beyond the charge to this committee, but specific comments are used as examples. Furthermore, it should be noted that not every comment made by reviewers requires that the drafts be modified to address the comments adequately. The committee agreed with most of the reviewers’ comments, but some were thought to be irrelevant or beyond the scope of the ATSDR draft as the committee interpreted it. This section focuses on how well ATSDR responded to the comments that the committee thought were pertinent.
Before the preparation of the 2007 draft, a 2004 draft and other interim drafts (not reviewed by the committee) underwent internal and external review. Reviews were conducted by EPA staff, the IJC, state public-health officials, staff of other parts of ATSDR and other external reviewers; in addition, an expert-panel discussion was held as another form of review. On the basis of those reviews, the 2007 draft was prepared. Reviewers commented on many aspects of the 2004 draft—its presentation, the timeliness of some of the information, the inclusion or exclusion of specific hazardous-waste sites in the AOCs, typographic errors, inaccurate statements, and so on. Reviewers also commented that it was a worthwhile endeavor to catalog environmental data on the Great Lakes region.
A number of reviewers raised concerns about the datasets used in the 2004 draft. They expressed concern about the appropriateness of the data, the potential confounding factors for
human disease outcomes, the possibility of alarming the public, and the inadequate consideration of the potential for actual exposures. And some recommended further review by states in which the AOCs are located, which ATSDR appears to have obtained.
ATSDR addressed comments that were straightforward to address, such as typographic errors or the need to add text (for example, to correct factual errors) recommended by reviewers. ATSDR also added data, as suggested, on chemical discharges into surface water permitted under the NPDES. In contrast, ATSDR did not address reviewer comments that might alter the direction of the draft’s focus. For example, no additional health outcomes or morbidity data were used, the draft was not edited to “present the strongest, most reasonable information only,” and chemicals other than the IJC-critical pollutants were not consistently added. (Depending on how the IJC request was interpreted, adding chemicals might not always have been appropriate.)
Efforts to make the draft more accessible to a broad, scientifically oriented audience were evident. In response to the reviews, the consistency of the presentation of demographic data was improved in the 2007 draft. Discussion as to why some information was or was not included is still lacking, although such discussion would have greatly helped the reader to understand the draft.
In the 2007 draft ATSDR included an executive summary to make it more accessible, updated information (such as, the maps of the AOCs and the current remediation status of sites), added some explanation of the term completed exposure pathway, and added further discussion of the limitations of the draft; all those changes addressed some reviewers’ comments. However, a more detailed discussion of the Great Lakes Water Quality Agreement, as requested by one reviewer, would have been helpful in providing the context of the IJC request. Moreover, the purpose of the draft is still not clear in the 2007 version despite at least one reviewer’s suggestion that it be clarified.
Some reviewers criticized the 2004 draft with respect to which hazardous-waste sites were and were not discussed; some changes in which sites were discussed are evident in the 2007 draft. The criteria for inclusion are described in the 2007 draft but, as noted previously, are ambiguously documented. Judging by the changes in the 2008 draft, however, unambiguous criteria were never rigorously applied. ATSDR also did not clarify details regarding the TRI data, as requested by one reviewer.
Reviewers suggested that rankings among the AOCs be provided to facilitate setting of priorities among activities and resources, and that the 2007 draft include a section for each AOC to integrate and summarize information from ATSDR’s health evaluations and other assessments. The 2007 draft, however, does not include any information to address those comments. How ATSDR handled exposure pathways was also criticized, and this was not substantially improved in the 2007 draft.
VALIDITY OF DIRECTORS’ OFFICES’ CONCERNS REGARDING 2007 REPORT DRAFT
This section provides the committee’s evaluation of the scientific validity of the concerns raised by ATSDR regarding the 2007 draft, as specified in Task 3 of the charge.
After the preparation of the 2007 report draft, the Offices of the Directors of ATSDR and CCEHIP had concerns about it, and ATSDR prepared a statement (ATSDR, 2008c) expressing those concerns about its content. The concerns included how data were used, how counties were compared, and the designation of a county health indicator as “elevated” if it was above the median value for its peer-county group. The agency stated further that it had concerns, for example, that some of the data used did not directly relate exposures to changes in measures of health indicators, that the statistics presented overstated health links; and that some reviewer comments had not been addressed. It also noted that there were contradictions between the 2004 and 2007 drafts.
This committee concluded that most of the concerns expressed by ATSDR regarding the 2007 draft were valid, especially as related to the health data used, the contaminant data used as indicators of exposure, and the juxtaposition of the health and exposure data that could suggest associations or causal relationships. As with the Offices of the Directors, the committee concluded that the lack of statistical analysis in the 2007 draft was problematic. In particular, ATSDR’s use of a median value as a cut-point, above which the rate of health effects is considered “elevated” is inappropriate given the lack of analysis.
RESPONSIVENESS TO REVIEWERS’ COMMENTS ON 2007 REPORT DRAFT
This section provides the committee’s evaluation of ATSDR’s response to reviewer comments on the 2007 report draft, and addresses the committee’s Task 5. That is, were the reviewers’ comments on the draft 2007 report adequately addressed in the 2008 draft?
As discussed previously, because of concerns of the Offices of the Directors of ATSDR and CCEHIP (ATSDR, 2008c), the 2007 draft underwent review, including review by some of those who had reviewed previous drafts, and the 2008 draft of the report was prepared. Of the four external reviewers, all cited some problems with the 2007 draft. Some comments, however, were in conflict. Three of the reviewers believed that the 2007 draft should be released with the qualifiers that it contained or with additional qualifications as to the conclusions that could be drawn from the data. As discussed above in the “Health-Related Data,” many concerns were addressed in the 2008 draft by removing the CHSR data. The committee agrees with the removal of the CHSR data as they were used in the 2007 draft; however, as discussed previously, the absence of health data limits the ability to contribute to evaluating public health implications of hazardous materials in AOCs. In addition, some reviewers commented on the contaminant data and the need for further discussion of availability of data. Those comments were not addressed.
SCIENTIFIC SOUNDNESS OF 2008 REPORT DRAFT
This section provides the committee’s assessment of the scientific soundness of the 2008 draft, as specified in Task 6.
The 2008 draft’s description of methods, definition of terms, and summary of existing data are clearer than those of the 2007 draft. The 2008 draft also identifies more clearly the boundaries of the AOCs and the sites and populations examined. All the CHSR data, with the questionable methods used in previous drafts to analyze them and any resulting inappropriate or misleading conclusions, have been removed, and the scientific basis of this draft is thereby
improved. However, as a document confined to summarizing existing data that still have substantial limitations, the 2008 draft adds little to the understanding of Great Lakes AOC contaminants.
The 2008 draft’s executive summary states that “in response to the IJC request, this report summarizes previously-published public health evaluation documents and chemical release information for the 26 U.S. AOCs and 54 counties that are in close geographic proximity to those AOCs.” It also states that “this report emphasizes the critical pollutants (within the constraints imposed by using existing data) but also presents information on other pollutants, when such information is available and relevant.” The draft does compile information from those documents and release data, however, it falls short of the scope indicated in those statements. As discussed above under “Contaminant Data,” there are still serious problems, including a lack of completed pathways, temporal issues, the limited number of chemicals reported on and, in some cases, the lack of a rationale for inclusion and exclusion of chemicals. In addition, summary presentations and synthesis of the enumerated data are still lacking.
SUGGESTED IMPROVEMENTS AND OTHER COMMITTEE CONCERNS
In addition to providing specific comments on the various drafts and ATSDR’s response to reviewers’ comments in Task 7, the committee was tasked to identify any outstanding areas that need improvement and areas about which the committee has concerns.
As stated previously in the section on “Overarching Comments on the Report Drafts,” many of the problems with the drafts seem to stem from the lack of a clear statement of purpose and delineation of the approach chosen to accomplish it. Because of those overarching problems and the other problems outlined in this letter report, the committee does not make recommendations for the improvement of the 2008 draft, but offers recommendations on how to approach similar tasks in the future.
As stated previously, one of the difficulties in reviewing the various drafts is that little guidance is provided to the reader as to how the authors approached their task, and decided on a method. The information available and resources required to address a scientific question adequately should be taken into consideration in future planning processes. As discussed in EPA’s Guidelines for Ecological Risk Assessment, the National Research Council’s report entitled Science and Judgment in Risk Assessment, and the 1997 Presidential/Congressional Commission on Risk Assessment and Risk Management report (U.S. EPA, 1998; NRC, 1994; PCCRARM, 1997), delineating the approach to or framework for a project, including the questions to be answered and the rationale for choosing the given approach to answer them, would facilitate the understanding of the project’s context and scope, the choice of methods and data, and the conclusions, for similar future endeavors, even if they are exploratory or hypothesis-generating studies. The approach would include a thorough literature review, defining the project scope (for example, criteria for inclusion and exclusion of literature, datasets, and chemicals to be considered), evaluation of possible analyses and methods, and the rationale for the choice of analyses and methods used. Careful study design would also include consideration of the validity of the data for the stated purpose and determination of the adequacy of statistical power to observe associations. The use and presentation of the approach or work plan for similar reports could lead to improved consistency of methods and analysis within and between reports,
clarity in ATSDR’s options for conducting such projects, and the rationale for the various choices of datasets and analytic methods that were made.
The planning stages should include consideration of who should be involved in the project. The ATSDR drafts might have benefited from broader and earlier participation of other federal agencies (such as EPA) and state and local governments in the process than appears to have occurred. In addition, greater interaction with the IJC than is evident in the draft might have been helpful. One fundamental concern is that ATSDR did not explicitly state how the agency interpreted the original IJC request. It may be that the wrong question was asked or that it was impossible to answer the question, given the nature of exposure and health surveillance data available in the United States. Discussions with the IJC to clarify the purpose and what could or could not be accomplished with available data might have avoided some problems.
Questions regarding whether historical pollution and current industrial activity are having adverse effects will probably remain of interest, and for the foreseeable future answers will be imprecise. Where possible, such findings as “we can’t see a relationship” or “we can’t rule out an effect” should be modified to or framed as statements like “this uncertainty means that the effect, if it exists, is weaker than” and inserting information about what magnitudes or kinds of effects could be detected, using “power to detect” information.
The committee agrees that investigating whether there is a relationship between contaminants and health outcomes is important for areas around the Great Lakes. However, the committee identified substantial limitations in the 2007 ATSDR draft. It noted changes in the 2008 draft, but important limitations remain. On the basis of its evaluation, the committee has the following conclusions with respect to its specific tasks (see Box 1):
Evaluate the appropriateness of the datasets used and the scientific quality of the data analysis and presentation, and the conclusions drawn from the draft July 2007 report.
No justification or support was provided for the selection of the datasets used in the 2007 draft, the data analysis and presentation were insufficient, and key conclusions were either not clearly stated or overstated and were presented in a manner that was not supported by the data summarized in the document.
The committee considered the summary of the ATSDR health evaluations (e.g., Public Health Assessments, Health Consultations) in the 2007 draft report to be partly responsive to the IJC request “to provide the Commission information on public health assessments that it has conducted on hazardous waste sites located within any of the 33 United States AOCs.” However, the 2007 draft did not provide suitable information on current contaminant concentrations and potential health effects in the AOCs. Although the datasets used contain valid information and are potentially useful for certain purposes, other potentially useful datasets were not considered, and no justification or support is provided for the selection of the datasets used in the report draft. The appropriateness of that use cannot be fully evaluated in the absence of a clear statement of the task undertaken in the draft report, and the approach to that task, that provide the rationale for investigating particular adverse health outcomes and particular contaminants. With respect to data analysis and presentation, only descriptive data analyses were presented in the report draft, and the presentation lacked sufficient interpretation or synthesis. No statistical
analysis was included to provide a basis for presenting and synthesizing data from multiple sources in the same tables or to quantify uncertainties. The committee believed that the data were summarized and described in a manner that could encourage the reader to reach conclusions not supported by evidence. Key conclusions in the 2007 draft were either not clearly stated or, as in the executive summary, overstated.
Determine whether the peer review comments to the draft April 2004 report were adequately addressed in the draft July 2007 report.
ATSDR was responsive to nearly all comments that noted factual errors, but the most fundamental criticisms of design and interpretation—such as the appropriateness of the data, potential confounding factors, and inadequate consideration of actual exposures—were not substantively addressed.
Assess the scientific validity of the concerns raised by ATSDR regarding the draft July 2007 report.
The committee concurred with the major scientific concerns expressed by the Office of the Director of ATSDR and the Office of the Director of the Coordinating Center for Environmental Health and Injury Prevention (CCEHIP), as detailed in ATSDR 2008c.
Evaluate the appropriateness of the datasets used, and the scientific quality of the data analysis and presentation, and the conclusions drawn in the draft April 2008 report.
A clear statement of purpose and delineation of the approach chosen to accomplish that purpose were absent, as was the rationale for inclusion or exclusion of information. The removal of the health data avoided problems with the use of those data, but limited the utility of the report draft and problems with the use of the contaminant data remained. The draft did not contain any statistical analysis, interpretation, or synthesis of the information. Some conclusions went beyond the information presented in the draft.
As in the 2007 draft, the datasets used in the 2008 draft contained valid information and are potentially useful for certain purposes. The decision to drop the use of the Community Health Status Report (CHSR) data from Health Resources and Services Administration (HRSA) was an acceptable solution to problems delineated in criticisms of the 2007 draft by some reviewers and by the Offices of the Director of ATSDR and CCEHIP. Although the 2008 draft was more focused, a clear statement of purpose and a delineation of the approach chosen to accomplish that purpose were both still absent, so the appropriateness of data selected for inclusion beyond those in ATSDR’s health evaluation documents could not be adequately evaluated. Moreover, the removal of the health data left the committee concerned about the limited utility of the 2008 draft, and about whether it responded adequately to the original IJC request. In the 2008 draft, no analyses, either descriptive or statistical, were presented, nor did the draft contain interpretation or synthesis of the information. The committee also noted that the rationale for inclusion or exclusion of information remained unclear and unstated. The conclusions section of the 2008 draft summarized some of the information in the report, but some conclusions stated in the executive summary went beyond the information presented in the draft, and some did not correspond to the conclusions section.
Evaluate whether subsequent reviewers’ comments to the draft July 2007 report were adequately addressed in the April 2008 report.
The reviewers had disparate and sometimes contradictory comments on the 2007 draft, which made it impossible for ATSDR to satisfy all reviewer comments. The removal of the
CHSR data pre-empted any concerns of reviewers regarding those data, but narrowed the potential utility of the report draft by excluding any consideration of health outcomes beyond the health data compiled in ATSDR’s existing health evaluations. Reviewer comments on the contaminant data and the need for further discussion of availability of data were not addressed.
Determine if the draft April 2008 report is scientifically sound.
It is difficult to comment on the overall scientific soundness of the 2008 draft. The draft is a compilation of existing documents (from ATSDR and select other sources), but in the absence of further analysis or integration of this information it does not add substantially to scientific understanding of contamination in the AOCs or Great Lakes region or of the potential health effects of such contamination.
If considered in the context of a narrow interpretation of the IJC request to compile information from ATSDR’s documents, the data from those documents are summarized. However, even as a document confined to summarizing selected existing contaminant data, it has substantial limitations. Given the issues discussed in this letter report with regard to the Toxic Release Inventory and National Pollutant Discharge Elimination System data as “indicators of exposure” in the AOCs and the lack of information in the 2008 draft on other potential sources of contaminants in the AOCs or opportunities for population exposures, the 2008 draft does not add substantially to the understanding of contamination in the AOCs.
Identify any outstanding areas that need improvement and/or that the committee may have concerns about.
The drafts each lack clear statements of purpose and delineation of methods chosen to address it. Because of those overarching problems and the problems outlined in this letter report, the committee does not make recommendations for the improvement of the 2008 draft, but makes recommendations on how to approach similar tasks in the future.
Future projects should be initiated with a process that begins with identification of the research questions to be answered or the tasks (taking into account the importance of the questions and whether information is available to answer them) and then develops and documents a detailed approach to answering those research questions. The approach would include a thorough literature review, definition of the project scope (for example, criteria for inclusion and exclusion of literature, datasets, and chemicals to be considered), evaluation of possible analyses and methods, and the rationale for the choice of analyses and methods that will be used. Any other suitable entities available for partnering, such as other federal agencies or state governments, would be engaged as early in the process as possible. Review comments and the agency’s responses to them would be documented.
ATSDR (Agency for Toxic Substances and Disease Registry). 2004. Public Health Implications of Hazardous Substances in the Twenty-Six U.S. Great Lakes Areas of Concern. Source: CDC.
ATSDR. 2007. Public Health Implications of Great Lakes Areas of Concern (AOC). Source: CDC.
ATSDR. 2008a. Selected Information on Chemical Releases within Great Lakes Counties Containing Areas of Concern (AOC), public comment draft 2008.
ATSDR. 2008b. Reviewers’ comments on ATSDR documents. Available at http://www.atsdr.cdc.gov/grtlakes/reviews.html.
ATSDR. 2008c. Statement of Scientific Concerns about the Draft Report, Public Health Implications of Hazardous Substances in the Twenty-Six U.S. Great Lakes Areas of Concern. Source: CDC.
ATSDR. 2008d. Available at http://www.atsdr.cdc.gov/.
Clayton, C.A., Pellizzari, E.D., and Quackenboss, J.J. 2002. National Human Exposure Assessment Survey: analysis of exposure pathways and routes for arsenic and lead in EPA Region 5. J Expo Anal Environ Epidemiol 12(1):29-43.
Elliott, S.J., Eyles, J., and DeLuca, P. 2001. Mapping Health in the Great Lakes Areas of Concern: A User-Friendly Tool for Policy and Decision Makers. Environ Health Perspect 109(Suppl 6):817-826.
EPA (U.S. Environmental Protection Agency). 1998. Guidelines for Ecological Risk Assessment. U.S. Environmental Protection Agency, Risk Assessment Forum, Washington, DC, EPA/630/R095/002F.
IJC (International Joint Commission). 2001.Letter from Gerald Galloway to Henry Falk, December 5, 2001.
McElroy, J.A., Kanarek, M.S., Trentham-Dietz, A., Robert, S.A., Hampton, J.M., Newcomb, P.A., Anderson, H.A., and Remington, P.L. 2004. Potential exposure to PCBs, DDT, and PBDEs from sport-caught fish consumption in relation to breast cancer risk in Wisconsin. Environ Health Perspect. 112(2):156-162.
NRC (National Research Council). 1994. Science and Judgment in Risk Assessment. Washington, DC: National Academy Press.
PCCRARM (Presidential/Congressional Commission on Risk Assessment and Risk Management). 1997. Risk Assessment and Risk Management in Regulatory Decision-Making, Final report to Congress, Volume 2.
Schneider, J.C. 1977. History of the Walleye Fisheries of Saginaw Bay, Lake Huron, Fisheries Research Report No 1850, Michigan Department of Natural Resources.
U.S. EPA. 1998. Guidelines for Ecological Risk Assessment. U.S. Environmental Protection Agency, Risk Assessment Forum, Washington, DC, EPA/630/R095/002F.
Weisskopf, M.G., Anderson, H.A., Hanrahan, L.P., Kanarek, M.S., Falk, C.M., Steenport, D.M., and Draheim, L.A.; Great Lakes Consortium. 2005. Maternal exposure to Great Lakes sport-caught fish and dichlorodiphenyl dichloroethylene, but not polychlorinated biphenyls, is associated with reduced birth weight. Environ Res. 97(2):149-162.