Review of ATSDR's Great Lakes Report Drafts: Letter Report (2008)

In March 2008, the US Centers for Disease Control and Prevention (CDC) of the Department of Health and Human Services asked the National Academies to respond to questions, listed below, about a series of report drafts that were prepared by the Agency for Toxic Substances and Disease Registry (ATSDR)¹ focusing on the regions in or near US areas of concern (AOCs, see Appendix Figure A-1), which are defined geographically on the basis of beneficial-use impairment or inability to support aquatic life.² In response to that request, the Institute of Medicine (IOM) has prepared the attached report. This cover letter provides a brief overview of the context of the IOM report and its conclusions on the ATSDR drafts.

ATSDR compiled its drafts in response to a 2001 letter request from the International Joint Commission (IJC, see Appendix B). In the 2001 letter, the IJC requested ATSDR’s assistance in “evaluating the public health implications of the presence of hazardous materials in Great Lakes Areas of Concern (AOC’s) by providing information on ATSDR’s public health assessments of hazardous waste sites within these AOC’s.” The letter further specified that it “would be most helpful if ATSDR could identify evaluated sites within each AOC, the Hazard Category assigned to each site, any relevant demographic information available to ATSDR concerning the populations at risk, completed exposure pathways identified, and the priority substances following these pathways.” The first draft, dated April 2004, contained both health data and contaminant data. That draft was reviewed internally (within ATSDR) and externally, and a revised report draft was prepared between 2004 and July 2007. The Office of the Director of ATSDR and the Office of the Director of the Coordinating Center for Environmental Health and Injury Prevention (CCEHIP), however, had concerns about the methods used and the conclusions drawn in the 2007 report draft, and held up the release of the report pending further review. In February 2008, an unofficial copy of the July 2007 draft appeared on a nongovernmental organization’s Web site. The Offices of the Director of ATSDR and CCEHIP subsequently outlined their concerns about the report draft and posted them and the report draft to their Web site, and CDC asked the National Academies to conduct this study. ATSDR

prepared a third draft in April 2008, which was also posted to its Web site. The 2008 draft no longer contained the health data that had been included in the previous draft.

In response to CDC’s request, in April 2008 IOM appointed a committee of 11 experts in toxicology, risk assessment, risk communication, exposure assessment, public health, biostatistics, and epidemiology to conduct this study. In addition to reviewing the 2007 and 2008 report drafts, the National Academies was asked to respond to whether comments on the drafts from reviewers and the Offices of the Director of ATSDR and CCEHIP were adequately addressed. CDC requested that the National Academies produce a letter report for public dissemination within 4 months. The present letter report fulfills that request. The IOM committee had the following materials available to it:

• The letter from the IJC with the original request to ATSDR (IJC 2001; see Appendix B).

• The three report drafts (ATSDR 2004, ATSDR 2007, ATSDR 2008a).

• The internal and external reviewers’ comments (ATSDR 2008b).

• Other supporting documentation provided by CDC and ATSDR (Appendix C).

The committee met twice. At its first meeting in May 2008 in Washington DC, an open session was held at which CDC officials presented the charge to the committee. The open session included an open-microphone period, but no other interested parties asked to address the committee.

This letter report presents background information on the ATSDR report drafts and a detailed evaluation. In summary, the committee identified substantial limitations in the 2007 draft. It noted changes in the 2008 draft report, but important limitations remain. The committee’s conclusions are presented here verbatim from the conclusions section of the letter report. The body of the letter report elaborates on the support for those conclusions, with examples. On the basis of its evaluation, the committee offers the following conclusions with respect to its specific tasks:

1. Evaluate the appropriateness of the datasets used and the scientific quality of the data analysis and presentation, and the conclusions drawn from the draft July 2007 report.

   No justification or support was provided for the selection of the datasets used in the 2007 draft, the data analysis and presentation were insufficient, and key conclusions were either not clearly stated or overstated and were presented in a manner that was not supported by the data summarized in the document.

The committee considered the summary of the ATSDR health evaluations (e.g., Public Health Assessments, Health Consultations) in the 2007 draft report to be partly responsive to the IJC request “to provide the Commission information on public health assessments that it has conducted on hazardous waste sites located within any of the 33 United States AOCs.” However, the 2007 draft did not provide suitable information on current contaminant concentrations and potential health effects in the AOCs. Although the datasets used contain valid information and are potentially useful for certain purposes, other potentially useful datasets were not considered, and no justification or support is provided for the selection of the datasets used in the report draft. The appropriateness of that use cannot be fully evaluated in the absence of a clear statement of the task undertaken in the draft report, and the approach to that task, that provide the rationale for investigating particular adverse health outcomes and particular contaminants. With respect to data analysis and presentation, only descriptive data analyses were presented in the report draft, and the presentation lacked sufficient interpretation or synthesis. No statistical

analysis was included to provide a basis for presenting and synthesizing data from multiple sources in the same tables or to quantify uncertainties. The committee believed that the data were summarized and described in a manner that could encourage the reader to reach conclusions not supported by evidence. Key conclusions in the 2007 draft were either not clearly stated or, as in the executive summary, overstated.

2. Determine whether the peer review comments to the draft April 2004 report were adequately addressed in the draft July 2007 report.

   ATSDR was responsive to nearly all comments that noted factual errors, but the most fundamental criticisms of design and interpretation—such as the appropriateness of the data, potential confounding factors, and inadequate consideration of actual exposures—were not substantively addressed.

3. Assess the scientific validity of the concerns raised by ATSDR regarding the draft July 2007 report.

   The committee concurred with the major scientific concerns expressed by the Office of the Director of ATSDR and the Office of the Director of the Coordinating Center for Environmental Health and Injury Prevention (CCEHIP), as detailed in ATSDR 2008c.

4. Evaluate the appropriateness of the datasets used, and the scientific quality of the data analysis and presentation, and the conclusions drawn in the draft April 2008 report.

   A clear statement of purpose and delineation of the approach chosen to accomplish that purpose were absent, as was the rationale for inclusion or exclusion of information. The removal of the health data avoided problems with the use of those data, but limited the utility of the report draft and problems with the use of the contaminant data remained. The draft did not contain any statistical analysis, interpretation, or synthesis of the information. Some conclusions went beyond the information presented in the draft.

As in the 2007 draft, the datasets used in the 2008 draft contained valid information and are potentially useful for certain purposes. The decision to drop the use of the Community Health Status Report (CHSR) data from Health Resources and Services Administration (HRSA) was an acceptable solution to problems delineated in criticisms of the 2007 draft by some reviewers and by the Offices of the Director of ATSDR and CCEHIP. Although the 2008 draft was more focused, a clear statement of purpose and a delineation of the approach chosen to accomplish that purpose were both still absent, so the appropriateness of data selected for inclusion beyond those in ATSDR’s health evaluation documents could not be adequately evaluated. Moreover, the removal of the health data left the committee concerned about the limited utility of the 2008 draft, and about whether it responded adequately to the original IJC request. In the 2008 draft, no analyses, either descriptive or statistical, were presented, nor did the draft contain interpretation or synthesis of the information. The committee also noted that the rationale for inclusion or exclusion of information remained unclear and unstated. The conclusions section of the 2008 draft summarized some of the information in the report, but some conclusions stated in the executive summary went beyond the information presented in the draft, and some did not correspond to the conclusions section.

5. Evaluate whether subsequent reviewers’ comments to the draft July 2007 report were adequately addressed in the April 2008 report.

   The reviewers had disparate and sometimes contradictory comments on the 2007 draft, which made it impossible for ATSDR to satisfy all reviewer comments. The removal of the

CHSR data pre-empted any concerns of reviewers regarding those data, but narrowed the potential utility of the report draft by excluding any consideration of health outcomes beyond the health data compiled in ATSDR’s existing health evaluations. Reviewer comments on the contaminant data and the need for further discussion of availability of data were not addressed.

1. Determine if the draft April 2008 report is scientifically sound.

   It is difficult to comment on the overall scientific soundness of the 2008 draft. The draft is a compilation of existing documents (from ATSDR and select other sources), but in the absence of further analysis or integration of this information it does not add substantially to scientific understanding of contamination in the AOCs or Great Lakes region or of the potential health effects of such contamination.

If considered in the context of a narrow interpretation of the IJC request to compile information from ATSDR’s documents, the data from those documents are summarized. However, even as a document confined to summarizing selected existing contaminant data, it has substantial limitations. Given the issues discussed in this letter report with regard to the Toxic Release Inventory and National Pollutant Discharge Elimination System data as “indicators of exposure” in the AOCs and the lack of information in the 2008 draft on other potential sources of contaminants in the AOCs or opportunities for population exposures, the 2008 draft does not add substantially to the understanding of contamination in the AOCs.

7. Identify any outstanding areas that need improvement and/or that the committee may have concerns about.

   The drafts each lack clear statements of purpose and delineation of methods chosen to address it. Because of those overarching problems and the problems outlined in this letter report, the committee does not make recommendations for the improvement of the 2008 draft, but makes recommendations on how to approach similar tasks in the future.

Future projects should be initiated with a process that begins with identification of the research questions to be answered or the tasks (taking into account the importance of the questions and whether information is available to answer them) and then develops and documents a detailed approach to answering those research questions. The approach would include a thorough literature review, definition of the project scope (for example, criteria for inclusion and exclusion of literature, datasets, and chemicals to be considered), evaluation of possible analyses and methods, and the rationale for the choice of analyses and methods that will be used. Any other suitable entities available for partnering, such as other federal agencies or state governments, would be engaged as early in the process as possible. Review comments and the agency’s responses to them would be documented.

This letter report contains conclusions about the 2007 and 2008 drafts, but it should not be interpreted as an endorsement of any future documents. The committee did not conduct a detailed technical review of the ATSDR documents. Such a detailed review, which would include verifying the accuracy of the data presented in the drafts, was beyond the scope of the charge to the committee. The committee believes that investigating whether there is a relationship between contaminants and health outcomes is important for areas around the Great Lakes; this report contains no such investigation. The committee was charged with evaluating the ATSDR drafts, not with determining whether health concerns are associated with contamination in the Great Lakes area. Evaluating the appropriateness of the request from IJC and the adequacy

of resources available to ATSDR for completing the drafts was outside the charge to the committee.

This letter report reflects the consensus of the committee and has been reviewed inaccordance with IOM review procedures.

This page intentionally left blank.