In 2011, the National Research Council (NRC) released the report Review of the Environmental Protection Agency’s Draft IRIS Assessment of Formaldehyde. The report provided a scientific review of the toxicological review of formaldehyde drafted by the US Environmental Protection Agency (EPA) for its Integrated Risk Information System (IRIS). Chapter 7 of the NRC report also suggested changes in the general process used to develop IRIS assessments and suggested a roadmap for making revisions if EPA decided to do so. In response, EPA announced plans to work with its Science Advisory Board to address the committee’s suggestions and recommendations. In 2011 testimony before a subcommittee of the US House of Representatives, the assistant administrator of EPA’s Office of Research and Development outlined the approach that EPA planned to take in response to the NRC recommendations.1 On December 23, 2011, the Consolidated Appropriations Act, 2012 (Public Law 112- 74) was signed into law; the House report (112-151) accompanying the act stated that “EPA shall incorporate, as appropriate, based on chemical-specific datasets and biological effects, the recommendations of Chapter 7 of the National Research Council’s Review of the Environmental Protection Agency’s Draft IRIS Assessment of Formaldehyde into the IRIS process.” To ensure that EPA adequately considers the NRC recommendations, Congress requested that NRC assess the scientific, technical, and process changes that EPA is making. As a result of that request, NRC convened the Committee to Review the IRIS Process, which drafted the present report.
EPA created the IRIS program in 1985 to provide information on human health effects that could arise from chronic exposures to environmental contaminants. A primary goal of IRIS is to increase the consistency of assessments being conducted throughout the agency. Accordingly, the IRIS program develops toxicologic assessments of various chemicals that include hazard and dose-response characterizations. The assessments also include toxicity values (reference values and unit risks) that can be used with exposure estimates to derive quantitative risk estimates.
Over the years, federal and state agencies and other entities have come to rely on IRIS assessments for setting regulatory standards and establishing exposure guidelines. IRIS assessments provide information needed to evaluate risks at the local level and are also considered authoritative internationally. In recent years, however, questions have been raised about the scientific basis of reference values and unit risks reported in some IRIS assessments. The IRIS program has also been criticized for the time that it takes the agency to complete assessments. Some assessments have taken more than a decade to develop and have gone through multiple cycles of revision and review, which further delay their acceptance. For example, the develop-
1EPA’s IRIS Program: Evaluating the Science and Process behind Chemical Risk Assessment, 2011: Hearing before the Subcommittee on Investigations and Oversight of the Committee on Science, Space, and Technology, House of Representatives, 112th Cong., 1st Sess., July 14, 2011.
ment of the most recent draft IRIS assessment of formaldehyde began in October 2004, and the draft was released to the public in June 2010. The US Government Accountability Office concluded in 2008 that “the IRIS database is at serious risk of becoming obsolete because EPA has not been able to routinely complete timely, credible assessments or decrease its backlog of 70 ongoing assessments” (GAO 2008, p.1).
Over the last decade, EPA risk-assessment guidance documents have reiterated EPA’s policy of evaluating and integrating evidence with an approach that is consistent, comprehensive, balanced, and reproducible (EPA 2002, 2003, 2004, 2005). However, NRC committees have conducted several reviews of some of the more complex and challenging IRIS assessments in the last decade and have identified methodologic problems and pointed out deficiencies in EPA’s approaches. For example, the NRC committee that reviewed the dioxin reassessment found problems with the noncancer assessment and stated that “EPA does not use a rigorous approach for evaluating evidence from studies and the weight of their evidence” (NRC 2006, p. 47). The NRC committee that reviewed the tetrachloroethylene assessment offered similar criticisms and emphasized that “the overall impression is that data are presented to support a positive association between tetrachloroethylene and cancer, and that studies that found no such association are criticized or minimized” (NRC 2010, p. 85).
In June 2010, EPA released the IRIS formaldehyde assessment. Recognizing the complex nature of the assessment and its importance as the basis of risk calculations and regulatory decisions for this high-production chemical, EPA asked NRC to review the assessment and answer questions related specifically to the derivation of the inhalation reference concentration (RfC) and unit risk values. The NRC committee convened to conduct that task released its report in 2011. It identified problems similar to those expressed by earlier NRC committees and concluded that “the draft was not prepared in a consistent fashion; it lacks clear links to an underlying conceptual framework; and it does not contain sufficient documentation on methods and criteria for identifying evidence from epidemiologic and experimental studies, for critically evaluating individual studies, for assessing the weight of evidence, and for selecting studies for derivation of the RfCs and unit risk estimates ” (NRC 2011, p. 4). As noted, the committee provided specific recommendations for revision of the IRIS formaldehyde assessment. It also made general suggestions for improvement of the IRIS process and provided a roadmap for its revision as guidance if EPA decided to move forward with changes in the process.
After release of the NRC formaldehyde report (NRC 2011), Congress held several hearings to examine the objectivity and credibility of IRIS assessments and the program.2,3 On July 12, 2011, EPA (2011) emphasized its commitment to respond to the recommendations in the NRC formaldehyde report and to improve the IRIS program further. Figure 1-1 highlights EPA’s actions and demonstrates its commitment to improve IRIS since release of the NRC formaldehyde report.
As requested by Congress, EPA gave relevant congressional committees a progress report in June 2012 that described a phased approach to implementing the NRC recommendations. EPA noted that its first action was “streamlining documents, increasing transparency and clarity, and using more tables and figures to present information and data in assessments” (EPA 2012a, p. 12). Specific improvements that were described in the EPA progress report included developing
2EPA’s IRIS Program: Evaluating the Science and Process behind Chemical Risk Assessment, 2011: Hearing before the Subcommittee on Investigations and Oversight of the Committee on Science, Space, and Technology, House of Representatives, 112th Cong., 1st Sess., July 14, 2011.
3Chemical Risk Assessment: What Works for Jobs and Economy? 2011: Hearing Before the Subcommittee on Environment and the Economy, Committee on Energy and Commerce, House of Representatives, 112th Cong., 1st Sess., October 6, 2011.
a new document structure with an executive summary to highlight major toxicologic findings and a preamble to outline approaches for identifying and evaluating studies, weighing evidence, selecting studies for deriving toxicity values, and deriving toxicity values. EPA also noted that literature-search strategies and criteria for evaluating studies would be explicitly described in new assessments and that it was developing a framework for reaching conclusions on noncancer effects, although it indicated that this effort would be implemented in a later phase. EPA emphasized, however, that it would be using more systematic approaches for analyzing data. Finally, EPA noted that it was expanding efforts for early peer and stakeholder consultation by hosting public workshops on various issues and that it was forming the Chemical Assessment Advisory Committee (CAAC) under the auspices of its Science Advisory Board. The purpose of the CAAC is to advise the agency on specific assessments and possibly on broader issues. Regarding peer consultation, EPA held a public stakeholder meeting in November 2012 to hear the needs of IRIS users and their views on improvements needed in the IRIS program (EPA 2012b).
FIGURE 1-1 Timeline of events since release of the NRC report, Review of the Environmental Protection Agency’s Draft IRIS Assessment of Formaldehyde. The timeline does not include all the meetings that have been held concerning IRIS and chemical-specific assessments.
On January 30, 2013, EPA provided materials to the present committee that outlined its actions in response to each of the recommendations offered in Chapter 7 of the NRC formaldehyde report (NRC 2011) and indicated the status of implementation of each change. It also provided the new IRIS document template, the draft preamble, a draft handbook for the IRIS assessment development, and several chemical-specific examples of its new approaches. The committee was later given a fact sheet issued on July 31, 2013 (EPA 2013) that restated EPA’s commitment to improving IRIS assessments and that highlighted some of the improvements that it had described previously, such as the new document structure and the use of systematic-review methods. It also noted changes to enhance productivity and transparency, such as focusing staff on fewer assessments and introducing stopping rules for the inclusion of new data or scientific issues. EPA described several other program improvements, including planning and scoping meetings for each assessment; making public at early stages critical pieces of draft assessments, such as literature searches, evidence tables, and dose-response figures; providing a forum to receive public comments; and ensuring opportunities for the public to make comments on draft assessments once released. Finally, the committee was updated on August 20, 2013, on the implementation of several NRC recommendations and was given additional chemical-specific examples of assessment documents. The materials provided to the committee are listed in Table 1-1 and discussed and reviewed in greater depth in the chapters that follow.
In 2011, the Institute of Medicine (IOM) released the report Finding What Works in Health Care: Standards for Systematic Review. As defined by the IOM report, systematic review is “a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar but separate studies” (IOM 2011, p. 1). Although the report was written in the context of comparative-effectiveness research, systematic-review methods have been used for decades in a variety of fields, from agriculture to education (Light and Pillemer 1984; Chalmers et al. 2002). Systematic reviews might or might not result in a quantitative summary of the data, such as an effect estimate. A specific approach to summarize quantitatively somewhat more homogeneous information is often referred to as meta-analysis.
|Part 1: Status of Implementation of Recommendations||Status report on implementation
IRIS toxicologic review template
Preamble to IRIS toxicologic reviews
Example of directions for contractors
Information on Comment Tracker Database
Information on scoping for IRIS assessments
Draft handbook for IRIS assessment development
|January 30, 2013|
|Part 2: Chemical-Specific Examples||Seven chemical-specific examples of implementation of various NRC recommendations||January 30, 2013|
|EPA Fact Sheet||Summary of enhancements of IRIS program||July 31, 2013|
|E-mail from IRIS program director||Responses to NRC committee inquiries
Attachments: Tables documenting implementation of NRC recommendations from draft ammonia, trimethylbenzene, and benzo[a]pyrene assessments
|August 20, 2013|
|IRIS Toxicological Review of Benzo[a]pyrene||Draft IRIS assessment of benzo[a]pyrene and supplemental materials||August 20, 2013|
|IRIS Toxicological Review of Methanola||Draft IRIS noncancer assessment of methanol||September 30, 2013|
aThe IRIS methanol assessment was not sent directly to the committee but was released while the committee was conducting its review and therefore was used as an example of implemented changes.
The materials and examples provided by EPA indicate that the agency is incorporating systematic-review principles as it makes changes in the IRIS process. Figure 1-2 shows systematic review as the present committee envisions its use in the context of the IRIS process. As noted above, one might be able to use meta-analysis to summarize quantitatively the results of the systematic reviews of each data stream (human, animal, and mechanistic). Meta-analysis is also one analytic approach that could be used to integrate evidence across data streams for hazard identification (see Chapter 6) and that could be used for combining data or dose-response estimates of individual studies to derive toxicity values (see Chapter 7).
The terms weight of evidence (WOE) and WOE analysis are sometimes used interchangeably with systematic review, and there is often confusion surrounding the meanings of the terms. However, the committee views a WOE analysis as a judgment-based process for evaluating the strength of evidence to infer causation, that is, as one approach to integrating evidence for hazard identification. The meaning and use of WOE analysis are discussed further in Chapter 6 of this report.
The committee, which was convened, included experts in epidemiology, toxicology, dose-response modeling, risk assessment, systematic review, and risk communication (see Appendix A for biographic information on the committee). As noted earlier, it was asked to assess the scientific, technical, and process changes that EPA is making in the IRIS process. The verbatim statement of task is provided in Box 1-1.
The committee held six committee meetings to accomplish its task. Open sessions were held during the first and second meetings in which the committee heard from the sponsor on changes being made in the IRIS process and from staff of the National Toxicology Program on changes being made in its chemical-assessment program. During the third meeting, the committee held a workshop that involved participants from academe, government agencies, and private organizations to address approaches used to evaluate and integrate evidence for use in an IRIS assessment (see Appendix B for the workshop agenda). In each open session, interested parties were allowed to address the committee. The committee reviewed materials provided by EPA that document changes that it has made or is planning to make in the IRIS process and materials submitted by interested parties.
FIGURE 1-2 Systematic review in the context of the IRIS process. The committee views public input and peer review as integral parts of the IRIS process, although those activities are not specifically noted in the figure.
A committee of the National Research Council (NRC) will assess the scientific, technical, and process changes being implemented by the U.S. Environmental Protection Agency (EPA) for its Integrated Risk Information System (IRIS). Specifically, the committee will review the IRIS process and the changes being implemented or planned by EPA and will recommend modifications or additional changes as appropriate to improve the scientific and technical performance of the IRIS program. The committee will focus on the development of the IRIS assessments rather than the review process that follows draft development. Because several reviews of IRIS assessments have expressed concerns about EPA’s weight-of-evidence analyses, the committee will review current methods for evidence-based reviews and recommend approaches for weighing scientific evidence for chemical hazard and dose-response assessments.
The present report is organized into eight chapters and three appendixes. Chapter 2 provides an overview of some general issues associated with IRIS assessments. Chapter 3 describes the need for problem formulation to define the systematic-review questions and protocol development to describe the methods used in the systematic reviews. Chapters 4 and 5 address evidence identification and evidence evaluation, respectively. Chapter 6 discusses evidence integration for hazard identification, and Chapter 7 evaluates methods for deriving toxicity values. Chapter 8 presents some overall findings and considerations for future directions. Appendix A provides biographic information on the committee, Appendix B is the agenda of the committee’s workshop, and Appendix C provides some background information on Bayesian analysis.
Chalmers, I., L.V. Hedges, and H. Cooper. 2002. A brief history of research synthesis. Eval. Health Prof. 25(1):12-37.
EPA (U.S. Environmental Protection Agency). 2002. Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by the Environmental Protection Agency. EPA/260R-02-008. Office of Environmental Information, U.S. Environmental Protection Agency, Washington, DC [online]. Available: http://www.epa.gov/quality/informationguidelines/documents/EPA_InfoQualityGuidelines.pdf [accessed Aug. 13, 2013].
EPA (U.S. Environmental Protection Agency). 2003. A Summary of General Assessment Factors for Evaluating the Quality of Scientific and Technical Information. EPA 100/B-03/001. Science Policy Council, U.S. Environmental Protection Agency, Washington, DC [online]. Available: http://www.epa.gov/stpc/pdfs/assess2.pdf [accessed Aug. 13, 2013].
EPA (U.S. Environmental Protection Agency). 2004. An Examination of EPA Risk Assessment Principles and Practices. EPA/100/B-04/001. Office of the Science Advisor, U.S. Environmental Protection Agency Washington, DC [online]. Available: http://www.epa.gov/osa/pdfs/ratf-final.pdf [accessed Aug. 13, 2013].
EPA (U.S. Environmental Protection Agency). 2005. Guidelines for Carcinogen Risk Assessment. EPA/630/P-03/001F. Risk Assessment Forum, U.S. Environmental Protection Agency, Washington, DC [online]. Available: http://www.epa.gov/raf/publications/pdfs/CANCER_GUIDELINES_FINAL_3-25-05.PDF [accessed Aug. 13, 2013].
EPA (U.S. Environmental Protection Agency). 2011. EPA Strengthens Kay Scientific Database to Protect Public Health. EPA News: July 12, 2011 [online]. Available: http://yosemite.epa.gov/opa/admpress.nsf/48f0fa7dd51f9e9885257359003f5342/a3fcd60838197067852578cb00666c4d!OpenDocument [accessed Aug. 13, 2013].
EPA (U.S. Environmental Protection Agency). 2012a. EPA’s Integrated Risk Information Program, Progress Report and Report to Congress. Office of Research and Development, U.S. Environmental Protection Agency [online]. Available: http://www.epa.gov/iris/pdfs/irisprogressreport2012.pdf [accessed Aug. 13, 2012].
EPA (U.S. Environmental Protection Agency). 2012b. Integrated Risk Information System Program: Summary Report from November 2012 Public Stakeholder Meeting [online]. Available: http://www.epa.gov/iris/publicmeeting/stakeholders-kickoff/ [accessed Aug. 13, 2013].
EPA (U.S. Environmental Protection Agency). 2013. Enhancements to EPA’s Integrated Risk Information System Program [online]. Available: http://www.epa.gov/iris/pdfs/irisprocessfactsheet2013.pdf [accessed Aug. 13, 2013].
GAO (U.S. Government Accountability Office). 2008. Toxic Chemicals: EPA’s New Assessment Process Will Increase Challenges EPA Faces in Evaluating and Regulating Chemicals. GAO-08-743T. Washington, DC: U.S. Government Accountability Office [online]. Available: http://www.gao.gov/assets/120/119860.pdf [accessed Aug. 13, 2013].
IOM (Institute of Medicine). 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: National Academies Press.
Light, R.J., and D.B. Pillemer. 1984. Summing Up: The Science of Reviewing Research. Cambridge MA: Harvard University Press.
NRC (National Research Council). 2006. Health Risks from Dioxin and Related Compounds: Evaluation of the EPA Reassessment. Washington, DC: National Academies Press.
NRC (National Research Council). 2010. Review of the Environmental Protection Agency’s Draft IRIS Assessment of Tetrachloroethylene. Washington, DC: National Academies Press.
NRC (National Research Council). 2011. Review of the Environmental Protection Agency’s Draft IRIS Assessment of Formaldehyde. Washington, DC: National Academies Press.