The US Environmental Protection Agency (EPA) clearly has embraced the need for revision of its Integrated Risk Information System (IRIS). It has begun to implement changes that follow the general guidance given in Chapter 7 of the report Review of the Environmental Protection Agency’s Draft IRIS Assessment of Formaldehyde (NRC 2011). The present report has reviewed and evaluated those changes and has provided additional suggestions for further improving the process. This chapter highlights several recommendations that should receive high priority, considers issues that extend over the full process, and presents suggestions directed at ensuring that the IRIS program provides the best possible assessments in the future.
The IRIS program provided substantial materials to the committee that documented the general strategy for the continuing revisions of its process (see Table 1-1) and various materials that constitute the building blocks of the revision: the draft preamble (EPA 2013a, Appendix B) and the draft handbook (EPA 2013a, Appendix F). EPA also provided examples of assessments in which elements of the revisions have been implemented (EPA 2013b,c). Although the committee recognizes that EPA has not yet completed its revisions of the methods used for IRIS assessments, it is able to assess the general approach taken by the agency and the trajectory of change in the assessment approach. The committee commends EPA for its substantive new approaches, continuing commitment to improving the process, and successes to date. Overall, the committee expects that EPA will complete its planned revisions in a timely way and that the revisions will transform the IRIS program.
EPA has responded to many of the suggestions made in the 2011 NRC formaldehyde report at a particularly challenging time for the IRIS program. Its response to the formaldehyde report acknowledges the current context and makes changes, not only in scientific methods but in the underlying principles and in public processes. Critical changes in leadership positions have occurred since the release of the NRC formaldehyde report. Kenneth Olden, the newly named director of the EPA National Center for Environmental Assessment, has made a far-reaching effort to engage the full array of stakeholders, including the general public, in providing input into the changes being made. The revisions embrace stakeholder engagement in all relevant phases of the process. Under its acting director, Vincent Cogliano, the IRIS program has moved forward steadily in planning for and implementing changes in each element of the assessment process. The committee is confident that there is an institutional commitment to completing the revisions of the process even as the program continues through the current transition phase and is closely watched by both stakeholders and Congress.
As reviewed in the preceding chapters, the committee found that appropriate revisions of all elements of the IRIS assessment process (see Figure 1-2) were underway or planned. The preamble represents a useful general framework for the assessments, and the committee recommends that this document and the draft handbook be completed and reviewed. The committee
also found that the proposed format for the assessments should enhance “user friendliness” and transparency. The evidence tables and data displays in the new documents are moving to the standard practice for systematic reviews.
The present committee offers a long series of findings and recommendations in this report. The recommendations can be categorized as those directly relevant to current revisions of the process, those related to future refinements of the methods, or those calling for research related to the assessment process that will provide results for guiding the further evolution of the methods used in the process. The committee urges that high priority be assigned to those in the first category and has listed them in Box 8-1. A timetable should be developed for the other recommendations. For the longer term, as discussed below, EPA will need to establish procedures to ensure that there is continuing refinement of the assessment methods. In addition, the IRIS program will need resources to conduct or commission focused methodologic research.
The present committee has looked retrospectively at the methods and performance of the IRIS program and evaluated the changes that have been implemented. The 2011 NRC formaldehyde report cited a number of “lessons learned.” Here, the committee offers several that are deemed critical for ensuring that the IRIS program provides the best possible assessments.
• Assessment methods should be updated in a continuing, strategic fashion. The 2011 NRC formaldehyde report found that state-of-the-art approaches that had widespread application in other fields, such as systematic review, were not being used in the IRIS assessment process. Even as the IRIS program undergoes revision, consideration needs to be given to how methods relevant to all elements of the process will evolve continuously. The revisions in progress should include consideration of how relevant progress in risk assessment and other domains will be tracked and incorporated into the IRIS assessment approach.
• Inefficiencies in the IRIS program need to be systematically identified and addressed. Although the present committee recognizes that factors beyond the program itself create delay, it urges the IRIS program to consider systematically how delay occurs so that it can be anticipated and addressed. Some of the most controversial assessments have had long histories with multiple cycles of revision and review. Some assessments that have been delayed have involved review of substantial bodies of evidence and continuing publication of relevant evidence. EPA has examined the timing of its process and has proposed principles for stopping rules related to such issues as literature identification and the addition of important new documents to assessments. Although principles have been offered, their application in practice could prove challenging, and monitoring of adherence to stopping rules will be needed. Collaborating with other agencies, such as the National Toxicology Program, to avoid duplication of effort is another important efficiency-promoting activity.
• Evolving competences that reflect new scientific directions are needed. The conduct of an IRIS assessment necessarily involves multiple scientific disciplines, and as research methods and data streams change, EPA management will need to ensure that the chemical-assessment teams and chemical-assessment support teams have appropriate expertise and training. The IRIS program needs continuing evaluation of its expertise in relation to changing scientific contexts.
BOX 8-1 Recommendations Directly Relevant to Current Revisions
• EPA needs to complete the changes in the IRIS process that are in response to the recommendations in the NRC formaldehyde report and specifically complete documents, such as the draft handbook, that provide detailed guidance for developing IRIS assessments. When those changes and the detailed guidance, such as the draft handbook, have been completed, there should be an independent and comprehensive review that evaluates how well EPA has implemented all the new guidance. The present committee is completing its report while those revisions are still in progress.
• EPA should provide a quality-management plan that includes clear methods for continuing assessments of the quality of the process. The roles of the various internal entities involved in the process, such as the chemical-assessment support teams, should be described. The assessments should be used to improve the overall process and the performance of EPA staff and contractors.
• EPA should establish a transparent process for initially identifying all putative adverse outcomes through a broad search of the literature. The agency should then develop a process that uses guided expert judgment to identify the specific adverse outcomes to be investigated, each of which would then be subjected to systematic review of human, animal, and in vitro or mechanistic data.
• EPA should include protocols for all systematic reviews conducted for a specific IRIS assessment as appendixes to the assessment.
• The current process can be enhanced with more explicit documentation of methods. Protocols for IRIS assessments should include a section on evidence identification that is written in collaboration with information specialists trained in systematic reviews and that includes a search strategy for each systematic-review question being addressed in the assessment. Specifically, the protocols should provide a line-by-line description of the search strategy, the date of the search, and publication dates searched and explicitly state the inclusion and exclusion criteria for studies.
• To advance the development of tools for assessing risk of bias in different types of studies (human, animal, and mechanistic) used in IRIS assessments, EPA should explicitly identify factors that can lead to bias in animal studies—such as control for litter effects, dosing, and methods for exposure assessment—so that these factors are consistently evaluated for experimental studies. Likewise, EPA should consider a tool for assessing risk of bias in in vitro studies.
• When considering any method for evaluating individual studies, EPA should select a method that is transparent, reproducible, and scientifically defensible. Whenever possible, there should be empirical evidence that the methodologic characteristics that are being assessed in the IRIS protocol have systematic effects on the direction or magnitude of the outcome. The methodologic characteristics that are known to be associated with a risk of bias should be included in the assessment tool. Additional quality-assessment items relevant to a particular systematic-review question could also be included in the EPA assessment tool.
• Although additional methodologic work might be needed to establish empirically supported criteria for animal or mechanistic studies, an IRIS assessment needs to include a transparent evaluation of the risk of bias of studies used by EPA as a primary source of data for the hazard assessment. EPA should specify the empirically based criteria it will use to assess risk of bias for each type of study design in each type of data stream.
• To maintain transparency, EPA should publish its risk-of-bias assessments as part of its IRIS assessments. It could add tables that describe the assessment of each risk-of-bias criterion for each study and provide a summary of the extent of the risk of bias in the descriptions of each study in the evidence tables.
• The risk-of-bias assessment of individual studies should be carried forward and incorporated into the evaluation of evidence among data streams.
• EPA should continue to improve its evidence-integration process incrementally and enhance the transparency of its process. It should either maintain its current guided-expert-judgment process but make its application more transparent or adopt a structured (or GRADE-like) process for evaluating evidence and rating recommendations along the lines that NTP has taken. If EPA does move to a structured evidence-integration process, it should combine resources with NTP to leverage the intellectual resources and scientific experience in both organizations. The committee does not offer a preference but suggests that EPA consider which approach best fits its plans for the IRIS process.
• EPA should expand its ability to perform quantitative modeling of evidence integration; in particular, it should develop the capacity to do Bayesian modeling of chemical hazards. That technique could be helpful in modeling assumptions about the relevance of a variety of animal models to each other and to humans, in incorporating mechanistic knowledge to model the relevance of animal models to humans and the relevance of human data for similar but distinct chemicals, and in providing a general framework within which to update scientific knowledge rationally as new data become available. The committee emphasizes that the capacity for quantitative modeling should be developed in parallel with improvements to existing IRIS evidence-integration procedures and that IRIS assessments should not be delayed while this capacity is being developed.
• EPA should develop criteria for determining when evidence is sufficient to derive toxicity values. One approach would be to restrict formal dose-response assessments to when a standard descriptor characterizes the level of confidence as medium or high (as in the case of noncancer end points) or as “carcinogenic to humans” or “likely to be carcinogenic to humans” for carcinogenic compounds. Another approach, if EPA adopts probabilistic hazard classification, is to conduct formal dose-response assessments only when the posterior probability that a human hazard exists exceeds a predetermined threshold, such as 50% (more likely than not likely that the hazard exists).
• EPA should clearly present two dose-response estimates: a central estimate (such as a maximum likelihood estimate or a posterior mean) and a lower-bound estimate for a POD from which a toxicity value is derived. The lower bound becomes an upper bound for a cancer slope factor but remains a lower bound for a reference value.
The committee has looked beyond the approach for revising the process and the timespan of its recommendations. As EPA completes the current revisions, it needs to consider developing a strategic plan for continuous updating of the IRIS methodology. The strategic plan should be sufficiently flexible to consider a variety of approaches that incorporate advances in fields relevant to the IRIS program. For example, such a strategic plan should address
• Using data from emerging technologies of molecular toxicology.
• Incorporating new statistical methods.
• Applying advances in data retrieval and text-mining.
The committee urges that the strategic plan consider the human and technologic resources that are needed to carry out the IRIS assessments and to support methodologic research and the implementation of new approaches. The program is already challenged by the number of assessments that it conducts and by the need to increase the pace at which assessments are completed. It is now faced with the additional burden of fundamental revisions of its methods. EPA needs to evaluate carefully the demands on its staff and the load carried by its contractors and consultants. For now, sufficient financial and staff resources need to be available to complete the revisions of the process; for the future, increased capacity is needed for methodologic work and incorporation of modifications into the assessment approach.
Consideration will also need to be given to how changes in the IRIS assessment process will be reviewed before implementation. The committee suggests that EPA consider whether its Chemical Assessment Advisory Committee or Science Advisory Board could be used for this purpose. Peer review should be an integral part of the revision process, and the IRIS system should be sufficiently dynamic to reflect relevant advances.
EPA (U.S. Environmental Protection Agency). 2013a. Part 1. Status of Implementation of Recommendations. Materials Submitted to the National Research Council, by Integrated Risk Information System Program, U.S. Environmental Protection Agency, January 30, 2013 [online]. Available: http://www.epa.gov/iris/pdfs/IRIS%20Program%20Materials%20to%20NRC_Part%201.pdf [accessed November 13, 2013].
EPA (U.S. Environmental Protection Agency). 2013b. Part 2. Chemical-Specific Examples. Materials Submitted to the National Research Council, by Integrated Risk Information System Program, U.S. Environmental Protection Agency, January 30, 2013 [online]. Available: http://www.epa.gov/iris/pdfs/IRIS%20Program%20Materials%20to%20NRC_Part%202.pdf [accessed December 19, 2013].
EPA (U.S. Environmental Protection Agency). 2013c. Toxicological Review of Benzo[a]pyrene (CAS No. 50-32-8) in Support of Summary Information on the Integrated Risk Information System (IRIS), Public Comment Draft. EPA/635/R13/138a. National Center for Environmental Assessment, Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC. August 2013 [online]. Available: http://cfpub.epa.gov/ncea/iris_drafts/recordisplay.cfm?deid=66193 [accessed Nov. 13, 2013].
NRC (National Research Council). 2011. Review of the Environmental Protection Agency's Draft IRIS Assessment of Formaldehyde. Washington, DC: National Academies Press.
