Models in Environmental Regulatory Decision Making
THE NATIONAL ACADEMIES PRESS
Washington, D.C.
www.nap.edu
THE NATIONAL ACADEMIES PRESS
500 Fifth Street, NW Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This project was supported by Contract No. 68-C-03-081 between the National Academy of Sciences and the U.S. Environmental Protection Agency. Additional support was provided by the U.S. Department of Transportation. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for this project.
International Standard Book Number-13: 978-0-309-11000-6
International Standard Book Number-10: 0-309-11000-9
Additional copies of this report are available from
The National Academies Press
500 Fifth Street, NW Box 285 Washington, DC 20055 800-624-6242 202-334-3313 (in the Washington metropolitan area) http://www.nap.edu
Copyright 2007 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council.
COMMITTEE ON MODELS IN THE REGULATORY DECISION PROCESS
Members
CHRIS G. WHIPPLE (Chair),
ENVIRON, Inc., Emeryville, CA
M. BRUCE BECK,
University of Georgia, Warnell School of Forest Resources, Athens
CLAYTON J. CLARK II,
University of Florida, Gainesville
ROBERT T. CLEMEN,
Duke University, Fuqua School of Business, Durham, NC
JUDITH A. GRAHAM,
American Chemistry Council, Arlington, VA
LOUIS J. GROSS,
University of Tennessee, Institute for Environmental Modeling, Knoxville
WINSTON HARRINGTON,
Resources for the Future, Washington, DC
PHILIP HOWARD,
Syracuse Research Corporation, Environmental Science Center, Syracuse, NY
KIMBERLY L. JONES,
Howard University, Washington, DC
THOMAS E. MCKONE,
University of California, Berkeley, and the Lawrence Berkeley National Laboratory, Berkeley, CA
NAOMI ORESKES,
University of California, San Diego
SPYROS N. PANDIS,
University of Patras, Patra, Greece
LOUISE M. RYAN,
Harvard School of Public Health, Department of Biostatistics, Boston, MA
MICHAEL L. STEIN,
University of Chicago, Statistics Department, Chicago, IL
WENDY E. WAGNER,
University of Texas School of Law, Austin
Staff
K. JOHN HOLMES, Project Director
MATTHEW RUSSELL, Associate Staff Officer
RUTH E. CROSSGROVE, Senior Editor
MIRSADA KARALIC-LONCAREVIC, Manager, Technical Information Center
RADIAH A. ROSE, Senior Editorial Assistant
Sponsor
U.S. ENVIRONMENTAL PROTECTION AGENCY
U.S. DEPARTMENT OF TRANSPORTATION
BOARD ON ENVIRONMENTAL STUDIES AND TOXICOLOGY1
Members
JONATHAN M. SAMET (Chair),
Johns Hopkins University, Baltimore, MD
RAMÓN ALVAREZ,
Environmental Defense, Austin, TX
JOHN M. BALBUS,
Environmental Defense, Washington, DC
DALLAS BURTRAW,
Resources for the Future, Washington, DC
JAMES S. BUS,
Dow Chemical Company, Midland, MI
COSTEL D. DENSON,
University of Delaware, Newark
E. DONALD ELLIOTT,
Willkie, Farr & Gallagher LLP, Washington, DC
MARY R. ENGLISH,
University of Tennessee, Knoxville
J. PAUL GILMAN,
Oak Ridge Center for Advanced Studies, Oak Ridge, TN
SHERRI W. GOODMAN,
Center for Naval Analyses, Alexandria, VA
JUDITH A. GRAHAM,
American Chemistry Council, Arlington, VA
WILLIAM P. HORN,
Birch, Horton, Bittner and Cherot, Washington, DC
JAMES H. JOHNSON, JR.,
Howard University, Washington, DC
WILLIAM M. LEWIS, JR.,
University of Colorado, Boulder
JUDITH L. MEYER,
University of Georgia, Athens
DENNIS D. MURPHY,
University of Nevada, Reno
PATRICK Y. O’BRIEN,
ChevronTexaco Energy Technology Company, Richmond, CA
DOROTHY E. PATTON (retired),
Chicago, IL
DANNY D. REIBLE,
University of Texas, Austin
JOSEPH V. RODRICKS,
ENVIRON International Corporation, Arlington, VA
ARMISTEAD G. RUSSELL,
Georgia Institute of Technology, Atlanta
ROBERT F. SAWYER,
University of California, Berkeley
LISA SPEER,
Natural Resources Defense Council, New York, NY
KIMBERLY M. THOMPSON,
Massachusetts Institute of Technology, Cambridge
MONICA G. TURNER,
University of Wisconsin, Madison
MARK J. UTELL,
University of Rochester Medical Center, Rochester, NY
CHRIS G. WHIPPLE,
ENVIRON International Corporation, Emeryville, CA
LAUREN ZEISE,
California Environmental Protection Agency, Oakland
Senior Staff
JAMES J. REISA, Director
DAVID J. POLICANSKY, Scholar
RAYMOND A. WASSEL, Senior Program Officer for Environmental Sciences and Engineering
KULBIR BAKSHI, Senior Program Officer for Toxicology
EILEEN N. ABT, Senior Program Officer for Risk Analysis
KARL E. GUSTAVSON, Senior Program Officer
K. JOHN HOLMES, Senior Program Officer
ELLEN K. MANTUS, Senior Program Officer
SUSAN N.J. MARTEL, Senior Program Officer
STEVEN K. GIBB, Program Officer for Strategic Communications
RUTH E. CROSSGROVE, Senior Editor
OTHER REPORTS OF THE BOARD ON ENVIRONMENTAL STUDIES AND TOXICOLOGY
Toxicity Testing in the Twenty-first Century: A Vision and a Strategy (2007)
Sediment Dredging at Superfund Megasites: Assessing the Effectiveness (2007)
Environmental Impacts of Wind-Energy Projects (2007)
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget (2007)
Assessing the Human Health Risks of Trichloroethylene: Key Scientific Issues (2006)
New Source Review for Stationary Sources of Air Pollution (2006)
Human Biomonitoring for Environmental Chemicals (2006)
Health Risks from Dioxin and Related Compounds: Evaluation of the EPA Reassessment (2006)
Fluoride in Drinking Water: A Scientific Review of EPA’s Standards (2006)
State and Federal Standards for Mobile-Source Emissions (2006)
Superfund and Mining Megasites—Lessons from the Coeur d’Alene River Basin (2005)
Health Implications of Perchlorate Ingestion (2005)
Air Quality Management in the United States (2004)
Endangered and Threatened Species of the Platte River (2004)
Atlantic Salmon in Maine (2004)
Endangered and Threatened Fishes in the Klamath River Basin (2004)
Cumulative Environmental Effects of Alaska North Slope Oil and Gas Development (2003)
Estimating the Public Health Benefits of Proposed Air Pollution Regulations (2002)
Biosolids Applied to Land: Advancing Standards and Practices (2002)
The Airliner Cabin Environment and Health of Passengers and Crew (2002)
Arsenic in Drinking Water: 2001 Update (2001)
Evaluating Vehicle Emissions Inspection and Maintenance Programs (2001)
Compensating for Wetland Losses Under the Clean Water Act (2001)
A Risk-Management Strategy for PCB-Contaminated Sediments (2001)
Acute Exposure Guideline Levels for Selected Airborne Chemicals (five volumes, 2000-2007)
Toxicological Effects of Methylmercury (2000)
Strengthening Science at the U.S. Environmental Protection Agency (2000)
Scientific Frontiers in Developmental Toxicology and Risk Assessment (2000)
Ecological Indicators for the Nation (2000)
Waste Incineration and Public Health (2000)
Hormonally Active Agents in the Environment (1999)
Research Priorities for Airborne Particulate Matter (four volumes, 1998-2004)
The National Research Council’s Committee on Toxicology: The First 50 Years (1997)
Carcinogens and Anticarcinogens in the Human Diet (1996)
Upstream: Salmon and Society in the Pacific Northwest (1996)
Science and the Endangered Species Act (1995)
Wetlands: Characteristics and Boundaries (1995)
Biologic Markers (five volumes, 1989-1995)
Review of EPA’s Environmental Monitoring and Assessment Program (three volumes, 1994-1995)
Science and Judgment in Risk Assessment (1994)
Pesticides in the Diets of Infants and Children (1993)
Dolphins and the Tuna Industry (1992)
Science and the National Parks (1992)
Human Exposure Assessment for Airborne Pollutants (1991)
Rethinking the Ozone Problem in Urban and Regional Air Pollution (1991)
Decline of the Sea Turtles (1990)
Copies of these reports may be ordered from the National Academies Press
(800) 624-6242 or (202) 334-3313
PREFACE
The use of computational models is an essential element of the environmental regulatory process. The complex relationship between environmental emissions, the quality of the environment, and human and ecological impacts are linked by modeling in the regulatory process. The U.S. Environmental Protection Agency (EPA) may make a scientific determination of basic environmental goals, such as how clean our air and water need to be to protect human health and the environment. But determining how those goals can be met while simultaneously allowing for basic economic services, such as transportation, energy, and agriculture, requires that we examine the links, for example, between the auto emission standards and the attainment of ambient air quality standards or between the point sources of water pollution and the quality of water. The spatial and temporal scales on which environmental controls and environmental quality are linked generally do not allow for an observational approach to understand the links between economic activity and environmental quality. These linkages are made by modeling.
The task undertaken by this committee for the National Academies was to assess evolving scientific and technical issues related to the development, selection, and use of computational and statistical models in the regulatory process at EPA. In this report, the committee provides advice concerning management, evaluation, and use of models at the agency. Through public workshops and other means, the committee has considered cross-discipline issues related to model development and use, performance evaluation, peer review, uncertainty, and quality assurance–quality control. The committee assessed scientific and technical criteria that should be considered in deciding whether a model and its results could serve as a reasonable basis for environmental regulatory activities. It also examined case studies of model development, evaluation, and application as a basis for arriving at guiding principles.
This report has been reviewed in draft form by persons chosen for their diverse perspectives and technical expertise in accordance with procedures approved by National Research Council (NRC) Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards of objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following for their review of this report: George V. Alexeeff, California EPA; Eula Bingham, University of Cincinnati; John Bredehoeft, the Hydrodynamics Group; E. Donald Elliott, Willkie, Farr & Gallagher, LLP; Paul Gilman, Oak Ridge Center for Advanced Studies; James Hammitt, Harvard Center for Risk Analysis; Michael Koerber, Lake Michigan Air Directors Consortium; Charles Lucas, American International Group, Inc. (retired); Virginia McConnell, Resources for the Future, Inc.; Jana Milford, University of Colorado and Environmental Defense; Lee Mulkey, University of Georgia; Kenneth Reckhow, Duke University; and Scott Zeger, Johns Hopkins University.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor
did they see the final draft of the report before its release. The review of this report was overseen by John Bailar, University of Chicago (retired), and David Allen, University of Texas. Appointed by the NRC, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the committee and the institution.
The committee received oral and written presentations from the following individuals:
Gary Foley, U.S. Environmental Protection Agency
Tom Voltaggio, U.S. Environmental Protection Agency
Albert McGartland, U.S. Environmental Protection Agency
S.T. Rao, U.S. Environmental Protection Agency
Joseph Merenda, U.S. Environmental Protection Agency
Jim Weaver, U.S. Environmental Protection Agency
David Burden, U.S. Environmental Protection Agency
Tim Wool, U.S. Environmental Protection Agency
Leslie Shoemaker, Tetra Tech, Inc.
Jim George, Maryland Department of the Environment
Cecilia Ho, Federal Highway Administration
Harry Kitch, U.S. Environmental Protection Agency
Timothy Miller, U.S. Geological Survey
Jennifer Sass, Natural Resources Defense Council
Scott Slaughter, Center for Regulatory Effectiveness
Adam Finkel, University of Medicine and Dentistry of New Jersey and Princeton University
Gene Tierney, U.S. Environmental Protection Agency
H. Christopher Frey, North Carolina State University
Margo Schwab, Office of Management and Budget
John Graham, Rand Graduate School
Rob Howard, Bechtel-SAIC, LLC
Sheila Jasanoff, Harvard University
Daniel Krewski, McLaughlin Centre for Population Health Risk Assessment
Jan M. Zielinski, McLaughlin Centre for Population Health Risk Assessment
Tim Ramsay, McLaughlin Centre for Population Health Risk Assessment
Richard T. Burnett, McLaughlin Centre for Population Health Risk Assessment
George Leavesley, U.S. Geological Survey
Sam Napolitano, U.S. Environmental Protection Agency
Elliot Lieberman, U.S. Environmental Protection Agency
M. Granger Morgan, Carnegie Mellon University
Pasky Pascual, U.S. Environmental Protection Agency
Barbara Petersen, Exponent, Inc. and Durango Software, LLC
James D. Schaub, U.S. Department of Agriculture
Woodrow Setzer, U.S. Environmental Protection Agency
Harvey Clewell, Centers for Health Research
Rory Conolly, U.S. Environmental Protection Agency
Richard Morgenstern, Resources for the Future
Robert Perciasepe, Audubon Society
Kenneth Reckhow, Duke University
Paul Gilman, Oak Ridge Center for Advanced Studies
The committee and I thank all of these individuals for their contributions. A complete list of dates, titles, and presenter names can be found in Appendix B.
The committee and I are also grateful for the assistance of the NRC staff in the preparation of this report. K. John Holmes played a key role in preparing this report as project director. We also thank Raymond Wassel, senior program director of environmental sciences and engineering in the Board on Environmental Studies and Toxicology (BEST), and the other staff members contributing to this report: James Reisa, director of BEST; Steven Gibb, program officer for strategic communications; Ruth Crossgrove, senior editor; Matthew Russell, associate staff officer; Mirsada Karalic-Loncarevic, manager of the Technical Information Center; and Radiah Rose, senior editorial assistant.
As chair, I thank all the members of the committee for their expertise and dedicated effort throughout the study.
Chris Whipple, Chair
Committee on Models in the Regulatory
Decision Process
FIGURES
1-1 |
Appendix J curve, |
|||
1-2 |
Typical EKMA diagram, |
|||
1-3 |
An orrery or physical model of the solar system, |
|||
1-4 |
The use of a mouse model for estimating human health risks, |
|||
1-5 |
Examples of dose-response models for estimating lifetime risk for male bladder cancers due to arsenic in drinking water for various exposed populations, |
|||
1-6 |
Components and functional arrangements of the IEUBK model that predict blood lead levels in children, |
|||
2-1 |
Basic modeling elements relating human activities and natural systems to environmental impacts, |
|||
2-2 |
Basic elements of risk assessment from the National Research Council’s Red Book, |
|||
2-3 |
Elements of advanced mechanistic approaches to health risk assessment, |
|||
2-4 |
Sources of information for setting various NAAQS, |
|||
2-5 |
Flow chart of general regulatory requirements for models used at the regulatory design and promulgation stage, |
3-1 |
Iterative steps in model development proposed by Jakeman et al. (2006), |
|||
3-2 |
Conceptual model for assessing eutrophication in the European seas linking nutrient enrichments and its direct and indirect effects in the ecosystem, |
|||
4-1 |
Stages of a model’s life cycle, |
|||
4-2 |
Range of probabilities that people assign to different words absent any specific context, |
|||
4-3 |
(a) Hypothetical distribution representing uncertainty in number of lives saved by a policy; (b) distribution representing uncertainty in value of a statistical life, |
|||
4-4 |
Unconditional posterior distribution for net benefit of policy, |
|||
4-5 |
Conditional distributions of net benefit assuming different amounts for the value of a statistical life, |
|||
5-1 |
(a) Individual dose-response models, and (b) overall dose-response model fitted using the Bayesian model averaging approach, |
TABLES
2-1 |
Examples of EPA’s Web Sites Containing Model Descriptions for Individual Programs, |
|||
2-2 |
Examples of Major EPA Documents That Incorporate a Substantial Amount of Computational Modeling Activities, |
|||
2-3 |
Examples of Substantive Legislative Directions for EPA Models, |
|||
3-1 |
MOBILE Model Revisions, |
|||
4-1 |
QA/QC Checks for Model Code, |
BOXES
S-1 |
Task Statement, |
|||
1-1 |
Ozone Modeling and the Irregular Swings Between Policy and Science, |
|||
2-1 |
Incorporating Human Behavior into Environmental Models, |
|||
2-2 |
Risk Assessment for Arsenic in Drinking Water, |
|||
2-3 |
The Development of the Requirement for Regulatory Impact Analysis for Major Federal Rules, |
|||
2-4 |
Elements of External Peer Review for Environmental Regulatory Models, |
|||
2-5 |
The Different Types of Science Advisory Panels at EPA, |
|||
3-1 |
Basic Steps in Modeling Development Process, |
|||
3-2 |
Interdependence of Models and Data from Measurements, |
|||
4-1 |
Attributes That Foster Accuracy, Precision, Parsimony, and Transparency in Models, |
|||
4-2 |
Individual Elements of Model Evaluation, |
|||
4-3 |
To Calibrate or Not To Calibrate, |
|||
4-5 |
Life-Cycle Evaluation of Models for Assessing Persistence and Long-Range Transport Potential, |
|||
4-6 |
Retrospective Analysis of Model Predictions, |
5-1 |
Example of a Generic Model for Application to Specific Settings, |
|||
5-2 |
Appendix W: EPA’s Guidelines on Air Quality Models, |
|||
5-3 |
Arsenic in Drinking Water: Model Selection, |
|||
5-4 |
Use of Multiple Models of Varying Complexity for Estimating Mercury in Fish, |
|||
5-5 |
Model Training and Support, |
|||
5-6 |
Confidence Building in Models Through Transparency, |
|||
5-7 |
Proprietary Components of Environmental Models, |