National Academies Press: OpenBook
« Previous: Front Matter
Suggested Citation:"Summary." National Research Council. 2007. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget. Washington, DC: The National Academies Press. doi: 10.17226/11811.
×

Summary

In January 2006, the Office of Management and Budget (OMB) released a draft bulletin that proposes technical guidance for risk assessments produced by the federal government. The bulletin defines risk assessment broadly, states several goals for risk assessment, and proposes general risk assessment and reporting standards and special standards for influential risk assessments. The stated intent of the bulletin is “to enhance the technical quality and objectivity of risk assessments prepared by federal agencies by establishing uniform, minimum standards,” and it follows several other influential documents issued by OMB, including the Information Quality Guidelines, the Information Quality Bulletin on Peer Review, and Circular A-4, which pertains primarily to benefit-cost analysis and cost-effectiveness analysis. Recognizing the potential impact on federal agencies, OMB—with the Environmental Protection Agency (EPA), the U.S. Department of Agriculture (USDA), the Department of Defense (DOD), the Department of Energy (DOE), the Department of Health and Human Services (DHHS), the Department of Labor (DOL), and the National Aeronautics and Space Administration (NASA)—asked the National Research Council (NRC) to conduct an independent review of the bulletin. In response to that request, NRC convened the Committee to Review the OMB Risk Assessment Bulletin, which prepared this report.

COMMITTEE’S CHARGE AND APPROACH TO ITS CHARGE

The committee was asked to conduct a scientific and technical review of the proposed bulletin and to determine whether it meets OMB’s objective to “enhance the technical quality and objectivity of risk as-

Suggested Citation:"Summary." National Research Council. 2007. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget. Washington, DC: The National Academies Press. doi: 10.17226/11811.
×

sessments prepared by federal agencies.” In performing its task, the committee was asked to comment, in general terms, on how the guidance will affect the practice of risk assessment in the federal government, to identify critical elements that might be missing from the guidance, and to assess whether there are scientific or technical circumstances that might limit applicability of the guidance. In addition, the committee was asked whether OMB appropriately incorporated recommendations from previous reports of the NRC and other organizations into the proposed risk assessment guidance.

To accomplish its task, the committee held a large public meeting during which it heard presentations from the study sponsors and other invited speakers from private industry, universities, trade associations, and environmental groups. The committee reviewed numerous documents cited in the bulletin and reviewed public comments submitted to OMB on the bulletin. The committee also requested information from the federal agencies on their risk assessment practices and their view of the potential impact of the bulletin on current practices. The committee reviewed both the bulletin and the accompanying supplementary information, and reference to “the bulletin” in this summary includes both the bulletin and the supplementary information.

Although this report touches on some statutory, policy, and budgetary issues, it is not a comprehensive review of all potential impacts of the bulletin. Rather, it is primarily a review of the science involved and the technical applications of the bulletin. Furthermore, much of the language used (and the examples provided) in the bulletin is related to human health risk assessment and not engineering, ecologic, or behavioral risk assessment. The committee recognizes that each of these fields has generated risk assessment methods that address specific interests. However, the committee was tasked with reviewing the bulletin and not providing a comprehensive treatment of risk assessment, so its comments focus mainly on human health risk assessment, as did the OMB bulletin.

COMMITTEE’S REVIEW

Consistency with NRC and Other Reports

The general thrust of the bulletin appears to be consistent with many of the themes and recommendations in reports by previous NRC committees and other expert organizations. The bulletin emphasizes the

Suggested Citation:"Summary." National Research Council. 2007. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget. Washington, DC: The National Academies Press. doi: 10.17226/11811.
×

need to define objectives clearly and to ensure that assessments yield results that are both faithful to underlying scientific knowledge and useful for decision-making. The committee, however, is concerned that the bulletin is inconsistent with previous recommendations in a number of ways, including its presentation of a new definition of risk assessment, its omission of discussion of the important role of default assumptions and clear criteria to modify or depart from defaults, its proposal of risk assessment standards related to activities traditionally regarded as risk management activities, and its requirement for formal analyses of uncertainty and presentation of “central” or “expected” risk estimates. In several respects, the bulletin attempts to move standards for risk assessment into territory that is beyond what previous reports have recommended and beyond the current state of the science. Such departures from expert studies are of serious concern, because any attempt to advance the practice of risk assessment that does not reflect the state of the science is likely to produce the opposite effect.

Definition of Risk Assessment and the Bulletin’s Goals

The bulletin defines risk assessment as “a scientific and/or technical document that assembles and synthesizes scientific information to determine whether a potential hazard exists and/or the extent of possible risk to human health, safety or the environment.” That definition conflicts with long-established concepts and practices that have defined risk assessment as a process that involves hazard identification, hazard characterization or dose-response assessment, exposure assessment, and risk characterization. The definition in the bulletin is too broad and encompasses not only traditional risk assessments but the components of risk assessment. Such a definition, which captures a variety of analyses under the same name, could cause great confusion. Moreover, several standards proposed in the bulletin are not applicable to individual components of risk assessment or other types of documents that might be classified as risk assessment under the proposed definition.

The bulletin defines five goals of risk assessment that are related to problem formulation, completeness, character of risk assessment, resources expended, and peer review and public participation. Taken as a whole, the five goals indicate that a risk assessment should be tailored to the specific need for which it is undertaken; balanced in scope, time, and cost with the importance of the issue; and peer-reviewed and released for

Suggested Citation:"Summary." National Research Council. 2007. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget. Washington, DC: The National Academies Press. doi: 10.17226/11811.
×

public comment. The goals mostly emphasize efficiency, rather than quality, in the conduct of risk assessment. Thus, the goals do not all support the primary purpose of the bulletin—“to enhance the technical quality and objectivity of risk assessments.”

Proposed Standards for Risk Assessment

The bulletin proposes seven standards for general risk assessment—one of which refers to risk assessments for regulatory analysis—and nine special standards for influential risk assessments. The committee found this structure problematic, because one may not know at the outset whether an analysis will constitute an “influential” risk assessment. Furthermore, arbitrarily separating risk assessment into two broad categories (general and influential) ignores the continuum of risk assessment efforts. The committee reviewed each standard and provides comments on them in this report. In general, the committee found many of the standards to be unclear or flawed. Standards on presentation of specific information, uncertainty, and adversity of health effects exemplify the problems.

Several standards require the presentation of “a range of plausible risk estimates” that includes “central or expected estimates.” The discussion regarding this requirement is incomplete and confusing. Those numerical quantities are meaningful only in the context of some distribution that arises when variability and uncertainty are taken into consideration. A central estimate and a risk range might be misleading in situations when sensitive populations are of primary concern. Thus, the choice of summary statistics cannot be a blanket prescription but must reflect the specific context.

Standards for influential risk assessments require a formal characterization of uncertainty. However, the description of uncertainty and variability in the bulletin is oversimplified and does not recognize the complexities of different types of risk assessments or the need to tailor uncertainty analysis to a given agency’s particular needs. Furthermore, there is no scientific consensus to support the bulletin’s universal prescriptions for how uncertainty should be evaluated. In the absence of clear guidance regarding the conduct of uncertainty analysis, there is a serious danger that agencies will produce ranges of meaningless and confusing risk estimates, which could result in risk assessments of reduced rather than enhanced quality and objectivity.

Suggested Citation:"Summary." National Research Council. 2007. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget. Washington, DC: The National Academies Press. doi: 10.17226/11811.
×

Finally, for influential risk assessments, the bulletin states that “where human health effects are a concern, determinations of which effects are adverse shall be specifically identified and justified.” The bulletin’s definition of adverse effect implies a clinically apparent effect, which ignores a fundamental public-health goal to control exposures well before the occurrence of any possible functional impairment of an organism. Dividing effects into “adverse” and “nonadverse” ignores the scientific reality that adverse effects may be manifest along a continuum. The committee concludes that the bulletin’s treatment of adverse effects is too simplistic and restrictive and ignores important factors in determining appropriate effects to evaluate, the scientific information available, and an understanding of the underlying biochemical mechanisms for an effect of interest.

Omissions from the Bulletin

Omission of several relevant topics limits the utility of the bulletin as balanced and comprehensive risk assessment guidance. Specifically, OMB has proposed a bulletin addressing risk assessment in the federal government; however, the bulletin focuses mainly on biologic systems, with an emphasis on human health risk assessment. The vast majority of examples it presents (and the authorities cited) apply to toxicologic and other human health end points. By reducing risks to human health risks, as important as they may be, OMB commits a serious error in neglecting risk assessment of technology and engineered structures. Those are of vital importance to such agencies as DOE, DOD, and NASA and therefore to the general public and the economic vitality of the United States. The bulletin’s incomplete and unbalanced approach to engineering risk assessment (as well as ecologic and other types of risk assessment) contradicts its stated objective of improving the quality of risk assessment throughout the federal government. Unless all risk assessment disciplines are considered, any government-wide guidance on risk assessment would be unacceptable.

Furthermore, the bulletin gives little attention to sensitive populations, the often pivotal role of risk assessment policy in choices regarding default options, the integral role of risk communication, and standards for risk assessments submitted by outside parties for use in the rulemaking process. With reference to risk communication, the committee agrees with previous NRC reports that view risk communication as a dia-

Suggested Citation:"Summary." National Research Council. 2007. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget. Washington, DC: The National Academies Press. doi: 10.17226/11811.
×

logue with users of risk assessment throughout the process that helps to ensure its relevance and credibility and does not see it as a one-way, end-of-the process activity. The bulletin also fails to explain the basis for exempting risk assessments associated with licensing and approval processes.

Perhaps the most glaring omission is the absence of criteria and information for gauging the benefits to be achieved by implementing the bulletin (that is, a benefit-cost analysis). Although OMB has implied that the agencies currently do not meet the standards that it seeks to establish, it has not established a baseline of each agency’s risk assessment proficiency, including the extent to which generally satisfactory and highquality risk assessments are produced or how some agencies fall short of the specified standards. Specifically, OMB has not established which agencies do not appear to know what good practices are and which agencies do not have the ability, resources, or incentives to meet the standards. Similarly, OMB has not identified the costs that could be encountered in implementing the bulletin. Thus, OMB has not determined the impact of the bulletin on federal agencies.

Impact on Risk Assessment Practices in the Federal Government

Although OMB did not construct a baseline reflecting current agency risk assessment practices, the committee concludes on the basis of agency comments and its own knowledge of risk assessment practices that some aspects of the bulletin could be beneficial but that the costs— in terms of staff resources, timeliness of completing risk assessments, and other factors—are likely to be substantial. Overall, the committee concludes that the potential for negative impacts on the practice of risk assessment in the federal government, although varied and uncertain to some extent, would be very high if the currently proposed bulletin were implemented.

COMMITTEE’S CONCLUSIONS AND RECOMMENDATIONS

On the basis of its review, the committee concludes that the OMB bulletin is fundamentally flawed and recommends that it be withdrawn. Although the committee fully supports the goal of increasing the quality and objectivity of risk assessment in the federal government, it agrees

Suggested Citation:"Summary." National Research Council. 2007. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget. Washington, DC: The National Academies Press. doi: 10.17226/11811.
×

unanimously that the OMB bulletin would not facilitate reaching this goal. The committee also agrees that OMB should encourage the federal agencies to describe, develop, and coordinate their own technical risk assessment guidance. Therefore, the committee recommends that, after additional study of current agency practices and needs, a different type of risk assessment bulletin be issued by OMB. That bulletin should outline goals and general principles of risk assessment designed to enhance the quality, efficiency, and consistency of risk assessment in the federal government. It should direct the agencies to develop technical guidance that would implement the general principles, be consistent with the individual agencies’ legislative mandates and missions, and draw on the expertise that exists in federal agencies and other organizations. The technical guidance developed or identified by the agencies should be peer-reviewed and contain procedures for ensuring compliance with the guidance within the agencies. Although OMB should determine whether the technical guidance developed by the agencies fully addresses the general principles, the committee recommends that development and peer review of agency technical guidance be left to the agencies. The committee strongly recommends that federal agencies addressing similar hazards or risks work together to develop common technical guidance for risk assessment; that would help to achieve the appropriate consistency among agencies in risk assessment practices.

The committee arrived at its position after deliberate consideration of many factors. The committee began with the working assumption that its role would be to recommend modifications, if necessary. After digging deeply into the bulletin and after extensive discussion, the committee reluctantly came to its conclusion that the bulletin could not be rescued.

Risk assessment is not a monolithic process or a single method. Different technical issues arise in assessing the probability of exposure to a given dose of a chemical, of a malfunction of a nuclear power plant or air-traffic control system, or of the collapse of an ecosystem or a dam. Thus, one size does not fit all, nor can one set of technical guidance make sense for the heterogeneous risk assessments undertaken by federal agencies. Although the bulletin generally acknowledges that diversity and attempts to meet it with frequent references to “where appropriate” or “where feasible,” the bulletin does not reflect an adequate understanding of the many risk assessment disciplines, particularly those devoted to analyzing the risks of engineered structures and natural systems. Its narrow focus on human health risk assessment makes it inappropriate as

Suggested Citation:"Summary." National Research Council. 2007. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget. Washington, DC: The National Academies Press. doi: 10.17226/11811.
×

across-the-board guidance for all risk assessments conducted throughout the federal government. Furthermore, as stated above, the committee strongly recommends that technical guidance be produced by the individual agencies and that agencies dealing with the same or similar hazards work together to produce common guidance to ensure an appropriately consistent approach.

The committee agrees that there is room for improvement in risk assessment practices in the federal government and that additional guidance would help “to enhance the technical quality and objectivity of risk assessments prepared by federal agencies.” However, the committee concludes that OMB should limit its efforts to stating goals and general principles of risk assessment. The details should be left to the agencies or expert committees appointed by the agencies, wherein lies the depth of expertise to address the issues relevant to their specific types of risk assessments.

Suggested Citation:"Summary." National Research Council. 2007. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget. Washington, DC: The National Academies Press. doi: 10.17226/11811.
×
Page 1
Suggested Citation:"Summary." National Research Council. 2007. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget. Washington, DC: The National Academies Press. doi: 10.17226/11811.
×
Page 2
Suggested Citation:"Summary." National Research Council. 2007. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget. Washington, DC: The National Academies Press. doi: 10.17226/11811.
×
Page 3
Suggested Citation:"Summary." National Research Council. 2007. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget. Washington, DC: The National Academies Press. doi: 10.17226/11811.
×
Page 4
Suggested Citation:"Summary." National Research Council. 2007. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget. Washington, DC: The National Academies Press. doi: 10.17226/11811.
×
Page 5
Suggested Citation:"Summary." National Research Council. 2007. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget. Washington, DC: The National Academies Press. doi: 10.17226/11811.
×
Page 6
Suggested Citation:"Summary." National Research Council. 2007. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget. Washington, DC: The National Academies Press. doi: 10.17226/11811.
×
Page 7
Suggested Citation:"Summary." National Research Council. 2007. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget. Washington, DC: The National Academies Press. doi: 10.17226/11811.
×
Page 8
Next: 1 Introduction »
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget Get This Book
×
Buy Paperback | $84.00 Buy Ebook | $64.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Risk assessments are often used by the federal government to estimate the risk the public may face from such things as exposure to a chemical or the potential failure of an engineered structure, and they underlie many regulatory decisions. Last January, the White House Office of Management and Budget (OMB) issued a draft bulletin for all federal agencies, which included a new definition of risk assessment and proposed standards aimed at improving federal risk assessments. This National Research Council report, written at the request of OMB, evaluates the draft bulletin and supports its overall goals of improving the quality of risk assessments. However, the report concludes that the draft bulletin is "fundamentally flawed" from a scientific and technical standpoint and should be withdrawn. Problems include an overly broad definition of risk assessment in conflict with long-established concepts and practices, and an overly narrow definition of adverse health effects—one that considers only clinically apparent effects to be adverse, ignoring other biological changes that could lead to health effects. The report also criticizes the draft bulletin for focusing mainly on human health risk assessments while neglecting assessments of technology and engineered structures.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!