Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget (2007)

In January 2006, the Office of Management and Budget (OMB) released a draft bulletin that proposes technical guidance for risk assessments produced by the federal government. The bulletin defines risk assessment broadly, states several goals for risk assessment, and proposes general risk assessment and reporting standards and special standards for influential risk assessments. The stated intent of the bulletin is “to enhance the technical quality and objectivity of risk assessments prepared by federal agencies by establishing uniform, minimum standards,” and it follows several other influential documents issued by OMB, including the Information Quality Guidelines, the Information Quality Bulletin on Peer Review, and Circular A-4, which pertains primarily to benefit-cost analysis and cost-effectiveness analysis. Recognizing the potential impact on federal agencies, OMB—with the Environmental Protection Agency (EPA), the U.S. Department of Agriculture (USDA), the Department of Defense (DOD), the Department of Energy (DOE), the Department of Health and Human Services (DHHS), the Department of Labor (DOL), and the National Aeronautics and Space Administration (NASA)—asked the National Research Council (NRC) to conduct an independent review of the bulletin. In response to that request, NRC convened the Committee to Review the OMB Risk Assessment Bulletin, which prepared this report.

The committee was asked to conduct a scientific and technical review of the proposed bulletin and to determine whether it meets OMB’s objective to “enhance the technical quality and objectivity of risk as-

sessments prepared by federal agencies.” In performing its task, the committee was asked to comment, in general terms, on how the guidance will affect the practice of risk assessment in the federal government, to identify critical elements that might be missing from the guidance, and to assess whether there are scientific or technical circumstances that might limit applicability of the guidance. In addition, the committee was asked whether OMB appropriately incorporated recommendations from previous reports of the NRC and other organizations into the proposed risk assessment guidance.

To accomplish its task, the committee held a large public meeting during which it heard presentations from the study sponsors and other invited speakers from private industry, universities, trade associations, and environmental groups. The committee reviewed numerous documents cited in the bulletin and reviewed public comments submitted to OMB on the bulletin. The committee also requested information from the federal agencies on their risk assessment practices and their view of the potential impact of the bulletin on current practices. The committee reviewed both the bulletin and the accompanying supplementary information, and reference to “the bulletin” in this summary includes both the bulletin and the supplementary information.

Although this report touches on some statutory, policy, and budgetary issues, it is not a comprehensive review of all potential impacts of the bulletin. Rather, it is primarily a review of the science involved and the technical applications of the bulletin. Furthermore, much of the language used (and the examples provided) in the bulletin is related to human health risk assessment and not engineering, ecologic, or behavioral risk assessment. The committee recognizes that each of these fields has generated risk assessment methods that address specific interests. However, the committee was tasked with reviewing the bulletin and not providing a comprehensive treatment of risk assessment, so its comments focus mainly on human health risk assessment, as did the OMB bulletin.

The general thrust of the bulletin appears to be consistent with many of the themes and recommendations in reports by previous NRC committees and other expert organizations. The bulletin emphasizes the

need to define objectives clearly and to ensure that assessments yield results that are both faithful to underlying scientific knowledge and useful for decision-making. The committee, however, is concerned that the bulletin is inconsistent with previous recommendations in a number of ways, including its presentation of a new definition of risk assessment, its omission of discussion of the important role of default assumptions and clear criteria to modify or depart from defaults, its proposal of risk assessment standards related to activities traditionally regarded as risk management activities, and its requirement for formal analyses of uncertainty and presentation of “central” or “expected” risk estimates. In several respects, the bulletin attempts to move standards for risk assessment into territory that is beyond what previous reports have recommended and beyond the current state of the science. Such departures from expert studies are of serious concern, because any attempt to advance the practice of risk assessment that does not reflect the state of the science is likely to produce the opposite effect.

The bulletin defines risk assessment as “a scientific and/or technical document that assembles and synthesizes scientific information to determine whether a potential hazard exists and/or the extent of possible risk to human health, safety or the environment.” That definition conflicts with long-established concepts and practices that have defined risk assessment as a process that involves hazard identification, hazard characterization or dose-response assessment, exposure assessment, and risk characterization. The definition in the bulletin is too broad and encompasses not only traditional risk assessments but the components of risk assessment. Such a definition, which captures a variety of analyses under the same name, could cause great confusion. Moreover, several standards proposed in the bulletin are not applicable to individual components of risk assessment or other types of documents that might be classified as risk assessment under the proposed definition.

The bulletin defines five goals of risk assessment that are related to problem formulation, completeness, character of risk assessment, resources expended, and peer review and public participation. Taken as a whole, the five goals indicate that a risk assessment should be tailored to the specific need for which it is undertaken; balanced in scope, time, and cost with the importance of the issue; and peer-reviewed and released for

public comment. The goals mostly emphasize efficiency, rather than quality, in the conduct of risk assessment. Thus, the goals do not all support the primary purpose of the bulletin—“to enhance the technical quality and objectivity of risk assessments.”

The bulletin proposes seven standards for general risk assessment—one of which refers to risk assessments for regulatory analysis—and nine special standards for influential risk assessments. The committee found this structure problematic, because one may not know at the outset whether an analysis will constitute an “influential” risk assessment. Furthermore, arbitrarily separating risk assessment into two broad categories (general and influential) ignores the continuum of risk assessment efforts. The committee reviewed each standard and provides comments on them in this report. In general, the committee found many of the standards to be unclear or flawed. Standards on presentation of specific information, uncertainty, and adversity of health effects exemplify the problems.

Several standards require the presentation of “a range of plausible risk estimates” that includes “central or expected estimates.” The discussion regarding this requirement is incomplete and confusing. Those numerical quantities are meaningful only in the context of some distribution that arises when variability and uncertainty are taken into consideration. A central estimate and a risk range might be misleading in situations when sensitive populations are of primary concern. Thus, the choice of summary statistics cannot be a blanket prescription but must reflect the specific context.

Standards for influential risk assessments require a formal characterization of uncertainty. However, the description of uncertainty and variability in the bulletin is oversimplified and does not recognize the complexities of different types of risk assessments or the need to tailor uncertainty analysis to a given agency’s particular needs. Furthermore, there is no scientific consensus to support the bulletin’s universal prescriptions for how uncertainty should be evaluated. In the absence of clear guidance regarding the conduct of uncertainty analysis, there is a serious danger that agencies will produce ranges of meaningless and confusing risk estimates, which could result in risk assessments of reduced rather than enhanced quality and objectivity.

Finally, for influential risk assessments, the bulletin states that “where human health effects are a concern, determinations of which effects are adverse shall be specifically identified and justified.” The bulletin’s definition of adverse effect implies a clinically apparent effect, which ignores a fundamental public-health goal to control exposures well before the occurrence of any possible functional impairment of an organism. Dividing effects into “adverse” and “nonadverse” ignores the scientific reality that adverse effects may be manifest along a continuum. The committee concludes that the bulletin’s treatment of adverse effects is too simplistic and restrictive and ignores important factors in determining appropriate effects to evaluate, the scientific information available, and an understanding of the underlying biochemical mechanisms for an effect of interest.

Omission of several relevant topics limits the utility of the bulletin as balanced and comprehensive risk assessment guidance. Specifically, OMB has proposed a bulletin addressing risk assessment in the federal government; however, the bulletin focuses mainly on biologic systems, with an emphasis on human health risk assessment. The vast majority of examples it presents (and the authorities cited) apply to toxicologic and other human health end points. By reducing risks to human health risks, as important as they may be, OMB commits a serious error in neglecting risk assessment of technology and engineered structures. Those are of vital importance to such agencies as DOE, DOD, and NASA and therefore to the general public and the economic vitality of the United States. The bulletin’s incomplete and unbalanced approach to engineering risk assessment (as well as ecologic and other types of risk assessment) contradicts its stated objective of improving the quality of risk assessment throughout the federal government. Unless all risk assessment disciplines are considered, any government-wide guidance on risk assessment would be unacceptable.

Furthermore, the bulletin gives little attention to sensitive populations, the often pivotal role of risk assessment policy in choices regarding default options, the integral role of risk communication, and standards for risk assessments submitted by outside parties for use in the rulemaking process. With reference to risk communication, the committee agrees with previous NRC reports that view risk communication as a dia-

logue with users of risk assessment throughout the process that helps to ensure its relevance and credibility and does not see it as a one-way, end-of-the process activity. The bulletin also fails to explain the basis for exempting risk assessments associated with licensing and approval processes.

Perhaps the most glaring omission is the absence of criteria and information for gauging the benefits to be achieved by implementing the bulletin (that is, a benefit-cost analysis). Although OMB has implied that the agencies currently do not meet the standards that it seeks to establish, it has not established a baseline of each agency’s risk assessment proficiency, including the extent to which generally satisfactory and highquality risk assessments are produced or how some agencies fall short of the specified standards. Specifically, OMB has not established which agencies do not appear to know what good practices are and which agencies do not have the ability, resources, or incentives to meet the standards. Similarly, OMB has not identified the costs that could be encountered in implementing the bulletin. Thus, OMB has not determined the impact of the bulletin on federal agencies.

Although OMB did not construct a baseline reflecting current agency risk assessment practices, the committee concludes on the basis of agency comments and its own knowledge of risk assessment practices that some aspects of the bulletin could be beneficial but that the costs— in terms of staff resources, timeliness of completing risk assessments, and other factors—are likely to be substantial. Overall, the committee concludes that the potential for negative impacts on the practice of risk assessment in the federal government, although varied and uncertain to some extent, would be very high if the currently proposed bulletin were implemented.

On the basis of its review, the committee concludes that the OMB bulletin is fundamentally flawed and recommends that it be withdrawn. Although the committee fully supports the goal of increasing the quality and objectivity of risk assessment in the federal government, it agrees

unanimously that the OMB bulletin would not facilitate reaching this goal. The committee also agrees that OMB should encourage the federal agencies to describe, develop, and coordinate their own technical risk assessment guidance. Therefore, the committee recommends that, after additional study of current agency practices and needs, a different type of risk assessment bulletin be issued by OMB. That bulletin should outline goals and general principles of risk assessment designed to enhance the quality, efficiency, and consistency of risk assessment in the federal government. It should direct the agencies to develop technical guidance that would implement the general principles, be consistent with the individual agencies’ legislative mandates and missions, and draw on the expertise that exists in federal agencies and other organizations. The technical guidance developed or identified by the agencies should be peer-reviewed and contain procedures for ensuring compliance with the guidance within the agencies. Although OMB should determine whether the technical guidance developed by the agencies fully addresses the general principles, the committee recommends that development and peer review of agency technical guidance be left to the agencies. The committee strongly recommends that federal agencies addressing similar hazards or risks work together to develop common technical guidance for risk assessment; that would help to achieve the appropriate consistency among agencies in risk assessment practices.

The committee arrived at its position after deliberate consideration of many factors. The committee began with the working assumption that its role would be to recommend modifications, if necessary. After digging deeply into the bulletin and after extensive discussion, the committee reluctantly came to its conclusion that the bulletin could not be rescued.

Risk assessment is not a monolithic process or a single method. Different technical issues arise in assessing the probability of exposure to a given dose of a chemical, of a malfunction of a nuclear power plant or air-traffic control system, or of the collapse of an ecosystem or a dam. Thus, one size does not fit all, nor can one set of technical guidance make sense for the heterogeneous risk assessments undertaken by federal agencies. Although the bulletin generally acknowledges that diversity and attempts to meet it with frequent references to “where appropriate” or “where feasible,” the bulletin does not reflect an adequate understanding of the many risk assessment disciplines, particularly those devoted to analyzing the risks of engineered structures and natural systems. Its narrow focus on human health risk assessment makes it inappropriate as

across-the-board guidance for all risk assessments conducted throughout the federal government. Furthermore, as stated above, the committee strongly recommends that technical guidance be produced by the individual agencies and that agencies dealing with the same or similar hazards work together to produce common guidance to ensure an appropriately consistent approach.

The committee agrees that there is room for improvement in risk assessment practices in the federal government and that additional guidance would help “to enhance the technical quality and objectivity of risk assessments prepared by federal agencies.” However, the committee concludes that OMB should limit its efforts to stating goals and general principles of risk assessment. The details should be left to the agencies or expert committees appointed by the agencies, wherein lies the depth of expertise to address the issues relevant to their specific types of risk assessments.