Appendix E
Questions for Federal Agencies from the Committee and Agency Responses to Questions



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget Appendix E Questions for Federal Agencies from the Committee and Agency Responses to Questions

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget This page intentionally left blank.

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget BACKGROUND INFORMATION ON NRC REVIEW OF THE OMB RISK ASSESSMENT BULLETIN The National Research Council’s Committee to Review the OMB Risk Assessment Bulletin has been tasked with conducting a scientific review of the proposed Risk Assessment Bulletin released by the Office of Management and Budget (OMB). More specifically, the committee was asked to determine whether the application of the proposed guidance will meet OMB's stated objective to “enhance the technical quality and objectivity of risk assessments prepared by federal agencies.” The committee will evaluate generally the impact of the Bulletin on risk practices, identify possible omissions from the Bulletin, and determine whether there are circumstances that might limit applicability. To address its charge, the committee is hoping that the agencies will assist it by responding to the questions below. QUESTIONS FOR ALL AGENCIES POTENTIALLY AFFECTED BY THE OMB BULLETIN General questions about current risk assessment practices Please provide a brief overview of your current risk assessment practices. Specifically, do you conduct probabilistic risk assessment? Is there a common approach to both risk assessments and uncertainty analysis? How do you currently address uncertainty and variability in your agency's risk assessments? Please identify any substantial scientific or technical challenges that you may encounter when conducting risk assessments for your agency. What is your current definition of risk assessment, and what types of products are covered by that definition? About how long (that is, from initiation of the risk assessment to delivery to the regulatory decision maker) does it take to produce the various types of risk assessments? Questions about OMB’s definition of risk assessment and applicability Using the definition of risk assessment described in the OMB Bul-

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget letin, are there work products that would now be considered risk assessments that were not previously considered risk assessments? If so, what are they? Questions about type of risk assessment (tiered structure) In your agency, is there currently a clear demarcation between risk assessments used for regulatory analysis and those not used for regulatory analysis? Is this clear at the outset of the risk assessment? In your agency, is there currently a clear demarcation between “influential risk assessment” used for regulatory purposes and other risk assessments used for regulatory purposes? Is this clear at the outset of the risk assessment? Questions about impact of the Bulletin on agency risk assessment practices If applicable, please specify provisions in the Bulletin that can be expected to have a substantial positive effect on the quality, conduct, and use of risk assessments undertaken by your agency. If applicable, please specify provisions in the Bulletin that can be expected to have a substantial negative effect on the quality, conduct, and use of risk assessments undertaken by your agency. If your agency followed the procedures described in the Bulletin, would it affect the time course for production of the risk assessment (that is, the time required from initiation of the risk assessment to delivery to the regulatory decision maker)? If so, please explain why? One of the Bulletin’s reporting standards states the need to be scientifically objective by “giving weight to both positive and negative studies in light of each study’s technical quality.” Please give an example of how this would be implemented by your agency or department. Does your agency use risk assessments conducted by external groups? Would it be helpful to you if risk assessments submitted to your agency by external groups, such as consultants and private industry, met the requirements proposed in the OMB Bulletin?

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget ADDITIONAL QUESTIONS FOR SPECIFIC AGENCIES DOE What are DOE’s current overall challenges regarding risk assessment? Specifically, please address DOE sites that have to be remediated (e.g., Hanford); DOE facilities (e.g., research and test reactors and processing plants); special projects (e.g., Yucca Mountain); and other sites (e.g., Pantex). How will the OMB Bulletin impact the quality, conduct, and use of risk assessments in these cases? EPA Regarding pesticides specifically, what risk-assessment activities will be covered by the Bulletin and what risk-assessment activities will be exempted? Does EPA have any examples of the application of the 1996 requirements of the Safe Drinking Water Act, as described on page 13 of the Bulletin? Can any examples be provided to the committee? If none are available, can EPA provide an explanation? Does EPA have a working definition of “expected risk” or “central estimate?” The agency indicated in its 1986 cancer guidelines (51 FR 33992-34003) that central estimates of low-dose risk, based on “best fit” of the observed dose-response relationship, were meaningless—that “fit” in the high-dose region provided no information about “best fit” in the region of extrapolation. The newer cancer guidelines appear to adopt the same thinking. Has the Agency changed its view on this point? If so, why? FDA Dr. Galson indicated at the public meeting that there were problems with the application of OMB requirements to certain types of assessments. Can FDA suggest specific language to exclude those problematic assessments from OMB requirements, rather than just offering examples of those assessments? In other words, how would FDA describe in general terms the types of assessments it would like to see excluded?

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget QUESTIONS FOR OMB Dr. Graham discussed the recent perchlorate evaluation as an example that would have benefited from this Bulletin. Does the Bulletin support using a “precursor” of an adverse effect or other mechanistic data as the basis of a risk assessment, as was recommended in the National Academies’ perchlorate review. Is it correct that those submitting data and risk assessments to the government to obtain product registrations, approvals, and licenses are excluded from the requirements of the Bulletin? Will the Bulletin require further review by OMB staff of risk assessments that have been peer reviewed in accordance with established peer review procedures and standards, including publication in a reputable peer reviewed journal? Public participants in the risk assessment and rulemaking processes—industry groups, environmental groups, other governmental entities, individual scientists—often provide risk assessments for agency consideration. Will these outside assessments be held to the same standards as agency-generated assessments, that is, to the requirements in the Bulletin? The 1983 NRC report Risk Assessment in the Federal Government: Managing the Process treats “risk assessment” as a term of art that covers four distinct analyses (hazard identification, dose-response assessment, exposure analysis, and risk characterization), each typically based on a number of separate studies and analyses. The OMB Bulletin defines “risk assessment” to apply to “any document” that “could be used for risk assessment purposes, such as an exposure or hazard assessment that might not constitute a complete risk assessment as defined by the National Research Council.” What is the advantage of defining risk assessment in this way? The Bulletin discusses the importance of risk assessors interacting with decision-makers. What safeguards will be built into the process to protect the scientific process from being framed by the decision-maker instead of the science?

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget Appendix E Agency Responses to Questions* Consumer Product Safety Commission Department of Defense Department of Energy Department of Health and Human Services Department of Housing and Urban Development Department of Interior Department of Labor Department of Transportation Environmental Protection Agency National Aeronautics and Space Administration Office of Management and Budget *Agencies that were sent the committee’s questions but did not provide responses: Department of Homeland Security Nuclear Regulatory Commission Department of Agriculture

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget This page intentionally left blank.

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget Below are responses developed by the U.S. Consumer Product Safety Commission’s (CPSC) staff to the questions posed by the National Research Council in its scientific review of the proposed Risk Assessment Bulletin released by the Office of Management and Budget. (Note: These comments are those of the CPSC staff, have not been reviewed or approved by, and may not necessarily represent the view of, the Commission.) General questions about current risk assessment practices Question: Please provide a brief overview of your current risk assessment practices. Specifically, do you conduct probabilistic risk assessment? Is there a common approach to both risk assessments and uncertainty analysis? How do you currently address uncertainty and variability in your agency's risk assessments? CPSC Staff Response In general, the CPSC staff performs risk assessments addressing a variety of hazards, including toxicity, electrical, fire and burn, and mechanical hazards. Depending on staff and agency needs, CPSC staff conducts all manner of analyses, both qualitative and quantitative. Some analyses constitute complete risk assessments, while others deal with one or more individual steps of risk assessment, e.g., hazard identification or exposure assessment. The toxicological risk assessment practices used by the CPSC staff are described in the CPSC Chronic Hazard Guidelines (FR 57: 46626-46653, 1992). The guidelines include sections on cancer, neurotoxicity, reproductive-developmental toxicity, exposure, bioavailability, and acceptable risk. The staff uses either probabilistic methods or sensitivity analysis to assess uncertainty or variability. The approach to evaluating uncertainty and variability is determined by the analyst on a case-by-case basis, based on the purpose of the risk assessment and availability of data. Question: Please identify any substantial scientific or technical challenges that you may encounter when conducting risk assessments for your agency. CPSC Staff Response When performing toxicological risk assessments staff may encounter a variety of technical and scientific challenges, such as the lack of complete toxicity or exposure data, or the lack of methodologies to develop such data. These challenges are addressed on a case-by-case basis, and may include performing exposure assessment studies, such as migration and emissions studies, and developing novel laboratory methods. The staff also nominates chemicals for further toxicological testing by the National Toxicology Program. Consider, for example, the CPSC staff’s risk assessment of diisononyl phthalate (DINP), which is a plasticizer used in teethers and toys made from polyvinyl chloride. CPSC convened a Chronic Hazard Advisory Panel (CHAP)1 to address the toxicity and potential 1 Convening a CHAP is a statutory mandate before CPSC can regulate products based on chronic toxicity of a substance, 15 U.S.C. 2077 and 2080(b).

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget risks from DINP, especially the human relevance of rodent tumors induced by peroxisome proliferation. Lack of exposure data for DINP in children’s products led to the conduct of observational studies of children’s mouthing behavior, as well as the development of methods to measure the mitigation of DINP from certain toys, and laboratory analysis of toys in the market to determine the proportion that contained DINP. Question: What is your current definition of risk assessment, and what types of products are covered by that definition? CPSC Staff Response The staff defines risk assessment following the definition of the National Research Council (1983), in which a risk assessment encompasses hazard identification, dose-response assessment, exposure assessment, and risk characterization. Depending on the agency’s needs, the staff may complete one or more of these steps for a particular task, but a risk assessment generally consists of all four steps. The definition applies to all consumer products under CPSC jurisdiction, and includes a variety of toxicological and physical hazards. However, the CPSC’s Chronic Hazard Guidelines (57:46626-46653, 1992) were developed primarily to address chronic toxicity. Question: About how long (that is, from initiation of the risk assessment to delivery to the regulatory decision maker) does it take to produce the various types of risk assessments? CPSC Staff Response The length of the risk assessment process is highly variable, depending on the intended use of the assessment, e.g., for screening or priority setting, or regulatory analysis; the needs of the decision maker; factors such as the availability of data and the amount, quality, and complexity of available data; and the need for public comment and peer review. The simplest assessments may be completed in a matter of days, while more involved analyses take months or years, especially if the agency must perform extensive studies to assess exposure or convene a CHAP. Questions about OMB’s definition of risk assessment and applicability Question: Using the definition of risk assessment described in the OMB Bulletin, are there work products that would now be considered risk assessments that were not previously considered risk assessments? If so, what are they? CPSC Staff Response Using the definition in the OMB Bulletin, almost every work product prepared by the CPSC staff could be considered a risk assessment. This would include: Injury or fatality reports; The agency budget, which employs “risk-based” decision making; Product Safety Assessments—short-turnaround assessments of specific products; Toxicity reviews; and

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget Routine testing of products, such as toys and fireworks, for compliance with standards. Work products from the CPSC’s Directorate for Epidemiology might especially be affected by the expanded definition of risk assessment contained in the Bulletin. For the most part, these work products provide information on injuries and fatalities associated with consumer products and, under the Bulletin’s definitions, would be considered either risk assessments or work products that contain data that are used in risk assessments. Examples include hazard sketches (estimates of the number of product-related injuries and descriptions of injury scenarios), estimates of consumer product-related injuries and deaths as part of Product Safety Assessments, and analyses supporting Commission briefing packages that are associated with regulatory activities. Some of these work products contain estimates of risk in the form of injuries or deaths per unit exposure. Exposure may be defined as products in use or per unit population possibly subdivided by age group. Exposure-based analyses are more commonly found in staff work products where there are a large number of injuries or deaths. They are less common when there are relatively few casualties and/or valid exposure measures are not available. In those cases, it is likely that most readers would conclude that the risk is small regardless of the exposure measure selected. Questions about type of risk assessment (tiered structure) Question: In your agency, is there currently a clear demarcation between risk assessments used for regulatory analysis and those not used for regulatory analysis? Is this clear at the outset of the risk assessment? CPSC Staff Response There is no clear demarcation between risk assessments used for regulatory analysis and those not used for regulatory analysis. Moreover, the importance to the agency of a specific risk assessment is not necessarily determined only by whether it is used to support a regulation. For example, in the staff risk assessment of DINP in children’s products, it was determined that the risk was low and no regulations were pursued. Nonetheless, it was important to perform the best risk assessment possible to be reasonably certain that the products (soft plastic toys) were not hazardous. The intended use of a staff risk assessment is usually clear at the outset, e.g., responding to public petitions, evaluating the impact of a regulation, or supporting the development of voluntary standards. In the event that staff objectives or agency needs change during the process, adjustments are made.

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget “Margin” in design and operational parameters is an important issue for space exploration. “Margin” in a key parameter is the difference between the value of that key parameter in some operational state and the value of that parameter at which failure will occur. Designers incorporate margin to reduce the chance of failure. Unfortunately, the provision of physical margin in space vehicles is very costly (e.g., extra material strength or shielding adds weight which, in turn, reduces payload delivery capacity to orbit). The determination of the adequacy of margin in a given situation must be key to developing realistic PRA models. Transforming design margins into a probabilistic framework to support PRAs poses significant challenges. Question 3: What is your current definition of risk assessment, and what types of products are covered by that definition? NASA Response: NASA NPR 7120.5 C [6] defines risk assessment as an evaluation of a risk item that determines (1) what can go wrong, (2) how likely is it to occur, and (3) what the consequences are. As stated in response to Question 1, since NASA defines the concept of risk broadly, the subjects of its risk assessments are also broad to include one or more of the three basic program execution domains: System technical performance Program cost Program schedule NASA considers risk assessment as a necessary element of the risk management process [7] which is required by all programs and projects that provide aerospace products or capabilities—i.e., flight and ground systems, technologies, and operations for space and aeronautics. Question 4: About how long (that is, from initiation of the risk assessment to delivery to the regulatory decision maker) does it take to produce the various types of risk assessments? NASA Response NASA uses risk assessment throughout a program’s or project’s life cycle, from initial stages of formulation where concepts and preliminary design ideas are developed, through fielding and operation to decommissioning. Since a risk assessment evolves and is updated over the life of the project or program, it can be considered as a “living” risk model with no fixed dates for their final delivery. The level of detail associated with a risk assessment model is dependent on the availability of design and operational information and the nature of the application for which the risk model is intended.

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget For large programs, where the assessments were conducted after the spacecraft were fielded, i.e., Space Shuttle, the risk assessment required several years to complete. For nuclear missions, where probabilistic and risk techniques are used as part of the safety analyses, the risk assessment is conducted as the mission is planned, the spacecraft and launch vehicles are constructed, and reviews and approvals are attained. The completion of nuclear mission safety analyses require about 3-5 years. Risk assessments of conceptual designs used to perform trades studies and sensitivity analyses to optimize safety, mission profile and operations have been conducted in several months. These types of risk assessments are typically conducted at a high level. Questions about OMB’s definition of risk assessment and applicability Question 5: Using the definition of risk assessment described in the OMB Bulletin, are there work products that would now be considered risk assessments that were not previously considered risk assessments? If so, what are they? NASA Response One significant area of change relates to our internal policies and directives. OMB Circular A-123, Management’s Responsibility for Internal Control, requires management to perform risk assessments to identify internal and external risks that may prevent the organization from meeting its objectives. The results of those risk assessments would be used to identify control activities that could be implemented to ensure agency objectives are met. The OMB Bulletin indicates that influential risk assessments are those that the agency reasonably can determine will have a clear impact on private sector decisions. For NASA, where the majority of our budget is applied to contracted activity, most of our internal controls do not impact the decisions of the private sector. If the special standards for Influential Risk Assessments were applied to every risk assessment performed to determine if an internal policy or directive was required, this could dramatically impact the time to develop, implement, and modify the internal controls. It is not clear, given the emphasis that the Bulletin places on regulatory matters and public use of the risk assessments, that risk assessments performed for internal NASA decision-making purposes need to have this level of regulation. Questions about type of risk assessment (tiered structure) Question 6: In your agency, is there currently a clear demarcation between risk assessments used for regulatory analysis and those not used for regulatory analysis? Is this clear at the outset of the risk assessment? NASA Response NASA is not a regulator, but rather a user with direct stakes in the technical and programmatic risk metrics that support decision making. NASA never uses its risk models for any regulatory application. Unlike regulatory agencies, NASA owns and

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget operates the subjects of risk assessment. This has several implications for NASA’s approach to risk assessment and risk management. Because NASA is interested in technical performance (e.g., safety, mission success), as well as programmatic performance (e.g., cost and schedule), its risk assessments need to address both, preferably in an integrated fashion. At regulatory agencies, the need for regulatory stability and transparency creates an incentive to standardize and hold static the technical approaches used in quantitative risk assessment. At NASA, quite the opposite, because of the application of novel technologies in new environments, there is a need to advance the state-of-the-art in quantitative risk assessment to support decision making aimed at optimizing safety and likelihood of mission success (see Response to Question 2). In this connection, NASA is continuously developing new risk assessment techniques and, as such, needs the flexibility to push the envelope on probabilistic methods and applications. For example, methodological enhancements are needed and are being planned for implementation to handle the dynamic nature of space flights in risk assessment of space missions. In general, the technical and programmatic risk assessments conducted within NASA would meet the five aspiration goals as described in Section III of the proposed Bulletin (Problem Formulation; Completeness; Effort Expended; Resources Expended; and Peer Review and Public Participation). The significant exception in most cases would involve public participation that would normally not be required in the same sense as a regulatory process affecting the livelihood of the private sector would be. In the case of environmental compliance and remediation, NASA responds to regulating the agency’s requirements in the development of risk assessments for site-specific environmental remediation. These risk assessments reflect the direction, requirements, and processes required by the regulating agency. Question 7: In your agency, is there currently a clear demarcation between “influential risk assessment” used for regulatory purposes and other risk assessments used for regulatory purposes? Is this clear at the outset of the risk assessment? NASA Response This question is not applicable to NASA because technical and programmatic risk assessments performed within NASA are not used for any regulatory purposes. If a regulating agency asks NASA to perform a risk assessment, NASA will perform the assessment in compliance with the technical procedures of the regulating agency. Questions about impact of the Bulletin on agency risk assessment practices Question 8: If applicable, please specify provisions in the Bulletin that can be expected to have a substantial positive effect on the quality, conduct, and use of risk assessments undertaken by your agency.

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget NASA Response By and large, risk assessments to examine technical risk within NASA meet the provisions that are cited within the Bulletin. The largest benefit to NASA with respect to the performance of risk assessment in the technical arena would be the added emphasis that a higher level external (OMB) requirement provides. Implementation of risk assessment, particularly probabilistic risk assessment, to analyze technical risks is a relatively new activity. If the OMB requirements were to apply to NASA, being able to cite an external requirement reinforces the existing risk assessment requirements established within NASA. Question 9: If applicable, please specify provisions in the Bulletin that can be expected to have a substantial negative effect on the quality, conduct, and use of risk assessments undertaken by your agency. NASA Response The largest potentially detrimental aspect of this Bulletin upon NASA relates to the scope of the Bulletin. The Bulletin indicates that the scope of the document covers risk assessments disseminated by Federal agencies (See “The Requirements of This Bulletin”, page 8). This wording infers that the requirements of the Bulletin apply to risk assessments that are prepared specifically for or are likely to be provided external to the agency. This wording is consistent with the significant emphasis that the Bulletin places on use of risk assessments to support definition and implementation of regulations. Later in the definitions section of the Bulletin, however, the scope is significantly broadened when the Bulletin indicates that these rules would apply to any risk assessment document that is made available to the public by the agency or that is subject to release under the Freedom of Information Act. If the Bulletin applies to any internal risk assessment performed within NASA that is releasable under the Freedom of Information Act, there could be a substantial burden to meet all of the requirements contained within the Bulletin. Two examples bear notation. Within NASA technical areas, many of the benefits of performing risk assessments do not lie in the completion and release of a formal risk assessment report but are realized by performing the risk assessment process among the participants in the design process. The Bulletin indicates that one of its goals is to have the risk assessors engage in an iterative dialogue with the decision makers who will use the assessment. The inference in this text is that the decision maker receives a report at the end of the risk assessment process and then makes a decision7. In the use of risk assessment within NASA, it is often the “give and take” with the participants in the risk assessment process that causes design and operational changes to be made to control risks, often at engineering levels lower than that of the ultimate decision maker. Emphasis on delivery of a report rather than the beneficial effects through pursuing the discipline of the process can have a negative effect on the ultimate impact of the risk assessment. 7 Risk assessments related to the launch of nuclear or radioactive materials are conducted in this manner; however, other technical risk assessments within NASA are not of this nature.

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget A second example has to do with internal controls as defined in OMB Circular A-123, Management’s Responsibility for Internal Control. This OMB Circular requires management to perform risk assessments to identify internal and external risks that may prevent the organization from meeting its objectives. The results of those risk assessments then are to be used to identify control activities that can be implemented to ensure agency objectives are met. The OMB Bulletin indicates that influential risk assessments are those that the agency reasonably can determine will have a clear impact on private sector decisions. In NASA, where the majority of the budget is applied to contracted activity, most of our internal controls have no direct application to the private sector; however, they do influence private sector activities. If the special standards for Influential Risk Assessments were applied to every risk assessment performed to determine if an internal policy or directive was required, this could impact the time to develop, implement, and modify the internal controls. Given the emphasis that the Bulletin places on regulatory matters and public use of the risk assessments, it is not clear that risk assessments performed for internal purposes need to have this level of requirement placed upon them. Question 10: If your agency followed the procedures described in the Bulletin, would it affect the time course for production of the risk assessment (that is, the time required from initiation of the risk assessment to delivery to the regulatory decision maker)? If so, please explain why? NASA Response NASA does not provide its programmatic or technical risk assessments to regulatory decision makers so there would be no impact in that area; however, it should be noted that even with the internal risk assessments that are performed to assess technical risk the final delivered report may not be the most important aspect of the risk assessment. NASA accrues much of the benefit from performing its technical risk assessment because of the iterative work performed by the risk assessors in conjunction with the engineers, logisticians, and analysts during the design process. Risks identified during the process are often resolved well before a final report is completed. As stated earlier, in the case of environmental, safety and health risk assessments, NASA’s conduct of site-specific risk assessments reflects the direction and requirements of the regulating agency. NASA must comply with the requirements imposed by the regulating agency and would be subject to any schedule changes or other impacts generated by the regulatory agency’s conformance with the Bulletin. Question 11: One of the Bulletins’s reporting standards states the need to be scientifically objective by “giving weight to both positive and negative studies in light of each study’s technical quality.” Please give an example of how this would be implemented by your agency or department.

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget NASA Response NASA is adopting a risk-informed decision-making process which is supported by two major activities: (1) risk assessment and (2) deliberation. NASA considers the deliberation activity as a crucial part of the decision-making process since it evaluates risk assessment results and scrutinize the results to ensure that they are meaningful and the risk models used as the basis of the results are technically sound and traceable [8]. The deliberation activity involves all affected stakeholders that may include, as appropriate, program/project manager, astronauts, NASA workforce, engineering organizations, and safety and mission assurance organizations8. Typically these deliberations take place in several forms depending on the context and the nature of decision situation. For example, deliberations can take place as part of the risk assessment peer review activities [9, 10] or as part of the design review or flight readiness review activities [11]. Question 12: Does your agency use risk assessments conducted by external groups? Would it be helpful to you if risk assessments submitted to your agency by external groups, such as consultants and private industry, met the requirements proposed in the OMB Bulletin? NASA Response Large-scale risk assessment projects within NASA are often conducted jointly by several groups that include both NASA civil service and contractor analysts. The involvement of contractors in the conduct of risk assessments is necessary because NASA contracts out the majority of its mission execution activities to aerospace sector companies. The domain and discipline knowledge of the external groups who are involved in the development and operation of various aspects of NASA’s mission is needed in order to develop realistic risk models. Because of the multi-group and multi-discipline nature of risk assessments and to ensure technical quality and consistency, NASA has developed risk assessment requirements and procedures (e.g., the PRA Procedures Guide) that must be met by all parties involved (internal and external) in the conduct of NASA’s risk assessments. NASA’s internal use of risk assessments is primarily for non-regulatory purposes and often requires significant innovation in application; therefore, the OMB Bulletin would be of limited help in conducting our risk assessments. REFERENCES 1 Space Shuttle Probabilistic Risk Assessment, Volume II, Rev. 1: Model Integration Report, Johnson Space Flight Center, January 2005. 2 Probabilistic Risk Assessment of the International Space Station: Phase II – Stage 7A Configuration, Volume II – Data Package, Futron Corporation, 2000. 3 NASA’s Exploration Systems Architecture Study, NASA-TM-2005-214062, November 2005. 4 NASA Smart Buyer Study, 2004 5 Pluto/New Horizons Interagency Nuclear Safety Review Panel Safety Evaluation Report of August 2005. 8 The safety and mission assurance organizations are independent from the program/project management organizations and provide independent perspective on risk-related issues that affect safety and mission success.

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget 6 NASA NPR 7120.5C “NASA Program and Project Management Processes and Requirements,” March 2005 7 NASA NPR 8000.4, “Risk Management Procedural Requirements,” April 2002. 8 NASA NPR 8715.3: Chapter 2 (system Safety) of NASA General Safety Program Requirements (updated version pending release) 9 Final Report of the Independent Peer Review Panel on the Probabilistic Risk Assessment of the Space Shuttle, Prepared for NASA Headquarters Office of Safety and Mission Assurance, April 4, 2005. 10 Report of the Independent Peer Review Panel on the Probabilistic Risk Assessment of the International Space Station Phase II – Stage 7A Configuration, Prepared for NASA Headquarters Office of Safety and Mission Assurance, June 2002. 11 Space Shuttle Flight Readiness Review.

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget EXECUTIVE OFFICE OF THE PRESIDENT OFFICE Of MANAGEMENT AND BUDGET WASHINGTON, D.C. 20503 Dr. Ellen Mantus Project Director National Research Council Division on Earth and Life Sciences Board on Environmental Studies and Toxicology 500 Fifth Street, NW Washington, DC 20001 Dear Dr. Mantus: Enclosed with this letter are the Office of Management and Budget's (OMB’s) responses to questions that the National Academies of Sciences (NAS) submitted to OMB on June 28, 2006. We hope these responses will be helpful to the National Research Council Committee as it reviews the OMB Proposed Risk Assessment Bulletin (Proposed Bulletin). Additionally, your request to OMB asked for “copies of all comments that are submitted by federal agencies on the OMB Bulletin, if possible.” At this point in time, OMB has not received any official comment letters on the Proposed Bulletin from Federal agencies that conduct risk assessments. However, staff of one agency did send us comments marked “internal deliberative.” Additionally, we have received a comment letter from the Small Business Administration’s (SBA’s) Office of Advocacy, which is available on that office’s website at http://www.sba.gov/advo/laws/comments/omb06_0608.html. If you should need further information from OMB, please contact Dr. Nancy Beck at 202-395-3258. Sincerely, Steven D. Aitken Acting Administrator Office of Information and Regulatory Affairs Enclosure

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget 1. Dr. Graham discussed the recent perehlorate evaluation as an example that would have benefited from this Bulletin. Does the Bulletin support using a “precursor” of an adverse effect or other mechanistic data as the basis of a risk assessment, as was recommended in the National Academies’ perehlorate review? OMB response: While the Proposed Risk Assessment Bulletin (Proposed Bulletin) does not speak to specific use of a precursor effect, there is no language in the Proposed Bulletin that precludes the use of a “precursor” of an adverse effect or other mechanistic data as the basis of a risk assessment. Further, Section V, subsection 7 (page 20) of the preamble of the Proposed Bulletin discusses the standard for characterizing human health effects: “[I]t may be necessary for risk assessment reports to distinguish effects which are adverse from those which are non-adverse?” Additionally, Section V, subsection 7 (page 25) of the text of the Proposed Bulletin notes the importance of describing the ramifications of the choice of effect: “Where human health effects are a concern, determinations of which effects are adverse shall be specifically identified and justified based on the best available scientific information generally accepted in the relevant clinical and toxicological communities.” 2. Is it correct that those submitting data and risk assessments to the government to obtain product registrations, approvals, and licenses are excluded from the requirements of the Bulletin? OMB response: The Proposed Bulletin does not apply to risk assessments performed with respect to individual agency adjudication or permit proceedings (including a registration, approval or licensing) unless the agency determines that: (i) compliance is practical and appropriate and (ii) the risk assessment is scientifically or technically novel or likely to have precedent-setting influence on future adjudications and/or permit proceedings. (Proposed Bulletin, Section II, subsection 2(b), page 23). This exemption applies regardless of who generated the data and the risk assessment. The OMB Information Quality Guidelines (67 FR 8460 Feb 22, 2002) do not cover adjudicative processes. The OMB Final Information Quality Bulletin for Peer Review (70 FR 2677 Jan 14, 2005) (Peer Review Bulletin) also includes an exemption for “individual agency adjudication or permit proceedings (including a registration, approval, licensing, site-specific determination), unless the agency determines that peer review is practical and appropriate and that the influential dissemination is scientifically or technically novel or likely to have precedent-setting influence on future adjudications and/or permit proceedings.” The exemption used in the Proposed Bulletin is consistent with the exemption in the Peer Review Bulletin.

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget 3. Will the Bulletin require further review by OMB staff of risk assessments that have been peer reviewed in accordance with established peer review procedures and standards, including publication in a reputable peer reviewed journal? OMB response: The Proposed Bulletin does not require OMB review of any risk assessment. However, under existing authorities and procedures, OMB might review a risk assessment. For example, risk assessments that axe part of regulatory impact analyses might be reviewed under Executive Order 12866. Additionally, Section III, subsection 5 (page 23) states: “The agency shall follow appropriate procedures for peer review and public participation in the process of preparing the risk assessment.” Agencies should rely on the Peer Review Bulletin to determine appropriate peer review procedures. 4. Public participants in the risk assessment and rulcmaking processes – industry groups, environmental groups, other governmental entities, individual scientists – often provide risk assessments for agency consideration. Will these outside assessments be held to the same standards as agency-generated assessments, that is, to the requirements in the Bulletin? OMB response: The Proposed Bulletin applies to risk assessments that are made publicly available by an agency, regardless of whether the agency conducted the risk assessment. If third-party submissions are to be used and made publicly available by Federal agencies, it is the responsibility of the Federal Government to make sure that such information meets relevant standards. 5. The 1983 NRC report Risk Assessment in the Federal Governraent: Managing the Process treats “risk assessment” as a term of art that covers four distinct analyses (hazard identification, dose-response assessment, exposure analysis, and risk characterization), each typically based on a number of separate studies and analyses. The OMB Bulletin defines “risk assessment” to apply to “any document” that “could be used for risk assessment purposes, such as an exposure or hazard assessment that might not constitute a complete risk assessment as defined by the National Research Council.” What is the advantage of defining risk assessment in this way'? OMB response: The Proposed Bulletin used a risk assessment definition that “applies to documents that could be used for risk assessment purposes, such as an exposure or hazard assessment that might not constitute a complete risk assessment…” (Proposed Bulletin, Section I, page 8). Many of these individual documents are relied upon by Federal agencies and used in important, and often economically significant, regulatory decisions made by Federal agencies as well as other

OCR for page 155
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget decision makers. The accuracy, quality, clarity, transparency, and utility of these documents could be improved by meeting, as appropriate, the quality standards outlined in the Proposed Bulletin. As we stated in the OMB Press Release accompanying the Proposal Bulletin, “Transparent and accurate risk assessments are necessary for agencies and other decision makers to make wise risk management decisions during the formation of agency rules and policy decisions.” Additionally, if these individual documents are prepared in a manner consistent with the Proposed Bulletin, this may avoid additional work when these activities are combined to create a comprehensive risk assessment document at a later point in time. 6. The Bulletin discusses the importance of risk assessors interacting with decision-makers. What safeguards will be built into the process to protect the scientific process from being framed by the decision-maker instead of the science? OMB response: In Section III, subsection 1 (page 10) of the preamble, the Proposed Bulletin sets forth an aspirational goal of an iterative dialogue between risk assessors and agency decision maker(s). This type of dialogue “will help ensure that the risk assessment serves its intended purposes and is developed in a cost-effective manner.” (Proposed Bulletin, Section III, subsection 1, page 10). The standards proposed in the Proposed Bulletin are designed to ensure the quality and objectivity of the scientific process and the science.