Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
3 The Design of Risk Assessments Risk Assessment as a Design Challenge Risk assessment is sometimes used to describe a process and sometimes to describe the product of a process. The dual use can create confusion, but it also serves as a reminder that the task of improving risk analysis necessarily requires attention both to desirable quali- ties of the process and to desirable qualities of the product. Given that there are inevitable constraints on efforts to assess risk and multiple objectives to be met, the selection of ap- propriate elements of process and the specification of required elements of the final product constitute a complex design challenge. Well-designed risk-assessment processes create products that serve the needs of a com- munity of consumers, including risk managers, community and industrial stakeholders, risk assessors themselves, and ultimately the public. Multiple interpretations of the word design apply to our presentation. One of the primary goals of design reflects the overall utility of a product to its end users. A second key aspect of design is the assurance of technical qual- ity. Many of the technical aspects of quality may not be apparent to end users, but they are important prerequisites that provide the foundation for the quality of a decision-support product. Finding the appropriate mix of technical quality and utility, given constraints, is the essence of design of a decision-support product. The Decision-Making Environment and the Importance of Process Many decision-making situations involving matters of public heath and environmental risk have five common elements: the desire to use the best scientific methods and evidence in informing decisions, uncertainty that limits the ability to characterize both the magnitude of the problem and the corresponding benefits of proposed interventions, a need for timeliness in decision-making that precludes resolving important uncertainties before decisions are re- quired, the presence of some sort of tradeoff among disparate adverse outcomes (which may be health, ecologic, or economic outcomes, each affecting a different set of stakeholders), 65
66 SCIENCE AND DECISIONS: ADVANCING RISK ASSESSMENT and the reality that, because of the inherent complexity of the systems being managed and the long-term implications of many decisions (such as cancer latency, changes in the structure of ecosystems, or multiple simultaneous sources of exposure), there will be little or no short- term feedback as to whether the desired outcome has been achieved by the decisions. The combination of uncertainty in the scientific data and assumptions (the âinputsâ) and inability to validate assessment results directly or to isolate and evaluate the impact of a resulting decision (the âoutputsâ) creates a situation in which decision-makers, the scientific community, the public, industry and other stakeholders have little choice but to rely on the overall quality of the many processes used in the conduct of risk assessment to provide some assurance that the assessment is aligned with societal goals. Those challenging properties of the decision-making environment may be considered particularly acute for many health and environmental decisions, but they are by no means new to decision-makers generally. The academic discipline of decision analysis under un- certainty, among others, has a rich literature on which to draw for methods and findings (Morgan et al. 1990; Clemen 1996; Raiffa 1997). The importance of attention to process is entirely compatible with the theory of the management sciences that defines a good decision under uncertainty as one that uses the most appropriate processes and methods to assemble and interpret evidence, to apply the decision-makerâs values properly, and to make timely choices with available resources rather than defining a good decision only according to its (apparent) outcomes. This attention to process is also compatible with arguments for the inclusion of more deliberative approaches to assessment and decision-making. As such, the most appropriate processes and methods in a given situation may be an appropriate balance of deliberative and analytic methods, as advocated in NRC (1996). Risk Assessment as a Decision-Support Product The process of risk assessment involves generation of a number of individual products that are combined to form a final product (which is often referred to as âthe risk assess- mentâ). The final product of a risk assessment process is most often understood to be a report. The present committee suggests that the product of a risk assessment should be considered to include not only the report but various subproducts, such as computational models and other information that is assembled during the process. The subproducts have different uses and serve a variety of audiences. For example, a computational model with a user-friendly interface may be at least as valuable in informing decision-making as the techni- cal report most often associated with the term risk assessment. In addition, such subproducts as dose-response assessments typically have value that transcends a particular decision-sup- port application and may be used in thousands of future decision-support situations. It is also useful to consider that risk assessments and individual subproducts experience a life cycle (consisting, for example, of conception, design, development, testing, use, maintenance, obsolescence, and replacement) that should be explicitly recognized. The products of risk assessment may be thought of as, among other things, communica- tion products. Their value lies in their contribution to the objectives of the decision-making function, including their effects on the primary decision-maker and other interested parties who participate in the decision or otherwise use the information that the products convey. Although the effort expended in the process is largely scientific, the critical final process in risk assessment is ultimately communication.
THE DESIGN OF RISK ASSESSMENTS 67 The Quality of Risk Assessment Includes Both Process and Product Attributes The decision-making environment associated with health and environmental risk man- agement compels the various users of risk assessment to value and scrutinize the assessment process. In addition, risk assessment is understood to result in a set of final products whose specific attributes are critical for meeting their objectives. In a sense, it may be neither pos- sible nor appropriate to separate the process from the product. The situation is somewhat analogous to that of other products whose quality is more readily scrutinized with respect to the process that is used rather than through scrutiny of detectable qualities of the final prod- uct. For example, the safety aspects of the quality of complex engineered systems, medical devices, and foods are increasingly scrutinized with respect to the quality of the process that generates and maintains them rather than judged solely on the basis of measurable quali- ties of the final product. Similarly, the final products of a risk assessment have a mixture of detectable and undetectable qualities, and both the final product and the underlying process must be considered in judging the overall quality. Given the demands of health and environmental decision-making, perhaps the most ap- propriate element of quality in risk-assessment products is captured in their ability to improve the capacity of decision-makers to make informed decisions in the presence of substantial, inevitable and irreducible uncertainty. A secondary but surely important quality is the abil- ity of the assessment products to improve other stakeholdersâ understanding and to foster and support the broader public interests in the quality of the decision-making process (for example, fairness, transparency, and efficiency). Those attributes are difficult to measure, and some elements of quality often cannot be judged until some time after the completion of the risk assessment. Formative and Iterative Design of Risk Assessments For the committeeâs purposes, the term design implies adopting a user-centered per- spective to craft both an assessment process and a decision-support product that achieves the objectives of supporting high-quality decision-making while working within inevitable constraints. Accordingly, an important part of the early design process is the understanding and weighing of all the objectives, recognition of constraints, and explicit acknowledgment of the need for tradeoffs. Design will inevitably occur throughout the risk-assessment process, and flexibility and iteration will be important aspects of the overall process design. Like any complex product designed in a complex environment, the process and product may need to be redesigned as objectives and constraints inevitably change and in response to new knowledge. While recognizing the iterative nature of risk-assessment planning, the committee strongly encour- ages increasing attention to design in the formative stages of a risk assessment. Such a shift in attention is recognized by the Environmental Protection Agency (EPA 2004a). It is also captured in guidance documents for ecologic risk assessment and cumulative risk assessment (EPA 1992, 1998, 2003). In those applications, EPA has adopted two tasks labeled planning and scoping and problem formulation. The two tasks are examples of early design activi- ties, and the committee believes that they should be formalized, applied more consistently in risk-assessment activities, and, perhaps most important, result in concrete outputs detailing the rationale and findings of the early design process. The tasks are described in more detail later in this chapter.
68 SCIENCE AND DECISIONS: ADVANCING RISK ASSESSMENT Design Considerations: Objectives, constraints, and tradeoffs As in any complex design problem, the process of design is intended to find the best solu- tion to achieve multiple simultaneous and competing objectives while satisfying constraints on the process or the end product. As decision-support and communication products for use in public decision-making, risk-assessment products inherit objectives from their parent domains of science and public policy. The objectives are not always compatible and, con- sidered individually, would influence the design in different and sometimes opposing direc- tions. In addition, general constraints on the process (such as resources and time) require that tradeoffs be made in pursuit of the objectives. The candidate objectives of risk assessment can, for present purposes, be separated into three categories, which are related to the inputs to the process (including evidentiary and participatory aspects), the process that transforms the inputs into risk-assessment products, and the impact of the products on decision-making. The objectives described below are examples that might be considered by EPA in designing risk-assessment processes and prod- ucts; clearly, it is the responsibility of EPA to interpret its mandate to choose and weigh the relative importance of different objectives. Objectives Related to Inputs Use of the Best Scientific Evidence and Methods A core aspect of health and environmental risk assessment is the universal desire to make use of the best scientific methods and the highest-quality evidence. Pursuit of that objective would lead EPA to acquire and interpret evidence by using established, trusted, and formal methods. The specifics underlying the notion of the âbest scienceâ are, not sur- prisingly, highly contested. Many attributes might define âbest,â and different parties will place considerably different weights on them. Even though the objective, simply stated, is superficially clear and uncontroversial, some aspects of the implementation are necessarily complex and controversial. In addition, pursuit of the best scientific understanding is inevi- tably resource-intensive and time-intensive, and this leads to conflict with other objectives and with constraints on resources. Inclusiveness of Scope For various reasons, human health risk assessment has traditionally focused on single cause-effect pathways that involve a single chemical and single identified adverse effect. The narrowness of scope is frequently questioned with respect to both its scientific merits and its relevance to decision contexts of considerably greater scope. The scope of consideration in health and environmental risk management would ideally be as large as possible. It can be argued that any limitation in scope constitutes a simplification of reality that must be recognized and justified because important parts of the total cause-effect network may have been missed. A narrow scope has the potential to distort the external validity of the conclu- sions and the associated decisions they support and thus to limit their applicability to the âreal world.â From a decision-support perspective, limitations in scope might create what is seen as highly imbalanced information support, supporting a particular concern with voluminous technical analysis while other concerns of great relevance to stakeholders (which cannot be readily dismissed on purely scientific grounds) remain largely or completely unaddressed
THE DESIGN OF RISK ASSESSMENTS 69 and explained only by the chosen scope of the risk assessment. For example, in situations where stakeholders are concerned about exposure from both food and water pathways, the provision of an elaborate risk assessment for waterborne exposure while providing only a cursory review for foodborne exposure may appear to be imbalanced with respect to the information needs. A somewhat more simplified risk assessment that includes both pathways may be preferable, if the foodborne pathway cannot be dismissed on strong grounds. Here, the objective of broadening of scope may compete with the desire to perform the âbestâ risk assessment on a single pathway. The desire to broaden the scope of human health risk assessment appears to be shared by EPA. Table 3-1 illustrates the expansion of the scope (in both risk assessment and deci- sion-making) to which EPA aspires, at least as far as can be inferred from its guidance for cumulative risk assessment. Some of the ânewâ characteristics are current practice in ecologic risk assessment. A critical dimension of scope (and a theme of Chapter 8 of this report) is the explicit inclusion of the various possible mitigation options that might be considered to reduce the risk that is being assessed. The scope would be expanded so that the assessment would provide not only estimates of existing risk but estimates of risk reduction associated with a variety of changes in the risk-generating system. To provide more complete information to the decision-maker, the decision-support products would ideally include (or be reasonably integrated with) estimates of the associated costs and any countervailing risks associated with the proposed mitigation options, as might be presented, for example, in a remedial action report under Superfund or in assessments that inform pesticide registration decisions. Additional elements of scope derive from the desire to support decision-makers other than EPAâs internal risk managers. The often-advocated goal of supporting local decision- makers, communities, and industrial stakeholders in a participatory decision-making model suggests the need for more customized decision-support tools on the basis of the nuanced information needs and value foci of other decision-makers. This implies either that the scope of the risk assessment increases to include those diverse needs and values or that separate assessments are conducted with different scopes and end points considered (with the associ- ated problems of compatibility). The concept of extended decision support can be taken further to support the broad array of decisions that EPA may not be directly involved in but ultimately is interested in their being risk-based, particularly for preventive risk management. Product and process- TABLE 3-1â Transition in EPA Human Health Risk-Assessment Characteristics According to EPA (1997) Old New Single end point Multiple end points Single source Multiple sources Single pathway Multiple pathways Single route of exposure Multiple routes of exposure Central decision-making Community-based decision-making Command and control Flexibility in achieving goals One-size-fits-all response Case-specific responses Single-medium-focused Multiple-media-focused Single-stressor risk reduction Holistic reduction of risk Source: EPA 1997.
70 SCIENCE AND DECISIONS: ADVANCING RISK ASSESSMENT development decisions that are made every day around the world and have short-term and long-term effects on human health and the environment may be the most important class of external decisions that would ideally be increasingly risk-informed. This class of decisions includes decisions based on life-cycle analysis and various related approaches with similar goals, in which risks are ideally reduced by design of energy and material flows in advance rather than by end-of-pipe mitigation strategies. Some of these preventative strategies may benefit from risk-assessment components (like dose-response information, or quantifica- tion of common exposure scenarios) without the need for an entire risk assessment to be completed. This might suggest that risk-assessment products be designed, prepared, and disseminated in a modular fashion to allow for the individual components to be used and reused by third parties making different types of decisions. Inclusiveness of Input A process that considers a broader evidence base and uses diverse methods to reach conclusions is generally preferred to one that is limited to a narrower evidence base or a narrower selection of methods. Breadth can be achieved by considering input from different academic disciplines and by including traditional knowledge and a variety of deliberative methods of arriving at conclusions about what can be considered to be âknown.â The ideal becomes problematic when disciplinary biases rightly or wrongly determine that input from some other sources of information lacks sufficient validityâaccording to criteria that are idiosyncratic in each disciplineâto be included as reliable input into a given analysis. Breadth can be seen as a potential threat to the integrity of the evidence base and of the conclusions derived from it. Because there is no universal standard for inclusion and weighing of evidence among disciplines (and often even within a discipline), resolution of the competing ideals of breadth and integrity of evidence requires careful attention to process. Integrity of Science-Policy Assumptions As a primary theme of both the Red Book (NRC 1983) and Science and Judgment in Risk Assessment (NRC 1994) and continuing in the present report, the careful application of science-policy assumptions (or âdefaultsâ) is critical for the integrity of the risk-assess- ment process. The use of defaults is necessary to complete risk assessments in the presence of substantial uncertainties and the embedded policy choices can have profound impacts on the risk-assessment findings and the associated decision-making functions. In addition to the science-policy assumptions that are easily recognizable, the process should take account of the presence of key subjective elements in evidence-gathering and integration that can influence the results of risk assessment. They may include a number of standard practices or conventions that are not normally recognized as elements of science policy. Objectives Related to Process Inclusiveness in Process Decision-making processes ideally are inclusive with respect to the participation and deliberation of affected and interested parties. In pursuit of that objective, risk-assessment processes would be structured to accommodate the needs of diverse stakeholders, includ- ing accepting their input at appropriate points, ensuring fairness in the influence of various
THE DESIGN OF RISK ASSESSMENTS 71 aspects of the design of the risk-assessment process and products (for example, input into its scope and access to information), fostering their desired level of understanding of the process, and meeting their specific information needs. Transparency It is both a scientific and a policy-making objective that the process of conducting a risk assessment and the risk-assessment products themselves be transparent. Transparency is a requirement that is always present, but it is rarely defined in operational terms. Some strict interpretations of transparency are akin to requirements for scientific reproducibility: that enough information is provided for a skilled analyst to be able to follow all the reasoning and independently reproduce the results. Transparency in risk-assessment models could be interpreted to mean that the computer code is entirely in the public domain (but may be executable only on specified computers) or to suggest that the models be publicly avail- able to be downloaded, complete with a user guide, and to be able to be run by individual interested users who lack advanced computer skills. In other interpretations, transparency would require that simplified versions of documents be produced to increase the number and diversity of parties that could follow the main arguments and understand the overall process of analysis and its conclusions. Given the lack of specificity in the operational definition of transparency, some effort is required during the early design period to achieve agreement among risk assessors and those seeking or responsible for ensuring transparency on the at- tributes that are sought and how they will be implemented. Compliance with Statutes and Administrative Law Requirements Some risk-assessment activities must comply with a variety of requirements imposed on federal policy-making activities, with the level of requirements depending on the risk as- sessment and the statutes that govern them. The nature and impact of these requirements is reviewed by NRC (2007). For example, EPA and other federal agencies are required by law to provide opportunity for public comment on proposed regulations and to take comments into account in making decisions. Some statutes have requirements for stakeholder partici- pation in various aspects of the risk-assessment and rule-making processes; others require peer review of particular categories of risk assessment. Other statutory provisions call for EPA Science Advisory Board meetings to be open to the public and for agency records to be made available to the public through the Freedom of Information Act. The administrative requirements regarding the risk-assessment process generally increase effort in the process, add costs, and affect the schedule. However, good practice would suggest that many of the required elements (such as peer review and stakeholder consultation) would often be included even if they were not required by statute or other administrative requirements. â As outlined in Chapter 2, the organic statutes administered by EPA include substantive standards and criteria bearing on risk-assessment activities specific to different EPA programs (such as those involving air and water). In addition, program-specific and agencywide guidelines detail principles and practices related specifically to the risk assessment process (Table D-1).
72 SCIENCE AND DECISIONS: ADVANCING RISK ASSESSMENT Objectives Related to Impact on Decision-Making Consideration of Uncertainty and Its Impacts A shared ideal in science and decision-making is that uncertainties in evidence be fully exposed and described. The task of confronting the implications of uncertainty is ultimately the domain of the risk manager, so it is important that key sources of uncertainties be de- scribed individually and in the context of their collective impact on the conclusions of the risk assessment. When the set of decision-maker options is known, an uncertainty analysis can be most profitably directed toward describing the impact of uncertainty on the consid- eration of these options. A difficult challenge in risk assessment is determining the best way to communicate the nature and magnitude of uncertainties. Analysis and judgment are required for focusing the discussion of uncertainty on important sources and describing the impacts of uncertainty in a manner that is relevant to the decision-making process. There are many potential uses of information about uncertainty for risk managers, including choices to delay or to expedite decision-making or to invest in research to reduce uncertainties. Assessing and communi- cating the utility of investing in additional information (such as conducting or considering more studies or gathering or formally eliciting expert input) are among the most challeng- ing aspects of risk assessment. Formal and less formal methods for assessing the value of information are discussed below. Control of âIatrogenic Riskâ in the Decision-Making Process There are a number of ways in which the process of assessing and managing risk can lead to an increase in riskâanalogous to the notion of iatrogenic risk in medicine (risk âcaused by the doctorâ). In the same way that a delay in diagnosis by a physician can increase risk to the patient, delays in the process of assessing risks may increase overall exposure to risk when decisions are delayed. In the presence of low risk, the increased risk may also come from the prolonged stress of being in a state of uncertainty with regards to health. The design of a risk-assessment process should balance the pursuit of individual attributes of technical quality in the assessment and the competing attribute of timeliness of input into decision-making. The critical process of triage, like other resource-allocation decisions in health care, must balance the needs of individual patients with those of others seeking attention. An overburdensome process of assessing individual risks can result in a lack of attention to other risks that deserve the attention of both risk assessment and risk management. Design must consider not only the needs of the individual assessment but the institutional role in simultaneously assessing and managing many other risks. Thus, the design of risk assess- ments should provide flexibility with respect to resource demands to foster balance in the management of multiple risks across the organization. The health-care analogy is readily extended to the issue of risk-risk tradeoffs. Physicians routinely consider side effects of their treatment decisions. They also need to consider the impacts of decisions that patients themselves make in response to information about risks. In the same way, health and environmental risk-assessment and risk-management processes need to consider the complete impact of risk-assessment products and decisions given their inevitable potential to inadvertently contribute to increased risk. Ideally, the design of a risk assessment takes into account foreseeable consequences of decisions, including substitution risks (for example, replacement of one source of hazard with another of similar, greater, or
THE DESIGN OF RISK ASSESSMENTS 73 unknown risk or diversion of waste from one waste stream to another), side effects of risk controls (for example, increase in risks due to disinfection byproducts in an effort to con- trol microbial hazards or development of resistance in pests, microorganisms, and invasive species), and other potential adverse outcomes associated with decisions taken by EPA or foreseeable decisions that might be taken by other stakeholders. It is also possible to extend the analogy to post-market surveillance for medicine to suggest that decisions based on risk assessments be monitored for the potential for unanticipated impacts (or the absence of anticipated impacts). Environmental protection agencyâs Current guidance RELATed TO Risk-Assessment Design The 1983 Red Book described the four key stages in the risk-assessment process as hazard identification, exposure assessment, dose-response assessment, and risk characteriza- tion (see Figure 3-1). In the intervening years, planning and scoping (a deliberative process that assists decision-makers in defining a risk-related problem) and problem formulation (a technically oriented process that assists assessors in operationally structuring the assessment) have emerged as additional distinct but related stages in both the human health and ecologic risk-assessment paradigms (EPA 1992, 1998, 2003, 2004a). Not all decisions require or are amenable to the results of a risk assessment. Decision- makers must first consciously identify risk assessment as an appropriate decision-support Is risk Issue assessment Problem the appropriate Non-Risk Concern decision support Considerations NO Objective tool? (Manager) YES PLANNING & SCOPING Manager, Stakeholder, (assessor) Dialogue Management Summary Risk Communication Options Identification & Public Comment Statement Risk Communication Peer-Review Hazard Identification DECISION Conceptual Model Technical Analysis ~ (Exposure Analysis, Dose-Response PROBLEM FORMULATION Manager, Assessor, (stakeholder) Analysis Analysis, Risk Characterization) Dialogue Plan Time FIGURE 3-1 Schematic representation of the formative stages of risk-assessment design. Dotted line in figure denotes that decisions informed by risk assessment will be influenced by nonrisk considerations. Source: Adapted from EPA 1998, 2003. Figure 3-1.eps
74 SCIENCE AND DECISIONS: ADVANCING RISK ASSESSMENT tool. If risk assessment is not selected as a tool, the decision-maker can be guided by a host of other, nonrisk-related considerations. Clearly, even decisions that are informed by the results of a risk assessment will be influenced by the same nonrisk-related considerations (as indicated by the dotted connection in Figure 3-1). Here, planning and scoping is used as described by EPA (2003, 2004a), and problem formulation is used as described by EPA (1998, 2003, 2004a). Planning and scoping are considered to constitute primarily a discussion between decision-makers (risk managers) and stakeholders in which assessors have a supporting role, and problem formulation involves a discussion between decision-makers and assessors (and technically oriented stakeholders) to develop a detailed technical design for the assessment that reflects the broad conceptual design developed in the scoping stage. As illustrated in Figure 3-1, planning and scoping determine which hazards and risk- mitigation options are of concern for the assessment and set boundaries for the assessment (that is, its purpose, structure, content, and so on). Box 3-1 lists some of the specific issues related to scope that may be discussed during this stage. Once planning and scoping are under way, problem formulation begins and runs in parallel with them. Discussions during this stage focus primarily on methodologic issues of the desired assessment, as illustrated in Box 3-2. It is important to note that communication between the two, now parallel stages, needs to occur for the assessment to be useful. The overarching purpose of the two critical, but often underused, stages of the risk-assessment process is to provide a clearer and more explicit connection between the decision-making context and the risk assessment that will inform the decision-maker. It also makes more explicit the relative roles of the decision- maker, stakeholders, and the risk assessor (EPA 2003, 2004a). Planning and Scoping In 1989, EPAâs guidance for Superfund provided several pages of guidance specific to the planning and scoping of a human health risk assessment (EPA 1989). Because assess- ment of complex ecologic systems challenged both decision-makers and assessors, it was BOX 3-1 Selected Elements of Scope Considered During Planning and Scoping â¢ Spatial and temporal scope options â¢ Direct hazards and stressors â¢ Mitigation-related hazards and stressors â¢ Sources â¢ Source-mitigation options â¢ Environmental exposure pathways â¢ Exposure-mitigation options â¢ Individual intake pathways â¢ Individual intake mitigations â¢ At-risk populations â¢ Populations at mitigation-related risk â¢ Direct adverse health outcomes â¢ Mitigation-related adverse health outcomes
THE DESIGN OF RISK ASSESSMENTS 75 BOX 3-2 Selected Methodologic Considerations in Problem Formulation â¢ Hazard-identification methods â¢ Stressor-characterization methods â¢ Source-characterization models and methods â¢ Environmental transport and fate models and methods â¢ Computational methods â¢ Uncertainty-characterization methods â¢ Intake and internal-dose models â¢ Dose-response models and methods â¢ Health-outcome measurement (risk measurement) methods â¢ Integrated cost-benefit methods â¢ Transparency, dissemination, and peer-review methods the ecologic risk-assessment community that ultimately championed the need to define the scope of a risk assessment and the need for discussion between decision-makers, assessors, and interested parties from the outset of an assessment. The need to scope an assessment and the need for assessors and managers to interact were discussed briefly in EPAâs 1992 framework for ecologic risk assessment (EPA 1992). NRC (1993) advocated for the integra- tion of ecologic risks into the 1983 Red Book paradigm, and expressed a need to extend this paradigm to include the need for interaction between risk assessment and management at the early stages of a risk assessment, based on experience in ecologic assessment. In 1996, a National Research Council committee commented on the importance of planning from the beginning of a risk assessment (NRC 1996). In 1998, EPA released its guidance for ecologic risk assessment, which superseded the 1992 framework document and provided a greatly expanded discussion of scoping and of the roles of assessors and decision-makers; it also drew a clear distinction between the goals and content of the planning and scoping stage and the problem-formulation stage. More recently, EPA has further articulated how critical planning and scoping are for the conduct of a successful risk assessment and has provided detailed guidance for their conduct (EPA 2003, 2004a). During planning and scoping, a team of decision-makers, stakeholders, and risk assessors identifies the issue (or concern, problem, or objective) to be assessed and establishes the goals, breadth, depth, and focus of the assessment. Once the decision to use a risk assessment has been made, this stage be- comes critical for developing a common understanding of why the risk assessment is being conducted, the boundaries of the assessment (for example, time, space, regulatory options, and impacts), the quantity and quality of data needed to answer the assessment questions, and how decision-makers will use and communicate the results. During this stage, deci- sion-makers charged with protecting health and the environment, in the context of other competing interests, can identify the kinds of information they need to reach their decisions, risk assessors can ensure that science is used effectively to inform decision-makersâ concerns, and stakeholders can bring a sense of realism and purpose to the assessment. This stage is a focal point for stakeholder involvement in the risk-assessment process and the point at which risk communication should begin (EPA 2003). The relevance of risk-assessment results to decision-making can be enhanced by the up-front involvement of decision-makers and stakeholders in setting goals, defining options, and defining the scope and complexity of an
76 SCIENCE AND DECISIONS: ADVANCING RISK ASSESSMENT assessment (Suter et al. 2003). Together, all can evaluate whether the assessment will help to address the identified problems (EPA 2004b). While a common plan for the risk assessment is one of the goals of these stages, reaching consensus on all aspects of the scope and conduct of a risk assessment among decision-makers and stakeholders representing diverse interests, will not always be feasible. In addition, it is not necessarily in the public interest to delay the risk assessment where consensus is difficult to achieve. The process requires a balance among the competing values of deliberative input into a risk assessment, timeliness in the risk assessment process, and the resource burden associated with these early stages. Early Identification of Decision-Making Options As discussed later in this chapter and further in Chapter 8, the utility of a risk assessment is greatly enhanced when it is constructed and carried out in the context of a clear set of op- tions under consideration by the decision-maker. Figure 3-1 explicitly includes identification of options as a critical element of planning and scoping. Although present EPA guidance (for example, on ecologic risk assessment, cumulative risk assessment, and air toxics) does not contain exact language calling for the explicit identification of decision-making options during the planning and scoping stage, it does allow preliminary consideration of regulatory or other management options. Existing EPA risk-assessment frameworks unquestionably contemplate consideration of options as they are related to decision-making, with plenty of interpretive room for arraying options if that is desired by or available to decision-makers and risk managers. For example (EPA 1998, p. 10), ârisk assessors and risk managers both consider the potential value of conducting a risk assessment to address identified problems. Their discussion explores what is known about the degree of risk, what management options are available to mitigate or prevent it, and the value of conducting a risk assessment com- pared with other ways of learning about and addressing environmental concernsâ (emphasis added). Not every issue faced by a risk manager will necessarily lend itself to âarraying options.â Some complex problems may also best be addressed by completing a thorough assessment of health risks and vulnerable populations prior to considering necessary control options. The Clean Air Act has used this approach to reduce air pollution concentrations over the past four decades. In the management of contaminated sediment, for example, it may be possible to examine the tradeoffs between various options, such as removal vs monitored natural recovery or capping versus hot-spot removal; in the case of soil contaminants for which no practical treatment options exist, options may be limited to various degrees of soil removal; and there may be instances in which the regulatory environment is so prescriptive as to preclude all but a few stipulated options. Although the planning and scoping stage is primarily deliberative, in that it involves extensive discussion between decision-makers and stakeholders and to a smaller extent with risk assessors, it is expected to produce tangible products that are critical for the performance of a credible and useful risk assessment (EPA 2003, 2004a). The primary product is a state- ment, with explanation, of why the assessment is being performed and what it will include and exclude (that is, how comprehensive it will be). Other products may be descriptions of those involved and their roles (for example, technical, legal, or stakeholder advisers), key agreements made and understandings reached among those involved, the resources (such as budgets, staff, data, and models) required by or available to the assessment, and the sched- ule to be followed (including provision for timely and adequate internal and independent external peer review). A statement (Box 3-3) often summarizes the end result of the planning
THE DESIGN OF RISK ASSESSMENTS 77 BOX 3-3 Planning and Scoping: An Example Summary Statement âAir toxics emissions may be causing increased long-term inhalation health risk (both cancer and noncancer concerns) to people in the immediate vicinity of Acme Refining Company. A modeling risk assessment will be performed to evaluate potential long-term human health impacts of inhalation ex- posures to all air toxics emitted by the facility. Inhalation risks for populations within 50 km of the Acme property boundary will be assessed under residential exposure conditions. Noninhalation pathways will not be assessed for either human or ecological receptorsâ (EPA 2004a, Chapter 5). and scoping process, describing the specific concerns that the risk assessment will address and generally what will be included in its purview. The problem-formulation stage, whose specific products are a conceptual model and an analysis plan, develops the specific technical details for the assessment laid out during planning and scoping. Problem Formulation The extension of the concept of âproblem formulationâ to human health risk assess- ment first emerged during a 1991 National Research Councilâsponsored risk-assessment workshop where the absence of such an activity in health risk assessment and the criticality of its use for ecologic risk assessment were discussed (NRC 1993). In 1992, EPA published Framework for Ecological Risk Assessment as the first statement of principles for ecologic risk assessments, including a further articulation of the concept of problem formulation (EPA 1992). The concept reached fruition in the agencyâs 1998 Guidelines for Ecological Risk Assessment, which superseded the 1992 framework document (EPA 1998). Those documents describe methods for conducting conventional single-species, chemical-based risk assessments and techniques for assessing risk to ecosystems from multiple exposures (or stressors) and multiple effects (or end points) (EPA 1991). For several reasons, ecologic risk assessments in the United States have generally placed a greater emphasis on problem formulation than have human-health risk assessments (Moore and Biddinger 1996). But by emphasizing completion of problem formulation early, the ecologic-risk framework provides a clear procedural advantage over the existing human health risk framework in achieving an assessment that can be used to inform a management decision. The advantage is derived from having decision-makers and stakeholders as active participants from the beginning of an assessment rather than passively awaiting receipt of the results. The problem-formulation stage sketches out the technical implications and decisions that are implied by the discussions that occur among decision-makers and stakeholders during planning and scoping so that risk assessors can proceed with the technical aspects of the as- sessment in a manner consistent with the decision context. This stage translates the results of the planning and scoping stage into two critical products: a conceptual model that explicitly identifies the stressors, sources, receptors, exposure pathways, and potential adverse human health effects that the risk assessment will evaluate and an analysis plan (or work plan) that outlines the analytic and interpretive approaches that will be used in the risk assessment. The general concern and approach articulated in the summary statement developed during scoping are given greater detail in a study-specific conceptual model. The model comprises both graphic illustrations (see Figure 3-2) and narrative descriptions that explicitly identify
78 FIGURE 3-2â Illustration of the scope of a risk assessment, indicating both pathways considered (bold lines) and pathways not considered. Source: EPA 2004a.
THE DESIGN OF RISK ASSESSMENTS 79 sources, contaminants of concern (stressors), exposure pathways, potential receptors, and adverse human health effects that the risk assessment is going to evaluate. The review of the conceptual model led to significant savings in the application of the model for calculating air dispersion, exposure and risk estimation. More than a third of the possible analyses were shown to be unnecessary to address the problem formulated in the planning and scoping discussion [EPA 2002, p. E-6]. For important risk assessments, particularly controversial or precedent-setting ones, it may be advisable that the scientific and technical credibility of the conceptual model be examined with a peer-review process. Although the conceptual model serves as a guide for determining what types, amount, and quality of data are needed for the assessment to ad- dress the issues and concerns of interest to decision-makers, the analysis plan matches each element of the conceptual model with the analytic approach that the assessors initially intend to use to develop data or otherwise represent that element. Box 3-4 lists some of the major elements of an analysis plan (from EPA 2004a). BOX 3-4 Major Elements of an Analysis Plan Sources How will information on the sources in the analysis (e.g., source location, important release parameters) be obtained and analyzed? Pollutants How will chemicals of potential concern (COPC) be confirmed and their emissions values be estimated? Exposure pathways How will the identified exposure pathways be assessed? How will ambient concentrations be estimated? Exposed populations(s) How will exposures to populations of interest be characterized? How will their exposure concentrations be estimated? What will be the temporal resolution? What sensitive subpopulations may be affected? End points How will information on the toxicity of the COPC be obtained (what are the data sources)? What risk metrics will be derived for the risk characterization? In addressing the above aspects of the analysis, the plan should also clearly describe the following: â¢ How will quality be ensured in each step (e.g., what will be included in the quality assurance/ quality control plans)? â¢ How will uncertainty and variability in the results be assessed? â¢ How will all stages of the assessment be documented? â¢ Who are the participants and what are their roles and responsibilities in the various activities? â¢ What is the schedule for each step (including milestones)? â¢ What are the resources (e.g., time, money, personnel) being allocated for each step? Source: EPA 2004a.
80 SCIENCE AND DECISIONS: ADVANCING RISK ASSESSMENT Recognition of the Need to Strengthen the Use of Formative Design Stages The specific nature and needs of the decision environment are often neglected in risk assessment, if there is no systematic approach (Crawford-Brown 1999). It is increasingly clear that even âthe highest-quality risk assessment is worthless if it does not address the needs of the decision-makerâ (Suter 2006, p. 4). EPA guidance documents make it evident that the agency recognizes, at least in theory, that â[planning and scoping] may be the most important step in the risk assessment processâ and that âwithout adequate [planning and scoping], most risk assessments will not succeed in providing the type of information that risk management needs to make a well founded decisionâ (EPA 2004a, p. 5-9). Similar ideas were also expressed in a report on EPA risk-assessment practices by GAO (2006). EPA has also observed that many of the shortcomings or failures of ecologic risk assessments can be traced to a weakness in or lack of problem formulation (CENR 1999). Both the planning and scoping and problem-formulation stages are necessary to ensure that the form and content of a risk assessment are determined by the nature of the decision to be supported. Both stages offer opportunities to reach some level of consensus on how to proceed (for example, with respect to regulatory context and objectives, scientific objectives, data needs, or reasonably expected limitations) in an assessment so that its results will be useful and informative to decision-makers. Those stages also offer excellent opportunities to give risk communication an early and pivotal role in the overall risk-assessment process rather than allowing it to become an afterthought. Although both planning and scoping and problem formulation can be challenging and time-consuming, the time and effort are usually well spent and have been shown to result in risk assessments that are more useful to and better accepted by decision-makers (EPA 2002, 2003, 2004a). The incorporation of those stages in the risk-assessment process is, however, still incon- sistent. For example, both stages are missing from EPAâs new cancer guidelines and from the current Office of Solid Waste and Emergency Response risk-assessment protocol for combus- tion facilities (EPA 2005a,b). Thus, although the stages are now widely acknowledged, at least conceptually, as critical for the success of a risk assessment (particularly for complex, controversial, or precedent-setting assessments) and guidance for their conduct is available, the question remains as to whether EPA or other public agencies, the regulated community, or their contractors are taking full advantage of them to focus, refine, and improve human health and ecologic risk-assessment efforts. The question warrants attention in that contin- ued inattention to the importance of planning and scoping and of problem formulation can be expected to yield human health risk assessments (by EPA and others) that fail to reach their full potential in providing support to decision-makers and others seeking solutions to environmental and health concerns. Incorporating Value-of-Information Principles in Formative and Iterative Design Scylla and Charybdis: Navigating the Twin Hazards of Uncertainty and Delay in Decision Support The combination of the magnitude and the practical irreducibility of key uncertainties and their impact on decision-making constitute the core challenge in efforts to achieve a â cylla S and Charybdis are two sea monsters of Greek mythology. They were located on opposite sides of a narrow strait such that they posed an inescapable choice in that avoiding Scylla (a six-headed monster) required passing too closely to Charybdis (presenting a whirlpool hazard) and vice versa.
THE DESIGN OF RISK ASSESSMENTS 81 robust yet practical approach to public-health and environmental risk management. The combination of uncertainty and the prospect of delaying important decisions constitute a key hazard in navigating the difficult waters of health and environmental decision-making. To an extent, it can be argued that the conflict is inherent in a decision-making environment that, while valuing a timely decision, places a large premium on the often-repeated yet ill- defined goal that the decision be âscientific,â âbased on sound science,â or âbased on the best available science.â The nature of the conflict can be understood by recalling that the scientific process of seeking the truth, by design and to its great credit, has no natural end point. Also by design, the training of scientists and such embedded traditions as applying tests of statistical significance instill the value of prudence and the âdue-processâ tasks of peer review, replication, and scientific debate before a conclusion can be said to be based in science. The idea that there are risks (for example, prolonged exposure to a hazard, or stress in the community awaiting an assessment of health risks) that may be associated with waiting for a particular study to be completed or for a scientific consensus to emerge is not readily incorporated into the standard scientific paradigm. The lack of established âstopping criteriaâ in science contributes to the conflict wherein any attempt to put an end to or otherwise constrain scientific inquiry and debate to meet regulatory or legal deadlines or, perhaps most problematically, to achieve an abstract notion of timeliness can lead to the accusation that the corresponding decision is âunscientific.â In pursuing the goal of timely decision-making, there is an inherent conflict between meeting the requirements associated with the goal of knowing and the requirements associated with the more pragmatic goal of deciding. Protection of the public and protection of the scientific knowledge base from Type 1 errors (that is, avoiding false positives) are not equivalent goals. That fact, somewhat obvi- ous when considered carefully, is a fundamental source of tension that is not sufficiently acknowledged or confronted directly in risk assessment, risk communication, and risk man- agement practices. Navigating (as opposed to resolving) the conflict between those goals is best addressed through its careful consideration by both risk managers and risk assessors in the formative and iterative design of risk assessment. To confront that challenge, risk managers must see themselves as managing uncertainty and delay as well as managing risk. Managing under uncertainty requires diverse strategies that address different aspects of the overall decision-making process, including investments to collect, store, and manage infor- mation; investments to improve the knowledge base, that is, to generate new knowledge; formalization of the processes used to collect, use, and process information; formalization of processes to calculate and communicate uncertainty; adjustment of the risk-assessment process to mitigate the practical impact of the uncertainty on the analytic process; adjust- ment of the decision-making process to accommodate the consideration of the uncertainty; and adjustment of the timing of decision-making in both directionsâto delay or to expe- diteâwhen uncertainty is acknowledged to be sufficiently great. It is important to note that the day-to-day work of uncertainty management should not be considered the sole domain of analytic experts. It is primarily the responsibility of the risk manager to prescribe and implement appropriate accommodations in the overall decision- making process if the analytic efforts aimed at supporting decisions under great uncertainty are to have the desired impact and to ensure that risks associated with delays in decision- making are balanced by the likelihood and magnitude of any benefit that is believed to be associated with proposed enhancements of the knowledge base or with the process of risk assessment. Choosing a strategy involves important tradeoffs because any strategy to deal with uncertainty will be incomplete and imperfect. The committee believes that one of the dominant pathways to improving risk analysis involves correctly matching the uncertainty- management strategy to the particular demands and resources of the decision-making envi-
82 SCIENCE AND DECISIONS: ADVANCING RISK ASSESSMENT ronments in and outside EPA. This issue is discussed in greater detail in Chapter 4. Ideally, the matching process would be expanded to consider the many other decision-makers that make use of EPAâs analytic products. Value of Information: What Makes Information Valuable? A fundamental aspect of decision-making under uncertainty involves the inevitable choice between making an immediate decision with the information and analysis available and delaying the decision while, for example, more raw information is collected, a more refined analytic product is prepared, or consultations with affected parties are conducted. Even if delay is not the primary concern, the direct and indirect costs of acquiring the infor- mation will often need to be considered. As the most generic analytic framework for valuing information in the context of deci- sions, value-of-information (VOI) analysis provides a set of methods for optimizing efforts and resources to gather, to process, and to apply information to help decision-makers achieve their objectives. The application of VOI analysis is illustrated schematically in Figure 3-3. The Process of Quantitative Value-of-Information Analysis The decision-theoretic process to quantitatively value information begins with analyzing the best option available to the decision-maker in a certain state of uncertainty. This serves as a baseline scenario with respect to information available to the decision-maker. The process then systematically considers when and how the decision-makerâs preferred option might be changed if the decision-maker was able to incorporate additional information into the decision that was not available in the baseline scenario. This new information is expected to either eliminate or reduce the extent of a source of uncertainty. In VOI analysis, the decision-maker is assumed to change the preferred option only when there would be a change in the net expected benefits. Accordingly, in addition to consider- ation of how likely it is that the preferred decision would change, the process measures how much of an increase in benefit would be expected given the additional information. The net (or expected) value of gathering information to resolve or reduce uncertainty is calculated by weighing the increase in benefits associated with each potential outcome of the information collected by the probability of each outcome. This weighing process includes assigning the value zero (that is, representing no increase in benefits) for situations where the information gathered does not change the decision-makerâs preferred option. A critical part of understanding the concept of VOI analysis is to differentiate scientific and decision-analytic perspectives on the value of information. In research proposals and in the literature, scientists often describe proposed studies as valuable with respect to enhancing the overall knowledge base, perhaps with a suggestion that it will inform important deci- sions. Conversely, the decision-analytic notion of VOI is entirely decision-centric. In a VOI analysis, an information source is valued solely on the basis of the probability and magnitude of its potential impacts on a specific decision at a specific time with a specific state of prior knowledge. Therefore, it is a common and expected result of VOI analysis to estimate that an information source, which may otherwise be considered valuable as a general scientific matter, has little or no value in support of a particular decision. This happens when the spe- cific decision is not sensitive to the resolution of the uncertainty that the information source addresses. Considering this situation in Figure 3-3, the arrow indicator, which denotes that option C is preferred given currently available information, would not be moved much by this source of new information.
THE DESIGN OF RISK ASSESSMENTS 83 Decision-Makerâs Valuation of Uncertain Uncertain Uncertain Risk and Cost Information C1 Information C2 Information C3 Outcomes Risk Reduction A Uncertain & Cost Model B Preferred Information R1 Option C Uncertain Information R2 D Baseline Decision Risk Model Support Model E Uncertain Information R3 Uncertain Information R4 Additional Information X1 â¦ Additional Information Xn Net Benefit Metric Information Opportunities Value-of-Information Analysis Estimate of Simulated Impact Survey Capital Cost of Information Unc. Reduction (R1) Study (C1) on Preferred Option and Net Benefit Expert Elicit. Operating Cost Indirect Risks and (R2) Study (C2) Costs of Study Bioassay Control Direct Costs Net Benefit (R3) Effectiveness (C3) of Information of Information FIGURE 3-3 Schematic of the application of value-ofÂ-information analysis to assess the impacts of additional studies in a specific decision context. Information opportunities that address uncertainties in Figure 3-3.eps the baseline model are considered with respect to the changes they would have on the decision-makerâs preferred decision option and the associated change in net benefits. The analysis may also consider any direct costs (for example, financial) and indirect costs (for example, the health or economic impacts of delayed decision-making) associated with the information opportunity. The valuation of information is ultimately driven by the decision-makerâs values with respect to the distribution of risks and costs, including any costs associated with delayed decisions. Experience in the Application of Value-of-Information Methods The applications of VOI methods in environmental health decision-making might be characterized as sporadic and somewhat academic (Yokota and Thompson 2004). In the academic literature, there has been a considerable interest in the use of VOI techniques to evaluate various activities within toxicity testing (Lave and Omenn 1986; Lave et al. 1988; Taylor et al. 1993; Yokota et al. 2004). Recently Hattis and Lynch (2007) applied a VOI framework to assess the expected effect of improved human pharmacodynamic or pharma- cokinetic variability information on doses deemed to be protective for noncancer effects. VOI methods have been employed to estimate value of sampling information in the context of environmental remediation (Dakins et al. 1996), and in an assessment of information value in the context of alternate control policies for source water protection in a watershed impacted by agricultural runoff (Borisova et al. 2005). Other applications can be found in
84 SCIENCE AND DECISIONS: ADVANCING RISK ASSESSMENT the value of improved exposure information in the case of drycleaning operations (Thomp- son and Evans 1997), and the value of genetic screening options related to prevention of beryllium disease (Bartel et al. 2000). There is evidence of sporadic interest and research aimed at employing VOI methods at EPA. For example, Messner and Murphy (2005) present an analysis of VOI about the quality of source water in the context of decisions about investments in drinking water treatment plants. In other applications, EPA staff and contractors have applied VOI principles in as- sessing the value of environmental information systems and human exposure information across a class of regulatory decisions (IEc 2000; Koines 2005). Prospects for Formal Value-of-Information Analysis at EPA VOI analysis has a number of benefits in support of decision-making compared with the more common scientific characterization of the potential value of a study. The intuitive and idiosyncratic views of individual scientists and decision-makers tend to place high value on information from their own discipline while diminishing the value of information from other disciplines. Scientists from all disciplines may devalue information that is not scien- tifically interesting (for example, that would not be publishable in a scientific journal) even if it substantially reduces a critical uncertainty in a risk assessment and the knowledge has considerable potential to affect the decision-makerâs choice of the best option. In contrast, VOI analysis could provide both a more context-specific and a more objective assessment of the decision-centric value of a piece of information or, by extension, the value of an information system to a class of decisions that might use it. Despite the potential benefits, it is important to note that a VOI analysis is not considered to be generally superior to the use of expert scientific judgment about the importance of a scientific investigation; rather, it answers a much narrower question about the importance of a study for the outcome of a specific decision and is not appropriate as a general measure of the scientific merit and broader utility of a study. For example, in the context of some specific decision, a VOI analysis might place great value on a small survey to estimate the fraction of businesses using a near-obsolete technology and very little value on a large, well-designed, and broadly important scientific study when considering only the narrow purposes of the specific decision at hand. The decision-makerâs preferences for options (perhaps in choosing among options B, C, and D in Figure 3-3) may be very sensitive to the level of uncertainty in risk reductions and the costs that would be imposed on businesses by a decision that would, for example, forbid the continued use of the older technology. In both the risk estimation and the cost estimation, the number of such businesses may be an important consideration in this particular decision context. Conversely, a scientific study that would contribute to the understanding of the risk and may reduce the overall uncertainty in a broadly desirable and scientifically rigorous way may not be able to add information that changes the relative desirability of the specific options enough to change the decision-makerâs preferred choice. Clearly, there are many other scenarios in which scientific investigation is precisely what is required to differentiate adequately among available options. Despite the intellectual appeal of the formal VOI analytic framework and the ever-pres- ent need for a robust means of assessing information value, the formal VOI paradigm imposes a number of challenges that limit its practical and widespread use in the near term. The use of the formal VOI framework in environmental health applications has been extensively reviewed by Yokota and Thompson (2004). One of their findings relates to the somewhat academic status of VOI in this field:
THE DESIGN OF RISK ASSESSMENTS 85 Rigorous VOI analyses provide opportunities to evaluate strategies to collect information to improve EHRM [environmental health risk-management] decisions. This review of the methodology and applications shows that advances in computing tools allow analysts to tackle problems with greater complexity, although the literature still lacks ârealâ applica- tions, probably due to a number of barriers. These barriers include the lack of guidance from EPA and others on criteria for standardizing EHRM risk and decision analyses, the lack of consensus on values to use for health outcomes, the lack of default distributions for frequently used inputs, and inexperience of risk managers and communicators with using probabilistic risk results. There are important considerations in addition to the barriers expressed above. â¢ VOI computation can be technically challenging, particularly when one is trying to evaluate imperfect information, which is almost always the relevant case. â¢ Its analytic formality does not lend itself to being combined with the more common deliberative approaches of determining the potential value of information. â¢ The approach presumes that the analyst can fully describe the change in a decision- makerâs choices in response to new information. This condition is not very realistic (or at least is rarely the case) and is particularly problematic when the decision-making process is not rule-driven or whenever the VOI analyst is forced to speculate as to the behavior of the decision-maker in response to new information. â¢ The impact of the new information must be characterized with respect to the result- ing change in a probability distribution that describes the current level of uncertainty, which may not be formally characterized as a probability distribution. â¢ Very few technical or policy analysts or decision-makers have had any exposure to this type of analysis, suggesting a considerable burden of training. â¢ The âvalueâ assigned in a VOI analysis is itself, ultimately, an uncertain quantity. A key challenge for uncertainty management in EPA and elsewhere is the need to design the risk assessment to support decisions with respect to an explicit array of candidate op- tions that the decision-maker is likely to consider. Without these options, it is not possible to assert a formal decision-centric valuation of information; indeed, in this case, a formal VOI analysis cannot even be attempted. A key potential side effect is the perpetuation of âincompleteâ risk assessments. The perpetuation side effect is a natural result in the absence of a well-characterized decision-support context, including a concrete array of decision op- tions, because there will always be a scientific rationale, as opposed to a decision-centric rationale, to continue to gather information, perform or review new studies, and to improve technical aspects of a risk assessment. The committee recognizes both the advantages of VOI analysis for risk assessment and risk management as well as the presence of continuing barriers to the use of formal and computational VOI analysis in EPA. As a result, there is likely to be only a small propor- tion of risk assessments and decision contexts that meet the criteria where a formal VOI is possible (for example, having clear decision rules and prior estimates of uncertainty) and for which the stakes are high enough to make a VOI analysis cost-effective. Alternative VOI Methods for Diverse Decision-Making Contexts As an alternative that is applicable to a larger proportion of decision contexts, the com- mittee believes that EPA would benefit from developing and applying a structured but less quantitative method for assessing the value of new information that captures the essential
86 SCIENCE AND DECISIONS: ADVANCING RISK ASSESSMENT reasoning embodied in VOI analysis. The essential reasoning in the formal VOI approach is based on the explicit characterization of a direct causal link between a specific source of new information, the predicted change in the behavior of a decision-maker given this new information, and the resulting improvement with respect to the decision-makerâs objectives that can be expected in the presence of the additional knowledge. Essentially, the process of valuation would involve the presentation of a qualitative or semiquantitative argument (as opposed to formal computation) that describes the causal relationship between the knowledge that might come from the considered source of information and the potential for improved decision outcomes. The process could also consider the potential for risk in delaying the decision until the information is available and is adequately incorporated into the decision-support products (either risk assessments or cost assessments). An example of the development and application of a structured semiquantitative VOI method, including a discussion of the complementary role of these methods, can be found in Hammitt and Cave (1991). Valuing Methodologic and Procedural Improvements in Risk-Assessment Design Earlier in this chapter, the committee described the rationale for placing a great premium on aspects of process in risk assessment. When all the combinations of choices of scope and technical, consultative, and quality-control methods are considered with the variations in the intensity of their application, it could be argued that there are an uncountable number of ways in which a risk assessment could be constructed. Such flexibility is generally welcome and has the potential to make risk assessment relevant to the broadest possible array of ap- plications, but it can be problematic. The essentially deliberative process of matching opportunities to enhance the risk- assessment process with the objectives of achieving high-quality decision support may be facilitated by using a decision-centric evaluation model that characterizes the impact of any proposed enhancements to the risk assessmentâand its manifestation in the form of a risk-assessment product with corresponding attributesâon the desired objectives of the decision-making function. The committee encourages the development of such an evalua- tion framework for methodologic improvements in risk assessment that instills some of the concepts of decision-analytic value of information. A schematic of such an evaluation model is illustrated in Figure 3-4. The proposed evaluation framework would expand the consideration of the casual relationship between risk-assessment activity and the quality of decision-making in two respects. It would be structured to assist in the relative valuation of the many attributes of risk-assessment processes and products that need to be considered in the formative and itera- tive design process. By relaxing the formality of the VOI approach, it could include a broader set of decision-making objectivesâsuch as transparency, timeliness, integration with other decision inputs, and compatibility with stakeholder participationâthat are less tangible and quantifiable but nonetheless critically important in determining the overall decision-support value of a given activity or effort. An important aspect of instilling the benefits that are analogous to VOI analysis will be in drawing explicit causal linkages, even if expressed qualitatively, between risk-assessment design options and the ultimate impact on the decision-making environment. In this way, the potential for the âvalue-of-methodsâ approach is limited in an analogous way by one of the barriers in the formal VOI approach. In VOI analysis, the analyst must know the decision- makerâs valuation of risk assessment or other quantitative outcomes in sufficient detail as to predict a change in the decision-makerâs behavior in response to new information (that is,
THE DESIGN OF RISK ASSESSMENTS 87 Decision-Makerâs Stakeholder Peer Review Methods to Valuation of Attributes Involvement Approach Achieve Timeliness of Quality in Public Decision-Making Quality of Risk and Cost Uncertainty Assessment Process Characterization Variability Characterization Baseline Design of Quality of Risk Quality of Risk Assessment and Cost Measurement Decision Support Process and Product Dose-Response Methods â¦ Exposure Modeling Methods Transparency Communication Baseline Assessment Methods Methods of Decision-Support Enhancement Opportunities Value-of-Methods Analysis Detailed Analysis Expanded Projected Impact Expected Benefit of Susceptible Uncertainty of Enhancement on to DM Process Population Quantification Quality of Decision Support Expand Scope to Multi-Stage Include Risks Public and Expected Detriments of Substitutes Peer Review to DM Process Enhanced More Complex Stakeholder Exposure Assessment Direct Resource Net Impact Consultation Model Costs of Enhancement of Enhancement FIGURE 3-4 Schematic of an analysis of the value of various methodologic opportunities (or âvalue of methodsâ analysis) to enhance the risk-assessment process and products. The structure mimics the Figure 3-4.eps standard VOI approach, but focuses on different impacts. In contrast with VOI analysis, the valuation of these opportunities is derived from the value system that specifies the desirable attributes of the over- all process of public-health and environmental decision-making. Whereas VOI analysis considers the impact of information on the decision outcome (the âendsâ), this type of analysis would consider the impact of diverse risk-assessment methods on the overall quality of decision support (the âmeansâ). predicting their choice among available options, or their choice in setting a single number within a continuum). In the value-of-methods approach, the analyst who is contemplating the value of a particular risk-assessment method (for example, in choosing among a qualitative, quantitative scenario-based, or fully probabilistic characterization of uncertainty) requires some way to characterize the change in the decision-support environment that corresponds to each of these alternative methods. Further, the analyst would need to know how much the different changes in the decision support environment are valued based on the capacity of the decision-making process to take advantage of the method, and the institutional values of the desirable qualities of decision-making. In order to remove this potential barrier, this expression of the valued attributes of decision support would be made highly context spe- cific (for example, having very different objectives for community-level decision support as compared to a national standard-setting process) and would be agreed to and documented in the formative stages of risk-assessment design.
88 SCIENCE AND DECISIONS: ADVANCING RISK ASSESSMENT Weight-of-Evidence and Hazard Classification: An Example of a Value-of-Methods Question The phrase weight of evidence (WOE) is used by EPA and other scientific bodies to describe the strength of the scientific inferences that can be drawn from a given body of evi- dence. In its most common applications in EPA, WOE is used to characterize the hazardous (toxic or carcinogenic) properties of chemicals on the basis of an integrated analysis of all relevant observational and experimental data. It is increasingly used to describe the strength of evidence supporting particular modes of (toxic) action (MOAs) and dose-response rela- tionships. Because scientific evidence used in WOE evaluations varies greatly among chemi- cals and other hazardous agents in type, quantity, and quality, it is not possible to describe the WOE evaluation in other than relatively general terms. It is thus not unexpected that WOE judgments in particular cases can vary among experts and that consensus is sometimes difficult to achieve. Perhaps the most formal WOE activity undertaken in EPA concerns the classification of carcinogens. The weighing of evidence from epidemiology and experimental studies pertain- ing to specific chemicals or chemical mixtures that may be carcinogenic involves substantial agency resources and can lead to controversy and extended debate. One distinction made in EPA carcinogen classification is whether the available evidence is sufficient to establish causality for humans (that is, whether a substance can be labeled as a âknownâ human carcinogen) or falls short and indicates that the agent is a âlikelyâ hu- man carcinogen. Causal relationships can be more straightforward to establish in well-done clinical and (in animals) experimental studies, but an individual observational (epidemiology) study typically can establish only a statistical association. A larger body of epidemiologic evidence can be sufficient to rule out bias and confounding with sufficient confidence to support a causal relationship; with experimental evidence, it may be sufficient to establish causality in humans. The weighing of such evidence can be controversial, so such institutions as EPA, the International Agency for Research on Cancer (IARC), the Institute of Medicine (IOM), and the National Toxicology Program (NTP) have developed practices and classifi- cation schemes to aid the process of reaching conclusions about the overall evidence. NTP, IOM, and IARC convene expert bodies to undertake WOE analyses of carcinogenicity data; EPA relies on peer review by expert groups, such as its Science Advisory Board, to vet staff findings on carcinogenicity evidence. The committee notes that in some cases there does not appear to be substantial value in the agencyâs making distinctions between certain carcinogenicity classifications. Whether a chemical is âcarcinogenic in humansâ or âlikely to be carcinogenic in humansâ generally has no important influence on the ultimate quantification of risk and the use of risk estimates in decision-making. In many regulatory contexts, known human carcinogens may be treated no differently from âlikelyâ human carcinogens: risks are estimated for all substances for which there is sufficiently convincing evidence of carcinogenicity, irrespective of whether human causality has been established, and the risk estimates are not adjusted according to the WOE classification. As a result, once the available evidence, either epidemiologic or experimental, is judged sufficient to establish that a given finding of toxicity or carcinogenicity is potentially relevant to humans, there may not be the need for further distinctions in classification, except in some circumstances as a communication tool. Unless clear reasons are brought forward at some stage, such as in the formative design stage of risk assessment, to support the need for such a definitive human causality assessment, the committee sees no reason for the agency to
THE DESIGN OF RISK ASSESSMENTS 89 spend time and resources to fine-tune the hazard classification in order to settle the question of whether the agent is a likely or known cause of the effect in humans. However, the systematic consideration of evidence in WOE analyses remains important as a matter of good scientific practice. Thus, whether the accumulated evidence is sufficient to consider a substance potentially hazardous to humans or is sufficient to support a given MOA requires a weighing of individual studies and pieces of evidence, and this practice should continue. The committee recommends that the agency remain mindful of cases in which fine distinctions have little or no impact on the overall use of risk information. WOE classification provides an example of distinctions between the formal VOI analysis and the less formal value-of-methods analysis. The fact that these finer distinctions in WOE classification are not used further in risk assessment or in any apparent decision rule used by EPA suggests placing no value on the exercise to seek these distinctions, when the potential benefit is viewed purely from a formal VOI analysis perspective (as illustrated in Figure 3-3). But a WOE classification that distinguishes known from likely carcinogens may be deemed by EPA to be required in support of other values associated with risk assessment practice (for example, using a âgood scientific practicesâ argument, or as the basis for a simplified means of communication of the epistemic status of a claim of carcinogenicity). WOE is an example of how EPA may benefit from a structured characterization (as described above and illustrated in Figure 3-4) of the exact role of a resource-intensive method in supporting the broader goals of public-health and environmental decision-making, which would include, among many other aspects, the use of good scientific practices and consideration of good communication practices. The method would require a more explicit valuation of important attributes of quality in decision support. Conclusions â¢ The nature of health and environmental risk management places great demands on both the processes and the products of risk assessment. In reviewing the history and many objectives of risk assessment, the committee finds that a more aggressive formative design stage is critical for the future success of risk assessment. The design should reflect the many objectives of the decision-making function and maintain this focus throughout the life cycle of the assessment. â¢ The key role of design in risk assessment is captured in current EPA guidance for ecologic risk assessment and cumulative human health risk assessment and embodied in the tasks of planning and scoping and problem formulation. â¢ A key design consideration for risk assessment lies in the potential for a poorly designed risk-assessment process to contribute to increased risk by a number of pathways. These include the potential to contribute to excessive delays in decision-making, to divert assessment and management attention from competing hazardous concerns, to contribute to ill-informed substitution of one risk for another, and to create barriers to inclusion or acceptance of risk assessments by various stakeholders. â¢ Decisions to invest in additional information to support a risk assessment are stan- dard and important in risk management. The investment can be in the form of direct costs, resource costs, or delay. Standard scientific rationales for asserting that a study is important may be misleading when considered from a purely decision-centric perspective. The commit- tee acknowledges the potential for a key beneficial role of VOI analysis in providing an objec- tive measure of the potential impact of new information on a particular decision. A number of barriers to application of formal VOI methods limit its general applicability. However, the
90 SCIENCE AND DECISIONS: ADVANCING RISK ASSESSMENT underlying structure of VOI analysis in expressing an explicit causal link between informa- tion, decision-maker behavior, and decision-making objectives is broadly applicable. It can be extended to guiding a number of design decisions at the formative and later stages of risk- assessment design. A value-of-methods analysis would provide an approach for considering the impact of opportunities, in the form of specific activities or methods, to enhance a risk assessment with respect to the overall quality of decision support and for considering any costs associated with the activity or methods. The approach could be applied to assess the value of current or proposed risk-assessment activities, for example, in weighing the value of advanced methods of uncertainty analysis, weight-of-evidence methods, or the development of complex computational models. The approach could also be applied to assess the benefit of procedural methods, such as stakeholder consultations, more intensive peer reviews, or methods to achieve greater transparency. RECOMMENDATIONS â¢ The committee recommends that EPA strengthen its commitment to risk-assessment planning. That can be achieved by formally including the requirement for formative and iterative design of risk assessments that is user-centric and maintains focus on informing decisions. â¢ The committee recommends formalizing and implementing planning and scoping and problem formulation in human health risk assessment and ensuring their continued and intensive application in ecologic risk assessment. Important elements of formalization would include specification of concrete documentary and related communication products that would be expected as the outcomes of these formative design stages, and consideration of the feasibility and benefits of explicitly arraying decision-making options as early as pos- sible in the process in order to focus the analytic tasks in the risk-assessment process. â¢ The committee recommends that EPA design risk assessments with due consideration of the potential for risk-assessment processes to contribute to unintended consequences, such as delays in risk-based decision-making that may prolong exposure to risk, diversion of attention away from other important risks within EPAâs mandate, and the potential for uninformed risk-risk substitutions. â¢ The committee recommends that EPA consider the adoption of formal VOI methods for highly quantified and well-structured decision-making problems, particularly those with very high stakes, clear decision rules, and the possibility of substantial risks associated with delays in decision-making. For the great majority of decisions that are not readily amenable to formal VOI analysis, the committee recommends that EPA develop a structured evalua- tion method that exploits, in a less quantitative fashion than formal VOI analysis, a causal understanding of the impact of new information in specific decision-making situations. The committee further recommends that EPA consider an extension of the structured evaluation method, conceptually related to VOI analysis, to assess the potential value of diverse meth- odologic options in risk assessment with respect to improving the overall quality of decision support. References Bartel, S.M., R.A. Ponce, T.K. Takaro, R.O. Zerbe, G.S. Omenn, and E. M. Faustman. 2000. Risk estimation and value-of-information analysis for three proposed genetic screening programs for chronic beryllium disease prevention. Risk Anal. 20(1):87-100.
THE DESIGN OF RISK ASSESSMENTS 91 Borisova, T., J. Shortle, R.D. Horan, and D. Abler. 2005. Value of information for water quality management. Water Resour. Res. 41, W06004, doi:10.1029/2004WR003576. CENR (Committee on Environment and Natural Resources). 1999. Ecological Risk Assessment in the Federal Government. CENR/5-99/001. Committee on Environment and Natural Resources, National Science and Technology Council, Washington, DC. Clemen, R.T. 1996. Making Hard Decisions: An Introduction to Decision Analysis, 2nd Ed. Boston: Duxbury Press. Crawford-Brown, D.J. 1999. Risk-Based Environmental Decisions: Methods and Culture. New York: Kluwer. Dakins, M.E., J.E. Toll, M.J. Small, and K.P. Brand. 1996. Risk-based environmental remediation: Bayesian Monte Carlo Analysis and the expected value of sample information. Risk Anal. 16(1):67-79. EPA (U.S. Environmental Protection Agency). 1989. Risk Assessment Guidance for Superfund, Vol. 1. Human Health Evaluation Manual Part A. EPA/540/1-89/002. Office of Emergency and Remedial Response, U.S. Environmental Protection Agency, Washington, DC. December 1989 [online]. Available: http://rais.ornl. gov/homepage/HHEMA.pdf [accessed Jan. 11, 2008]. EPA (U.S. Environmental Protection Agency). 1991. Ecological Assessment of Superfund Sites: An Overview. EPA 9345.0-05I. Office of Solid Waste and Emergency Response, U.S. Environmental Protection Agency, Washing- ton, DC. ECO Update 1(2) [online]. Available: http://www.epa.gov/swerrims/riskassessment/ecoup/pdf/v1no2. pdf [accessed Jan. 11, 2008]. EPA (U.S. Environmental Protection Agency). 1992. Framework for Ecological Risk Assessment. EPA/63-R-92/001. Risk Assessment Forum, U.S. Environmental Protection Agency, Washington, DC. EPA (U.S. Environmental Protection Agency). 1997. Guidance on Cumulative Risk Assessment, Part 1 Planning and Scoping. Science Policy Council, U.S. Environmental Protection Agency, Washington, DC. July 3, 1997 [online]. Available: http://www.epa.gov/brownfields/html-doc/cumrisk2.htm [accessed Jan. 14, 2008]. EPA (U.S. Environmental Protection Agency). 1998. Guidelines for Ecological Risk Assessment. EPA/630/R-95/002F. Risk Assessment Forum, U.S. Environmental Protection Agency, Washington, DC. April 1998 [online]. Avail- able: http://oaspub.epa.gov/eims/eimscomm.getfile?p_download_id=36512 [accessed Feb. 9, 2007]. EPA (U.S. Environmental Protection Agency). 2002. Lessons Learned on Planning and Scoping for Environmental Risk Assessments. Science Policy Council Steering Committee, U.S. Environmental Protection Agency, Wash- ington, DC. January 2002 [online]. Available: http://www.epa.gov/OSA/spc/pdfs/handbook.pdf [accessed Jan. 11, 2008]. EPA (U.S. Environmental Protection Agency). 2003. Framework for Cumulative Risk Assessment. EPA/600/P- 02/001F. National Center for Environmental Assessment, Risk Assessment Forum, U.S. Environmental Protec- tion Agency, Washington, DC [online]. Available: http://cfpub.epa.gov/ncea/cfm/recordisplay.cfm?deid=54944 [accessed Jan. 4, 2008]. EPA (U.S. Environmental Protection Agency). 2004a. Risk Assessment and Modeling-Air Toxics Risk Assessment Library, Vol.1. Technical Resources Manual. EPA-453-K-04-001A. Office of Air Quality Planning and Stan- dards, U.S. Environmental Protection Agency, Research Triangle Park, NC. EPA (U.S. Environmental Protection Agency). 2004b. Risk Assessment Principles and Practices: Staff Paper. EPA/100/B-04/001. Office of the Science Advisor, U.S. Environmental Protection Agency, Washington, DC. March 2004 [online]. Available: http://www.epa.gov/osa/pdfs/ratf-final.pdf [accessed Jan. 9, 2008]. EPA (U.S. Environmental Protection Agency). 2005a. Guidelines for Carcinogen Risk Assessment. EPA/630/ P-03/001F. Risk Assessment Forum, U.S. Environmental Protection Agency, Washington, DC. March 2005 [online]. Available: http://cfpub.epa.gov/ncea/cfm/recordisplay.cfm?deid=116283 [accessed Feb. 7, 2007]. EPA (U.S. Environmental Protection Agency). 2005b. Human Health Risk Assessment Protocol for Hazardous Waste Combustion Facilities. EPA530-R-05-006. Office of Solid Waste and Emergency Response, U.S. En- vironmental Protection Agency, Washington, DC [online]. Available: http://www.weblakes.com/hh_protocol. html [accessed Jan. 11, 2008]. GAO (U.S. Government Accountability Office). 2006. Human Health Risk Assessment: EPA Has Taken Steps to Strengthen Its Process, but Improvements Needed in Planning, Data Development, and Training. GAO-06- 595. Washington, DC: U.S. Government Accountability Office [online]. Available: http://www.gao.gov/new. items/d06595.pdf [accessed Jan. 10, 2008]. Hammitt, J.K., and J.A.K. Cave. 1991. Research Planning for Food Safety: A Value of Information Approach. R-3946-ASPE/NCTR. RAND Publication Series [online]. Available: http://www.rand.org/pubs/reports/2007/ R3946.pdf [accessed Jan. 11, 2008]. Hattis, D., and M.K. Lynch. 2007. Empirically observed distributions of pharmacokinetic and pharmacodynamic variability in humansâImplications for the derivation of single point component uncertainty factors providing equivalent protection as existing RfDs. Pp. 69-93 in Toxicokinetics in Risk Assessment, J.C. Lipscomb, and E.V. Ohanian, eds. New York: Informa Healthcare.
92 SCIENCE AND DECISIONS: ADVANCING RISK ASSESSMENT IEc (Industrial Economics, Inc.). 2000. Economic Value of Improved Exposure Information, Review Draft. EPA Contract Number: GS-10F-0224J. Industrial Economics, Inc., Cambridge, MA. Koines, A. 2005. Big Decisions: Initial Results of Benefit-Cost Analysis. Presentation at EPAâs Environmental In- formation Symposium 2005: Supporting Decisions to Achieve Environmental Results, November 30, 2005, Las Vegas, NV [online]. Available: http://www.epa.gov/oei/proceedings/2005/pdfs/koines.pdf [accessed Aug. 2, 2008]. Lave, L.B., and G.S. Omenn. 1986. Cost-effectiveness of short-term tests for carcinogenicity. Nature 324(6092):29-34. Lave, L.B., F.K. Ennever, H.S. Rosenkranz, and G.S. Omenn. 1988. Information value of the rodent bioassay. Nature 336(6200):631-633. Messner, M. and T.B. Murphy. 2005. Reducing Risk of Waterborne Illness in Public Water Systems: The Value of Information in Determining the Optimal Treatment Plan. Poster presentation at EPA Science Forum 2005: Collaborative Science for Environmental Solutions, May 16-18, 2005, Washington, DC [online]. Available: http://www.epa.gov/sciforum/2005/pdfs/oeiposter/messner_michael_illness.pdf [accessed Aug. 5, 2008]. Moore, D.R.J., and G.R. Biddinger. 1996. The interaction between risk assessors and risk managers during the problem formulation phase. Environ. Toxicol. Chem. 14(12):2013-2014. Morgan, M.G., M. Henrion, and M. Small. 1990. Uncertainty: A Guide to Dealing with Uncertainty in Quantita- tive Risk and Policy Analysis. Cambridge, MA: Cambridge University Press. NRC (National Research Council). 1983. Risk Assessment in the Federal Government: Managing the Process. Washington, DC: National Academy Press. NRC (National Research Council). 1993. Issues in Risk Assessment. Washington, DC: National Academy Press. NRC (National Research Council). 1994. Science and Judgment in Risk Assessment. Washington, DC: National Academy Press. NRC (National Research Council). 1996. Understanding Risk: Informing Decision in a Democratic Society. Wash- ington, DC: National Academy Press. NRC (National Research Council). 2007. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget. Washington, DC: The National Academies Press. Raiffa, H. 1997. Decision Analysis: Introductory Lectures on Choices under Uncertainty. New York: McGraw-Hill. Suter II, G.W. 2006. Ecological Risk Assessment, 2nd Ed. Boca Raton, FL: CRC Press. Suter II, G.W., S.B. Norton, and L.W. Barnthouse. 2003. The evolution of frameworks for ecological risk assess- ment from the Red Book ancestor. Hum. Ecol. Risk Assess. 9(5):1349-1360. Taylor, A.C., J.S. Evans, and T.E. McKone. 1993. The value of animal test information in environmental control decisions. Risk Anal. 13(4):403-412. Thompson, K.M., and J.S. Evans. 1997. The value of improved national exposure information for perchloroethy- lene (Perc): A case study for dry cleaners. Risk Anal. 17(2): 253-271. Yokota, F., and K.M. Thompson. 2004. Value of information analysis in environmental health risk management decisions: Past, present, and future. Risk Anal. 24(3):635-650. Yokota, F., G. Gray, J.K. Hammitt, and K.M. Thompson. 2004. Tiered chemical testing: A value of information approach. Risk Anal. 24(6):1625-1639.