3

Risk Analysis Concepts and Terms

This chapter describes the Corps's progress in its risk analysis applications, the methods and terms the Corps employs in those applications, and provides recommendations regarding standardization of risk terminology and concepts.

Water resources project planning involves many types of uncertainties. Some of these relate to the natural environment, such as the variability of precipitation, stream flow, and river stage. Others relate to the performance of engineered systems, such as the reliability of levees, pumps, locks, and gates, or to variations in transit times of barges. Still others relate to the economic value of floodplain property, the probability distribution used to describe flood frequency, or the costs of alternative transportation modes.

The Corps has made significant strides in the use of risk analysis. In the committee's judgment, it would be advantageous for the Corps to consistently use terms describing uncertainty and to standardize risk and uncertainty concepts throughout its civil works programs. This would result in a clearer understanding of risk analysis issues among Corps personnel, most of whom work across program areas. It would also facilitate communication with other federal agencies, consultants, contractors, and the public.

UNCERTAINTY

The term “uncertainty” is used by different people to mean different things. A review of Corps documents describing risk-related planning



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 40
RISK ANALYSIS AND UNCERTAINTY IN FLOOD DAMAGE REDUCTION STUDIES 3 Risk Analysis Concepts and Terms This chapter describes the Corps's progress in its risk analysis applications, the methods and terms the Corps employs in those applications, and provides recommendations regarding standardization of risk terminology and concepts. Water resources project planning involves many types of uncertainties. Some of these relate to the natural environment, such as the variability of precipitation, stream flow, and river stage. Others relate to the performance of engineered systems, such as the reliability of levees, pumps, locks, and gates, or to variations in transit times of barges. Still others relate to the economic value of floodplain property, the probability distribution used to describe flood frequency, or the costs of alternative transportation modes. The Corps has made significant strides in the use of risk analysis. In the committee's judgment, it would be advantageous for the Corps to consistently use terms describing uncertainty and to standardize risk and uncertainty concepts throughout its civil works programs. This would result in a clearer understanding of risk analysis issues among Corps personnel, most of whom work across program areas. It would also facilitate communication with other federal agencies, consultants, contractors, and the public. UNCERTAINTY The term “uncertainty” is used by different people to mean different things. A review of Corps documents describing risk-related planning

OCR for page 40
RISK ANALYSIS AND UNCERTAINTY IN FLOOD DAMAGE REDUCTION STUDIES activities suggests that a consistent set of terms can be readily developed, and this set should be agreeable to most of those involved in risk analyses (Table 3.1). This set of terms, as discussed below, is consistent with the usage of others, including Morgan and Henrion (1990), Haimes (1998), and the Corps's Institute for Water Resources (USACE, 1992a,b). The term uncertainty is normally used to describe a lack of sureness about something or someone, ranging from just short of complete sureness to an almost complete lack of conviction about an outcome. Doubt, dubiety, skepticism, suspicion, and mistrust are common synonyms. Each synonym expresses an aspect of uncertainty that comes to play in risk analysis. Uncertainty with respect to natural phenomena means that an outcome is unknown or not established and is therefore in question. Uncertainty with respect to a belief means that a conclusion is not proven or is supported by questionable information. Uncertainty with respect to a course of action means that a plan is not determined or is undecided. In many, but not all, situations a lack of sureness can be described by probability distributions. The definition of uncertainty found in the Principles and Guidelines is that uncertainty describes only situations wherein the lack of sureness is not describable by probabilities. This narrow definition is no longer commonly used. The term uncertainty should be used to describe situations without sureness, whether or not described by a probability distribution. Generally speaking, uncertainty can be attributed to two sources: (1) the inherent variability of natural processes (“natural variability ”), or (2) incomplete knowledge (“knowledge uncertainty”). These two sources arise for different reasons and are usually evaluated in different ways (Morgan and Henrion, 1990). Moser (1998) and a National Research Council committee (NRC, 1996) describe these two types of uncertainty as follows: Natural variability—sometimes called aleatory uncertainty—deals with inherent variability in the physical world; by assumption, this “randomness” is irreducible. The word aleatory comes the Latin alea, meaning a die or gambling device. In the water resources context, uncertainties related to natural variability include things such as stream flow, assumed to be a random process in time, or soil properties, assumed to be random in space. Natural variability is also sometimes referred to as external, objective, random, or stochastic uncertainty. Knowledge uncertainty—sometimes called epistemic uncertainty— deals with a lack of understanding of events and processes, or with a lack of data from which to draw inferences; by assumption, such lack of

OCR for page 40
RISK ANALYSIS AND UNCERTAINTY IN FLOOD DAMAGE REDUCTION STUDIES TABLE 3.1 Alternative Terms from the Professional Literature Describing Categories of Uncertainties Type of Variability Terms from Literature Natural Variability Aleatory uncertainty External uncertainty Objective uncertainty Random uncertainty Stochastic uncertainty Knowledge Uncertainty Epistemic uncertainty Functional uncertainty Internal uncertainty Subjective uncertainty knowledge is reducible with further information. The word epistemic is derived from the Greek “to know.” Knowledge uncertainty is also sometimes referred to as functional, internal, or subjective uncertainty (see Box 3.1). IACWD (1981) provides an example of the distinction between natural variability and knowledge uncertainty found in flood–frequency calculations, wherein the frequency curve (i.e., probability distribution) describes natural variability, and the error bounds about the curve (i.e., uncertainty in the parameters of the probability distribution) reflect knowledge uncertainty. Natural variability is presumed to be an uncertainty of the world, a natural or inherent randomness. Knowledge uncertainty, in contrast, is presumed to be an uncertainty of the mind, a function of models and data. Although the distinction between natural variability and knowledge uncertainty is both convenient and important, it is at the same time hypothetical. The division of uncertainty into a component related to natural variability and a component related to knowledge uncertainty is attributable to the model developed by the analyst. Consider flood frequency. In the future—at least in principle—the sophistication of atmospheric models might improve sufficiently such that flood time series could be modeled and forecast with great accuracy. All the uncertainty currently ascribed to natural variation might become knowledge uncertainty in the modeling, and thus reflect incomplete knowledge rather than randomness. Modeling assumptions may cause “natural randomness” to become knowledge uncertainties, and vice versa. In its risk analysis framework, the Corps should be clear about which

OCR for page 40
RISK ANALYSIS AND UNCERTAINTY IN FLOOD DAMAGE REDUCTION STUDIES BOX 3.1 Evaluating Knowledge Uncertainties There are several examples from civil engineering in which risk analyses have involved the assessment of natural variability and knowledge uncertainty. One area of considerable development is earthquake engineering. Here, the assessment of seismic hazards and performance of civil, mechanical and electrical systems has involved comprehensive probabilistic analyses in which full analyses of natural and knowledge uncertainties are conducted as part of seismic risk studies. These analyses are performed for critical facilities such as nuclear power plants, chemical weapon demilitarization facilities, insurance risk assessments, and lifeline systems (e.g., water, gas, communication, and transportation). In probabilistic seismic hazard assessments, assessment of knowledge uncertainties has significantly matured in two areas (although seismic hazard analysis is fundamentally an earth science endeavor, probabilistic modeling of seismic hazards was initiated and has been largely advanced by civil engineers (e.g., Budnitz et al., 1997; Cornell, 1968; and Cornell and Vanmarcke, 1969)). These are the development of models for estimating the spatial and temporal rate of earthquake occurrences (seismic source characterization), and the prediction of earthquake ground motions. The characterization of the seismic source is particularly challenging. While considerable historic and instrumental data are available, estimates of the spatial and temporal rate of earthquake occurrences must rely on scientific evaluations and probabilistic assessments in which indirect evidence of the potential for future earthquake occurrences are gathered and evaluated. As part of these evaluations, formal elicitations are conducted with the earth scientist who must quantitatively evaluate knowledge uncertainties in modeling the location, magnitude and frequency of future earthquake occurrences. variables it treats as natural variability, which it treats as knowledge uncertainty, and why and how it makes this distinction. Furthermore, the Corps should establish a risk analysis framework that permits quantification of each source of uncertainty and properly incorporates each uncertainty in the analysis. Differences in the effects of these sources of uncertainty on risk calculations can be large. For example, variations in stream flow, treated as natural variability, average out in a calculation from one year to the next (high flows in one year balance against low flows in another). In contrast, uncertainty in the mean annual flow parameter, treated as knowledge uncertainty, introduces a systematic effect

OCR for page 40
RISK ANALYSIS AND UNCERTAINTY IN FLOOD DAMAGE REDUCTION STUDIES into a calculation. If the mean flow is overestimated in one year, it is overestimated in every year of the calculation. It is not always obvious which uncertainties in a risk analysis should be ascribed to natural variability and which should be ascribed to knowledge uncertainty. Although most engineers and planners are familiar with natural variability, they are often less familiar with knowledge uncertainty. To understand how knowledge uncertainty enters a risk analysis, one might think of the analysis as being built upon a mathematical model describing the behavior of the natural world. Mathematical relationships in this model include parameters that determine how output varies with input—for example, the stability of a levee as water rises behind it. In its simplest form, knowledge uncertainty can be thought of as comprising uncertainty in the appropriate parameter values for the model, combined with uncertainty in the model itself. Parameter uncertainty relates to the accuracy and precision with which parameters can be inferred from field data, judgment, and the technical literature. Model uncertainty relates to the degree to which a chosen model accurately represents reality. Parameter uncertainty derives from statistical considerations and is usually described either by confidence intervals when using traditional (frequentist) statistical methods, or by probability distributions when using Bayesian statistical methods. Data uncertainties, which are the principal contributors to parameter uncertainty, include (1) measurement errors, (2) inconsistent or heterogeneous data sets, (3) data handling and transcription errors, and (4) nonrepresentative sampling caused by time, space, or financial limitations. Model uncertainty can result from the use of surrogate variables, from excluded variables, and from approximations and the use of the incorrect mathematical expressions for representing the physical world. An NRC committee argued that model uncertainty should be addressed with sensitivity analysis (NRC, 1994); however, this view is not unanimously shared by the scientific community. Another type of knowledge uncertainty might be called decision model uncertainty, which describes an inability to understand the objectives that society holds important or to understand how alternative projects or designs should be evaluated. Such uncertainty, for example, would include uncertainty in discount rates and the appropriate length of planning horizons. The Corps's risk analysis approach in flood damage reduction studies is mandated in ER 1105-2-101, Risk-Based Analysis for Evaluation of Hydrology/Hydraulics, Geotechnical Stability, and Economics in Flood

OCR for page 40
RISK ANALYSIS AND UNCERTAINTY IN FLOOD DAMAGE REDUCTION STUDIES Damage Reduction Studies (USACE, 1996a), and further discussed in EM 1110-2-1619, Risk-Based Analysis for Flood Damage Reduction Studies (USACE, 1996b). While this latter document provides a clear definition of parameter uncertainty and model uncertainty, neither document discusses fundamental differences between natural variability and knowledge uncertainty. The distinction between natural variability and knowledge uncertainty is particularly important for flood damage calculations of expected annual damage (EAD) of the form found in the Corps's risk analysis procedure. Such calculations of expected annual damage lead to different numerical results depending upon which uncertainties—natural variability, knowledge uncertainty, or both—are included in the probabilistic averaging. In the Corps's method, expected annual damage is calculated by averaging natural variations among floods and in levee performance. Thus, the expected annual damage so calculated contains no contribution from knowledge uncertainty. To incorporate knowledge uncertainty, a probability distribution is specified over expected annual damage. This probability distribution over expected annual damage reflects the influence of parameter uncertainties in the flood–frequency distribution, stage–discharge curve, and stage–damage function. The expectation of expected annual damage itself reflects only natural variability, while the probability distribution of expected annual damage reflects only knowledge uncertainty. Due to nonlinearities in the calculations, this procedure of separately treating natural variability and the knowledge uncertainty can lead to different results compared to the approach of incorporating both types of uncertainty from the beginning. An NRC committee that reviewed flood risk management in the American River (California) basin (NRC, 1995) recommended that the Corps be clearer about which variables it treats as natural variability in the computation, which it treats as knowledge uncertainty, and why it makes the choice it does. CONSISTENCY ACROSS PROGRAM AREAS Risk analysis is based upon (1) the magnitude and likelihood of consequences, (2) defined risk acceptance criteria, and (3) a balance between implementation costs and avoided costs (Moser, 1998). In addition, such analyses should provide insight and understanding of likely failure modes and of significant economic issues.

OCR for page 40
RISK ANALYSIS AND UNCERTAINTY IN FLOOD DAMAGE REDUCTION STUDIES Corps documents use a variety of terms to describe what in this report is called risk analysis (Table 3.2). Among these are risk analysis, risk-based analysis, and risk and uncertainty analysis. All of these use probability to assess likelihoods of events occurring. The terms appear to be used interchangeably to describe efforts involving probabilistic analyses. Flood damage reduction studies use “risk-based analysis” and “risk and uncertainty.” Rehabilitation studies have often used “risk-based analysis.” Environmental and ecosystem restoration studies typically use “risk and uncertainty analysis.” “Risk analysis” is the more general term that includes risk assessment and risk management (NRC, 1983) and sometimes also includes hazard identification, risk characterization, and risk communication (NRC, 1994, 1996). The Corps should adopt “risk analysis” as the most general term. For Corps water resources project planning purposes, no distinction should be made between risk analysis, risk-based analysis, and risk and uncertainty analysis. “Risk” is generally understood to describe the probability that some undesirable event occurs, and is sometimes used to describe the combination of that probability and the corresponding consequence of the event. The Corps measures risk by the probability that system operation is undesirable (e.g., the probability that a levee fails or that an ecosystem restoration project fails to meet a standard). The complement of risk is reliability, the probability that a system operates without failing. In an economic risk analysis, the consequences of undesirable performance are also computed (e.g., expected flood damage). An important document in the Corps's rehabilitation program area is Tools for Risk Based Economic Analysis (USACE, 1999c). In describing what constitutes a risk analysis, this document presented only knowledge (parameter) uncertainty. The discussion neglects natural variability. This is noteworthy because natural variability was the only uncertainty TABLE 3.2 Terms employed in Corps program areas Program Area Term Used Term Sometimes Used Risk analysis course Risk analysis Rehabilitation Risk-based analysis Flood damage reduction Risk-based analysis Risk and uncertainty analysis Environmental restoration Risk and uncertainty analysis Dam safety Risk analysis

OCR for page 40
RISK ANALYSIS AND UNCERTAINTY IN FLOOD DAMAGE REDUCTION STUDIES included in the other risk analysis programs described in that publication (rehabilitation for hydropower and for locks, channel improvements in waterways, and dike maintenance). Efforts to develop risk analysis for environmental and ecosystem restoration projects have proceeded carefully in clarifying terminology and the conceptualization of uncertainty (USACE, 1996c). The Corps refers to a taxonomy suggested by Morgan and Henrion (1990) for categorizing different kinds of quantities in modeling (USACE, 1996c, 1996d). That taxonomy is then used to categorize instead different types of uncertainty (USACE, 1996c). The Corps correctly applies the taxonomy to quantities (USACE, 1996d); however, much of the confusion is retained in another Corps document (USACE, 1996c). The terminology and concepts that underlie the use of risk analyses across Corps program areas are not always well documented and not always consistently applied. There would be clear advantages to having a consistent, well-documented conceptual framework and a consistent set of terms to support those analyses. With relatively little effort, this situation can be improved by adopting a set of terms similar to those in Figure 3.1. RISK ANALYSIS AND DECISION MAKING Improved decision making is emphasized in the Corps's risk analysis literature, and there is widespread interest in how these tools can be used more effectively. Table 3.3 shows four civil works program areas using risk analysis, and the performance metrics important to each. The choice of decision criteria is generally related to risks to human welfare or to large economic losses. Many flood damage reduction studies and projects implicitly include some risk to human life. Such risk is described by the annual exceedance probability. Yet the primary decision criterion employed by the Corps, as specified by the Principles and Guidelines, is national economic development (NED). Supplemental criteria are the conditional nonexceedance probability for various design events and the expected annual damages (EAD). The current analytical approach does not address the question of which uncertainties are the more important. Environmental restoration projects typically do not focus upon loss of life or on reducing flood damages. However, such projects have inherently low reliability, because habitat suitability models are often poorly developed and investment levels tend to be modest (USACE,

OCR for page 40
RISK ANALYSIS AND UNCERTAINTY IN FLOOD DAMAGE REDUCTION STUDIES FIGURE 3.1 Taxonomy of uncertainties in risk analysis. 1997a). Adaptive management has been suggested as one approach for addressing uncertainties associated with ecological complexity (such as in Corps's environmental restoration efforts in the Florida Everglades and in Missouri River dam operations). It appears that formal risk analyses have been uncommon in environmental restoration studies. Rehabilitation studies are typically not concerned with loss of life or even with large economic loss. As mandated by the Principles and Guidelines, national economic development serves as the primary decision criterion in rehabilitation studies. But the NED criterion can be supplemented by other performance metrics because, if expected costs of alternatives are essentially equal, a plan that minimized disruption is generally preferred. Risk analyses that explain the dynamics of a system and explain opportunities for interventions that improve system operation can be useful. At a minimum, risk analysis should identify which uncertainties are the most important. Across these four areas, basic analyses have been formulated to compute primary and secondary criteria. To achieve the objective of using risk analyses to improve decision making, the remaining challenge is to compute other criteria that provide insight into system operation and into where cost-effective changes can be made to improve performance. It is similarly important to determine which uncertainties are important. Does risk analysis aid decision making in flood damage reduction studies? The new risk and uncertainty analysis method developed for flood damage reduction studies is different from earlier methods in that it includes a wider range of parameter uncertainties in the stochastic Monte Carlo analysis that generates expected project damages. It is thus important that the distributions describing parameter uncertainty be appropriate; otherwise, the expected annual damages criterion upon which projects are selected and justified will be distorted. This concerned both an NRC committee (NRC, 1995) and Stedinger (1997), who challenged the description of uncertainty in the parameters of the flood–frequency distribution.

OCR for page 40
RISK ANALYSIS AND UNCERTAINTY IN FLOOD DAMAGE REDUCTION STUDIES TABLE 3.3 Performance Metrics in Corps Program Areas Using Risk Analysis Program Area Primary Criteria Secondary Criteria Threat to Human Life Other Criteria Rehabilitation Expected costs (NED) Low Annual failure rates Hours/year for unscheduled outage Trip transit time Environmental Restoration1 Change in habitat units Expected costs Cost per habitat unit Distribution of Habitat Units Low Probability exceeds standard Flexibility: ability to make adjustments Dam Safety2 Annual loss of life Failure probability Expected costs (NED) High Cost per life saved Flood Damage Reduction3 Expected damages (NED) Probability of flooding (AEP) Medium Conditional non- exceedance probabilities Estimated Annual Damage quantiles 1   The current criterion for ecosystem restoration projects is cost effectiveness or incremental cost analysis. The incremental analysis generally ignores uncertainty except when it is reflected in the value of specified characteristics of a site (e.g., water fluctuation, average temperature, annual maximum pH). The Corps (e.g., 1997a, p. 93–98) suggests use of the mean habitat unit change, minimum and maximum changes, and the distribution of the change as output from a risk and uncertainty study. The Corps is conducting additional research in this area. 2   These are the anticipated performance metrics for this area. 3   The failure probability is the probability of flooding, also called the annual exceedance probability (AEP). Expected damages are used in calculating the National Economic Development (NED) objective.

OCR for page 40
RISK ANALYSIS AND UNCERTAINTY IN FLOOD DAMAGE REDUCTION STUDIES appropriate; otherwise, the expected annual damages criterion upon which projects are selected and justified will be distorted. This concerned both an NRC committee (NRC, 1995) and Stedinger (1997), who challenged the description of uncertainty in the parameters of the flood–frequency distribution. The question has arisen whether adding parameter uncertainty to flood damage reduction calculations leads to potentially different project decisions or to greater insight into project performance. Some have noted that because the primary decision criterion is average expected annual damages, parameter uncertainty should have little impact. Stedinger (1997), for example, showed that with small sample sizes and high levels of protection, hydrologic parameter uncertainty can significantly increase expected damages; yet, Al-Futaisi and Stedinger (1999) found that adding hydrologic parameter uncertainty to the design process had little effect. Thus, while including uncertainty in economic analyses may impact performance indices, it may not impact the designs selected. The influence of uncertainty on the expected value of performance criteria depends upon the nonlinearity of the models being used. A small uncertainty in the flow–stage relationship, or in the stage at which a levee fails, for example, can make a large difference in the reliability of a levee system and thus in project decisions. Anecdotal evidence from the American River (California) project suggests that risk analysis led to potentially significant changes in the operating rules for Folsom Reservoir, based on the capacity of a flood bypass and levee system downstream (M. Burnham, U.S. Army Corps of Engineers, personal communication, 1999). Further study is needed to assess how risk analysis can best be used in making project decisions for flood hazard damage reduction.