Click for next page ( 115


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 114
4 Risk Methodology The U.S. Army Corps of Engineers (USAGE) has adopted new risk and uncertainty analysis procedures for project evaluation that explicitly include un- certainties in the hydrology, hydraulics, and economics of a planning study (USAGE, EC 1105-2-205, 1994) (hereafter referred to as EC 1105-2-2051. This procedure represents an extension of the traditional paradigm for flood control project planning and community flood protection evaluation. USACE observed that the new risk and uncertainty methodology is similar to present practice but differs in that uncertainty is explicitly quantified and integrated into the analysis (USAGE, EC 1 105-2-205, 1994). The 1994 Alternatives Report (USAGE, Sacramento District, 1994a) indi- cated that USAGE's analysis now considers "varying degrees of uncertainty in the causes of flooding, such as inflow to Folsom Reservoir, regulated outflow- frequency relationships for Folsom Dam, river stages, and levee stability." The methodology computes the risk of flooding due to combinations of hydrologic events, hydrologic parameter uncertainty, uncertainty in stage-discharge rela- tions, and levee performance. This change in methodology is important to the American River Watershed Investigation (ARWI) because the ongoing evaluation of flood control alterna- tives for the basin by the Sacramento District is one of the first applications of the approach, and almost certainly the most complex application yet attempted by USACE. The risk and uncertainty methodology specifically addresses many assumptions in the 1991 ARWI that were subject to controversy, and which the committee was charged to review. Whether the controversy will be resolved remains to be seen. 114

OCR for page 114
RISK METHODOLOGY 115 In particular, assumptions about levee freeboard for American River basin levees are replaced by a distribution on the stage at which levees fail. Likewise, hydrologic uncertainty that was described by an expected probability adjustment, and assumptions about delays between the beginning of the flood and increased releases, are now described by explicit probability distributions. Issues that were in contention have not disappeared; what some viewed as conservative values have been replaced by probability distributions, which may also be contested. For decades, civil engineers have realized that it is not practical to protect communities in the floodplain from all conceivable floods (Foster, 1924; Riggs, 1966~. Such protection measures would be prohibitively expensive, even if they were practicable. Communities and individuals who choose to locate in flood- prone areas will generally be exposed to some risk of flooding. However, it is often economically advantageous to provide protection from flood events that have a 1 in 50, 1 in 100, or a 1 in 500 chance of occurring in any year, depending on the value of the property at risk, the chance of loss of life, and the costs of flood risk reduction opportunities. Derivation of probability distributions to describe the possible magnitude of flood flows has been a practice in civil engi- neering since the early part of the century. They provide a description of hydro- logic risk. When a particular flood flow with a 1 in T chance of being exceeded in any year serves as a design flood for a project, USACE has said that the project provides a T-year "level of protection." The new USACE risk and uncertainty methodology explicitly introduces into the planning process consideration of hydrologic, hydraulic, and economic uncertainty. Before, the USACE planning procedure selected a level of protec- tion corresponding to perhaps the 1 in 250 chance event (often called the 250- year flood), and then determined the corresponding design flood flow. Use of an expected probability correction did incorporate hydrologic uncertainty into flood risk estimates (Beard, 1960, 1978; Stedinger, 1983a). Alternative hydraulic flood control structures including levees, flood storage capacity in dams, and channel improvements, in addition to flood-proofing efforts, were selected to control a flood of that magnitude. In the evaluation of flood control projects, there are a number of uncertain- ties that make it difficult to determine whether a specified flood can be passed safely. For example, flood control dams might have surcharge capacity that was not included in the flood routing calculations. Levees are a more common con- cern. Levee failure depends on factors such as the structural integrity of the levee embankment, possible scour and undercutting, variation in the state of levee repair, and other factors in addition to high water levels. Hydraulic predictions of the flood stage associated with different flow rates may also be in error. Levee failure stage predictions and stage-discharge relationships are affected by survey- ing inaccuracies in the measurements of channel geometry and riverbed eleva- tions, errors in estimation of flow resistance, simplifications in hydraulic routing

OCR for page 114
6 FLOOD RISK MANAGEMENT AND THE AMERICAN RIVER BASIN calculations, waves and wave effects, and possible settling of levees that affect crest elevation. Risk-based analysis of hydrologic and hydraulic engineering problems has been and is an active area of research (Davis et al., 1972; Tung and Mays, 1981; Ha~mes and Stakhiv, 1986, 1989, 1990; Duckstein et al., 1987; USACE, 1992a,b; Haimes et al., 1993; Taylor et al., 1993~. In most risk analysis applications, the risks of concern arise from the distributions of annual flood peaks, rainfall depths, and other hydrologic variables (Mays and Tung, 1992~. In a few cases, project performance is described probabilistically (Duckstein and Bernier, 1986; Chow et al., 1988, section 13.4; Mays and Tung, 1992~. Uncertainty in structure perfor- mance was important in several studies addressing dam rehabilitation and dam safety (McCann et al., 1985; Goicoechea et al., 1987; Von Thun, 1987; Stedinger et al., 1989; Bowles, 1990; see also NRC, 19851. There are relatively few applications where risk analyses have considered the natural variability in hydrologic and hydraulic variables as well as the uncer- tainty in the parameters of fitted flood-flow frequency curves and calculated stage-discharge relationships, and in economic quantities; these analyses might best be described as risk and uncertainty analyses to make the distinction clear. The Bayesian~ framework that is appropriate for hydrologic uncertainty has been employed in proposals to include hydrologic parameter uncertainty in planning studies (Benjamin and Cornell, 1970; Duckstein et al., 1975; Vicens et al., 1975; Wood, 1978; Stedinger, 1983a). The USACE use of expected probability adjust- ments is one way to include parameter uncertainty in flood control project evalu- ation. RISK AND UNCERTAINTY: A PRIMER Uncertainties, Safety Factors, and the Meaning of Level of Protection USACE traditionally has included safety factors in its design of facilities and the specification of operating policies to address important hydraulic uncertain- ties in flood control planning calculations. Surcharge storage in reservoirs might be one safety factor. For levees, engineers have required that the design flood The statistical literature includes several methods for dealing with parameter estimation, statisti- cal inference, and decisionmaking. Bayesian statistical methods treat unknown statistical parameters (the population mean, population variance, or a probability or quartile) as random variables whose probability distributions reflect the degree to which the value of a parameter can be resolved from available sample information as well as prior beliefs and other sources of information. With the traditional statistical procedures employing standard confidence interval estimators and classical hypothesis tests, such parameters are treated as if they have fixed (but unknown) values, and prob- ability distributions describing the sample-to-sample variability of sample statistics and parameter estimators are the focus of the analyses. The topic is addressed in more detail in the text.

OCR for page 114
RISK METHODOLOGY 117 pass through the levee system with some specified freeboard. Such a safety factor enables the engineer and the planning agency to be confident that in an actual flood event approximating the design storm, there will be sufficient chan- nel capacity to pass that flood flow without the levees failing from overtopping or excessive stages. In planning studies, encroachment within levee freeboard might be treated as sufficient to cause levee failure, even though in an actual flood failure might not necessarily occur at that stage. From an economic perspective, one can ask how much freeboard is justified economically to increase project reliability (Davis, 1991 J. The practice of including freeboard in design suggests that at the design flood associated with a target probability, called the "level of protection," there will often be some residual safety factor before actual flooding would occur. If there is, then the true chance of levee failure resulting in major flooding is less than the specified target probability. The question arises as to what was meant by the traditional "level of protection." Should it have been viewed as (1) an esti- mate of the chance of flooding due to levee overtopping or breaching, or was it simply (2) the exceedance probability of the design flood that a reservoir and levee system was designed to pass with some safety margin? Generally, evacuation plans would begin before a levee breached or was overtopped. Thus the "level of protection" could be viewed as the probability that the design event would be exceeded and thus that emergency measures would be required, even though widespread flooding might not occur. What seems clear is that there is confusion on this issue. Although calcu- lated levels of protection might appear to address (2) above, their use to estimate expected damages suggests that they are often used as an answer to (1~. This has led to the conventional wisdom that USACE projects provide more protection than acknowledged because safety factors built into levee design and reservoir operating policies appear to add an additional increment of safety. If this conven- tional wisdom is true, then by lowering the apparent benefit-cost ratios this prac- tice may have worked against some proposals to provide needed flood protection. For example, if levees can almost always pass flood flows that encroach within the specified design freeboard, they actually provide protection from larger floods than has been assumed in many analyses. However, the inclusion of safety factors in reservoir-levee system design to compensate for hydraulic uncertainty may not be sufficient to actually decrease the risk of levee system failure or levee overtopping. If levee settlement in one location ensures that a levee system failure will occur before the design flood event is reached, excess channel capacity or extra freeboard at other locations will not improve system reliability. In a levee system, failure occurs at the weakest point. However, if in a flood event a reservoir operator can vary releases in response to actual developments within the channel-levee system, it is possible that variation in reservoir operations taking advantage of excess surcharge stor

OCR for page 114
8 FLOOD RISK MANAGEMENT AND THE AMERICAN RIVER BASIN age could avoid levee system failures due to other deficiencies within some range of hydrologic loading. Planners and engineers also realize that the condition of levees and some equipment degrades with time. Safety factors are a reasonable way for designers of flood control works to ensure that over time a system can continue to pass the design flood without levee overtopping or breaching. However, it may not be immediately clear how safety factors included in different components of a reser- voir-channel-levee system interact to affect overall system reliability. Definitions for Risk and Uncertainty USACE will be wrestling for some time with the implementation of its new risk and uncertainty methodology. Of concern will be both a consistent scientific methodology, and a vocabulary and style for the presentation of the results to technical audiences and the public. The choice of words is very important be- cause they help us distinguish one concept or idea from another. In this regard, the terms "risk" and "uncertainty" can cause problems because different authors have ascribed to them significantly different ideas. Risk has been used to convey each of the following meanings (USAGE, 1992a, pp. 10-1 1~: 1. The idea of hazard, when something is described as being "at risk." 2. The expected losses or risk related to a venture. 3. The probability of some outcome, such as the risk that a levee will be overtopped. All three definitions attribute to risk a probabilistic character related to the possibility of an adverse and unwanted event in a particular system. Risk may be due solely to physical phenomenon or to the interaction between man-made systems and natural events. The tea uncertainty has been given a broad and sometimes conflicting range of meanings. There is a literature wherein the term uncertainty is used to describe events for which objective probabilities are not available (USAGE, 1992a). On the other hand, it could simply to be used to describe situations that are not certain; USACE (1992a) stated that "uncertainty means simply the lack of certainty. It is the reality of inadequate information. When information is imprecise or absent, that is uncertainty." The USAGE's guidelines provide the following operational definitions of risk and uncertainty (USAGE, 1992a, p. 123: Risk: The potential for realization of unwanted, adverse consequences; estima tion of risk is usually based on the expected result of the conditional probability of the occurrence of event multiplied by the consequence of the event, given that it has occurred.

OCR for page 114
RISK METHODOLOGY Uncertainty: Uncertain situations are those in which the probability of potential outcomes and their results cannot be described by objectively known probabili- ty distributions, or the outcomes themselves, or the results of those outcomes are indeterminate. Those guidelines indicate that actual uncertain planning situations are lo- cated on a continuum between situations of known risk (where the probability distributions of interest are well specified) and situations characterized by uncer- tainty (where those distributions are hardly specified at all; USACE, 1992a'. 119 A Distinction Between Risk and Uncertainty Although the cited distinctions between risk and uncertainty are some times useful, they are not the distinctions that are needed for our discussion of the USACE methodology for risk and uncertainty analysis. Of particular concern here with regard to the USACE risk and uncertainty methodology are: models of natural and operational variability and randomness, including probability distributions describing flood flows, event-to-event variability in stage-discharge relationships and reservoir operations, and variability in flood damages due to factors not captured by flood stage, and . uncertainty representing limited understanding of system processes and the lack of accuracy with which the parameters in models describing natural processes can be specified, including the parameters of a probability distribution, the cross-sections used to derive a stage-discharge curve, and the value and the count of the number of dwellings in a protected portion of the floodplain. In some cases one may be uncertain as to which of several competing models to employ, such as alternative probability distributions. Uncertainty refers to our lack of understanding of characteristics of nature that we conceptualize as being fixed at any given time. Ideally, the values of various model parameters could be determined. However, due to data limitations there are generally residual errors in our understanding of those characteristics of nature that cannot be eliminated with reasonable levels of effort. The first situation is referred to here as natural variability or randomness in the indicated process. The second situation is referred to as "specification error," or simply uncertainty. This use of uncertainty to describe lack of knowledge is not strictly consistent with the operational definition for the term suggested in USACE (1992a), although it may be consistent with the way the term is used. This use is consistent with the definitions adopted by other groups (IS O TAG 4, 1993; Taylor and Kuyatt, 1993; NRC, 1994~.

OCR for page 114
120 FLOOD RISK MANAGEMENT AND THE AMERICAN RIVER BASIN Sources of Uncertainty Recently, in a report on risk assessment of hazardous air pollutants, the National Research Council (NRC, 1994) recommended making a clear distinc- tion between parameter uncertainty, which is associated with the parameters of a particular model, and uncertainty as to the appropriate model, or model uncer- tainty. The report noted that parameter uncertainty often is described by continu- ous parameter ranges (NRC, 1994) that result in corresponding uncertainty inter- vals associated with predictions; however the choice among competing health risk models generally corresponds to distinct and mutually exclusive choices. The authors observed that "indiscriminately" combining the two types of uncer- tainty in health risk assessment could result in the calculation of average health risks and uncertainty ranges that are inconsistent with any of the alternative models. The report recommended that parameter uncertainty be evaluated sepa- rately for each competing model. Hydrologists face similar issues when choos- ing between alternative flood flow probability distributions or between methods for calculation of stage-discharge relations. Hydrologists often consider what can be classified as a third type of uncer- tainty, which arises due to model imprecision, or model prediction error. Thus, even with the best parameters, operational hydrologic models may fail to pre- cisely predict flood stages at some locations in a system; such model prediction errors are another source of uncertainty in the analysis of flood projects. The error here is not due to natural variability, which might be best described explic- itly, or to a failure to have the best set of model parameters, which is described by model parameter uncertainty, but is instead due to lack of model accuracy and thus is a source of uncertainty associated with model predictions. Such predic- tion errors can be thought of as a type of model uncertainty, because if one had a more accurate model, such errors might be eliminated. However, better models in most cases would have greater data requirements, requiring a finer spatial description of channel cross-sections and roughness coefficients with fewer lumped representations of watershed and channel characteristics. In fact, most operational models deliberately employ simplifications and lumped representa- tions of natural processes to restrict the parameter space to a manageable dimen- sion so that available data are sufficient for model calibration. Thus uncertainty due to model prediction error often reflects both data/parameter limitations and model uncertainty. In this discussion, model prediction error is included with other parameter and model uncertainties. A FRAMEWORK FOR RISK AND UNCERTAINTY ANALYSES A framework is needed to understand the structure of risk and uncertainty analysis efforts for flood protection project evaluation and to understand the relative roles of the natural variability of flood volumes, reservoir operations,

OCR for page 114
RISK METHODOLOGY / Q _ / In '~ ~ ~ 3 / i a ~\ Reservoir Outflow 121 - S(O) :> Failure /PL(S) D(S) us no a River Stage Possible values Possible reservoir Outflow peak 0 Probability levee Stage alone used to of annual flood operation determines determines breaches is estimate damages D. flows Q . outflow peak O for downstream stage S determined by Dotted line is without any inflow peak Q. at points of interest. stage S. levee overtopping or breach. FIGURE 4.1 Deterministic and stochastic processes contributing to flood risk. Perfor- mance of a flood control system involving both reservoirs and downstream levees can depend on deterministic and stochastic components. Possible values of the inflow peak for any year are described by a tree with branches, as are reservoir operations during that event, because both are described as random processes. The transformation of the out- flow peals O to downstream stage 5 is described by a deterministic relationship, though there is uncertainty associated with parameters of that relationship. Likewise, damages are described as a deterministic function of river stage for the levee breach/overtopping case, and the case without levee failure or over topping. Levee failure is probabilistic and occurs with a probability pit which depends on the stage S. hydraulic system performance, stage-discharge errors, and uncertainty in hydro- logic, hydraulic, and economic parameters. Figure 4.1 provides a conceptual model for describing hydrologic risk, variation in reservoir operations, use of a river stage-reservoir outflow relationship, levee reliability, and finally estimates of the economic damages that would result should a levee fail. Several of these relationships are stochastic, while others are described by deterministic relation- ships. The committee developed the event tree in Figure 4.1 to describe how the volume distribution of the largest flood volume in a year is transformed first into a river stage distribution and eventually into a damage distribution. This event tree can be used to evaluate the probability that flood protection works are over- whelmed and flooding occurs at some damage site, called the annual failure probability (AFP). Likewise, it can serve as the basis for calculating the expected annual damages (EAD), which would be the foundation of the economic evalua- tion of project performance. In Figure 4.1, a process is modeled either as being deterministic or as having some random component reflecting natural process variability. To understand the impact of specification errors or uncertainty in parameters of the selected discharge-frequency model or in economic parameters, it is useful to note that for each step in Figure 4.1 there is a set of parameters that define the relationship or model employed at that step. For example, in the first step the flood flow frequency relation requires specification of the parameters of that distribution;

OCR for page 114
22 FLOOD RISK MANAGEMENT AND THE AMERICAN RIVER BASIN these are often taken to be the mean, variance, and skewness coefficient of the logarithms of the flows. Likewise, the variability in reservoir operation at the second step will be described by some selected probability distribution, which will also have parameters. In this presentation, uncertainty analysis focuses on the parameters of the selected models; those models are assumed to accurately reflect the probability distributions of processes that are variable (such as the largest inflow in a year and the actual timing of reservoir releases in a future flood event) and of deterministic processes such as the stage associated with different channel flow, if only the correct or best values of the models' parameters could be determined. When the problem is structured as it is in Figure 4.1, one can identify the parameters of each of the models that determine the numbers that enter into calculation of risk and expected damages. One might then ask, how well or how precisely are those model parameters defined? Or, how uncertain are values of the project performance criteria AFP and EAD owing to the uncertainty or speci- fication errors in various parameters? There are several sources of variability in the economic damages that will be experienced in any year. Extreme variability results from the magnitude of the floods that may occur. Less important but still significant variability is intro- duced by flood hydrograph timing and shape, variations in reservoir operations, possible levee failure stage, and differences in the actual damages that would occur to a structure depending on the duration of flooding, wave attack, and differences in warning times; the effects of these factors are not captured by the specification of stage alone. Planners understand that this variability exists and so base their plans on AFP and EAD, which reflect the decision to average over the probability distributions describing annual maximum flood volumes and other variable processes. In structuring the problem, as has been done in Figure 4.1, engineers can also clarify how the various processes are thought to work. For example, the stage- discharge relation can be conceived of as being time-invariant or deterministic, so that a specific stage always corresponds to the same discharge. Then the relevant uncertainty would pertain to the precise functional relation between discharge and stage. Alternatively, there are certain stream reaches where the stage-discharge relation varies significantly over time because of channel changes, sediment movement, or the stages of tributaries or other streams with which the river of interest merges. Such stage-discharge relations hence might best be described by some random process. While in this second case the stage-dis- charge relation might best be described as a source of variation, there would still be uncertainty as to the best values of the parameters that describe that process. Economic damages depend on several factors, and some are deterministic while others are random. In particular, actual flood damages vary depending on flood duration, the presence of ice and sediment, wave action, and warning time. Flood damage uncertainties related to the number, types, and value of structures

OCR for page 114
RISK METHODOLOGY 123 in flood-prone areas would not change much from year to year, unless a major flood occurs. The source and nature of variability and uncertainty in levee performance present similar issues. USACE needs a clear framework for its risk and uncertainty calculations to be able to articulate and explain its treatment of such issues. Even so, it will not always be clear what should be described as variability and what to represent as uncertainty. Including Uncertainty in the Analysis Planners should know by how much the estimates of AFP and EAD might be in error. For example, a flood-frequency curve is based on a limited flood record. By how much might the parameters of the discharge frequency function be in error, and how big a change in AFP and EAD would result? Likewise, in deter- mining the stage-discharge relationship, a limited amount of effort goes into the surveying and the description of river cross-sections, geometry, and roughness coefficients: the hydraulic model has a limited amount of detail. What errors might this introduce into the evaluation of AFP and EAD? Similarly, limited effort is devoted to determining the value of property at risk in flood-prone areas. Additional effort could refine the data base describing the property at risk. Given a statistical description of the likely specification errors in economic and struc- tural survey data, a planner could quantify the magnitude of the corresponding errors in AFP and EAD. These questions can be addressed by sensitivity analysis procedures. The document Guidelines for Risk and Uncertainly Analysis in Water Resources Plan- ning (USAGE, 1992a), developed by the USACE Institute for Water Resources, defined sensitivity analysis as the technique of varying assumptions to examine the effects of alternative as- sumptions on plan formulation, evaluation and selection. This can include variation of model parameters as variation of benefit, cost and safety parame- ters. One of the important uses of sensitivity analysis is to investigate how different values of certain critical assumptions and parameters could result in changing the choice of the selected project and report recommendations. Sensi- tivity analysis is the systematic evaluation of the impacts on project formulation and justification resulting from changes in key assumptions underlying the anal- ys~s. Sensitivity analysis can be used to bracket forecasts, parameters, benefit and cost estimates, and other factors for which a range of values can be expected to occur. Generally, each model or process parameter is varied, one at a time, and the result observed (USAGE, 1992a'. However, there are often so many parameters in the models employed to evaluate flood protection projects that it would be difficult to integrate such one-at-a-time evaluations, or to decide how they should be incorporated into decisions (Moser, 1994~.

OCR for page 114
24 FLOOD RISK MANAGEMENT AND THE AMERICAN RIVER BASIN Describing Uncertainty Useful descriptions of uncertainty can be developed by describing the speci- fication errors or uncertainty in various economic parameters by probability dis- tributions. This must be done with care so that the resultant distributions truly reflect the probabilities planners should ascribe to the various parameters given the sample information at their disposal, general information they have about the processes of interest, and what is reasonable for the location in question. Then, using those probability distributions over the uncertain parameters, a statistical description of the uncertainty in AFP, EAD, and other performance criteria can be computed. For the purpose of developing a more mathematically precise notation for describing uncertainty, denote a possible set of model parameters for the event tree in Figure 4.1 by co. If the event tree in Figure 4.1 is evaluated with param- eters ce, let the resulting values of AFP and EAD be denoted AFP(co) and EAD(co). One can then ask what statistics should be calculated for the purposes of planning and project evaluation. A reasonable and simple procedure would be for planners and engineers to select their best estimate of ce, denoted here as Obese, and employ the value of AFP and EAD calculated with that best estimate: AFP(cubes~) and EAD(Cobes~) This is what is done in many studies. It is generally satisfactory when model parameter uncertainty is small. Alternatively, if a probabilistic description has been developed to describe the likelihood of different values of is, a different method could be employed. Just as EAD(co) is obtained by averaging over the probability distribution for annual floods, one could average the values EAD(co) over the probability distri- bution for co. The resulting descriptions of average flood risk and average eco- nomic losses are the average annual failure probability (denoted Avg~AFP]), and the average expected annual damages (denoted AvgtEAD]), where Avg{AFP] = Lover `,, {AFP(~) } Avg[EAD] = Eover i,, ~ EAD(co) ~ and where E denotes expectation over the indicated variable. The choice between AFP(cebes~) and Avg~AFP] and between EAD(cobes~) and AvgtEAD] reflects a philosophical choice in planning. The choice should also reflect how well plan- ners believe the available distribution for ~ has been specified. Even if Obese is simply the average value of ce, because of the nonlinear relationship between a probability distribution's parameters and exceedance probabilities, there will gen- erally be a difference between the two descriptions of AFP and EAD. Whether one uses average values of AFP and EAD or uses values of AFP

OCR for page 114
RISK METHODOLOGY 100 _ - ct i_ so _' .5, 50 o - ._ ED Ct ._ - o 153 1 1 1 1 -~ 50 100 200 300 500 year year year year year Return Period of Flood Peak FIGURE 4.5 Illustration of the trade-off between the return period of a flood peak and the reliability of the reservoir-levee system with possible flood flows associated with that return period. Application of Reliability Indices in the ARWI While the committee does not disagree with the analysis in Table 4.4 or Plate 12 of the 1994 Alternatives Report, it cannot see clearly what the public or most engineers would do with such information. There are several concerns: 1. It is not at all clear how one should conceptualize the 1 percent chance event given that it is not converted into a single flow estimate. Instead it is used to generate a set of flows reflecting the hydrologic uncertainty in the computed discharge-frequency relationship. This makes it very hard to anchor the analysis mentally or to know for certain to what it is applied. In the definition of reliabil- ity for the American River study taken from Ford (personal communication, September 19, 1994, quoted above), what is the particular "specified event" to which the chance of failure in Figure 4.5 or in Plate 12 of the 1994 Alternative Report refers? Use of critical historic flood events with known flood flow peaks would help resolve this conceptual vagueness. 2. The Sacramento District needs to clarify its reasons for wanting to calcu- late this reliability index shown in Table 4.4. If the overall probability of levee

OCR for page 114
154 FLOOD RISK MANAGEMENT AND THE AMERICAN RIVER BASIN failure, which describes the residual risk of flooding, is already known for levees of different heights, what does this other reliability calculation add? 3. The analysis does not indicate how much of the reliability (or likelihood of failure) is due to hydrologic uncertainty, how much to stage-discharge uncer- tainty, and how much to variability in levee failure stage. It is not clear what this reliability calculation reflects. The term reliability gave the committee the sense that it was a measure of how certain the Sacramento District is that the levee system would perform as intended. It was suggested that by using the new "reliability" index the Sacra- mento District is trying to tell the public that there is some uncertainty about how particular aspects of the project will perform. Ford (personal communication, September 19, 1994) wrote, "The term reliability, as used in the American River study, describes the likelihood that a proposed plan will perform as intended, given the occurrence of a specified event." Because the committee could find no explicit statement by the Sacramento District of what was intended, it had diffi- culty interpreting such statements. How should one define the intent of an exist- ing system? Because the numbers in Table 4.4, Plate 12, and Figure 4.5 also include the large uncertainty related to converting a "median" exceedance probability for a flood into the correct discharge, the committee found it very difficult to develop a useful interpretation of these numbers. It would be even more difficult for the public to interpret them. Proponents of careful risk communication warn of the pitfalls related to public misinterpretation of descriptions of risk (Plough and Krimsky, 1987; Slovic, 1987; NRC, 1989, 1994~. If each column in Table 4.4 corresponded to a discharge peak of a particular magnitude, corresponding to a historic flood or a selected design hydrograph, then one could interpret the calculated reliabilities as describing the consequences of stage-discharge estimation errors and levee performance uncertainty. For example, one could compute the reliability of the levee-reservoir system for a flood flow with a peak of 300,000 cfs or 500,000 cfs into Folsom Reservoir due to uncertainties in reservoir operation, stage-discharge relationships, and levee performance. One could also provide the estimated probabilities that these par- ticular peak flows are exceeded. For levee systems, a similar calculation would result by having the columns in Table 4.4 represent particular discharges (and perhaps the estimated probability each would be exceeded). 4. Calculations of project reliability may involve some difficulties that are not apparent. In reservoir-levee projects the characteristics of a critical event may depend on the capacity of the reservoirts) considered for different alterna- tives. For a levee-only system, it is the peak inflow that matters most. As one adds more storage, flood inflow volume becomes increasingly important. Thus one wants to select for the columns of a table such as Table 4.4, and for the graph

OCR for page 114
RISK METHODOLOGY 155 in Figure 4.5, events that for all alternatives are equally critical. It may be important that the public and the engineers who are reviewing project proposals understand how this is done. But this issue is completely hidden in a table such as Table 4.4, in which the character of the actual hydrologic event corresponding to each probability is obscure. 5. The explanation that is reproduced at the beginning of this section from EC 1105-2-205 also gives the sense that from Table 4.4 and Figure 4.5 one can determine the risk of flooding should a particular project be adopted. The re- sidual risk of flooding is certainly a primary concern. Ford (personal communi- cation, September 19, 1994) wrote, The importance of reliability is, to some extent, a function of the consequences of exceedance. If the consequences are great, then high reliability is necessary. For example, if overtopping a levee would inundate a high-density residential development to a depth of 25 feet without warning, high reliability is required. This discussion of the importance of reliability ignores the risk associated with the target flow. The reliability of the system as calculated by USACE is only part of the residual risk. It is the overall risk of flooding that is key, not the reliability of the system for particular events. That overall residual risk of flood- ing, described by AFP or Avg[AFP], and the expected annual damages (EAD) are certainly the two most important system performance criteria. A significant problem with the presentation of system reliability in Table 4.4 and Figure 4.5 is that reliability appears to address the residual risk of flooding, while it actually hides the true answer in a matrix of less meaningful numbers. To compute the actual risk of flooding (as described by AvgLAFP]), one would need to compute the average across all failure probabilities of the reliability of the system. There are insufficient numbers in the table to do this computation, and interested individuals should not have to do it themselves. Engineers and plan- ners should perform these calculations and provide the results. Table 4.4 and Figure 4.5 appear to reflect a desire to hold on to the old idea of "level of protection," expressed by the hydrologic return period T or exceedance probability for a design flood, while moving to a new risk analysis methodology that includes the idea of uncertainty and variability in other pro- cesses. Davis (1991) noted that traditionally projects were defined by the target "level of protection." The problem with the presentation of system reliability versus a target failure probability is that it fails to integrate those two sources of risk. 6. In the American River study, reliability is also used to demonstrate that the reliability of the levee network across the American-Sacramento River sys- tem is not impaired by a project. This is a legitimate concern and one that a risk analysis methodology should be able to address. The Sacramento District has demonstrated how its reliability index can be calculated at different points in the

OCR for page 114
56 FLOOD RISK MANAGEMENT AND THE AMERICAN RIVER BASIN system for different probability levels to demonstrate that "reliability" is not Impaired. The committee wondered if this is the most effective definition of reliability for that purpose. If reliability is used to ensure that for every flood level de- scribed by a cumulative probability p, throughout the river system the probability of flooding is not increased by a project, it is not necessary to include hydrologic discharge-frequency uncertainty. It is much simpler to specify a range of Folsom inflow hydrographs and evaluate the reliability for each. The "reliability of the system for a given inflow" is both simpler and more meaningful than the "reli- ability of the system for a given exceedance probability including our inability to determine the flow actually associated with that exceedance probability." In this regard, requiring reliability to remain the same for every hydrograph is a more demanding requirement than requiring that it not decrease for every median exceedance probability after averaging over hydrologic uncertainty in the frequency curve. The first approach requires that reliability not decrease at every flood flow; the second requires that reliability not decrease for averages over flow ranges. Keying on clearly specified flood hydrographs with their associated peak and volume seems to meet the requirement of ensuring that reliability not de- crease more rigorously than the approach the Sacramento District has adopted. It would also be clearer and easier to understand. Moreover, it is also easier to compute and relate to levee and channel system performance because those un- certainties will not be swamped by the potentially much larger hydrologic uncer- tainty. Overall, the committee applauds the USACE decision to adopt a risk-based planning methodology that better incorporates uncertainties in key variables. However, the committee does not believe that the definition for system reliability that was proposed in USACE guidelines and adopted by the Sacramento District is particularly effective at addressing the relevant issues. In many cases, it seemed unnecessary or misleading. Annual failure probability (AFP, Avg~AFP], or both) is likely to be the most straightforward and easily understood measure of residual flood risk. It could be supplemented by the vulnerability criteria dis- cussed in the risk communication section of Chapter 6. THE 1994 ALTERNATIVES REPORT The committee reviewed the 1994 Alternatives Report (USAGE, Sacramento District, 1994a) and found the document to be particularly confusing. The report provided a summary of its evaluation of different projects consisting of alterna- tive modifications of the system. Unfortunately, essential details of the analysis were omitted, so the committee could not determine what was actually done from reading the report. In particular, the committee could not determine the extent to which some criticisms of the 1991 analyses had been addressed.

OCR for page 114
RISK METHODOLOGY 157 The report also failed to associate with the estimated net benefits any mea- sure of overall uncertainty due to economic uncertainty, or hydrologic and hy- draulic uncertainties, as recommended by EC 1105-2-205. These uncertainties could be important given the modest benefit-cost ratios calculated for the alterna- tives considered. A very serious concern is how the report addressed issues of risk terminol- ogy and its reporting of flood risk. USACE now has two significantly different ways to calculate flood risk. On pages 8 and 9 of the 1994 Alternatives Report, they were both called "level of protection." No distinction was made between estimates of flood risk calculated with the traditional level of protection method- ology and those calculated with the new risk and uncertainty methodology. Throughout the report a host of different terms and phrases were used inter- changeably to describe these ideas. A layperson would have great difficulty sorting out the following jumble of terms: T-year level of protection, exceedance interval (p. 8), return period (p. 8), recurrence frequency (p. 9), control for Tyears (pp. 18 to 23), T-year flood (p. 9), T-year flood protection (p. 6), T-year protec- tion (pp. 27, 29), T-year return frequency (p. 34), expected exceedance (pp. 37, 39), expected level of protection (p. 57), annual recurrence (Plate 5), and flood event return period (Plate 12J. The report should use a few terms whose defini- tions are both clear and consistent with commonly accepted interpretations. The most common terms in the report are T-year level of protection, T-year protection, control for T years, and T-year flood protection. The use of the term "level of protection" to describe flood risk is inconsistent with the new USACE guidelines for risk and uncertainty analyses (EC 1105-2-205' and confuses the traditional and the new approaches to calculating flood risk. This terminology supports the erroneous idea that one and only one T-year flood occurs every T years. Actual statements in the report reinforce the error. On page 9, flood risk was described as a flood once in 78 years or 103 years, while the executive summary indicated that "levees could fail about once in every 78 years" and "the level of protection (or likelihood that levees would not fail) would be increased to about once in 100 years." These are exactly the analogies that should be avoided. With the new risk and uncertainty methodology, estimates of flood risk are no longer tied to a single T-year design flood, but can depend on different combi- nations of flood flows, operating decisions, and levee performance. Instead of stating that a project has a 200-year "level of protection" or protection for the 200-year flood, the Sacramento District should instead indicate that the annual risk of flooding is 0.5 percent per year, or the annual risk of flooding is 1 in 200 (see Stedinger et al., 1993, p. 18.3~. It is also informative to convert such annual risks into the risk of flooding over 25 to 50 year periods, reflecting the likely length of a mortgage or the anticipated economic life of structures and dwellings. When producing the 1994 Alternatives Report, the Sacramento District was under a great deal of pressure to revise its analysis of flood control alternatives to provide protection for people and property along the lower American River. Its

OCR for page 114
158 FLOOD RISK MANAGEMENT AND THE AMERICAN RIVER BASIN difficulties were increased by the need to use the new risk and uncertainty meth- odology being developed within the Sacramento District for the planning of flood protection projects. Inadequacies in the 1994 Alternatives Report reflect those pressures and constraints. The committee hopes that subsequent documents will more clearly describe how the analyses were conducted and will more clearly explain the basis for the risk and uncertainty analyses. THE PROMISE OF ECOLOGICAL RISK ASSESSMENT USACE has made a commendable effort to apply recently developed risk and uncertainty analysis to the engineering problems faced in minimizing the damage from floods. The question then arises: Should not the relevant ecologi- cal risk and uncertainty that may be the consequence of each of the proposed actions also be subjected to risk analysis? Applying ecological risk assessment to the major areas of uncertainty would be a daunting task. The following discus- sion highlights some of the advantages and disadvantages of such an approach. Development of the Paradigm Formal assessment of risk in ecological science and management is a rela- tively new development. Until very recently, EPA had not developed any guide- lines for risk assessment (Suter, 1993~. Thus far, the principal application of risk assessment to ecological problems has been in the context of considering impacts of hazardous chemicals in the environment, evaluating the risk of extinction of rare or endangered species, or providing management advice for commercial fisheries. Conceptually, there seems to be no reason that the process could not be applied to assessment of potentially adverse effects of water projects such as the ones considered here in the American River. However, the extension to such an analysis is controversial (Lackey, 1994) and probably will not be generally ac- cepted in the scientific community at this time. Ecological risk assessment has evolved slowly over the past two decades, but has received impetus from the National Research Council (NRC) paradigm for human health risk assessment: Risk Assessment in the Federal Government: Man- aging the Process (NRC, 1983~. EPA has recently released a Framework for Ecological Risk Assessment (EPA, 1992), along with a series of case studies (EPA, 1993, 1994~. These publications do not present final policy and proce- dures but are designed to stimulate discussion and development of a process that will be in flux for some time. EPA is developing formal guidelines for conduct- ing ecological risk assessments, which are expected to be released in late 1995 or early 1996. NRC has been in the forefront of such development, with reports on risk communication (NRC, 1989) and on issues in risk assessment, including a sig

OCR for page 114
RISK METHODOLOGY 159 nificant discussion of ecological risk (NRC, 1993~. Currently an NRC committee is conducting workshops designed to build consensus on the philosophy and methods for ecological risk analysis. In addition, many academic and industrial scientists are developing and evaluating the process (Suter, 19934. The debate about the extension of risk analysis to ecological problems has focused on several contentious points: The process is based on a human health paradigm; extension to ecological effects, particularly at the ecosystem level, is highly problematic. There is insuf- ficient understanding of ecosystem processes to predict outcomes with any cer- tainty. Risk assessment has the potential to produce a sort of an ecological triage, whereby particular processes and species thought to be important might receive attention at the expense of some potentially serious problems. Risk analysis may lead to a consideration of alternatives that is too nar- row, particularly if the focus is on the risk of a particular action versus that of no action. The analysis must consider the full range of alternatives, and benefits as well as risks of all the alternatives. The process can be tilted in favor of a particular action, given that uncer- tainty is great and the desired level of risk defined; the analysis may simply proceed until the desired endpoint is reached. In spite of these serious concerns, ecological risk analysis has had some success, leading to models that may provide templates for further development. A recent NRC report (NRC, 1993), in a section titled "A Paradigm for Ecological Risk Assessment," recognized significant problems in extension of the health risk approach from NRC (1983~. Nonetheless, that committee concluded that inte- grating ecological risk into the original framework is possible and that such an approach is preferable to developing a completely new framework. Key scien- tific issues limiting the application of ecological risk assessment include the following: Extrapolation across scales of space, time, and ecological organization. Estimating ecosystem-level response on the basis of laboratory or small-plot experiments is a particular concern. Quantification of uncertainty, including measurement uncertainty, natural variability in ecological systems, and inadequacy of models. . Validation of predictive tools. Substantial improvements are needed in the models fundamental to effective risk assessment. Valuation of outcomes. Analysis of both costs and benefits is essential, but generally accepted principles for valuation of ecosystems are not available.

OCR for page 114
60 FLOOD RISK MANAGEMENT AND THE AMERICAN RIVER BASIN Ecological Risk Assessment and the American River This committee believes that the Sacramento District has done a reasonably effective job of framing alternatives in the American River planning activities, particularly in the 1994 Alternatives Report. The recent organization of the Lower American River Task Force under the sponsorship of SAFCA has substan- tially improved communication among the various stakeholders in the basin. Hence there is the potential for appropriate use of ecological risk analysis. None- theless, there is little likelihood that such an analysis would be accepted by the scientific and lay community at this stage in the development of flood control proposals for the American River. One of the most contentious environmental issues faced by the committee is the assessment of the potential effects in the canyons of the North and Middle Forks above a proposed detention dam at Auburn. Great uncertainty surrounds estimates of the probability of mass soil failure and mortality of vegetation fol- lowing inundation. A case study example is available of risk analysis applied to a similar situation, modeling future losses of bottomland forest wetlands in Loui- siana in the face of increased flooding (EPA, 19931. However, this analysis was based on a substantial body of research in that region and on the application of a simulation model adapted for the specific area. No such base of knowledge is available for the American River canyon. Scientific understanding that would allow accurate modeling of the processes involved in hillslope failure and mortal- ity of vegetation is simply not available at this time, and most likely will not be available for years. Significant opportunities were missed when research failed to take advantage of the presence of the cofferdam upstream of the Auburn dam site, though the detention dam concept was not developed until after the dam breached in 1986. One field of resource management has a relatively long history of recogniz- ing uncertainty and may have lessons to provide as ecological risk assessment develops. Managers of marine and anadromous fisheries have long faced uncer- tainty. Stimulus for the development of more robust approaches to prediction in the face of incomplete knowledge has often come from the collapse of large fisheries (Ludwig et al., 1993~. The model of adaptive management advocated by Holling (1978) and Walters (1986) recognizes that uncertainty is a pervasive element of most resource management scenarios. The committee strongly rec- ommends that the water resource issues in the American River be managed in this adaptive context. Some important characteristics of this approach include the following: recognizing and communicating uncertainty, treating management as an experiment, and providing sufficient monitoring to allow managers to learn from the expe- rience gained from observing system behavior.

OCR for page 114
RISK METHODOLOGY 161 The current direction in models being developed for management advice emphasizes Bayesian analysis (Walters, 1986; Hilborn and Walters, 1992) and statistical decision theory (Frederick and Peterman, 1995~. Another trend in more traditional statistical analysis of natural resource issues has been a focus on statistical power analysis, particularly in the analysis of downward trends in resource abundance (Peterman, 1990~. The approach has promoted more explicit consideration of where the burden of proof properly lies. Incorporation of these concepts into ecological risk analysis should improve future decisions in a wide array of resource conflicts. CONCLUSION From its review of the material provided describing the new USACE risk and uncertainty analysis guidelines, and the 1994 Alternatives Report, the committee reached the following conclusions. Improvements in Planning Methodology. The USACE risk and uncer- tainty methodology is an innovative and timely development. The explicit recog- nition of modeling uncertainty should result in a better understanding of the accuracy of flood risk and damage reduction estimates. The committee applauds the USACE efforts to develop a better flood protection planning methodology incorporating both risk and uncertainty in hydrologic, hydraulic, and economic parameters and processes. However, USACE and the Sacramento District need to more carefully develop and articulate the structure of their risk and uncertainty methodology, employing an effective vocabulary for distinguishing among risk, variability, uncertainty, and system reliability for use with technical and public audiences. USACE leadership is encouraged to convene an intra-agency work- shop, including outside experts, to review the risk and uncertainty procedures, with special attention to the committee's concerns, and to recommend specific changes to the guidelines as necessary. Impact of Uncertainty on Performance Criteria. The proposed USACE risk and uncertainty methodology, which directly includes hydrologic uncertain- ties (and potentially other sources of uncertainty) in the calculation of average flood risk and the average annual flood damages that might be averted by a project, inflates those estimates. This upward bias is a concern if the methodol- ogy is adopted nationwide because it could distort the economic evaluation of projects. The committee did not have the resources to determine the actual distortion for the American River study. Descriptions of Project Performance. To avoid the problem of bias described in the recommendation above, and to simplify the analysis so that it can be more easily understood and is less dependent on hidden assumptions, the committee recommends that the primary descriptions of the expected annual flood damages and of the probability of flooding be based on best estimates of the

OCR for page 114
162 FLOOD RISK MANAGEMENT AND THE AMERICAN RIVER BASIN parameters of models defining the deterministic and significant random pro- cesses contributing to flood risk and flood damage. Descriptions of Project Performance Uncertainty. Best estimates of expected annual flood damages and the risk of flooding should be supplemented by descriptions of their uncertainty due to hydrologic, hydraulic, and economic uncertainties. Uncertainty can be described by a standard error or the distribu- tion of the likely values of the quantity of concern. The impact of uncertainty can be illustrated by computing the probability that the national economic devel- opment objective is negative, or various quartiles of its distribution. The ap- proach should be consistent with the requirement in USACE guidelines for risk and uncertainty analyses (EC 1105-2-205' that the estimate of NED benefits be reported both as a single expected value and on a probabilistic basis (value of the benefit and its associated probability) for each planning alternative. It is the committee's understanding that the American River study will not address eco . . . noetic uncertainties. Measures of System Performance Reliability. Estimates of expected annual flood damages and economic benefits associated with different projects, and the probability of flooding at different locations, are likely to be the primary criteria describing flood risk and economic impacts. It will often be useful to calculate other indices of system performance and the reliability of different components of the river channel and levee system. The committee questions in general the value of the system reliability index proposed by USACE documents and employed by the Sacramento District in the American River study. It seems to be an awkward combination of traditional and new concepts. In the case of the American River study, a reliability index did have an important role in demonstrating that different projects do not increase the risk of flooding in any reach of the American-Sacramento River system. Still, it is not clear that the adopted definition is the most effective or easily understood. How- ever, the Sacramento District's use of reliability does not affect the validity or accuracy of the study results and the calculations upon which they are based. Risk Analysis in USACE Alternatives Report. The committee re- viewed the risk and uncertainty analysis in the 1994 Alternatives Report. The report failed to associate with the estimated net benefits any measure of overall uncertainty due to economic, hydrologic, and hydraulic uncertainties. The com- mittee found the explanation and presentation of the results particularly confus- ing. No distinction was made between estimates of flood risk calculated with the traditional level-of-protection methodology and those calculated with the new risk and uncertainty methodology. Both were called "level of protection" and described by a variety of terms, which further contributed to the confusion. The most common terms in the report are control for T years, T-year level of protection, and T-year flood protection. The use of the term level of protection to describe flood risk is inconsistent with the new USACE guidelines for risk and

OCR for page 114
RISK METHODOLOGY 163 uncertainty analyses (EC 1105-2-205) and confuses the traditional and the new approaches to calculating flood risk. This terminology and phases appearing in the report fosters the erroneous idea that one and only one T-year flood occurs every T years. Moreover, with the new USACE risk and uncertainty methodology that was employed, failure is no longer related to a single T-year design flood being exceeded, but can depend on different combinations of flood flows, operating decisions, and levee perfor- mance. Instead of stating that a project has a 200-year level of protection, or protection for the 200-year flood, studies should indicate that the risk of flood- ing is 0.5 percent per year, or equivalently that the chance of flooding is 1 in 200 each year. The Promise of Ecological Risk Assessment. At this time the committee does not believe that the process of ecological risk analysis is sufficiently evolved, nor that there is sufficient knowledge of the ecological system, for this new tool to be applied usefully to problems of flood control in the American River basin. However, ecological risk assessment does provide a new approach that emphasizes the importance of uncertainty in the analysis of the conse- quences of various alternatives. The process will help select questions for inves- tigation and will be increasingly important in broadening the scope of future planning. USACE should follow this rapidly evolving approach and adopt it as soon as it shows promise of improving the decisionmaking process.