| ||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 30
Review of the Lake Ontario–St. Lawrence River Studies
Chapter 2
The Shared Vision Model
This chapter discusses the Shared Vision Model that includes a planning process, a simulation model, and a suite of associated models. The first section of the chapter introduces the Shared Vision approach including its planning framework and modeling efforts. The second section of the chapter evaluates the Shared Vision Model, beginning with its scientific qualities, followed by its integration and display of key information, and its overall sufficiency for informing decision-making. Documents provided for this review included:
SVM 1 – Evaluation Process Overview – a two-page broad description of the Shared Vision Model;
SVM 2 – Shared Vision Model for the Design and Evaluation of Alternative Regulation of Lake Ontario Releases into the St. Lawrence River – this document includes the “Toronto Board Room April 24.xls” spreadsheet file and 83 linked files. The linked files include:
Contextual Narratives that describe scenarios of economic impacts on commercial navigation, environmental effects, recreational boating and tourism, land use along the Lower Saint Lawrence River, and hydropower;
Regulation Plan descriptions; and
Approximately 70 Performance Indicator fact sheets;
SVM 3 – SVM Documentation Report – a six-page description of the structure of the model.
The contextual narratives were provided as background rather than for review, along with a document on “Preliminary Criteria and Metrics for the Plan Formulation and Evaluation Group” and an “Informal Decision Tool Program,” a spreadsheet file that provides an example of multi-attribute decision analysis. Staff members of the LOSLR Study made slide presentations and engaged in a three-hour question and answer period on the Shared Vision Model.
Shared Vision Planning and Modeling in the LOSLR Study
The LOSLR Study Board describes the Shared Vision Model (SVM) as “a unified computer simulation of the system” (SVM 1, p. 1; Werick, 2005). Its purpose is to evaluate previously formulated plans and alternative regulation plans as they are formulated. It is designed to be used by technical experts and stakeholders in a collaborative planning process for which its advocates have coined the term
OCR for page 31
Review of the Lake Ontario–St. Lawrence River Studies
“Shared Vision Planning.” An article reproduced on the U.S. Army Corps of Engineers’ Institute for Water Resources website describes shared vision planning as:
…deliberative, inclusive decision-making processes as the forum in which to debate how water resources will be used among competing ends. What is unique about shared vision planning, however, is how analytical technical expertise and analysis is integrated into a collaborative planning process. Through a structured planning process, an analytical computer model of the water resource system, called a shared vision model, is constructed with the participation of stakeholders. The shared vision model is designed to be used by stakeholders themselves to develop a mutually satisfactory water supply plan… (Stephenson, 2003, p. 1)
The U.S. Army Corps of Engineers introduced this approach to shared vision planning in its 1995 National Study of Water Management During Drought (IWR, 1995). The approach consists of three elements: (1) a structured planning process, similar to that described in the Economic and Environmental Principles and Guidelines for Planning Water and Related Land Resources (U.S. Water Resource Council, 1983) that states objectives and identifies problems and opportunities; formulates and evaluates alternatives; and displays effects of each alternative on each of the stated objectives; (2) a collaborative, negotiation process involving stakeholders; and (3) use of scientific and technical information through the use of computer models that accounts for the complex linkages among various elements of the water resource system under investigation (Stephenson, 2003). This last element of Shared Vision Planning is the Shared Vision Model.
The LOSLR Plan of Study is the official document that authorizes and defines the scope of the study (BACK 2). It was prepared by the IJC and approved by the two governments. The Plan of Study contains directives that are consistent with the three elements of a structured planning process, stakeholder collaboration, and computer simulation models described by Stephenson (2003). The Plan of Study mandated the use of a multiple-criteria rational planning model, similar to the one in the Principles and Guidelines, which guides water resource planning by federal agencies in the United States (U.S. Water Resources Council, 1983). The Plan of Study directed the LOSLR Study to investigate existing criteria within the IJC’s Orders of Approval, and to determine the requirements for establishing new criteria for improved operation of the Lake Ontario-St. Lawrence River system. Existing criteria were primarily related to economic effects and functional requirements, particularly of hydropower, commercial navigation, and flooding. In referencing the 1996 Scope of Work, the Plan of Study emphasizes the need to consider wetlands, other environmental factors, and recreational boating interests, which had not been addressed in the original regulation plan. The LOSLR Plan of Study called for formulation of alternative regulation plans and evaluation of them with respect to multiple criteria, including both existing and new criteria that may be added. These directives comprise a structured planning process, which is the first element of the shared vision approach.
The Plan of Study mandated the formulation and evaluation of regulation plans
OCR for page 32
Review of the Lake Ontario–St. Lawrence River Studies
using a collaborative, negotiation process involving a wide range of stakeholders. This process is the second element of the shared vision approach, it provided for creation of the LOSLR Study Board and assigned the Study Board overall responsibility for the Study. The Study Board’s 14 members, half from the U.S. and half from Canada, are predominantly government personnel, academics, other technical experts, and Native peoples. The Study Board was further authorized to establish bi-national work groups to support its work. The Plan of Study (BACK 2, p. 13) emphasized that:
Public consultation is critical to the assessment of plan criteria. It is recognized that progress in addressing water levels issues is dependent in large part on public understanding that most proposed solutions could have consequences for others. To achieve this understanding, it is recommended that the major interests and the relevant public be involved directly in the studies, by the formation of an Interest Advisory Group… The continuous involvement of all interests throughout the criteria review process is critical to the success of the study.
The Plan of Study (BACK 2, p. 76) also directed that computer simulation models would be used, i.e., the third element of Shared Vision Planning. It stated:
The evaluation of Lake Ontario regulation plans, the practicality of proposed criteria, and the hydrologic impacts on the interests, require computer simulation of water levels and flows of the Great Lakes–St. Lawrence River system downstream as far as Trois-Rivères, Quebec. …While separate computer models exist for different parts of the system, and substantial progress has been made in the development of a coordinated routing and regulation model for the upper lakes, additional work is required to develop and integrate Lake Ontario regulation plan(s) and St. Lawrence River components into the model to simulate water levels and flows of the entire system (including the Ottawa River and other tributaries).
The Plan of Study even specified the time step for the simulation: because Lake Ontario outflows are regulated on a weekly basis, the computer simulation would use a quarter-month time-step. In this specification is the recognition that effects dependent upon shorter time periods would not be captured by models using the quarter-month time-step. Therefore, the Plan of Study called for development of a supplemental, two-dimensional hydrodynamic model for the St. Lawrence River upstream of Cornwall-Massena from near Kingston to the Moses-Saunders dam to simulate short-term effects on commercial navigation, recreational boating, environmental questions, hydropower, and ice formation.
The Shared Vision Planning approach has been used in other venues prior to its use in the LOSLR basin, including the following large-scale studies : (1) Reservoir Operating Plan Evaluation (ROPE) Study for the Mississippi Headwaters (Cardwell et al., 2004); (2) the Rappahannock River Basin Commission Water Supply Planning Project
OCR for page 33
Review of the Lake Ontario–St. Lawrence River Studies
(Connor et al., 2004; Werick, 2000); (3) the Interstate Commission on the Potomac River Basin (ICPRB) Water Supply (Hagen and Kiang, 2003); and (4) the Alabama-Coosa-Tallapoosa-Apalachicola-Chattahoochee-Flint (ACT-ACF) River Basins (Palmer, 1998). These examples represent a subset of a much larger literature on adaptive water resources management that has relevance for this study (e.g., NRC, 2004).
The LOSLR Study is commended for building upon this shared vision planning approach, and for addressing the three criteria described by Stephenson (2003). The LOSLR Study attempts to integrate these criteria in an innovative water resources planning process in the Lake Ontario-St. Lawrence River basin. Although integrated water resources planning is increasing in scope and application (e.g., NRC, 2004), the norm is still for scientific and technical analyses to be conducted separately from one another and from various facets of public decision-making processes and for models to lack transparency. These issues will be discussed further in relation to the Shared Vision Model, which is introduced below.
Shared Vision Modeling
As noted earlier, the LOSLR Study uses the term Shared Vision Model to describe a (1) planning process, (2) suite of sub-models, and (3) core simulation model. This section describes the computer modeling. The structure of the SVM consists of the suite of models shown in Figure 2-1 and described in general terms in the LOSLR Documentation Report (SVM 3).
The heart of the SVM is a computer model (the “STELLA” model in Figure 2.1) that simulates behavior of the Lake Ontario-St. Lawrence River system to Batiscan, downstream of Montreal, for any given regulation plan and time series of inflows. The STELLA model gets its name from the fact that it was written in the STELLA computer language. A simplified flow diagram of the simulation model as the committee understood it is shown in Figure 2-2.
The decision or control variables in the model are given in a Plan of Regulation that governs releases to be made from Lake Ontario through the Moses-Saunders Dam under various system inflow conditions. That time series of releases from the dam is then used to route a time series of quarter-month system inflows through a water balance model for the Lake; the outputs are a time series of lake and river levels. Downstream tributary flows are added to the dam releases at points of confluence to calculate time series of river flows and stages at various locations.
Flows are used within the STELLA model to calculate lake elevations, power, and navigation benefits, which are forwarded to three other programs that evaluate impacts on coastal, environmental, and recreation systems. Results from all four models are then presented in a “Board Room” spreadsheet.
The Shared Vision Model also includes several research components that supported development of the simulation model and related calculations of economic and environmental effects. Hydrologic and hydraulic (H&H) research activities
OCR for page 34
Review of the Lake Ontario–St. Lawrence River Studies
FIGURE 2-1 Structure of the shared vision model. SOURCE: Werick (2005).
OCR for page 35
Review of the Lake Ontario–St. Lawrence River Studies
FIGURE 2-2 Committee interpretation of the Shared Vision Model.
produced time series of quarter-month system inflows adjusted to account for lake evaporation and other losses. Time series were generated for historical conditions, for historical conditions adjusted for climate change, and for stochastic series intended to represent probabilistically a wide range of possible future conditions. The hydrology and hydraulic research also produced estimates of ice and channel roughness and changes in temperature that may result from alternative plans. The Committee received a report on Lake Erie outflows (Fan and Fay, 2003) but did not receive other documentation of the time series of inflows to Lake Ontario used in the LOSLR study, and was not charged to review the hydrologic and hydraulic work.
Research was also undertaken to establish relationships between water levels and flows and important effects of regulation, namely municipal and industrial water supply (M&I), recreational boating (Rec Boating), the environment (Env), commercial navigation (Nav), coastal flooding and erosion (Coastal), and hydropower production (Hydro). The relationships established through these research activities were used to build three sub-models. One is the Flooding and Erosion Prediction System (FEPS), used to predict flooding and erosion along the coasts of Lake Ontario and the river above the regulating dam. FEPS takes a time series of
OCR for page 36
Review of the Lake Ontario–St. Lawrence River Studies
lake levels from the STELLA model (passed through the Board Room) to calculate a time series of (1) erosion of developed land that has no shoreline protection, (2) costs of maintenance and repair to land that has coastal shore protection structures, and (3) flood damages. A second sub-model, the River Shore Protection Model (RSPM), calculates impacts and costs of erosion of existing shore protection below the Moses-Saunders dam. The third sub-model, the Integrated Ecological Response Model (IERM), uses relationships established in the environmental research program to estimate a large number of environmental performance indicators resulting from a given regulation plan and a given time series of lake levels. FEPS and IERM are reviewed in chapters 3 and 4, respectively; the RSPM was not subject to this review.
The Board Room is a spreadsheet file. It takes the simulated time series of hydrologic attributes from the STELLA model and sends them to the FEPS, RSPM, and IERM models to estimate effects on performance indicators. It then receives results from those sub-models, and prepares statistical summaries as “…requested by decision makers and stakeholders, including average annual, worst case, persistence of bad performance, etc.” (SVM 3, p. 2). The Data Warehouse stores alternative time series of inflows for use in the SVM. The Control Panel is described as “…a spreadsheet file that helps the SVM user design new regulation plans…” (ibid, 4). It also provides the user with options from which to select a time series of inflows stored in the Data Warehouse as an input to the SVM. The Data Warehouse and Control Panel were not included in this review.
Brief Description of Regulation Plan Options
The primary purpose of the LOSLR study is to formulate regulation plan options for the releases from Lake Ontario and the upper St. Lawrence River through the Moses-Saunders dam. The Board Room includes information on seven plans formulated and evaluated in detail. Brief descriptions of the plans included in the Board Room are:
Plan 1958 D – the plan that went into operation in October of 1963;
Plan 1958 D with Deviations (Plan 1958DD) – Plan 1958D with deviations that the Control Board has been using in recent years to cope with outcomes it considered to be unacceptable;
Plan A (also known as Cornell V) – described by the Study Board as “the balanced economic plan,” (Available on-line at http://www.losl.org/PDF/17062005_NewsR-EN.pdf. Accessed August 7, 2005).
Plan B (also known as NaturalLy) – described by the Study Board as, “the balanced environmental plan” (Available on-line at http://www.losl.org/PDF/17062005_NewsR-EN.pdf. Accessed August 7, 2005).
Plan D – described by the LOSLR Study Board as, “the blended benefits plan” (Available on-line at http://www.losl.org/PDF/17062005_NewsR-EN.pdf. Accessed August 7, 2005).
OCR for page 37
Review of the Lake Ontario–St. Lawrence River Studies
Plan E – The Pre-project Plan with Ice Limits incorporates historic pre-project winter ice retardation effects on pre-project flow.
Plan 1998 – versions of the plan proposed in 1998 that was rejected by the IJC.
Outcomes from the current “Plan 1958 D with Deviations” (or Plan 1958DD) were used as a baseline against which other plans were compared. Although this review does not comment on the specific regulation plan options listed above, it does note here and discuss below that the plan formulation process appears to have focused on specifying release criteria to accommodate future hydroclimatic scenarios under current economic and environmental conditions. Plan formulation does not appear to have focused on provisions for monitoring, testing, evaluating, and, if necessary, adapting plans and criteria to changing conditions and priorities.
EVALUATION OF THE SVM OVERALL FINDINGS AND ACCOMPLISHMENTS
The overall strength of the SVM is its ability to support multi-stakeholder decision-making with data-driven comparisons of tradeoffs among the economic, environmental, and social effects of regulation plan options. The inter-disciplinary scope of inquiry of the SVM and its inclusive approach will likely serve as an important case study for other large-scale, multi-interest, water resources planning projects. These SVM strengths, however, are undercut by the poor model documentation that leaves unanswered many questions of the rationale behind unexpected modeling choices, methodology, treatment of model error and uncertainty, and whether the SVM in fact is integrating scientific and economic components in a technically and scientifically sound manner. A more detailed evaluation of the SVM follows that corresponds with the three charges in the statement of task: scientific foundations, integration and display of key information, and overall appropriateness and sufficiency of the SVM to inform regulation plan decision-making.
EVALUATION OF SCIENTIFIC METHODS, ASSUMPTIONS, AND SUPPORTED FINDINGS
Charge A of the statement of task focuses on the extent to which the studies and models reflect reasonable scientific methods, assumptions, and supported findings. Three evaluation criteria are used to assay the scientific foundations of the SVM: empirical foundations, quality assurance, and treatment of risk and uncertainty.
OCR for page 38
Review of the Lake Ontario–St. Lawrence River Studies
Criterion 1: Empirical Foundations
The primary purpose of the SVM is to convey how changes in lake level and river flow will impact a variety of other dependent conditions in the LOSLR system (e.g., wetlands, recreational boating, flood damages to property, etc.). To fulfill this purpose, the SVM is designed to draw from several empirical bases, the majority of which are either not subject to this NRC review or are reviewed in subsequent chapters of this report. Therefore, this section focuses on three specific empirical issues: lake level calculations, use of regression analysis for modeling the St. Lawrence system, and assumptions about related processes and trends within the STELLA model.
Single Lake Level Calculation
Water levels and flows in the LOSLR system are generated using hydrologic time series input from a limited number of points in the lake and the river. The input flows used by the SVM were generated from (1) historical records adjusted for current diversions, (2) historical flows adjusted for climate change, and (3) stochastic flow series intended to probabilistically represent a wide range of possible future conditions. Most input flow evaluations were based on 101 years of record (4,848 quarter-months). A stochastic sequence of flow for 50,000 years was generated and, from that sequence, 101-year records of very wet and very dry centuries were selected to evaluate plan performance under extreme conditions. IJC documents do not discuss how flow sequences were generated or why 101 years was chosen as the length of record. Stochastic time series would normally include multiple realizations that can be used to generate probabilistic model outputs, but documentation did not indicate that multiple realizations were used in the LOSLR application.
The STELLA model produces a single water elevation value for Lake Ontario for each quarter-month time period. This STELLA model value must be accurate for the SVM to function reliably because it is used in other models as input for wetlands, fauna, erosion, and bathymetric calculations. One problem in using a single value in the quarter-month time step is that Lake levels can be more dynamic than this value can capture. For example, the average change in water elevation of Lake Ontario may be as much as 20 cm in one week (http://www.glerl.noaa.gov/data/now/wlevels/levels.html). This large range in water elevation values is nontrivial because erosion and ecological models use the average values to calculate coastal and environmental responses, respectively. Although some of the sub-models generate variable water level conditions from the average values, any inaccuracies in the average water level values would propagate through the erosion and ecological model outputs and the SVM. It is therefore surprising that the SVM documents do not report on how the accuracy of average water level values is determined, or how sensitive the other models are to the accuracy of this value.
OCR for page 39
Review of the Lake Ontario–St. Lawrence River Studies
The problems of relying on a single water elevation value for Lake Ontario can be summarized in three points. First, water level data are available at multiple locations on Lake Ontario and at a finer time step than quarter-month period. The decision to calculate a single model output for a larger area and longer time period needs better justification. Second, the coastal and environmental models use these less precise single value quarter-monthly outputs to calculate finer-scale water level and wave processes in different areas of Lake Ontario and the Upper St. Lawrence River. Third, those finer-scale hydrologic scenarios are subsequently used to calculate a chain of environmental, coastal, and economic impacts. Thus, the decision to use a single lake-wide water level value needs fuller explanation and an analysis of the accuracy of a single water level value in the SVM and submodels.
Use of Regression for Modeling the St. Lawrence System
The STELLA model calculates river water levels and flows at a limited number of points based on regression equations (Fan and Fay, 2001, 2002, 2003). Regression analysis is straightforward and inexpensive, which may be the reasons it was chosen for use in the LOSLR work.
Several questions remain unanswered with respect to regression analysis in the SVM. First, why were regression analyses used to a greater extent than conventional hydraulic models? Given that hydraulic models are empirically feasible, and were recommended in the Plan of Study (BACK 2), it is unclear why the SVM relied more heavily on a different and less accurate water level calculation method. Second, how were the regression analyses developed, and how accurate were the results? Regression is a common statistical tool that is described by standard statistical descriptors, such as r2 and confidence intervals. These descriptors are not included in the SVM documents, and no measures of accuracy or reliability are presented to support the regression analysis approach. Without proper documentation of the development and accuracy of the regression analyses, and the relationships between the STELLA model and the hydrodynamic modeling, it is not possible to render a judgment about the adequacy or accuracy of the regression approach. In fact, regression could lead to inaccurate results, which spurs concerns about the use and application of regression analyses in the Shared Vision Model. The ways that lake and river hydraulics models and regression analysis were used with the SVM need additional justification to defend their selection and application.
Assumptions about Related Processes and Trends within the STELLA Model
The SVM has the ability to support multi-stakeholder decision-making using tradeoffs among economic, environmental and social effects of water level regulation. However, the reviewed documents suggest that related socioeconomic trends
OCR for page 40
Review of the Lake Ontario–St. Lawrence River Studies
and their possible feedback on system operation were not incorporated into the SVM. For example, population, land uses along the coastline, and demands for commercial navigation and recreational boating were specified externally. For the most part, values for these variables were fixed at current levels such that it is not possible to determine the effects of water level and flow regulation on these processes. Also omitted from the SVM analysis are the impacts of changes in externally specified water demands and land uses on the economic and environmental performance indicators modeled under alternative regulation plans. These deficiencies in socioeconomic analysis, forecasting, and scenario construction limit the long-term utility of the SVM, and the influence of dynamic socioeconomic factors on modeled output variables must be treated more fully.
Criterion 2: Quality Assurance and Quality Control
Model verification and sensitivity analysis are commonly used to compute and report model accuracy. Sensitivity analysis determines model parameter effects on model results and assesses model uncertainty, but sensitivity analyses of the SVM were not presented in the review documents. IJC documentation (SVM 3) states that validation and verification of the SVM involve a system of internal, self-assessments by the Technical Working Groups responsible for the development of each sub-model. Questions in these self-assessments included the following (SVM 3, p. 5):
“Is the underlying research trustworthy?”
“Is this the information you [stakeholders] were looking for?”
“Is this information being interpreted correctly?”
“Is a 101 year average a misleading metric?”
Although these questions are important, they indicate a different approach to validation, and it is unstated as to why these types of questions were used in lieu of traditional model validation, verification, and sensitivity analyses that would quantify inaccuracies due to underlying computational components, incorrect model choice, inappropriate approximations, or poor parameter estimation. Expanding the validation process to include these types of more traditional methods of model verification would yield greater levels of confidence in the SVM and its sub-models. To give the highest confidence that the SVM and its sub-models are accurate and perform as designed to meet model expectations, traditional validation and verification methods should be employed and the results should be presented for formal external peer-review.
Criterion 3: Treatment of Error and Uncertainty
Treatment of uncertainty in the SVM raised questions similar to the treatment
OCR for page 52
Review of the Lake Ontario–St. Lawrence River Studies
eral approaches. The Board Room spreadsheet includes a list of 21 screening factors used by the Study Board. Although it does not provide supporting information about how those factors were applied to specific options, it does indicate the importance attached to different evaluation criteria. It is still not clear whether the SVM was used to identify the possibility frontier of feasible alternatives.
The Possibility Frontier. A “possibility frontier” of efficient solutions (Cohon, 1978; Goicochea et al., 1982) can be depicted as a curved boundary that that separates technically feasible plans (inside the boundary) from technically infeasible plans (outside the boundary). In an illustrative case of two objectives, improving the environment and increasing net economic benefits, plans on the boundary are efficient, in the sense that it is not possible to improve the performance for one objective without decreasing performance of the other (Figure 2-4). A possibility frontier can also be developed for a multi-objective optimization problem, where the boundary is a 3- or more-dimensional surface. In either case, the possibility frontier can be used as a guide for those who are charged with deciding among efficient plans. Although the SVM does not identify a possibility frontier of efficient alternatives, it does enable evaluation of plans that are relatively better or worse for different indicators, regions, and interests, which the Board Room displays in detail. SVM materials do not document the process employed to formulate plans that increase one set of benefits without decreasing others (i.e., Pareto-optimal type solutions), or increase benefits for some without causing disproportionate loss to others (though see Leger et al., 2005).
Disproportionate Loss Criteria. The SVM presents board members’ screening factors in a level of detail that is commendable. The Study Board uses the criterion of no disproportionate losses, for instance, to eliminate water management plans that produce unacceptable damages for one or more sectors, regardless of benefits to other sectors. For example, IJC and LOSLR Study representatives reported that the economically most beneficial plan, particularly beneficial to hydropower, was rejected because it produced disproportionate losses to the environmental sector. The Board Room spreadsheet displays losses from selected plans, but it does not provide a record of the versions of those and other plans rejected as unacceptable, the progress made toward Pareto optimality through the SVM, the quantitative criteria for judgments of disproportionate loss, or changes in those criteria over time (though again see Leger et al., 2005). The IJC and the public would be informed by knowing the range of choice defined by both possibility frontier and loss criteria, as well as the record of tradeoff analysis made through SVM modeling. It is not clear from the SVM documentation which tradeoffs were made when various plans were screened, selected, or rejected. This information is crucial to create a transparent process. The SVM plan evaluation process should document the range of efficient and minimally acceptable plans considered in order to facilitate informed decisions.
OCR for page 53
Review of the Lake Ontario–St. Lawrence River Studies
FIGURE 2-4 The possibility frontier for regulation plans with two objectives.
Comparable Treatment of Economic and Environmental Aspects of Performance Indicators
The LOSLR Study treats economic and environmental effects differently. While it is acceptable to forgo a full multi-attribute utility theory model (that is, not put economic and environmental indicators on common footing, whether via multi-attribute utility models or via monetizing environmental indicators), environmental and economic indicators should be treated in ways that enable proper integration of the two types of indicators in decision making. For example, while environmental performance indicators were subjected to an arbitrary “10% rule” to assess their significance, economic indicators were not. There is no statistical or practical justification offered for the decision to use the “10% rule” to express confidence in the significance of environmental performance indicator ratios.
The environmental indicator ratios are problematic. Although ratios may be reasonably well-behaved when used to evaluate plans that represent minor deviations from a baseline plan, they do not allow expression of variability and confidence limits to assess whether differences between plans are significant. In addition, the ratios were constructed in a variety of ways (sometimes aggregating before and sometimes after forming ratios, using moving averages for some indicators, etc.), which makes it difficult to compare dissimilar metrics across plans consistently, leading to inconsistent and unstable ordering of plans (SAR 3A discusses these issues nicely).
The Economics Advisory Committee raised a similar question about the comparability (i.e., fungibility) of economic variables. Economic indicators,
OCR for page 54
Review of the Lake Ontario–St. Lawrence River Studies
although not a specific subject of this review, are a major aspect of the SVM, and the report of the Economics Advisory Committee (2004) was supplied to the Committee as a background document. The Committee shares the Economics Advisory Committee concerns about the treatment of economic factors in the SVM. It is not clear from SVM documents which of the Economics Advisory Committee recommendations have been implemented. For example, while the SVM has “net benefits” as an evaluation criterion, it does not follow standard procedures for evaluating the time streams of economic benefits associated with alternative plans. A generally accepted method for treating economic consequences over time is to use the present value of discounted benefits and costs over an economic time horizon. The present value of discounted stream of net benefits over T years (PVB) is defined as annual benefits (B) discounted by a rate that reflects economic time preference (r) (EAC, 2004, pp. 5-6):
In the SVM application, the time horizon is 101 years. Values of PVB can be calculated for each 101-year sequence of inflows. Expected values of PVB could be estimated as the average value of PVB for multiple sequences of inflows. In this application, the 50,000 years of simulated flow sequences would provide about 500 possible sequences and 500 possible values of PVB. This series could also be used to estimate a probability distribution for PVB.
By holding external economic factors at present levels, the Study appears to have estimated only values of current benefits, B0, under a range of possible annual sequences. This value does not capture effects of changes in economic factors and changes to physical and biological conditions over time.
The Study Board is aware of these issues (LOSLR, Year 3 Report, 2004, pp. 54-55). The coastal erosion performance indicator discounts future expenditures on shore protection, which vary under different hydroclimatic and water regulation scenarios (FEPS 7, pp. 23-7; and FEPS 14, p. 4). However, it is not clear how economic time preference has been addressed across different economic sectors and submodels in the SVM; and it is even less clear whether the present value of future environmental changes has been considered (e.g., as matters of intergenerational equity).
Because of problems with the way individual indicators were treated, and the different ways in which environmental and economic indicators were treated (e.g., the “10% rule”), the SVM output does not enable decision makers to comprehensively weigh trade-offs between environmental and economic indicators, as they will need to do in the absence of a fully formulated multi-attribute model.
OCR for page 55
Review of the Lake Ontario–St. Lawrence River Studies
Criterion 9: Balance of Scientific and Practical Professional Approaches
Evaluating the balance between scientific and practical professional approaches in the Shared Vision Model is contingent upon its performance under the other evaluation criteria, its constitutive models (FEPS, IERM, etc.), and its documentation. The constitutive models are discussed in chapters 3 and 4. This section focuses on the balance achieved within and among the other evaluation criteria. Each criterion has scientific and professional aspects, but the main scientific evaluation criteria employed in this review focus on empirical foundations, QA/QC, and treatment of error and uncertainty (Criteria 1, 2, and 3). Criteria 4 and 5 on study linkages and feedback, and spatial and temporal resolution, respectively, have strong scientific as well as practical professional roles in the LOSLR Study. Documentation, communication, and breadth of study scope criteria are common to both practical professional and scientific approaches. Two examples of the balance in the SVM are presented: one is drawn from the scientific criteria (Criterion 2) and the other from the practical professional side (Criterion 6).
Peer review is a standard scientific practice (Criterion 2). According to SVM materials, the SVM was to undergo two rounds of review: an intensive internal review and now this NRC review at the end of the study. The internal “peer” review is described in SVM documentation (BACK 6):
The Shared Vision Planning (SVP) process provided perhaps the most detailed and intensive collegial review in that all the research and data was incorporated into the SVM. All the scientific and technical information had to be encoded as algorithms in the model, and each part of the model was validated independently.
Through the course of this NRC review, however, it became clear that the SVM was likely neither “validated independently” nor presented for external peer review in refereed publications. Peer review among practitioners and scientists in refereed publications would have given strong indication as to whether the SVM has struck the important balance between the scientific and practical professional approaches.
Documentation is standard practice for scientists and practitioners alike. It is used to explain methods, analyses, and treatment of error and uncertainty within and across different audiences. Problems with SVM documentation have been noted (see Criterion 6). These problems contribute to a lack of clarity and understanding about why certain decisions were made, such as the choice of regression analyses or rationale for using a single lake water elevation. These decisions, if better defended in stronger documentation, could have shed light on the balance between scientific and practical professional approaches in the SVM. However, this balance cannot be duly assessed without improved documentation.
Remedying the lack of documentation and quality assurance is necessary to assess the overall balance between scientific and practical profes-
OCR for page 56
Review of the Lake Ontario–St. Lawrence River Studies
sional approaches in the SVM, and in some of its specific modeling choices and applications.
Criterion 10: Identification of Future Study Needs
The LOSLR Study was initially planned to span five years. Now at the end of its 5-year life, the enormous amount of work and resources expended on the SVM development will be put to the best use over a multi-decadal time line. The history of the current single static plan, Plan 1958DD, demonstrates the need for flexibility and adaptability in water regulation in this dynamic watershed. However, future study needs were not explicitly identified in SVM review materials, and the LOSLR Study would benefit from a long-term plan that identifies future needs and potential changes. In support of the SVM being used over a multi-decadal time scale to assess changes in performance indicator responses to water level management, three needs are identified: revision and maintenance of the SVM, further environmental scenario analysis (e.g., climate change); and development of a systems dynamics approach to water regulation.
Revision and Maintenance of the SVM
SVM materials did not disclose a plan to revise, maintain, and update data collection and modeling activities. Like the first iteration in most studies undertaken to support important and controversial political decisions, current versions of the SVM are based on numerous assumptions and simplified representations of the real system. As noted in earlier sections of this chapter, the SVM has several deficiencies: existing documentation is incomplete, and there are several important gaps in the modeling structure and database. As a first step, these revisions need to be made. After the revisions are made and a new regulation plan is selected, the new plan will need on-going evaluation to determine how well it satisfies performance objectives under conditions of dynamic change in the Lake Ontario-St. Lawrence River basin. The history of change in Plan 1958D/Plan 1958DD shows that a single, static plan cannot be maintained over multiple decades, and ability is needed to adapt plans to changing conditions. For the LOSLR Study to have a lasting effect, the SVM needs initial revision and then continued maintenance to keep current with changing conditions in the LOSLR basin.
Further Environmental and Socioeconomic Scenario Analysis
Scenarios of long-term environmental and socioeconomic change also need a prospective approach that can be adapted with advances in scientific understanding of the region. Climate change is one example of dynamic human-environment relationships that has environmental and socioeconomic implications. Most recent re-
OCR for page 57
Review of the Lake Ontario–St. Lawrence River Studies
search on climate change in the Great Lakes region indicates that the direction of possible effects on water levels and flows (up or down) is highly uncertain—as are the magnitude, timing, and spatial variability of change at regional and local scales. Although the potential impacts of climate change on the resources of Great Lakes basin have been extensively researched (Croley, 2003; Lofgren et al., 2002; Lofgren, 2003, 2004; Sousounis, 2002), and addressed to some degree in SVM scenarios of climate change, the need for further analysis of this and other long-term environmental processes may be anticipated. Similarly, as noted earlier, socioeconomic trends will likely have substantial relevance for future water level and flow regulation. The LOSLR Study takes an important step by examining scenarios of climate change and water regulation, but these and other environmental and socioeconomic scenarios will likely change and require further evaluation.
Toward a System Dynamics Approach
System dynamics simulation is a powerful tool for understanding complex feedback relationships among social and environmental components of a system (Simonovic, 1999; Simonovic and Rajasekaram, 2004; Sterman, 2000). The STELLA model is a system dynamics model as well as the modeling platform for the SVM. However, the STELLA model is not an immediate good fit for purposes of the SVM for several reasons. The value of system dynamics simulation lies in understanding the relationships within a complex system, and it is not typically used to predict the future behavior of a system. The STELLA model has capabilities that extend far beyond the simple water balance application for which it is being used in the SVM. The STELLA model, as developed and utilized in the Study has a very limited purpose: to calculate the response of the LOSLR hydraulic system to modification of rules governing release of water from Lake Ontario and the upper St. Lawrence River. The structure of the system is defined by the physical configuration of the network of rivers and lakes and the structures that regulate the flow of water through this system. This physical model of the hydraulic system is rigid. It is characterized by the same-state variables, and it does not add or subtract any elements or introduce any new interactions. For a given set of inputs (inflows to Lake Ontario, inflows from tributaries) the role of any model is to calculate flows and water levels at locations within the network where this information is required for other modeling purposes. Based on the structure of the SVM and its submodels, this task can be performed by a spreadsheet using mass balance equations.
The STELLA model could have been the modeling engine for all sub-models in the SVM, but for reasons unexplained, it is not being used that way. The important feature of the hydraulic network in the STELLA model is that it remains static: whatever is happening in other subsystems has no impact on how the water is being distributed through the network over time. On the other hand, changes in the operating rules (new plans) do have an effect both on the spatial and temporal distribution of water flows and levels and on how other subsystems behave and respond. Although the hydraulic system can be modeled independently of other subsystems, which may be the purpose for using the STELLA model for the SVM, this
OCR for page 58
Review of the Lake Ontario–St. Lawrence River Studies
rationale is not explained in the documentation.
Partial implementation of system dynamics simulation has the effect of under-utilizing the STELLA model in the SVM applications. The SVM should include feedback loops to account for treatment of exogenous dynamic variables such as socio-economic and demographic factors, but it does not in its current design. This limitation was previously discussed under Criterion 4. Had the SVM included these feedback loops, the STELLA model would be a suitable application to simulate those feedbacks. This partial implementation of the systems dynamics modeling approach (i.e., using the STELLA model and not including feedback loops) is a major conceptual limitation of the SVM as used in this study.
SUMMARY
The LOSLR study is commended for its incorporation of broader water management goals for the LOSLR system and its commitment to public participation. Technically, however, its formulation and evaluation of alternative water regulation plans using the SVM modeling system are not likely to serve the intended purpose as well as they could. The SVM does not document its exploration and evaluation of the range of choice among regulation plan options and associated mitigation measures. Its deficiencies in documentation and quality assurance are uncharacteristic of scientific and practical professional approaches, and it does not lay out priorities for future modeling improvements and applications. As discussed in criteria 1 through 7, some important linkages between the SVM and supporting empirical research, sub-models, and exogenous variables are found wanting.
In spite of these criticisms, the Shared Vision Model approach represents an enormous investment in science to support decision making, and that investment should be captured to the maximum extent. Doing so requires an ongoing commitment to maintain and augment the databases and the models developed during the LOSLR studies in an adaptive management framework.
Experience with the Plan 1958DD plan indicates that it is unrealistic to think that a single plan can be adopted and then implemented for years into the future, given the dynamic physical and social context of the LOSLR system. Whether the issue is climate change, population growth, or new technologies for energy production, any water regulation plan for the LOSLR will be need to adapt to the dynamism of the Lake Ontario-St. Lawrence River basin. The SVM modeling system provides the kernel for an adaptive management system, but only if resources are invested in repairing its deficiencies and managing it for long-term use.
Specific recommendations to improve the overall appropriateness and sufficiency of the SVM to inform decision making include the following:
The decisions to treat economic and environmental indicators differently should be justified in the SVM documentation or changed
Incompatibilities in temporal and spatial scales between the STELLA
OCR for page 59
Review of the Lake Ontario–St. Lawrence River Studies
model and SVM sub-models could introduce untold error and need fuller exploration and explanation
For the LOSLR Study to have a lasting effect, the SVM needs continued maintenance and updating to keep current with changing conditions in the LOSLR basin
A systematic process is needed to document the unfolding range of choice among alternative regulation plans.
Use of the STELLA model in the SVM represents a partial implementation of the systems dynamics modeling approach and is a major conceptual limitation of the SVM as used in the LOSLSR study.
REFERENCES
BACK 2. IJC (International Joint Commission). 1999. Plan of Study for Criteria Review in the Orders of Approval for Regulation of Lake Ontario-St. Lawrence River Levels and Flows. Available on-line at http://www.ijc.org/php/publications/html/pos/pose.html. Accessed September 2005.
BACK 6. Peer Review Triage Strategy (draft).doc.
Cardwell, H., B. Faber, and K. Spading. 2004. Reservoir Operating Plan Evaluation (ROPE) Study for the Mississippi Headwaters Collaborative Models for Planning in the Mississippi Headwaters. Available on-line at http://www.iwr.usace.army.mil/iwr/svp/rope.htm. Adapted from Conference Proceedings of the ASCE Environmental and Water Resources Conference, Salt Lake City, UT., June 28-July 1, 2004.
Cohon, J. L. 1978. Multiobjective Programming and Planning. New York: Academic Press.
Cohon, J. L., and D. H. Marks. 1975. A review and evaluation of multiobjective programming techniques. Water Resources Research 11(2):208-220.
Connor, J. L., L. Cartwright, and K. Stephenson. 2004. Collaborative water supply planning: A shared vision approach for the Rappahannock Basin in Virginia. World Water and Environmental Resources Congress 2004. In Sehlke, G., D. F. Hayes, and D. K. Stevens (eds.) Critical Transitions in Water and Environmental Resources Management. ASCE Environmental and Water Resources Conference, Salt Lake City, UT, June 27–July 1, 2004.
Croley II, T. E. 2003. Great Lakes Climate Change Hydrologic Impact Assessment. IJC Lake Ontario–St. Lawrence River Regulation Study. NOAA Technical Memorandum GLERL-126, Great Lakes Environmental Research Laboratory, Ann Arbor, MI, September 2003.
Fan, Y., and D. Fay. 2001. Variation of St. Lawrence River Hourly Water Levels About the Quarter-Monthly Mean. Hydrology and Hydraulics Technical Working Group. Lake Ontario–St. Lawrence River Study, International Joint Commission.
Fan, Y., and D. Fay. 2002. Final Report on the Development of Empirical Relationships to Estimate Water Levels of the St. Lawrence River from Montreal
OCR for page 60
Review of the Lake Ontario–St. Lawrence River Studies
to Trois Rivères. Great Lakes–St. Lawrence Regulation Office, Meteorological Service of Canada–Ontario Region, Environment Canada. March 28, 2002 (with corrections of October 2002).
Fan, Y., and D. Fay. 2003. Estimation of Lake Erie Outflows for Use in the Lake Ontario–St. Lawrence River Study. Great Lakes–St. Lawrence Regulation Office, Meteorological Service of Canada–Ontario Region, Environment Canada. December 20.
Goicochea, A., D.R. Hansen, and L. Duckstein. 1982. Multiobjective Decision Analysis with Engineering and Business Applications. New York: John Wiley & Sons.
Great Lakes Climate Change Hydrologic Impact Assessment I.J.C. Lake Ontario– St. Lawrence River Regulation Study . NOAA Technical Memorandum GLERL-126, September 2003.
Hagen, E., and J. Kiang. 2003. Interstate Commission on the Potomac River Basin Water Supply. Available on-line at http://www.iwr.usace.army.mil/iwr/svp/icprbstudy.htm. Accessed on. August 7, 2005.
IWR (Institute for Water Resources). 1995. National Study of Water Management During Drought: The Report to the US Congress IWR Report 94-NDS-12 (September).
Leger, W., F. Lupi, and W. Werick. 2005. Lake Ontario St. Lawrence River Study: Plan evaluation, rankings, and tradeoffs. Draft July 13, 2003. Copy on file at The National Academies, Water Science and Technology Board, 500 5th St., NW Washington, DC 20001.
Lofgren, B. M. 2003. Simulation of possible future effects of Greenhouse Warming on Great Lakes water supply using a regional climate model. Proceedings, 17th Conference on Hydrology, Long Beach, CA., February 8-13, 2003. Boston, MA: American Meteorological Society.
Lofgren B. M. 2004. Global warming effects on Great Lakes water: More precipitation but less water? In Proceedings of the 18th Conference on Hydrology, 84th Annual Meeting of the American Meteorological Society, Seattle, WA., January 11-15, 2004.
Lofgren, B. M., F. H. Quinn, A. H. Clites, R. A. Assel, A. J. Eberhardt, and C. L. Luukkonen. 2002. Evaluation of potential impacts on Great Lakes water resources based on climate scenarios of two GCMs. Journal of Great Lakes Research 28(4):537-554.
LOSLR Study (Lake Ontario-St. Lawrence River Study Board [LOSLR]). 2005. Options for Managing Lake Ontario and St. Lawrence River Water Levels & Flows: Final Report. July 1, 2005 Draft. Copy on file at The National Academies, Water Science and Technology Board, 500 5th St., NW Washington, DC 20001.
National Assessment Synthesis Team. 2000. Climate Change Impacts on the United States. A Report of the National Assessment Synthesis Team–US Global Change Research Program, 2000. Available on-line at http://www.usgcrp.gov/usgcrp/Library/nationalassessment/overview.htm. Accessed September 2005.
NRC (National Research Council). 2004. Adaptive Management for Water Resources Planning. Washington, DC: The National Academies Press.
OCR for page 61
Review of the Lake Ontario–St. Lawrence River Studies
NRC. 2005. Valuing Ecosystem Services: Toward Better Environmental Decision-Making. Washington, DC: The National Academies Press.
Palmer, R. N. 1998. A history of shared vision modeling in the ACT-ACF comprehensive study: A modeler’s perspective, presented at the 25th Annual Conference on Water Resources Planning and Management, Chicago, IL., June, 1998.
Palmer, R. N. 1999. Modeling Water Resources Opportunities, Challenges, and Trade-offs: The Use of Shared Vision Modeling for Negotiation and Conflict Resolution, Keynote Invited Presentation, presented at the 26th Annual National Conference, Water Resources Planning and Management Division of ASCE, Tempe, AZ., June, 1999.
Simonovic, S. P., and H. Fahmy. 1999. A new modeling approach for water resources policy analysis. Water Resources Research 35(1):295-304.
Simonovic, S. P., and V. Rajasekaram. 2004. Integrated Analyses of Canada’s Water Resources: A System Dynamics Model. Canadian Water Resources Journal 29(4):223-250.
Sousounis, P. J. 2002. Special issue on Global Warming. Journal of Great Lakes Research 28(4).
Stephenson, K. 2003. The what and why of shared vision planning for water supply. A paper prepared for the panel session on Collaborative Water Supply Planning: A Shared Vision Approach for the Rappahannock River Basin, Universities Council on Water Resources, Water Security in the 21st Century Conference. Washington, DC. July 30, 2003. Available on-line at http://www.iwr.usace.army.mil/iwr/svp/WaterSecurity03Text.pdf.
Sterman, J. D. 2000. Business Dynamics–Systems Thinking and Modeling for a Complex World. New York: McGraw-Hill.
SVM 1. LOSLR Study Board. n.d. Evaluation Process: The Shared Vision Model. Provided for review as SVM document.
SVM 2. Shared Vision Model for the Design and Evaluation of Alternative Regulation of Lake Ontario Releases into the St. Lawrence River. Includes Toronto Board Room April 24.xls.
SVM 3. Shared Vision Model for the Design and Evaluation of Alternative Regulation of Lake Ontario Releases into the St. Lawrence River. Documentation Report. February 15, 2005.
U.S. Water Resources Council. 1983. Economic and Environmental Principles and Guidelines for Water and Related Land Resources Implementation Studies. Available on-line at http://www.iwr.usace.army.mil/iwr/pdf/p&g.pdf. 48 CFR 10249 (March 10).
Werick, W. 2000. The future of shared vision planning. In Hotchkiss, R. H. and M. Glade (eds.) Water Resources 2000: Building Partnerships. Joint Conference on Water Resource Engineering and Water Resources Planning and Management 2000, Minneapolis, MN., July 30–August 2, 2000.
Werick, W. 2005. Presentations on shared vision planning and modeling in the LOSLR Study to the NRC committee. Niagara-on-the-Lake, Ontario, CA. June 11-13, 2005.
Wescoat, J. L. Jr. 1987. The ‘Practical Range of Choice’ in water resources geography. Progress in Human Geography (1987): 41-59.
OCR for page 62
Review of the Lake Ontario–St. Lawrence River Studies
White, G. F. 1964. Choice of Adjustment to Floods. Research Paper No. 93. Chicago: University of Chicago, Department of Geography.
Representative terms from entire chapter:
vision model
Items in cart [0]
