Review of the Lake Ontario-St. Lawrence River Studies (2006)

The Flooding and Erosion Prediction System (FEPS) is a sub-model in the SVM that forecasts the physical and economic impacts of water regulation options on coastal erosion and flooding. FEPS models wave and erosion physical processes of public concern: intermittent flooding, erosion of barrier beaches, sediment transport, and the economic damage associated with coastal processes along the LOSLR shoreline.

The FEPS model and the field of coastal erosion and flood damage prediction remain partly in the domain of research and partly in engineering practice. Engineering design analysis usually aims to exceed a minimum factor of safety, whereas predictive engineering analysis forecasts failure conditions that are based upon and reveal failure processes. The FEPS model(s) are in the realm of predictive engineering analysis.

This chapter describes and evaluates the LOSLR Study documents about the FEPS model and its sub-models. The first section describes the FEPS model and its accomplishments; subsequent sections evaluate FEPS in terms of its scientific and engineering foundations, its ability to integrate and display key information, and overall sufficiency to be used to inform decision making about regulation plans.

The FEPS model analyzes three coastal performance indicators (PIs) that are included in the Shared Vision Model:¹

1. Erosion PI – Lake Ontario/Upper St. Lawrence River

2. Flooding PI – Lake Ontario/Upper St. Lawrence River

3. Shore Protection PI – Lake Ontario/Upper St. Lawrence River

Three other coastal performance indicators were discussed in descriptive documents and contextual narratives, but they were not quantitatively modeled: Barrier Beaches and Dunes; Beach Access; and Sediment Budget.

The IJC presented the committee with several documents that represent the coastal processes work. The following documents were presented for review; the complete list of documents, including background documents and those distributed at committee meetings, are listed in Appendix B:

• FEPS 1 – Zuzek, P.J., and R.B. Nairn. n.d. Automated Lakewide Flooding Predictions and Economic Damages on Lake Ontario.

• FEPS 2 – Nairn, R.B., and P.J. Zuzek. n.d. Automated Lakewide Erosion Predictions and Economic Damages on Lake Ontario.

In addition, the Committee received fourteen background documents on the Coastal Performance Indicators (PIs), the Contextual Narratives associated with them, and oral presentations by FEPS and LOSLR staff at the open session meeting.

The condition of the materials submitted for review enabled an assessment of the general scientific validity and confidence in forecasting approach and results, but not modeling specifics (or model code). Thus, it is possible that some important details about FEPS have been omitted from this review.

The FEPS is a modular software system developed by Baird & Associates, and the overall modeling framework is fundamentally sound. It links bathymetric, topographic, and land use data in a GIS platform with engineering models intended to characterize the physical processes involved in coastal erosion and flooding, including wave generation through loss of shoreline, damage to structures, and shore protection costs (Figure 3-1). FEPS consists of several sub-models: a wave model (WAVAD), erosion model (COSMOS), and hazards models that estimate physical and economic damages of flooding and erosion. These models employ GIS tools and a relational database with results displayed in a user interface.

The modular structure of FEPS permits the replacement of individual submodels with alternative models as they are developed. Modularity also can support alternative sub-models when needed to perform a multiple-model analysis. The FEPS model as a whole, and its engineering sub-models, are relatively new. It was originally developed for the Lake Michigan Potential Damages Study to assess hydroclimatic and water level impacts, and it was adapted for the Lake Ontario and upper St. Lawrence River system in this study.

FEPS uses two physical sub-models, WAVAD and COSMOS to model physical erosion and flooding processes, respectively. The numerical WAVAD model generates wave fields in a lake basin with varying winds and bathymetry (Blomgren et al., 1997; Resio and Perrie, 1989). The WAVAD model was used to hindcast wave fields in Lake Ontario using hourly wind field data from 50 climate stations to generate waves on a 3 kilometer grid in the Lake Ontario basin (Scott et al., 2004).

Hourly wave heights and directions were predicted at each grid cell for 1961 to 2000 CE (FEPS 2, p. 4). The period of record was extended to 101 years using a random selection of years from the historic record applied to the 1900 to 1960 period (FEPS 7, p. 9).

COSMOS is a process-based numerical model used to calculate coastal sediment transport and morphology (Southgate and Nairn, 1993; Nairn and Southgate, 1993). The synthetic hourly wave data generated by WAVAD for perimeter grid points along the Lake Ontario coastline were used as inputs to the COSMOS submodel. Full details about COSMOS were not available for review because it is a proprietary model. FEPS modelers noted that in the LOSLR application, the COSMOS model effectively erodes but does not aggrade due in part to Lake Ontario shore conditions and in part “to the inability of modeling sandy shore evolution over long time periods (i.e., 101 years)” (FEPS 10, pp. 1, 3; Zuzek, 2005).

The outputs of the WAVAD model were combined with an historical Average Annual Recession Rate (AARR) to predict coastal recession using regression equations calibrated by the COSMOS model. The documentation (e.g. FEPS 1) indicates that separate regression coefficients were developed for a site in water level increments of 0.25 m. The regression coefficients were treated as a family of equations to be capable of predicting bluff recession for other hydrographs that were developed as the study progressed. Lake-bed profile shape is treated as being similar across a county, as is near-shore wave energy. Regression equation predictions are calibrated to within 10 percent of the COSMOS model predictions in order to reduce and simplify the computational effort to meet the IJC requirements.

Hazards modeling involved a combination of physical and economic analyses for erosion and flooding. This methodology was developed specifically for Lake Ontario.

Erosion hazards were modeled as physical failures of shore protection structures and economic investment in future shore protection. Erosion of reaches with shore protection was modeled separately from the erosion of unprotected shorelines described above. It entailed three failure modes: overtopping of structures (based on county design data, lake level and wave height modeled with WAVAD); downcutting (or undermining estimated with the COSMOS model); and degradation of different types of structures (e.g., aging of Level 1 or Level 2 revetment) (FEPS 9).

Economic impacts of shoreline erosion were quantified based on the cost of designing and constructing shore protection to mitigate the erosion hazard over the 101-year simulation period. This damage avoidance approach assumes that accelerated shoreline erosion would lead shoreline owners to construct shore protection

sooner and repair them more frequently, while plans with slower erosion rates would delay the cost of shore protection.

Flood damages were modelled as physical and economic impacts on coastal buildings. An algorithm was developed to determine flood damage from inundation and damage from waves striking a building. The damages are quantified directly from the depth of inundation and the wave energy, respectively, using 4^th order regression equations that relate the percentage of damage to flood elevation and to wave energy at the ground floor, respectively. The flooding algorithm does several things:

1. extracts relevant property parcel data from the database on a parcel by parcel basis

2. searches for lake elevation forecasts that exceed the main floor elevation of the structure on the parcel

3. calculates the percentage damaged using the Federal Emergency Management Agency’s depth-damage curves

4. searches through water levels at each individual parcel and identifies lake levels that produce standing water between the lake and the house, based on ground floor elevations

5. forecasts economic costs to the owners associated with these damages, by making assumptions about owner’s responses to the hazard.

A similar process is applied for wave damage, but wave damages are computed using the “damage equation published by the Ontario Ministry of Natural Resources and Environment Canada (1981) based on recorded damages during the 1973 high lake level period” (FEPS 2, p. viii).

Three methods were used to estimate flood damages. The first method assumes that flood damages occur to homes with depreciation after each event but no repairs to the structure. The second method adds estimated costs for repair of homes and contents after each flood event. The third method allows for expenditure on structural mitigation of flood damages, set as a proportion of building value, which the FEPS study regards as the most realistic model of behavior (FEPS 1, pp. ix, x; FEPS 8, p. 2).

This section assesses the degree to which the FEPS studies (A) reflect reasonable scientific methods, assumptions and supported findings; (B) integrate and display key information to evaluate tradeoffs for selecting among candidate regulation plans; and (C) are sufficient and appropriate for evaluating the regulation plan options and impacts of changes in water levels and flows.

The first charge in the statement of task asks the degree to which the scientific foundations of the FEPS model(s) are sound. Three criteria were used to address the charge: empirical and theoretical foundations, quality assurance and quality control, and treatment of error and uncertainty.

The FEPS models combine analyses of winds, waves, coastal erosion, shore protection, physical flood damages, and flood protection costs. Detailed databases were compiled at the county, township, reach, and parcel scales for the models (FEPS 7, pp. 4-5). While some of the data-sets have a high resolution and long stable records, others consist of shorter-term records and in some cases proxy data (e.g., shore protection costs are used as a measure of erosion damages). This range of data quality poses challenges for FEPS to achieve its objective of making 101-year forecasts of coastal erosion and flooding damages under alternative water level and flow regulation plans.

The FEPS model is based on physical models and empirical equations. Some conditions of the models or the underlying science weaken the empirical foundation of FEPS. Since the FEPS model involves physical models and empirical equations, explicit treatment of the empirical basis of the models is expected. In particular, weaknesses in the data and the empirical bases of the models should be explicitly addressed. For example, FEPS documentation provided few details about the data quality or explanatory power of COSMOS model and the regression models. Data quality issues are mentioned for some topics, such as shore protection structures (FEPS 9, pp. 17ff), and previous research by Baird & Associates and others is mentioned but not included or extensively cited in review materials.

Some of the science underlying the FEPS work is fragmented, semi-empirical, or not independently repeatable. In cases with inadequate empirical data, expert judgment was used to replace or supplement the problematic data. The flood damage models lack data on flooding impacts and the associated economic damages for lake levels above 76.0 m (249.3 ft) on Lake Ontario because the lake has never reached these elevations. In this instance, the modelers used their “…professional judgment to evaluate the predicted flooding damages” and these predictions were deemed “reasonable” (FEPS 15, page 5). It is not clear how the judgment of reasonableness was determined, but some measure is needed to convey how reliability of this judgment.

The 2-Dimensional WAVAD model for Lake Ontario uses historic data (1956–1987) to generate hourly wind fields, and then hourly wave predictions on a 3 km-by-3 km grid for the period from 1961 to 2000. This 2-D model is important to the LOSLR Study because it is used in conjunction with the coarse quarter-month time step of the SVM with finer time steps. However, the documents re-

viewed do not indicate exactly how these data are used in the 101-year simulation period to calculate flood and erosion damages.

The FEPS work uses regression equations, in part, to predict shoreline erosion. There are good reasons to use regression for this type of work: regression is straightforward and inexpensive. Still, the applicability of regression analysis for FEPS has limitations that make it problematic for use in the FEPS analyses. One major limitation is the variable quality of FEPS data that would be used as a predictor for complex coastal processes over the next 101 years. The FEPS documentation provided for review did not allow for the linear logic, quality of fit, or error of the regression models to be ascertained. The high values of R² can only be interpreted with respect to historical data used in the regression analysis, and not the 101-year simulation period. The limitations of the regression equation-based modeling are not adequately discussed (see Box 3-1) in the FEPS materials, and implications of using regression analysis for FEPS applications need further explanation and defense.

The decision to use regression analyses in the FEPS modeling apparatus needs detailed explanation and defense, including a discussion of the limitations and implications of using regressions for FEPS work.

The FEPS parcel database is impressive in its detail. It was reported to be complete for the USA part and approximately 75 percent complete for the Canadian part. The remaining 25 percent of parcels is a point of concern, as it has unknown implications on the final model output (this point was also raised in FEPS 4). Some of these issues are mentioned in the Contextual Narrative (FEPS 12) but not incorporated in the FEPS or SVM modeling. The database needs to be complete to ensure that the maximum range of scenarios is captured in the FEPS modeling effort.

The FEPS economic database is important in estimating flood impacts. In fact, an economic Performance Indicator was used in the SVM Board Room to evaluate operational alternatives. Documents were not provided on the economic database for this review, but according to information provided via oral presentations, the economic data are static. Considering the variation of real estate values and dynamics of the regional economy over the long simulation period, this assumption is unrealistic. Economic trends are discussed in the contextual narrative and a supplemental report, but they are not included in the modeling (FEPS12, pp. 3-5; and Christian J. Stewart Consulting, 2004).

The empirical damage equations are also based on very limited data sets. For example, the wave damage to buildings is based on as little as one data set from the 1973 high lake level period. FEPS has compiled extensive datasets that have long-term value, but FEPS documentation does not fully discuss empirical weaknesses or data quality issues that affect application and interpretation of these datasets in the Shared Vision Model. Failure to specify the bounds of the empiricism of the model and its calibration, as well as the lack of a clear statement of what was done for conditions outside those empirical bounds, are errors that should be corrected.

Quality assurance was expected to have been rigorous for the FEPS erosion and flooding results. One issue of QA/QC in the FEPS work is whether regression analysis calibrated against a deterministic 2-dimensional physically-based model COSMOS is an appropriate approach, and if so, whether the calibration and regression analysis was carried out correctly. Quality assurance procedures are also needed to translate coastal recession, property and infrastructure damage, and economic disruption into economic costs. Since the FEPS modeling endeavor is an integral part of the SVM, quality assurance of the integration of FEPS into the SVM is required.

Transparency of the FEPS models is needed to convey the coastal impacts of regulation plan options to the scientific, public, and decision-making sectors. The COSMOS model is described as “extensively tested and validated” with numerous published articles in “peer reviewed journals attesting to the accuracy and robustness of the COSMOS erosion predictions for sandy coastlines” (FEPS 10, year, p. 4). Testing, validation and peer review are appropriate means of QA/QC. In the case of COSMOS, the documentation should have described the extensive testing and listed the peer-reviewed journal articles that attest to COSMOS accuracy and robustness. Other measures, too, can be taken to ensure transparency, such as engaging outside experts to use informed judgment to confirm scientific integrity in determining assumptions, selecting models and data, and adjusting analyses. These measures can improve transparency through formal methods for eliciting expert judgment (e.g., Morgan and Henrion, 1990).

One aspect of quality control is for scientific peers to carry out the same calculations and arrive at the same answer. This is standard practice in the scientific and engineering fields. For this type of validation to occur, the calculation models must be specifically described and the data made available so that the calculations are repeatable. It is common practice for a study of this magnitude and complexity to depend on rigorous independent scientific peer review of each component of the FEPS modeling. The independent review process may also suggest alternatives to the existing methods that could improve efficiency, costs, and time.

The FEPS documents mention that some aspects of quality assurance were undertaken (e.g., FEPS 14, 5), and they include minutes from independent technical reviews in 2002 (FEPS 18) and a discussion paper in 2005 (FEPS4), but these documents do not provide details on specific quality assurance procedures. Overall, the quality assurance procedures for the FEPS work should be more fully specified and discussed in the review materials.

The analysis of uncertainties associated with data, methods, and models for coastal processes and hazards is important. Broadly speaking, uncertainty in the FEPS models can be characterized as follows (Hartford and Baecher, 2004, pp. 127-134; cf. NRC, 2000, pp. 40-45):

• Natural variability is associated with the “inherent” randomness of natural processes, manifesting itself as variability over time for phenomena that take place at a single location (temporal variability), or as variability over space for phenomena that take place at different locations but at a single time (spatial variability)

• Model uncertainty reflects the inability of a model or design technique to precisely represent a system's true physical behavior, or the inability to identify the best model, or a model that may be changing in time in poorly known ways (e.g., flood-frequency curve changes because of changing watershed)

• Parameter uncertainties result from an inability to accurately assess parameter values from test or calibration data, due to limited numbers of observations, and the statistical imprecision attendant thereto

• Data uncertainties, including (i) measurement errors, (ii) inconsistency and in homogeneity of data, and (iii) data handling and transcription errors, and (iv) inadequate representativeness of data sample due to time and space limitations

• Decision model uncertainty is also important in contexts such as the FEPS and Shared Vision Model, where decision-making objectives change over time (NRC, 2000, p. 44).

The methods used to deal with data uncertainty including gaps in the data should be available and their use justified. In particular, assumptions that gaps in the data are insignificant or of limited significance overall are unacceptable, and specific measures to address those gaps in the data would normally be required. The degree of resolution (error bounds) of all data should be specified and propagated through the modelling endeavour. Against this background, procedures are required to explain how the processes of interpolation and extrapolation have been embodied in the data analysis and modeling process.

Sensitivity studies involve determining the change in response of a model to changes in individual model parameter distributions or changes in the model. Sensitivity analysis is useful but not sufficient for characterizing model and parameter uncertainties. Thus, the output is ideally in the form of a probability distribution which specifies the likelihood of each possible result across the full range of possible results (Hartford and Baecher, 2004, p. 20). Probability distributions of parameters to which the analysis is sensitive are identified and their validity and accuracy verified to the extent that is realistic. Changes are made to probability distributions where justified.

The FEPS documents were expected to present a comprehensive theoretical and analytical treatment of the uncertainties in the models, parameters, and data. FEPS performance indicator summaries mention uncertainty (e.g., FEPS 4, p. 5), and the 2002 technical review recommends treatment of data and modeling uncertainties (FEPS 18, pp. 8, 12), but important details are not reported. For example, failure of shore protection structures is amenable to analysis using structural reliability methods (USACE, 2003, Voortman et al., 2002). The results of such an

analysis for shore protection structures would yield a theoretical fragility curve (Figure 3-2, as adapted from NRC, 2000, p. 70). Model uncertainty should have been dealt with explicitly, as should the propagation of uncertain data through uncertain models.

The uncertainty in the logarithmic regression equation is the most problematic aspect of the FEPS study because of its capacity to introduce and propagate error. All downstream modeling considerations are dependent, in part, on the logarithmic regression output. This dependence places critical importance on those output values being accurate; otherwise error will be propagated through the downstream models. Another problem stems from the use of the synthetic data set of monthly normal wave energy calculated by WAVAD and monthly recession calculated by COSMOS. The fit for the recession regression equation uses these data. Using synthetic data introduces unnecessary error (E1) on recession outputs, which subsequently are used as inputs for the regression analysis. Regression analysis produces its own error (E2) resulting in the aggregate error of E1+E2. The first type of error (E1) could presumably have been avoided by skipping the COSMOS process and using the observed recession data instead.

The uncertainty in the data used to derive the logarithmic relationship exhibits a great deal of scatter (Figure 3.6 of FEPS 7, p. 11, reproduced below, Figure 3-3). The horizontal axis extends to 250,000 J/m² and covers the range of the average yearly normal wave energy. However, the WAVAD model predicts January wave energy in excess of 600,000 J/m². The rate of change of the predicted recession decreases rapidly as wave energy increases. This implies that recession during severe winter storms will be similar to that for significantly lesser storms. From the materials presented for review, it was not possible to determine whether this feature of the model is reasonable. Thus calibrating the recession regression equations

FIGURE 3-2 Form of result of structural reliability analysis of shore protection structure.

to within less than 10% of the COSMOS model equations has little effect because the scatter around the COSMOS regression equations is greater than 10 percent. The uncertainty in parameters “a” and “b” must be significant, such that it may not be possible to make a reasonable distinction between the recession rates associated with different plans. As noted above, the problem of calibrating the model for conditions beyond the bounds of the empirical data introduces additional uncertainty that renders the forecasts questionable. The lack of specific treatment of uncertainties in the coastal recession regression model, especially parameters “a” and “b,” make objective discrimination between the coastal impacts of regulation plan options impossible.

The second charge in the statement of task focuses on the degree to which the FEPS models integrate and display coastal erosion and flooding information needed for a comprehensive evaluation and understanding of the tradeoffs for selecting among candidate regulation plans. Four criteria were used to evaluate this charge: linkages and feedbacks among related studies and models; treatment of spatial and temporal resolution and scale; documentation; and scientific communication of flooding and erosion outputs.

The FEPS model is one of several sub-models of the SVM. Linkages between FEPS and the other LOSLR models, particularly the integrated ecological response model and the SVM, are considered internal linkages. Exogenous linkages are those between FEPS and external variables, such as economic and demographic values. These two types of model and study linkages are discussed in this section.

The structure of the SVM and its sub-models presents an opportunity for holistic understanding of the Lake Ontario-St. Lawrence River system, as independent processes are modeled independently and then brought together for a single purpose in the SVM. Despite connections between bathymetry and wetlands processes, the FEPS model output was not connected to the integrated ecological response model (IERM) in the SVM. This is a missed opportunity to advance the science and better understand intricate relationships in the LOSLR system, as the FEPS model output would have informed the wetlands scientists with needed information about wetland bathymetry and thus the areal extent of wetland vegetation and species at risk habitats. Furthermore, linkages between FEPS and the

IERM could have simulated flooding and erosion effects on environmental variables. Similarly, FEPS documentation does not indicate how erosion analysis in Lake Ontario and the upper St. Lawrence River is related to or comparable with downstream analysis in the Lower St. Lawrence.

As discussed in the FEPS documentation, the model does not provide for sandy shore processes (see FEPS 7, 10 and 13). This omission limits the ability to assay flooding and erosion of barrier beach wetlands in the Shared Vision Model and Integrated Ecological Response Model (see Chapter 4). It also misses an opportunity to represent inherent feedbacks among erosion, vegetation, resistance to erosion, and water regulation plans. A regulation plan can affect erosion in ways that result in different types of vegetation and that in turn cause a change in erosion resistance. The FEPS analyses do not incorporate these dynamic feedback relationships. The lack of beach accretion in the FEPS model may have implications for other areas, uses, and values of Lake Ontario shore areas.

To their credit, the FEPS performance indicator summary and contextual narrative underscore some of these risks and mention separate analyses undertaken to address them (FEPS 10 and 12).

FEPS treats exogenous processes including coastal land use and economic development as static, which results in the same weaknesses as in the Shared Vision Model (noted in Chapter 2). Trends in coastal land use, economics, and hazards mitigation are discussed in a contextual narrative but not incorporated in the model (FEPS 12; Christian J. Stewart Consulting, 2004). Since coastal erosion and flood losses are just two dimensions of a dynamic and interdependent environmental system, the FEPS model should at least identify and ultimately incorporate causal relationships with the other dimensions of the system. The effects of coastal erosion and flooding should be incorporated as inputs to the environmental and economic models as well as the Shared Vision Model.

Also, the FEPS modeling currently does not include an analysis of the potential effects of future water regulation deviations, analogous to the deviations that have occurred under Plan 1958D, on flooding and erosion damages under alternative hydroclimatic scenarios and water regulation plans. The FEPS model includes explicit forward linkages but limited feed-back linkages among its sub-models and no linkages with associated ecological or other socioeconomic models. Along with alternations to the shoreline by riparian owners in response to coastal erosion and flooding that are inconsistent with the modeling assumptions, deviations from the selected plan, represent considerable uncertainties that should be incorporated in the analysis.

The range of temporal and spatial scales in the FEPS models is enormous. The temporal and spatial scales of inferences derived from the FEPS modeling endeavor are not currently explained in the FEPS documentation. Linking hourly wave data with quarter-monthly lake levels introduces issues of spatial and temporal resolution at the outset of the analysis. Questions pertaining to the resolution of the flooding and erosion forecasts should be addressed comprehensively and objectively in the documentation to avoid misinterpretation.

The FEPS modeling endeavor requires specific scales to be adopted for each of the sub-modeling efforts, making it necessary to scale the results of the analysis in both space and time. In Figure 3-4, “Time” reflects the temporal extent over which the erosion/damage parameter and planning horizon occur. “Spatial dimension” reflects the geographical extent and lake level fluctuation of those erosion/damage parameters and planning horizons. The FEPS process would be expected to illustrate where in this space-time context data are collected, processes modeled, and inferences made. The spatial and temporal resolution of elements in the FEPS model and sub models are illustrated conceptually in Figure 3.4. Describing the FEPS work in such a context would provide a basis for framing consultations, discussions, and debate around issues of modeling resolution and the sphere of applicability of FEPS results.

FIGURE 3-4 Spatial and temporal scales.

The SVM generates a 101-year simulation horizon in a series of single water elevations with a quarter-monthly time step (see Chapter 2) that is used in the FEPS algorithm. Both the single value (which is a spatially averaged value) and the quarter-month time step are too coarse for purposes of the SVM (see Chapter 2) and some environmental considerations (see Chapter 4). Similarly, the FEPS models are highly sensitive to actual lake elevation at the time of a storm event. In fact, the important parameter for the erosion model is the lake level at the time of the storm, not an averaged value over a 1-week period. Connections between the SVM and FEPS models appear to give inadequate treatment to the spatial and temporal variation of lake water levels within the time step important for proper evaluation of flood damages. The compatibility of data on different scales and their use to simulate future conditions raises a prominent concern. With reference to Figure 3-4, for example, the spatial and temporal scales of the economic damages are apparently restricted to:

• 1 km reaches of coastline ± an order of magnitude (FEPS 8, p. 20; FEPS 9, p. 25)

• 1 year (for damages scaled to the Average Annual Recession Rate; and for erosion damages defined by the year of investment in shore protection, FEPS 7, p 21).

The model documentation appropriately addresses issues of resolution. In its discussion, it identifies some limitations to the FEPS model. It specifically notes the lack of parcel-level detail that precludes more than an order of magnitude forecast at the reach scale (FEPS 8, p. 20), which means that FEPS cannot capture dramatic changes in local shoreline conditions that can occur during individual storms. The FEPS model, however, would be most useful under such conditions when it could convey predicted responses to interested parties, such as riparian and shoreline owners, the IJC, or others in the political process.

Relationships among spatial dimension, analytical effort, and the accuracy and precision of results are represented schematically in Figure 3-5. From the information available, it appears that the FEPS modeling endeavor is at the meso-scale spatially (FEPS 7, p. 3). The mathematical scaling in space and time of the “Families of Equations for Regional Zones” (FEqi), the “Local Wave Energy (Wei),” and the Annual Average Recession Rate (AAAR) measured over each 1 km reach is not specified. The FEPS modeling endeavor is considered to be of low- to medium-analytical rigor, due in part to the regression equations and in part because there are at least two higher levels of analysis available. The highest level of rigor is structural reliability modeling on a parcel lot-by-parcel lot basis using detailed wave and hydraulic models. The next highest level of rigor is multiple COSMOS-type modeling on a 1-km reach basis over the entire lake and river.

In addition, spatial or temporal intra-annual autocorrelation may occur. The annual recession value is calculated as the sum of individual recession events that occur during the year. For example, the performances of two sections of a shoreline reach may exhibit a long wave length of correlation in space, and thus exhibit

FIGURE 3-5 Spatial scale, analytical rigor, and accuracy and precision of results.

similar factors of safety against erosion. The FEPS review documents do not discuss these issues of spatial and temporal autocorrelation, or the effects of different storm sequences on shoreline evolution.

The treatment of spatial and temporal scaling requires considerable additional effort to enable a comprehensive evaluation and understanding of flooding and erosion hazards associated with regulation plan options.

Documentation needs to sufficiently inform a broad spectrum of users, especially for large, complex studies. It is an additional challenge to make this documentation navigable and interpretable to the full spectrum of users. A “map” of the documentation would provide a useful means of presenting studies of this nature. Documentation should completely describe both “what was done” and the justification for “why it was done the way it was done.” The documentation should permit the scientific community to review and if necessary reproduce all calculations. This means that all data must be provided and all models completely speci-

Documentation insufficiencies are noted throughout this chapter. Notable documentation concerns relate to:

• Description of the limitations of the empirical foundations for FEPS models

• Treatment of uncertainty and error

• Quality control and quality assurance of model results

• Inferences based on professional judgments

• Robust justification for the way that coastal erosion and flood damage calculates the various inputs to the SVM.

There is no final report for FEPS, and several of the individual items of documentation are clearly interim documents describing modeling intent as opposed to finalized details of the modeling effort. For example, FEPS 7 states “Predictive equations from the SVM are being developed based on the total monthly recession predicted by COSMOS” (p. 10). That the documentation would be highly conditional and written in the language of uncertainty would be an expectation for the FEPS process. The FEPS documentation should ultimately also be suitable for publication in the broader body of established scientific domains of coastal erosion, hydrodynamics, soil mechanics, structural reliability, economics, long-term policy and planning.

Although superior to documentation of the SVM, FEPS documentation presented and reviewed here is deemed inadequate for external scientific review, and for the judgments and conclusions to be defensible in public decision-making.

Results from the FEPS study can be interpreted in several ways over different spatial and temporal scales and warrant particular attention to communicating uncertainties at various scales. Protocols should be established to ensure that those relying on the results of the FEPS studies understand the strengths and limitations of the modeling endeavor, and the implications of interpreting the outputs in different ways.

Forecasting necessarily demands a tradeoff between the “security” of the forecast and the “informativeness” of the forecast. “Security” of the forecast pertains to the extent to which the forecast can be accepted in advance of the actual outcome, whereas “informativeness” pertains to the extent to which the forecast is genuinely insightful, providing definitive information. “Security” of forecasts increases as the forecasts become more general whereas “informativeness” increases as the forecasts become more specific. “Security” often comes at the expense of “informativeness” (Rescher, 1998). The FEPS study documentation does not discuss the nature of such forecasting tradeoffs, or the final balance that it took to those tradeoffs.

The process of communicating the accuracy, precision, reproducibility, and reliability of coastal erosion and flood damage prediction is extremely difficult, and considerable effort is needed to ensure that the results are not readily misinterpreted. The problem of properly representing risk and uncertainty in coastal erosion and flood damage prediction in a public decision process should be addressed. Forecasting can be difficult over the short to medium term, with the result that decision-makers and users of model results often form their own degree of belief in the credibility of the results. The communication of the tradeoff between the “security” and the “informativeness” of the forecast should be explicit and available to users to make an informed judgment as to model usefulness.

The results of the FEPS modeling can also be expected to be interpreted and used by owners of individual parcels of property, insurance companies, municipalities, courts at all levels, governments of bordering states and provinces, and the two national governments, as well as by international NGO’s and other parties with no direct interest in the regulation of Lake Ontario and the St. Lawrence River. Accordingly, the documentation and scientific communication should be prepared to meet the challenges posed by such a broad spectrum of users in such a way that the integrity of the studies is preserved. Effective scientific communication of FEPS model output requires a more thorough discussion of uncertainties associated with the flooding and erosion forecasts to enable a broad public understanding of tradeoffs among regulation alternatives.

The third charge of the statement of task is aimed at determining whether the IJC documents presented for review are appropriate and sufficient for use in selecting a water regulation plan option and determining impacts of changes in water levels. Three evaluation criteria are used to address this charge: study breadth, balance between scientific and practical inquiry, and the identification of future study needs.

FEPS encompasses a wide range of physical analysis and associated risk analysis for people and property. For example, FEPS analyzes poorly understood processes over a range of physical conditions (albeit not the full range) that could exist over the period of analysis and geographic scope of the Lake Ontario and upper St Lawrence River coastal zone. Under such circumstances, a heuristic approach would provide an effective framework for coastal erosion and flood damage prediction.

In other ways, the scope of the FEPS study remains narrow. Erosion hazards for environmental systems, public infrastructure, and wider economic activities are omitted from the model and only briefly discussed in conceptual narratives. The

FEPS focus on generating data that will be used in a standard deterministic economic cost comparisons may limit broader scientific understanding of the bases for water level regulation decisions.

Since the results of the FEPS analysis are utilized in the SVM and in public consultation, the FEPS documents should describe the design of interfaces between FEPS, other modeling endeavors, and the public consultation process to ensure that the modeling efforts are in harmony. Typically, these interfaces would include dynamic feedback from other models to permit comprehensive modeling of the system. The FEPS study takes a broad approach to erosion and flood hazards of water level regulation of Lake Ontario and Upper St. Lawrence River, but greater breadth is needed for a comprehensive, scientific analysis of the erosion and flood hazards associated with water-level regulation in the LOSLR.

The relationships between scientific inference and informed professional judgment should be elucidated. As noted above, protocols for qualifying experts and data should be defined and adherence to these protocols demonstrated. In the case of FEPS, the scope of the study and documentation should also indicate how precise and accurate the predictions are expected to be. It should demonstrate that each of the sub-models (wave, coastal erosion, failure of coastal protection structures, flooding and damage models) are representative of actual conditions. Different models will combine these considerations in different ways leading to different results. When modelling complex physical processes that are poorly understood such as coastal erosion and damages, the simpler the model the greater the difficulty in demonstrating the representativeness of complex processes (Pilkey and Cooper, 2004). Thus, modellers are challenged to demonstrate that (1) simplified erosion hazards models are sufficiently representative of the actual process and (2) model results lead to the same conclusion as detailed erosion hazards modelling.

Relationships between scientific research and practical scientific inquiry into matters of erosion, flooding and physical damage associated with flooding have become better defined in recent years with studies of this type and size becoming more common. The“EUrosion” project, (EUrosion, 2004a) is an example of a study that shows the understanding of coastal processes is still largely fragmented and semi-empirical (EUrosion, 2004b). The net result of such fragmentation, which is common to other developing sciences, is that different theories based on different concepts, assumptions and approaches result in different models with different levels of compatibility.

Balancing scientific and practical inquiry in FEPS should include specific discussion of:

1. The methods used to compensate for gaps in the physical data

2. The use of expert opinion throughout the FEPS modeling endeavor in relation to scientific norms

3. How the current weaknesses in the FEPS model might be improved through future research, testing, and regulatory adjustments, all of which would help refine the FEPS modeling process dynamically into the future.

The balance between scientific and dimensions of pragmatic inquiry is compromised by the impression of certainty that is conveyed throughout the documents. For example, the Erosion Performance Indicator document (FEPS 7) makes no reference to the uncertain state of scientific knowledge in coastal erosion and flood damage prediction, and the PI summary document mentions it only briefly (FEPS 14).

The FEPS model aims to provide coastal erosion and flood damage data for a deterministic economic analysis, and it appears that these issues of practicality dominated the design of the model. However, FEPS was not specifically developed for regulatory policy analysis, and it was first conceived for the Lake Michigan flood damages analysis. As a result, the model is limited in the damages it examines. The FEPS model considers flood and erosion damages to structures and shorelines but not to infrastructure, agriculture, or environment within an integrated approach, as has been the case in previous studies carried out for the IJC (2000) and others (e.g. Simonovic and Carson (2003)). Although pragmatic limits are necessary in studies of limited duration and funding, they should be explicitly mentioned along with their potential implications for decision-making.

FEPS attempts to balance scientific and professional methods of forecasting flooding and erosion hazards, but it does not describe the balance to be achieved or the extent to which that balance objective was achieved; and it does not sufficiently convey the challenges associated with this aim or the limitations of its approach

The FEPS databases formed during the LOSLR Study have value and provide a useful foundation for the future work. Integration of the property database, wave model, and damage calculation model while incomplete is an important step. Predictive capability would be expected to improve over time, and there is opportunity to refine the FEPS modeling process dynamically over time.

A long-term monitoring program would help assess the reliability of the FEPS model predictions using water regulation outcomes. Such a long-term monitoring program is required, and would address three specific needs:

• Monitoring actual erosion and flood damage and improving the models using reliable data.

  • Given that the FEPS model and its sub-models have not been demonstrated as being capable of precisely and accurately predicting coastal erosion and flood damages over the 101-year simulated planning horizon, a feed-

back process that updates the forecasts at regular intervals of 5 to 10 years would be expected

• A dynamic process to revise FEPS models

  • In the case of coastal erosion and flood damage prediction, future studies should be identified to analyze the water regulation plan that is adopted, including any deviations, as well as changes in coastal land use, land value, shoreline protection technologies, and other flood damage mitigation measures

• Promotion of the advancement of scientific research on coastal erosion, flooding, and damage reduction in the Lake Ontario and St.Lawrence River basin

  • It is clear from this review and complementary studies elsewhere that forecasting coastal erosion and flood damage is in its infancy, that the scientific knowledge is fragmented and largely incomplete, and that additional research in the lower Great Lakes is necessary.

The overall modeling framework for the Flooding and Erosion Predication System is impressive in many respects. It includes detailed parcel and reach analysis, and undertakes a lakewide analysis not previously attempted.

From a scientific perspective, and due in significant part to the lack of documentation, the reliance on regression equations that are poorly conditioned and inadequately justified, and the lack of formal treatment of uncertainty, the FEPS study as presented is insufficient for the purpose of enabling a comprehensive evaluation of tradeoffs among regulation plans. From the same perspective, the scientific robustness of the FEPS models remains to be demonstrated.

It is clear from the documentation that the regression analysis approach to modeling coastal erosion and forecasting flood damages is considered by the FEPS authors to be less than ideal. But this less than ideal approach was adopted because, “it is not practical for a parcel level application due to the geographic scope of the study, large data requirements, the manual nature of the desktop approach and the time required to generate answers. The study managers required an automated approach that produced robust and defensible results for the erosion impact assessment,” (FEPS 1, Nairn and Zuzek, p. 7). This review concurs that the regression analysis approach in the FEPS application is less than ideal, but considers that computational techniques are available that can be applied more extensively at a parcel lot level. The mathematical properties, limitations and implications of the regression equations should be clearly explained to the decision-makers. In particular, the analysts should demonstrate that the regression equations have a physical basis and that the potential problems of numerical instability have been addressed.

In light of the multiple scales and models used, a comprehensive analysis of uncertainties is needed. Further, multiple model analysis of the coastal erosion process would be a significant improvement over single model analysis.

One issue that remains, and which cannot be resolved given the design of the FEPS study, is the extent to which the results of the analysis as carried out would differ from those that would be obtained through application of best available science over the entire system. A second issue that remains and which may never be fully resolved is whether these differences matter in practical terms.

Against this background, the degrees of robustness and defensibility of the FEPS studies reviewed here have not been demonstrated or documented in a comprehensive way. The lack of explicit treatment of risk and uncertainty limits confidence in FEPS results for use in informing decision making, although, unfortunately, FEPS documentation and presentation may convey an impression of accuracy and precision in the data, models and interactions within and external to the system. In addition, the failure to identify those risks that society cannot afford to take, and those risks that society cannot afford to take too often, render the study incomplete.

Blomgren, S., H. Hanson, and M. Larson. 1997. Modeling of wave generation and transformation. Scientific report, Appendix A: Documentation. Subproject 4: Nearshore and coastline processes, 1 year progress report, Baltic Sea System Study

Christian J. Stewart Consulting. 2004. A Summary of Existing Land Use, Land Use Trends and Land Use Management Policies along the Lake Ontario-St. Lawrence River Shoreline: Implications for Future Water Level Management. Prepared for the Coastal Technical Working Group, International Joint Commission.

EUrosion Project. 2004a. Living with coastal erosion in Europe: Sediment and Space for Sustainability PART I—Major findings and Policy Recommendations of the EUROSION project Service contract B4-3301/2001/329175/ MAR/B3, Coastal erosion—Evaluation of the need for action. Directorate General Environment, European Commission.

EUrosion Project. 2004b. Living with coastal erosion in Europe: Sediment and Space for Sustainability. PART IV—A guide to coastal erosion management practices in Europe: Lessons Learned Service contract B4-3301/2001/ 329175/MAR/B3. Coastal Erosion—Evaluation of the need for action. Directorate General Environment, European Commission.

FEPS 1. Zuzek, P. J., and R. B. Nairn. n.d. Automated Lake wide Flooding Predictions and Economic Damages on Lake Ontario.

FEPS 2. Nairn, R. B., and P. J. Zuzek. n.d. Automated Lake wide Erosion Predictions and Economic Damages on Lake Ontario.

FEPS 3. FEPS Model.

FEPS 4. Sciremammano, F., and S. Renzetti. 2005. FEPS/Coastal Discussion Paper.

FEPS 5. Baird & Associates. Beach Access Performance Indicator Summary.

FEPS 6. Memorandum from Zuzek, P.J., and R. Roblin (Baird & Associates) to IJC Study Participants re: Integration of Beach User Economics into the Shared Vision Model. April 7, 2004.

FEPS 7. Baird & Associates. March 2004. Erosion Performance Indicator: Methodology and Shared Vision Model Application. Prepared for I.J.C. Plan Formulation and Evaluation Group.

FEPS 8. Baird & Associates. February 2005. Flooding Performance Indicator: Methodology and Shared Vision Model Application. Prepared for IJC Plan Formulation and Evaluation Group. Oakville, Ontario: Baird & Associates.

FEPS 9. Baird & Associates. February 2004. Shore Protection Performance Indicator: Methodology and Shared Vision Model Application. Oakville, Ontario: Baird & Associates.

FEPS 10. Barrier Beaches and Dunes PI Summary.

FEPS 11. Contextual Narrative for “Sediment Budget PI” Lake Ontario and the Upper St. Lawrence River.

FEPS 12. Contextual Narrative for Erosion, Flooding, and Existing Shore Protection PIs Lake Ontario and the Upper St. Lawrence River.

FEPS 13. Contextual Narrative for “Beach Access” and “Barrier Beaches & Dunes PIs” Lake Ontario and the Upper St. Lawrence River.

FEPS 14. Baird & Associates. Erosion Performance Indicator Summary.

FEPS 15. Baird & Associates. Flooding Performance Indicator Summary.

FEPS 16. Sediment Budget PI Summary.

FEPS 17. Baird & Associates. Existing Shoreline Protection Performance Indicator Summary.

FEPS 18. Minutes/Observations from FEPS Independent Technical Review 12/13/2002 and 7/26/2002 notes.

Hartford, D. N. D., and G. B. Baecher. 2004. Risk and Uncertainty in Dam Safety. London: Thomas Telford.

IJC (International Joint Commission) International Red River Basin Task Force. 2000. The Next Flood: Getting Prepared. Ottawa and Washington: Final Report to the International Joint Commission

Morgan, M. G., and M. Henrion. 1990. Uncertainty: A Guide to Dealing with Risk and Uncertainty in Quantitative Risk and Policy Analysis. New York: Cambridge University Press.

Nairn, R.B., and H. N. Southgate. 1993. Deterministic profile modeling of nearshore processes: Part 2, Sediment transport and beach profile development. Coastal Engineering 19:57-96.

Nairn, R. B., P. Zuzek, A. Morang, and L. E. Parson. 1997. Effectiveness of Beach Nourishment on Cohesive Shores, St. Joseph, Lake Michigan. USACE Technical Report CHL-97-15.

NIST (National Institute of Standards and Technology). 2005. Engineering Statistics Handbook. Available on-line at http://www.itl.nist.gov/div898/handbook/. Accessed October 2005.

NRC (National Research Council). 2000. Risk Analysis and Uncertainty in Flood Damage Reduction Studies. Washington, DC: National Academy Press.

NRC. 2001. Under the Weather: Climate, Ecosystems, and Infectious Disease. Washington, DC: National Academy Press.

Parson, L. E., A. Morang, and R.B. Nairn. 1996. Geologic Effects on Behavior of Beach Fill and Shoreline Stability for Southeast Lake Michigan. USACE Technical Report CERC-96-10. Vicksburg, MS: U.S. Army Corps of Engineers Waterways Experiment Station.

Pilkey, O.H., and J. A.G. Cooper. 2004. Society and sea level rise. Science 303:1781.

Rescher, N. 1998. Predicting the Future: An Introduction to the Theory of Forecasting. New York: State University of New York Press.

Resio, D., and W. Perrie. 1989. Implications of an F-4 equilibrium range for wind-generated waves. Journal of Phys. Oc. 19(2):193.

Scott, D, D. Schwab, P. Zuzek, and C. Padala. 2004. Hindcasting wave conditions on the North American Great Lakes. Available on-line at http://www.ocean-weather.com/8thwave/Papers/D4.pdf. Accessed October 2005.

Simonovic, S.P., and R.W. Carson. 2003. Flooding in the Red River Basin–Lessons from post flood activities. Natural Hazards 28:345-365.

Southgate, H. N., and R. B. Narin. 1993. Deterministic profile modeling of nearshore processes: Part 1, waves and currents. Coastal Engineering 19:27-56.

USACE (U.S. Army Corps of Engineers). 2003. Coastal Engineering Manual– Chapter 6, Reliability based design of coastal structures EM1110-2-1100 (Part VI).

Voortman, H. G., P. van Gelder, and J. K. Vrijling. 2002. Risk-based design of large-scale flood defence systems. 28th International Conference on Coastal Engineering. Pp. 2373-2385 In Solving Coastal Conundrums. July 7, 2002-July 12, 2002. Cardiff.

Zuzek, P. J. 2005. Presentation to the NRC Committee to Review the Lake Ontario-St. Lawrence River Studies. June 14 and June 15, 2005. Niagara-on-the-Lake, Ontario, Canada.