sensitive to Pmax (the maximum photosynthetic rate for a species), as are most models of this type, such that a thorough sensitivity analysis is advisable. Based on the available documentation, it appears that relatively little data or empirical parameterization has been taken from the SAV literature; a majority of the parameters are calibrated. A rigorous review of the final SAV model will seek examples where these selections are well founded in the SAV literature.
13. The SAV model grapples with the difficult challenge of handling maximum biomass per cell, as well as conversion of cells from one SAV species to another.
Simulating the processes of both colonization and conversion from one species to another is perhaps the most exciting and challenging aspect of the SAV model. The user-determined maximum aboveground biomass is set to limit biomass in a given cell. However, the model already includes self-shading and permits for negative growth as the main mechanisms that should, presumably, impose a more mechanistically derived limit on the maximum amount of biomass in a cell. Forcing a maximum biomass value can artificially help calibration because it simply cuts off biomass values that are too high without a biological reason. If the development team instead considers this limit to be related to colonization of adjacent cells, then this could be explicitly linked to the transition probabilities. Another option would be to have a variable translocation term, where more growth is allocated below ground as the above ground biomass in a cell becomes larger. For the species that float across the surface, thinking through whether it is necessary to have a rule that limits height to the water depth is advisable. Light is simulated with some detail throughout the water column, which may be critical for the range of species simulated in this model, some of which grow basally and some apically. However, it is not clear if this detailed water column light approach is matched with an equally detailed approach to modeling the SAV that takes into account the location of the meristem. Perhaps most importantly, the need for a translocation term suggests that further work may be needed on the rate process portions of the model.
The dispersal model is still under development and generally appears to be sound. Considering a cost to the parent biomass after dispersal to an adjacent cell seems like a reasonable adjustment that may be useful. Describing in greater detail whether the modelers consider the dispersal process to be related to sexual or asexual reproduction could also be helpful. Recognizing that the transition probabilities are currently under development, it is still important to provide more detail as to how they will be coupled to flow and the biomass dynamics portion of the model.
Model Calibration and Testing
Model testing is the estimation of model parameters (calibration) and testing (validation) of the model’s performance. The credibility of the modeling results depends, in large part, on how well the model can generate realistic behavior.
Fountain Darter
14. Calibration and validation of the FD model to date shows the model can reproduce the historical abundances, but additional confidence is needed to most effectively use the model for management purposes.
The strategy for calibration of the FD model was to vary the number of movement time steps needed to trigger mortality within the movement rules until simulated population abundance stayed near the maximum possible densities for 2003 to 2014. Additional simulations showed what happens if the density-dependent mortality is relaxed (see almost exponential increase) and if movement was simply random (extinction). The calibration results are reassuring, as they confirm the types of model behavior we expect and they demonstrate that the model can show other types of behavior if not properly constrained, but they could be more convincing. The follow-up validation analysis used the same approach, but with the model applied to other reaches than to which it was calibrated. Thus, the calibration and validation are based on this same strategy of the degree of agreement of simulated abundance hovering around the specified maximum densities over time.
It should be noted that the good agreement between predicted and observed abundances within the calibration is somewhat tautological. This is because in the model, FD densities are constrained to be less than the maximum densities in each cell; overcrowding kills them if they cannot move to a cell where there is room for more individuals. The maximum densities were set to observed densities by vegetation type. So the fact that the sum of FD densities (abundance) hovers near the sum of the maximum densities (abundance) is somewhat expected if the model was generating roughly realistic densities with some surplus production. This calibration approach would fail (e.g., predict extinction) if mortality was too high or reproduction was too low; the population would decrease and there is nothing in the model that triggers density-dependence (lowering of mortality or increasing in reproduction) at low densities. If the mortality and reproduction rates were set so that there is sufficient potential to produce adults in the model (e.g., reproduction greater than mortality), then the calibration approach used could be successful. The simulated abundance would try to exceed the specified maximum densities, which would trigger density-dependent mortality (i.e., higher mortality) and the simulated abundances would then decrease; with adjustment of the degree of density-dependent mortality, the simulated abundances would then hover near the summed maximum densities. Based on the calibration results to date, the Committee would characterize the model as being a good descriptor of FD abundances during the period of simulation, rather than being a tool for true prediction or forecasting. This is known by the model developers but it needs to be clearly understood by the general audience. The plots of simulated and maximum densities can mistakenly be interpreted as true model predictions that greatly agree with the maximum densities, which may (wrongly) lead to thinking the model is an excellent independent predictor of absolute abundance or can be used to forecast the response of abundance to large changes in flow. The model may indeed have such capabilities, but the calibration and validation done to date cannot be used to conclude that.
The calibration and validation can be strengthened by examining additional model outputs and years, and by quantifying the uncertainty associated with predictions. Some of this has been done by the model development team but could be better documented, more rigorously compared to the field and lab data, and additional outputs considered. For example, one could
examine the simulated spatial distributions and movement trajectories of individual FD in the model, and perform more in-depth contrasting of dynamics between years with extreme conditions. Scenarios can also be simulated that manipulate flow and SAV (habitat) conditions to then track how these progress through the FD processes and life stages, resulting in population-level responses. The presently used series of years can be manipulated to increase the interannual variation in environmental conditions. Some years can be adjusted or new single or a few years (e.g., drought, scour) inserted. Model responses at selected steps in this changed flow leading to a population response can be qualitatively compared to lab results and field data to confirm such intermediate effects are realistic. Propagating uncertainty and stochasticity through the FD model, while not adding to the validation credibility, would help in ensuring proper interpretation of model results and model differences predicted under different HCP scenarios.
Sensitivity analysis (model response to small changes in inputs) and uncertainty analysis (model response to realistic variations in inputs) can be used to identify key model inputs and the associated variability in model predictions. If key inputs such as parameter values can be identified, then field and lab studies can be designed to provide more certain estimates of these inputs. These revised estimates can then be inserted back into the models to reduce the uncertainty of the predictions. Furthermore, it is important to present not just individual value as model predictions but also the variability around those values. This aids in the comparison of model predictions to field data, as both have variances. Presenting the variability around predictions is also important to properly interpreting the results from running alternative management scenarios—that is, do these scenarios really lead to differences that go beyond the known levels of uncertainty.
15. The historical time period used for calibration had relatively similar environmental conditions from year-to-year, which limits the range of conditions of scenarios feasible for exploration by the model.
The 12 years used for calibration included a relatively narrow range of flow conditions. Lack of information on model performance outside of these conditions limits the scenarios that can be reliably examined by the model.
Submersed Aquatic Vegetation
16. Some calibration of the SAV model appears to have occurred, but the details are not provided in the interim report. More detail will be necessary in the final report.
Creating a framework for model documentation that covers goals, assumptions, justifications for parameterization, calibration, and verification for this (and future) versions of the SAV model is good practice and will aid in the longevity and application of the SAV model. Based on Table 13 in BIO-WEST (2015), several parameters have been calibrated for two species that have been the focus of initial SAV model efforts. However, descriptions of the calibration approach and results have not been provided. The model development team is strongly urged to provide detail regarding their calibration plans.
