Review of EPA's Environmental Monitoring and Assessment Program: Surface Waters (1994)

The Environmental Monitoring and Assessment Program (EMAP) of the U.S. Environmental Protection Agency (EPA) is an ambitious nationwide program to monitor the status and changes in condition of the nation 's ecological resources to provide information for regulators, managers, and policy makers. In 1990, EPA requested that the National Research Council (NRC) provide an ongoing review of its developing program. In response to that request, the NRC appointed the Committee to Review EPA's Environmental Monitoring and Assessment Program in 1991. Two other reports have been issued by the committee: (1) A Review of EPA's EMAP, Interim Report, 1992 and (2) A Review of EPA's EMAP: Forests and Estuaries, 1994. This third committee report reviews the EMAP-Surface Waters monitoring component in the context of the larger program. This report pays particular attention to the strengths and weaknesses of the overall program as they affect EMAP-Surface Waters. These program-wide issues fall into three major classes: assessment end points, indicators, and design. As part of its assigned task, the committee has also reviewed and commented on the Lakes Pilot Project and on early information available on the streams program. For this report, the committee reviewed documents received from EPA through April 15, 1994 (see Appendix 1).

The EMAP-Surface Waters group should be commended for its investigation into the critical ways different sources of variation will affect EMAP's ability to detect status and trends. The EMAP-Surface Waters Implementation pilot was reasonably organized, and logistical aspects of the operation were well planned. Execu-

tion of the field portion of the regional assessment of the pilot was successful and valuable experience was gained in the site selection process and in evaluating the logistical aspects of the program.

Although the committee applauds the fact that the pilot project appears to have been question-driven, it has concerns about the clarity and adequacy of some of the questions. In general, the committee believes the pilot study could be substantially improved, not just because it failed to address some of the questions and goals it set for itself, but also because those goals and questions are a very incomplete list of the fundamental issues that need to be addressed before the surface waters program is ready for full implementation. In particular, issues of coordination among resource groups, relationships between indicators and specific stressors, and ability to make inferences on scales ranging from single lakes to entire regions were not addressed. The committee recognizes that not every issue can be addressed in a single pilot study, but it is concerned that there appears to be no overall plan to address these issues sequentially in subsequent pilot studies. In addition, oversight and involvement of senior scientists from a central management team at EMAP Center might have enhanced the scientific rigor of the pilot study, improved the design, analysis and reporting phases of the pilot study, and produced more useful models for the full program.

The Surface Waters component of EMAP has responsibility for achieving EMAP goals for the nation's lakes, reservoirs, streams, and rivers. Surface Waters is one of eight EMAP resource groups. The other resource groups are: forests, estuaries, agroecosystems, arid lands, the Great Lakes, wetlands, and landscape ecology. EMAP-Surface Waters initial efforts emphasized lakes and reservoirs, and this portion of the program is more fully developed than the program for rivers and streams. For lakes and reservoirs a pilot project was conducted from 1991-1993 in the northeast area of the United States. A stream pilot project was conducted in the mid-Appalachian area in 1993.

EMAP-Surface Waters differs from most other surface water monitoring approaches in that it is statistically designed to apply information from a sample of lakes to the entire population of lakes on regional and national scales.

Objectives of EMAP-Surface Waters parallel those of the general EMAP program. The objectives of the Surface Waters Component (from D. McKenzie, EMAP Program Officer, personal communication, February 24, 1994) are to:

• estimate the current status, trends, and changes in selected indicators of condition of the nation's lakes, reservoirs, streams, and rivers on a regional basis with known confidence;

• estimate the extent (number and surface area of lakes and reservoirs, miles of rivers and streams) of the nation's lakes, reservoirs, streams, and rivers with known confidence;

• seek associations between selected indicators of natural and anthropogenic stresses and indicators of the condition of ecological resources; and

• provide annual statistical summaries and periodic assessments on the condition of the nation's lakes, reservoirs, streams, and rivers.

EMAP-Surface Waters has designated three assessment end points for the lakes portion of their program: biological integrity, trophic condition, and fishability.

The choice of assessment end points provides the foundation for the EMAP Lakes Program. This first step, therefore, is of critical importance. The NRC committee believes that the EMAP-Surface Waters' current selection of end points needs further definition and improvement. Of the three end points, biological integrity is the most problematic. As used by EMAP-Surface Waters, this term is vague and all-inclusive, conceptually subsuming the content of the other end points and all other more specific environmental problems in lakes. Such a broadly defined term may be useful in summarizing diverse data or in addressing the multiple issues related to environmental quality, but it is not

specific enough to be a useful end point upon which to design data monitoring activities. Therefore, the committee recommends that EMAP-Surface Waters (and other EMAP resource groups) use the term “appropriate biological diversity” instead of “biological integrity” as an assessment end point, as discussed in Chapter 2. This term is based on objective evaluations and depends on measurable, quantifiable reference systems, and its use should lead to the development of objective, quantifiable indicators.

The other two assessment end points for EMAP-Surface Waters are trophic condition and fishability. In theory, each could be defined in reasonably unambiguous ways, and straightforward means can be developed to measure them quantitatively. Nonetheless, the committee concludes that further efforts are needed to refine the definitions and measurement strategies for both end points.

In addition, although EMAP's financial resources will be limited, the committee thinks it is imprudent to exclude drinking water from consideration as a societal value in its surface water assessment program. The EPA and bills to amend the Clean Water Act both express and affirm the concept of holistic watershed planning and management. Also many impoundments and natural lakes are used both for recreation and for drinking water supplies. This is an example where close cooperation with EPA's Office of Water 305b program could be very beneficial.

Once the major “assessment end points” have been decided, the next critical task is to determine what measurements are necessary to assess these end points. Next, a conceptual model of how the particular system works with respect to the end points should be stated explicitly. Examination of the conceptual model leads to the selection of candidate indicators, which are tested in the field. The indicators are selected on the basis of known or suspected cause-effect relationships that are identified in the conceptual model. Until March 1994, EMAP provided no satisfactory program-wide guidelines for indicator selection strategy, and each resource group was left to fend for itself with

little or no guidance from EMAP-Center. As a result, use of conceptual models to drive indicator selection is not well developed in EMAP.

The committee is concerned that the conceptual model implicit in the EMAP-Surface Waters strategy document underestimates the complexity of freshwater ecosystems. There is no consideration of factors like biogeography, seasonal shifts in community structure with secondary nutrient depletion, competition, predation, or hydrologic factors. Therefore, the committee recommends that explicit conceptual models of the ecological systems being monitored be used to guide indicator development.

The committee shares the concern raised by EPA's peer review panel regarding heavy reliance on indices with unknown properties. Use of indices to describe complex ecosystems has some advantages but also some important disadvantages. The major advantage is the ability of an index to condense many parameters into a single number, which at first glance may be easier to understand. A major disadvantage is that the statistical properties of the index are often not well understood; moreover the indices are often nonlinear; that is, a change from 1 to 2 is not the same as a change from 2 to 3.

Rather than relying upon a univariate index with unknown statistical properties, it is possible to use the multi-response vector of the original parameters and apply multivariate statistical techniques for analysis, or exploratory data techniques involving better visualization of multi-dimensional data. Nonparametric multivariate procedures also exist for testing whether groups of multivariate data points are significantly different from each other (e.g. comparing disturbed to undisturbed areas). The committee recommends that EMAP-Surface Waters continue its efforts to develop indices using a number of different approaches including multivariate statistical and exploratory data analyses. In addition, appropriate new statistical methods should be incorporated as they become available.

The design for the Surface Waters component follows the overall EMAP design. The scheme for lakes is better developed

than that for streams, which have not received detailed consideration to date. About 3,200 lakes will be selected using a probability-based sampling scheme. A different subset of 800 of them will be sampled each year so that every lake is sampled once every four years. Lakes will be stratified into size classes so that large lakes (which are relatively rare compared to small lakes) are represented in the sample. The random selection of the lakes will occur in a way that maintains uniform spatial coverage nationally. The committee has identified several areas of concern regarding the general EMAP sampling design.

A watershed perspective is lacking in the sampling design. Because surface water systems are linked with their watersheds, the committee believes that the lack of a watershed perspective will severely limit the identification of likely causes of detected changes in the EMAP lakes. Without this watershed perspective, landscape characterization data cannot be used to evaluate the status of individual aquatic resource units. Thus, the committee recommends that a greater emphasis be given to concomitant measures of watershed characteristics. Remotely sensed data on land use and cover could be used to great advantage. Representatives of EMAP have recently indicated that they will be using a watershed approach for their data gathering, and the committee encourages this approach.

Another concern is that the sampling design may not be sensitive enough to detect a change in condition unless the change is very large and affects most lakes and streams in a region. There are many types of lakes and streams in many types of landscapes and each type has different sensitivities to a particular stress. The committee is not certain that enough sensitive lakes and streams will be included in the sample to detect a change due to a particular stress. Because lakes and streams will be sampled during one 9-week period, some measurements may not be made at the biologically most meaningful time, thus decreasing the sensitivity of the measurement.

In addition, the sampling design may have difficulty detecting changes in biological metrics over time. Variances of biological populations (and therefore community metrics) among lakes and within lakes over the course of a year are large.

EMAP-Surface Waters has been a leader in performing tests of statistical power to detect changes or differences with real limnological data collected by various state agencies. Power studies to date have examined primarily physical and chemical variables, because these data were relatively available. The committee recommends that similar tests with published or simulated biological (population and community level) data be vigorously pursued, because EPA indicates that these data will be important in formulating indicators similar to Karr 's Index of Biological Integrity.

It is not clear how useful the trends that EMAP may detect will be, and whether EMAP will be able to relate such trends to specific stressors is uncertain.

Because of the 4-year revisitation rate, the current design essentially does not allow for site-specific inferences to be made. Although it is not an explicit goal of EMAP to make site-specific inferences, the committee believes there is value in making site-specific inferences from well-chosen sites. This would augment the basic EMAP design and the added value could be achieved at small additional cost. Therefore, the committee recommends that a substantial number of sites be sampled annually. Some of these sites might be selected because they are known or suspected to be sensitive indicator lakes for selected stressors.

The committee recommends that additional power tests be performed to examine the ability of the current design to detect status and trends for quantiles in the tails of distributions (e.g., lower and upper 10th percentiles).

The surface water component of EMAP began its first year pilot study during the summer of 1991. Pilot activities included a regional sampling effort (EPA Region 1), a set of more focused indicator development studies, and an analysis of the effects of different types and magnitudes of variability on the ability to detect regional trends.

The regional assessment portion of the pilot study represents the first application of the general EMAP design to surface water

ecosystems. The EMAP-Surface Waters implementation pilot was reasonably organized, and logistical aspects of the operation were well planned.

Field execution of the regional assessment portion of the pilot was successful. Valuable experience was gained in the site selection process and in evaluating the logistical aspects of the program. However, a substantial portion of the data was not analyzed in time to meet deadlines for the pilot study report. This suggests that a larger investment in data analysis will be necessary if a larger scale implementation is to be completed in a timely fashion.

The design of the indicator development study was not as good as that of the regional assessment portion of the pilot. We believe the scope of this portion of the pilot was too ambitious given the financial resources available. The response of lakes to catchment disturbance, or even the ability of certain indicators to detect the response, is unlikely to be discerned without a much larger set of lakes selected specifically to address this question. With only four to six lakes per class it was unreasonable to expect to be able to see a strong signal between disturbance and the response variables.

Field sampling for the indicator portion of the pilot appeared to go smoothly. Useful variance estimates and time and cost estimates were obtained for many of the assemblage indicators. However, there appear to be major difficulties with the analyses of the indicator assemblage data. They include:

• Lack of planning and coordination.

• Lack of statistically sound hypothesis evaluation.

• Lack of any quantitative comparison between the various indicator variables measured.

The EMAP Surface Waters group should be commended for its investigation into the critical ways different sources of variation will affect EMAP's ability to detect status and trends.

EMAP-Surface Waters also began to conduct a pilot program on streams in the summer of 1993. It is difficult for the committee to evaluate this pilot study, because we have received almost no written information about it and what we received was not delivered to the committee until May, 1994. A conference call was held with several committee members and EMAP representatives at that time to discuss the pilot study.

Based on what we have been told, the sampling strategy to be used in the stream survey needs further development. Based on the limited information now available, the committee concludes that it was premature to embark on a stream pilot study at this time. The currently conceived sampling strategy appears inadequate to characterize stream quality either chemically or biologically. We recognize that not everything can be planned in advance, and there still is room for the trial-and-error approach in developing large-scale programs like EMAP. Nonetheless, the scale of financial and human resources required even for a pilot-level survey is sufficiently great that EPA must not only minimize the risk of error, but also maximize the likelihood that it will successfully address the critical issues necessary for planning a full-scale stream survey. We do not believe that EMAP is presently in this position.

The committee recommends that EMAP decide what its overall objectives are for assessing the status of the nation's rivers and streams. These objectives—and a strategy to achieve them— need to be developed within the context of existing federal monitoring programs. This should occur before EMAP proceeds with the development of stream pilot studies. The committee does not believe that the currently conceived sampling strategy is appropriate to characterize stream quality either chemically or biologically.

It is unclear to the committee to what extent there has been substantive involvement of the scientific community in the planning done to date for the streams pilot. Therefore, the committee recommends that EMAP-Surface Waters' scientists spend time developing a substantive planning document and continue their dialogue with stream scientists in other branches

of EPA, in other water-related federal agencies (U.S. Geological Survey, U.S. Fish and Wildlife Service, etc.), and in the academic community to better evaluate how the stream phase of EMAP should be designed.

Much routine water quality sampling done by state pollution control agencies on surface waters nationwide is funded through EPA's Office of Water under Section 305b of the Clean Water Act. Closer collaboration between the 305b program and EMAP has the potential to enhance the effectiveness of both programs while reducing the overall cost of federal monitoring programs for surface water quality.

Therefore, the committee recommends that EMAP-Surface Waters and the EPA Office of Water work together to insure that data collected under the 305b program can be useful not just for compliance monitoring (the primary focus of current programs in most states), but also to assess temporal and geographic trends in water quality.

Coordination among resource groups is especially important for the Surface Waters component of EMAP. Surface waters are affected by processes occurring within the terrestrial ecosystems in their watersheds. Currently, the Surface Waters group is analyzing riparian vegetation. However, it is not clear that the classification system being used is the same as that used by terrestrially focused resource groups. Without a closer interaction with the terrestrial components of EMAP, an opportunity for comprehensive understanding of how and why lakes may be changing is likely to be missed.

There is a continuing lack of a clearly defined procedure for defining and prioritizing the assessment questions that can and will be addressed by the program. These questions are critically important, because they will drive the sampling strategies and clarify the goals of EMAP-Surface Waters and the other resource groups.

A procedure should be developed to identify the most important assessment questions from a policy perspective, but at the same time ensure that it is scientifically feasible to address the questions. One possible approach is to formalize a planning structure that would be composed of guidance panels associated with each resource group. A central planning committee, composed of representatives from each of the “thematic” panels would then make the hard decisions about resource allocation and attempt to optimize efficiency and coordination between groups. Some aspects of such a planning structure already exist within EMAP. However, the committee believes it is critical that the guidance panels also include representatives of EMAP clients, i.e., policy makers and the larger scientific community. Most panel members should be external to EMAP and to EPA and they should be leaders in their areas of expertise. These panels should not duplicate the advisory and planning functions of the current peer-review panels or of EPA 's Science Advisory Board.