Review of EPA's Environmental Monitoring and Assessment Program: Overall Evaluation (1995)

EMAP's components can be characterized as being program-wide or pertaining primarily to one or more resource groups. This chapter reviews the program-wide components of EMAP—landscape characterization, indicators, and information management.

The Landscape Monitoring and Assessment Research Plan - 1994 (EPA, 1994a) sets forth a plan for assessing status and trends of landscape patterns using remote sensing and geographic information system methods. The plan proposes three steps: establishing a baseline condition, detecting changes and determining when and where declines in landscape condition are sensed, and assessing the association between landscape condition and stressors. Such a monitoring effort at regional and national scales is a valuable and crucial component of EMAP.

The Landscape Characterization Plan has not been available for review. The Chesapeake Bay Watershed Pilot Program suggests that the current Landscape Characterization program focuses on land cover mapping for all resources using thematic mapper data, apparently fulfilling the Landscape Ecology program's goal

of establishing a baseline condition. Since the Landscape Ecology and Landscape Characterization programs appear to be inextricably linked, they are evaluated together.

The assessment of land use and land cover is an extremely important activity for EMAP. A large body of current literature (e.g., Houghton et al., 1983; Turner et al., 1990; McDonnell and Pickett, 1993) suggests that human land use management practices are the most important factor influencing ecosystem structure and functioning at local, regional, and global scales. Land use can dramatically alter species composition, food-web structure, ecosystem carbon storage, and interactions between biota and the atmosphere. Monitoring of the spatial distribution of land cover will provide crucial information regarding our national environmental status.

The role of the landscape programs has changed since the initial phases of EMAP in which Landscape Characterization was described as a central and pivotal program that would provide necessary information for resource groups. With the help of EMAP-Landscape Characterization, resource groups would be able to determine spatial and temporal sampling resolution and key indicator variables. The current role of the landscape programs within EMAP is unclear. A section within the EMAP-Landscape Ecology plan called ''Integration with EMAP Resource Groups" only creates more confusion. For example, the most explicit statement is "In this example, a series of common habitat measures could be implemented among EMAP resource groups…." How will this series be implemented? By top-down decisions? If so from whom? How will information be exchanged? Exactly what personnel, data, concepts, and technology will be exchanged? Although Landscape Characterization is a program with strong potential, without a document describing its goals and

structure it is unclear how the program will relate to the rest of EMAP.

Although it is unclear that EMAP-Landscape Characterization is providing integration among all of the resource groups, it has interacted successfully with at least some of them. For example, the EMAP-Landscape Characterization group has worked with the Surface Waters resource group and has characterized the land cover in the watersheds surrounding each of the lakes sampled during the Surface Waters pilot study. These data will be essential in the Surface Waters group's attempt to understand the influence of the surrounding landscape on trophic status, biological diversity, and fishability defined by EMAP as: (the catchability and edibility of fish by humans and wildlife). Clearly, strong interactions between the Landscape Characterization group and each of the individual resource groups will enhance the groups' efforts to understand landscape-level processes that might affect individual resources.

The EMAP-Landscape Ecology plan is extremely well written, providing a coherent theoretical framework for assessing ecological status and trends at landscape scales. This document arguably has provided the clearest description of the connections among societal values, assessment questions, conceptual models of ecological phenomena, and indicator variables. Such a framework is crucial to the implementation of EMAP.

The following are concerns about the landscape programs.

• EMAP-Landscape Characterization and EMAP-Landscape Ecology are in very early stages relative to the resource groups, with an implementation plan to be developed in the next three years. It is difficult to see how the landscape programs can guide resource groups if they are so far behind in development.

• The major impetus, ideas, and follow-through of the Landscape Ecology program appear to rely upon scientists who are not a permanent part of EMAP. Clearly, it is ideal that one of the leaders in landscape ecology, R. V. O'Neill, along with Denise Shaw of EPA, is guiding the development of the program. But it is unclear whether any permanent personnel or resources, other than the hard-working Denise Shaw, are dedicated to the program.
• The focus of the EMAP-Landscape Ecology program is abstract. A major focus on theory is not in itself a concern. However, there should be a concurrent effort in EMAP to assess ecological functioning. Such an effort would require interaction and integration among resource groups at the empirical stage, relying on common sites and common indicator variables (see Chapter 2). EMAP-Landscape Ecology's current plans are to study the indicators of landscape structure discussed in the large body of theoretical work on landscape ecology. However, the relationship between these indicators and ecological functioning has yet to be demonstrated. Since individual resource groups focus on assessment end points, the Landscape Ecology program should emphasize ecological functioning.
• Landscape Characterization is not a well-defined program. This program is costing millions of dollars, and has the potential to be one of the most important synthesis and integration programs; yet there is no documentation of its purpose, its projected scope, or its connection to the resource group activities. Questions about the Landscape Characterization program's direction abound. Will thematic mapper data be used to classify the entire United States? Will other satellite imagery be used to assess the nation's status on an annual basis? What will be the result of the collaboration between EMAP and the United States Geological Survey's EROS Data Center? The data center could provide EMAP access to biweekly composite images of the United States at 1-km scales. Such images could provide a useful tool for assessing changes in the nation's ecological status. Will EMAP

• direct any resources toward this end, or toward combining these data with field data from the resource groups?

There appears to be an important gap between Landscape Ecology and Landscape Characterization in the area of geographic analysis. Large-scale geographic and temporal trends can be extremely useful for testing correlations, such as those between lake and terrestrial productivity, or atmospheric deposition and ecological indices. These correlations can provide strong inference for cause and effect relationships. Further, data from the EMAP sampling sites can be reasonably extrapolated with good geographic information and ecological algorithms. The committee suggests that such analysis belongs in landscape programs.

Providing landscape-level data surrounding each of the EMAP study sites seems to be a major reason for the existence of EMAP-Landscape Characterization. The benefits of the interactions between the individual resource groups and EMAP-Landscape Characterization will be diminished without a strong information management system that will allow the individual resource groups easy access to the land cover data. Therefore, EMAP needs to develop and maintain an administrative structure that demands close communication and interaction among EMAP-Landscape Characterization. EMAP-Information Management System, and each of the resource groups. This structure might take several forms; one possibility is to locate leaders from each of these groups in a central office.

A fundamental premise of the EMAP program is that the status of large and complex ecological systems can be monitored and assessed using a limited set of indicators, and biological indicators are being proposed as the key indicators within EMAP (see Chapter 2). Choosing appropriate indicators has been a

major focus of EMAP activity since the program began. Despite the obvious centrality of indicator development to EMAP, the completion of a comprehensive indicator strategy document has been slow in coming. An early version of a strategy document (Olsen, 1992) followed a major program reorganization. A new indicator strategy document has since been developed and was distributed in the spring of 1994 (Barber, 1994). This more recent document serves as the formal basis for the review of the overall indicator strategy of the EMAP program. This review also focuses some attention on the individual resource groups, since they develop indicators independently of one another.

The EMAP program has developed a four-part strategy for selecting the indicators it will use for nationwide ecological assessments. These four major steps are selection, evaluation, implementation, and re-evaluation. Each step has an associated list of tasks that together constitute the overall EMAP indicator strategy (Table 3-1). This strategy provides a succinct and useful starting framework for standardizing indicator development across resource groups. Earlier indicator documents (Hunsaker and Carpenter, 1990; Olsen, 1992) provided additional background information on EMAP indicator development, much of which has been refined and summarized in the current strategy document. The strategy document strongly and correctly emphasizes the iterative nature of indicator development. It makes clear the desirability of ongoing interactions between EMAP and the administrators, politicians, scientists, and the general public who use the assessment information, as well as the continual need to reassess and redevelop appropriate indicators. The strategy represents a reasonable and sequential procedure that could serve to coordinate what are at present a very diverse set of EMAP indicator development activities.

The Document. The current strategy document itself is a clear improvement on the original, and shows a significant sharpening and development of thought within EMAP with regard to biological indicators. A question remains, however, as to whether the current document is comprehensive enough to guide such a key element of the overall EMAP program. While it clearly presents an overall strategy for indicator development, it is still weak in terms of clarifying specific procedures and organizational arrangements that will ensure that program development is consistent with the vision of the strategy document.

It is unfortunate that five years into the development of the EMAP program there is little evidence of a clear program-wide set of procedures that standardize and coordinate indicator development. The fact that it has taken this long to produce the current indicator development strategy document reflects poorly on a program that claims to rest on innovative indicator development. The current strategy document can provide a reasonable starting point for such coordination. Because a considerable amount of indicator development work has already been done in the pilot studies of various resource groups, it is imperative that a central strategy and set of guidelines be available.

The Strategy in Theory and Practice. There are already substantial inconsistencies between the EMAP strategy and the actual indicator development taking place in the resource groups. For example, few if any of the indicators now being evaluated in the field have their links with assessment end points and potential stressors documented in scientifically defensible conceptual models, as specified in the development strategy. EMAP established an Office of Indicator Development and appointed an indicator coordinator at EMAP Center early in 1993. However, as of March 1994, the Indicator Development Strategy Document (Barber, 1994) makes no mention of this central coordinating office, the coordinator, or their roles in indicator development. Similarly, the status of an indicator development database, proposed by Olsen

(1992) to facilitate communication among resource groups during indicator development, has apparently not changed in two years. Barber (1994) reported that the database is still in the development stage. The divergence of theory and practice in this matter will only grow without strong central leadership. Individual resource groups have taken very different approaches to evaluation studies and appear to have different strategies and criteria for evaluating the performance of potential indicators (NRC, 1992; NRC, 1994a; NRC, 1994b). The conclusion is that there is still little operational coordination of indicator development within EMAP. However, the committee recently learned that the position of indicator coordinator has now been filled at EPA, which may lead to a more acceptable level of coordination.

Monitoring Philosophy. The EMAP approach to monitoring involves the identification by each resource group of sets of indicators to be used on a national scale. In most cases, these sets of indicators do not yet exist. This is therefore a good time to impose program-wide procedures for indicator selection and development. According to the strategy documents, the following two factors should constrain the selection and development process.

First, the process must be consistent with the EMAP Assessment Framework (Thornton, et al., 1994), which dictates that assessment questions be based on resource values. Resource values give rise to assessment questions; in EMAP these usually focus on resource status with respect to that value.

Second, the selection process should be science-based. That is, based upon current scientific theory and models of resource structure and functioning.

The indicator development strategy clearly calls for the use of explicit conceptual models as a basis for selecting potential indicators for field evaluation that are scientifically sound. These models will provide a scientifically defensible hypothesis showing how indicators are related to resource values and assessment end points. They will also show how indicators are linked to potential

stressors and other important aspects of ecosystems structure and functioning.

What is at issue here is the entire EMAP retrospective assessment approach (see Chapter 2), which draws its conclusions from arguments based on the weight of evidence. Documenting the mechanistic link between an indicator and its stressors and assessment end point is crucial to the retrospective assessment approach (Thornton et al., 1994). The problem is, while the assessment framework document discusses the issue of establishing cause and effect in some detail, the indicator strategy document says little on this key point. EMAP's retrospective approach will not succeed unless the development of conceptual models is taken seriously.

Put more simply, it is at this stage that science should be infused into the EMAP assessment protocols. Unfortunately, the example models used in both the strategy development and the assessment framework documents (and for the most part in the resource group planning documents reviewed in previous reports) are almost uniformly trivial. This raises questions as to whether EMAP is capable of implementing its own assessment strategy, and whether it is interested in pursing indicator development in a scientifically rigorous fashion. In many cases EMAP seems to confuse conceptual ecological models with graphic portrayals of rationales for indicator selection decisions that have already been made. Each EMAP resource group should develop one or more mechanistic conceptual models of its resource based on current scientific knowledge. These models should serve as explicit hypotheses about those aspects of ecosystem structure and functioning relevant to the assessment end points the group has chosen. The models must be detailed enough to include potential indicators, assessment end points, key variables, and factors causing the endpoint to vary across the landscape. The models must also include some hypothesis concerning mechanistic and functional relationships among key variables. Only from such models can the resource groups generate lists of potential indicators about which there is sufficient knowledge to interpret chang-

es in indicator status. It is necessary to link indicator selection to current understanding of ecological mechanisms if indicator development is going to be scientifically credible, and if the resulting monitoring data are to be useful in ecological assessment.

Evaluation of Indicators. The strategy document seems particularly weak in terms of laying out guidelines for indicator evaluation. Developing indicators of ecological condition as envisioned by EMAP is a challenging task. It is uncertain how much the anticipated indicators of ecological condition can be developed, and how reliable and easy to measure they will be. Therefore, to be successful, indicator development for EMAP will need to have a substantial research component.

There are many practical, program-wide questions that need to be answered. How do resource groups decide that an indicator works? What kinds of evidence are required? How does the program review the evidence? How often and in how many different settings should validation tests be performed? How many indicator variables should be measured for a given assessment endpoint? These are just some examples of issues in indicator evaluation that need to be answered by all of EMAP in a coordinated manner. The current indicator development strategy provides insufficient guidance in these matters. The document does contain a list of what are identified as criteria (see Barber, 1994, Table 2, page 37); but these really seem to be only a list of desirable general characteristics. A feature of a useful criterion is that it can be (at least relatively) unambiguously evaluated. The current document provides little information as to how EMAP will evaluate the performance of a potential indicator with respect to Barber's Table 2, nor does it discuss whether any or all of the criteria are essential.

There are numerous evaluation issues that need program-level guidance if the indicator selection strategy is going to yield the nationally applicable set of indicators EMAP envisions for each resource group. One particularly important issue, for example, is the explicit and early evaluation of the biological and statistical properties of all potential indicators. This should be given a high

priority. Such evaluations should include analyses of the properties of the mean, variance, and behavior of the index in power tests. (A power test is a test of the statistical power of an approach to detect change). If this cannot be done analytically, then simulation analyses should be performed. An example of such an investigation for Karr's Index of Biotic Integrity is that done by Fore et al. (1994).

Questions of Scale and Regionalization. The indicator strategy document wisely provides for a phased indicator evaluation (Table 3; Barber, 1994, page 41). Problems associated with spatial scale in indicator performance are likely to be common, as noted above, and careful testing in pilot and then demonstration projects before national implementation is likely to be the most efficient approach in the long run. Biological indicators useful at a national scale at present are quite rare. Biological indicators generally require region-specific interpretations. For example, green sunfish (Lepomis cyanellus) play an important role as in indicator of subnominal habitat for states of the central midwest. This species does not occur in colder rivers of the northern midwest, but white suckers (Catostomus commersoni) may provide a useful analog. Similarly, sweet gale (Myrica gale) is a common indicator of rheotrophic wetlands in the upper midwest, but is a common constituent of coastal ombrotrophic bogs in New England. If EMAP succeeds in developing a national set of indicators, this will be a major accomplishment indeed. EMAP's emphasis on indices and cumulative distribution function hopefully will compensate for large-scale spatial variations in an indicator's effectiveness. Karr's Index of Biotic Integrity (IBI) is an example of an index that might be used on a national scale, but it has to be adjusted from region to region. However, careful evaluation of each potential indicator over larger and larger areas and regions seems a wise approach. The ways in which the various resource groups deal with this problem will have important consequences for the selection of nationally implemented indicator metrics. Program-wide strategies for dealing with this issue should be developed now, in

time to be applied with some uniformity across the resource groups.

Indicator Development Across Resource Groups. This is another area in which more coordination is needed. Ideally, nationally-implemented indicators should have some correspondence across resource groups. At present, it is unclear whether or not the assessment questions in each resource group are similar enough to lead to parallel sets of indicators. Possible examples include indicators reflecting net primary productivity, biological diversity, and aesthetic value. Such symmetry among resource groups, while not essential to the basic EMAP objectives, would greatly enhance the scientific and analytical value of the data collected.

The indicator strategy (Barber, 1994) is a welcome if late addition to the EMAP program. It is clearly sufficient to begin the coordination that will be required to bring the indicator development activities of the resource groups into a more scientifically rigorous context. Having a documented strategy, however, is not a replacement for organizational structure and guidance in this area. The strategy document, in conjunction with a strong central office for indicator coordination, would be a key asset in the EMAP program, and would help to ensure that the massive amounts of data that EMAP proposes to collect will in fact be useful in future retrospective assessments. Without such coordination, the ad hoc nature of indicator development as it currently operates in the resource groups will weaken the value of whatever data the monitoring program generates.

An encouraging sign is that the position of indicator coordinator has recently been filled. This can be the first step in a more rigorous coordination of indicator development within EMAP. The peer review committees have been essential in providing a sounding board for the various resource groups over the past five years,

but it is clear the relationship between EMAP and these boards is too informal to provide the degree of coordination required. These boards have no formal oversight authority, and operate in isolation from each other and the overall EMAP effort. The role of peer review panels as currently constituted in the program should be expanded, but they are no substitute for much needed central coordination of indicator development inside EMAP itself.

The EMAP-Information System is a critical component of EMAP. It is essential for the success of the program that EMAP-Information System provide appropriate access to the data and information generated by the program, as well as the personnel, hardware and software resources necessary to support such access and processing.

EMAP-Information System should be viewed primarily as a scientific database system that also supports data analysis and modeling activities. The requirements for systems that support the storage of, access to, and analysis of large-scale collections of scientific information are not satisfied by the relational database systems currently available. Therefore, although these latter systems have proven to be the technology of choice for supporting many business activities, they have not proven adequate for supporting scientific activities, particularly modeling efforts. Scientific data typically involve very long transactions, are transformed for a variety of simulation processes and models, have heavy reliance on metadata, and are more complex in structure and organization. As a result of these fundamental differences, the EMAP-Information System must be carefully designed to facilitate the myriad future uses appropriate for scientific data that could easily be overlooked when trying to retrofit existing relational database technology for purposes for which it was not designed.

A scientific information system can be viewed as having two logically distinct components. The first component is the environment in which the users will operate and the second is the technical personnel, hardware and software to support this environment. The environment must be designed to support the users in their information processing tasks and permit users to carry out their tasks efficiently. There must be efficient support for database activities, including data input, data access, and data analysis. EMAP-Information System user requirements involve databases containing a large and heterogeneous collection of data sets, which are spatially distributed and spatially indexed. In relation to such a database, EMAP-Information System users should also have access to a large variety of analytical and visualization tools that may be applied to various subsets of the data, over various spatial scales, and at various stages of information processing.

EMAP-Information System users should also be able to integrate any tools they need into their environment. An investment in technical support staff and good hardware and software should ensure that the information management system enhances the productivity of the users and facilitates the accomplishment of EMAP's goals.

The information available to the committee for its evaluation of EMAP-Information System was a strategic plan for the information system and an initial rapid prototype system developed largely within the estuaries resource group. A set of white papers intended to lay out important details of the system and its design have not appeared. There is, however, a rapidly growing body of information concerning scientific database systems that provides standards for comparison. In particular, much information has been generated in relation to programs such as the Long-Term Ecological Research and National Science Foundation scientific database initiatives, and National Aeronautics and Space Administration programs such as the Applied Information Systems Program and the Earth Observing System project.

The list of serious concerns that follow are based on a visit to EMAP headquarters, discussions with EMAP-Information System staff, and the limited documentation available.

• The high level of abstraction in the strategic plan does not permit an adequate evaluation of the proposed system. In particular, many elements that one would expect to find in a strategic plan, such as a requirement analysis, system design, and other detailed studies and plans, are absent. Much of the effort underlying the document appears to have been focused on developing high-level concepts about the functionality of the system, its implementation, and its management in terms of the ''Zachman Framework". This framework, which was developed within EPA, takes a management point of view towards system specification. While a management perspective should not be ignored, such planning should be based on the viewpoint that EMAP-Information System is a scientific database system, rather than an information system focused on the needs of management. In particular, the planning should be largely focused on the design of an environment that is sensitive and responsive to user requirements and on the design of appropriate hardware, software and personnel for such an environment. Furthermore, the planning should follow a top-down strategy that begins with the highest-level requirements of scientific users, and gradually expands the details of these requirements.
• The strategic plan lacks almost any reference to recent activities in the area of scientific databases and supporting infrastructure. In particular, the plan makes a single reference to an early conference on scientific databases. There is no reference to database developments in programs that have similarities to EMAP, such as NSF's Long-Term Ecological Research program. There is no reference to emerging protocols and standards, such as the recent federal standard concerning spatially referenced data. There is no useful reference to the role in the EMAP-Infor-

• mation System that the Internet, World Wide Web, and other related developments will play, although these are affecting scientific activities and scientific database systems in important ways. It appears that EMAP-Information System is being developed in an environment that is effectively isolated from related developments occurring outside of EPA, despite the extensive activity in the area of scientific databases. It is not clear whether EMAP drew upon appropriate expertise in drafting the strategic plan.
• The plan gives no significant indication that user requirements, and their central role in EMAP-Information System, have been given adequate consideration. In particular, the plan does not include any useful analysis of user requirements. An appropriate basis for a strategic plan is a detailed analysis of the user requirements that are likely to arise in EMAP over the next two decades.
• The plan fails to specify important aspects of the user environment that relate, for example, to data access, data processing, and data visualization. Furthermore, the plan does not appear to provide a uniform approach over the resource groups with respect to such issues. Currently it appears that the plan allows much autonomy among the resource groups with respect to important decisions concerning issues such as query languages, data models, and data organization. Uniformity over the groups with respect to matters such as these may well be critical for the success of EMAP, since long-term coordination of both data and analyses of data across resource groups will be not only important but essential to achieve the degree of integration envisioned for EMAP.
• The plan fails to address the specifics of such complex and intensive data processing. For example, EMAP's success relies on the handling and integration of large, spatially indexed data sets, including images, with more standard observational data sets. The failure of the plan to specify important aspects of support for the users, such as technical support staff, hardware and software, may result from the plan's failure to describe the environment.

• The plan does not specify how the development and operation of EMAP-Information System will relate to other federal programs in terms of sharing resources and technologies (for example, making EMAP data available to the National Science Foundation Center for Ecological Analysis and Synthesis). While there is some interaction with a consortium of federal agencies involving a large, spatially indexed database located at the EROS Data Center, the links between such systems and EMAP-Information System are at best vague. They are particular challenges for EMAP-Information System in its ability to stay abreast of developments relating to the national information infrastructure.
• The plan appears to underestimate the resources required to design, construct, and test an information system that will prove adequate for EMAP in the long-term. For example, the plan appears to call for only 1 to 1.5 full-time positions over the next four-years to address the issue of spatially indexed data. With such inadequate resources it would be a surprise if an adequate or stable system were available to EMAP users by 1997.
• It is not clear that the rapid prototyping effort has a clear-cut goal. Such a system should be focused on understanding user requirements. The results of this understanding could then be incorporated into the main EMAP-Information System. The current prototyping effort, which involves an integration of data management systems such as ORACLE, ARC-INFO, and SAS, has very limited capabilities and is inefficient from a user's point of view. The system appears to be focused on resolving technical issues with the use of systems that are probably not appropriate for supporting large-scale scientific database activities in the long run.

The information provided for this review of EMAP-Information System is an insufficient basis for evaluating the adequacy of the system in its primary task of supporting the resource group scien-

tists. In particular, too many aspects of the system are specified at too high a level of abstraction for an appropriate judgment while many important aspects either are left unspecified or are severely underspecified. Based on our reading of the strategic plan and on our evaluation of the rapid prototype, the committee has very serious doubts as to whether the current approach to designing, implementing, and managing EMAP-Information System is an appropriate solution to the long-term data and information processing requirements of EMAP.