Review of EPA's Environmental Monitoring and Assessment Program: Overall Evaluation (1995)

EMAP is a program of many parts, but all the parts share common goals. The purpose of this chapter is to review those matters that apply to all parts of the program. They concern the design and implementation of monitoring; the sampling protocols; the development of indicators; integration among various parts of the program; program coordination within EMAP, within EPA, and within the federal government; and external scientific review.

Most ecological monitoring programs are driven by some explicit or implicit set of assessment questions. EMAP is no different. The topic of this section is the basic monitoring approach taken by EMAP—retrospective monitoring. This well-established monitoring approach is suited for many environmental problems, but not for all; it is essential that any evaluation of EMAP have a clear view of those kinds of environmental problems that EMAP is likely to help identify and those for which other monitoring approaches would be better.

Retrospective or effects-oriented monitoring is monitoring that seeks to find effects by detecting changes in status or condition

of some organism, population, or community. Examples include monitoring the body temperature of a person, monitoring the productivity of a lake, monitoring the condition of foliage in forests, and so on. It is retrospective in that it is based on detecting an effect after it has occurred. It does not assume any knowledge of cause-effect relationships, although the intention is usually to try to establish a cause if an effect is found. It is EMAP's general approach.

Predictive or stress-oriented monitoring is monitoring that seeks to detect the known or suspected cause of an undesirable effect (a stressor) before the effect has had a chance to occur or to become serious. Examples include monitoring the cholesterol level in a person's blood, monitoring the stress level along a geological fault, monitoring animal tissues for the presence of known carcinogens or other disease-causing compounds, and monitoring with a canary the toxic gas level in a mine. It is predictive in that the cause-effect relationship is known, so that if the cause can be detected early, the effect can be predicted before it occurs.

The EMAP Assessment Framework (Thornton et al., 1994) is the formal exposition of the assessment context for the EMAP program. As such, it is the fundamental statement of the philosophical and practical requirements of EMAP's data gathering activities. This is an especially critical element in the EMAP program development because in recent years EPA has been actively promoting ecological risk assessment (NRC, 1993b, RAF, 1992) as a new operating paradigm. Early EMAP documents were curiously silent about this larger EPA perspective, raising some question about the extent to which these initiatives were coordinated. In February 1994, EPA-EMAP released a document entitled Environmental Monitoring and Assessment Program Assessment Framework (Thornton et al., 1994). It is a welcome—if long overdue—addition to the voluminous descriptive literature EMAP has generated. It contains the most developed and lucid descriptions to date of the overall philosophy and approach behind the EMAP program. Together with the Indicator Development Strate-

gy, the Assessment Framework clearly outlines the rationales, approach, and objectives of the EMAP program.

The Assessment Framework document describes the proposed role of EMAP in terms of EPA's ecological risk assessment paradigm. It makes clear that the program is conceived of as having two distinctive roles in the ecological risk assessment process. As a supplement to the problem formulation phase of risk assessment, EMAP is to identify emerging problems that will require the attention of other programs within EPA to determine comparative ecological risk. EMAP's second task is to provide documentation of the success or failure of national risk management decisions. By reporting on long-term trends in environmental status, EMAP is expected to provide data on the effectiveness of regulatory decisions and risk management by the agency. Its purpose in this regard is to provide data to evaluate policy, thus helping to close the loop in the iterative Ecological Risk Framework (RAF, 1992). The authors argue effectively that, in this sense, EMAP is really an integral part of EPA's new operating paradigm.

The Assessment Framework document is the appropriate place for a full exploration of the benefits and shortcomings of retrospective risk assessment. Unfortunately, it provides no wide-ranging examination of this issue. Given the proposed scale of EMAP, EPA has the responsibility to provide the public with a more detailed analysis.

One of the most important features of the Assessment Framework is the discussion of the implications of EPA's decision to use a retrospective (effects-oriented) instead of predictive (stress-oriented) assessment model. The Assessment Framework describes both approaches in some detail and refers to them as being complementary, arguing that retrospective analysis will "become increasingly important as assessments of larger scale problems are conducted because it will become increasingly difficult to establish specific cause-effect relationships …" Because the whole of the EMAP monitoring strategy is based on retrospective analysis, EMAP needs to present a more rigorous exposition of its rationale for this strategic choice. It is not clear that the

quoted statement above is in fact true. For example, in the 1950s, the poor hatching success of birds of prey such as the bald eagle was a large-scale problem clearly caused by accumulation of DDT in the eggshells. It is significant that this cause-effect relationship between DDT and eggshell thickness was not established by a monitoring program such as EMAP, but by a research program driven by clearly stated hypotheses. Sound policy (the banning of DDT and subsequent recovery of eagles) was based on this cause-effect research.

Predictive and retrospective approaches are complementary, but they are not equally useful in every risk assessment. For situations in which the risk or consequence is severe (e.g., a pedestrian crossing a busy highway), effects-based risk assessment is an inappropriate strategy. When the risk is lower, either because the effect is weak, or it can be mitigated, retrospective analysis is appropriate and probably more cost-efficient (e.g., exposure badges in low radiation environments). The EMAP approach will be more useful for some types of ecological risks than for others. In considering the value of EMAP as a national monitoring program, it is important to understand what kinds of ecological risk EMAP is likely to provide useful data on, and what kinds of risks will require more traditional predictive, or stressor-based, analyses.

The Assessment Framework's assertion that spatial and temporal scales are the major variables determining whether or not EMAP can provide useful assessment data may not be correct. The usefulness of retrospective data in risk assessment at any scale varies with (1) the severity of risk, (2) the probability of detecting the effect (related to statistical power of sampling procedure), and (3) the time lag required for a mitigating response:

Application of this simple model to a list of real or potential environmental risks (Table 2-1) suggests that there are many risks for which retrospective analysis is appropriate, two for which it may

be appropriate, and many for which it is inappropriate. (These examples are ways to think about appropriate monitoring strategies, individual areas might need more detailed analyses.) It is noteworthy that many high-profile environmental risks fall into the latter category. The EMAP approach is no panacea, and it is important that claims by the program and expectations of the public be realistic. In general, when the probability of detecting an effect is high and the cost of failing to detect is not extremely large, EMAP's effects-based monitoring can provide useful input to the Ecological Risk Assessment process as indicated in the Assessment Framework document (Thornton et al., 1994). However, when the cost of failing to detect an effect early is high traditional predictive risk assessment, which emphasizes stressor monitoring and modeling, is a clearly preferable strategy.

The practical usefulness of EMAP's retrospective monitoring design depends on achieving a sufficiently high probability of detecting ecologically important effects. As discussed elsewhere in this and in previous reports of this committee, EMAP's current design might not have sufficient statistical power to detect ecologically important changes. The program's data-quality objective of detecting a change of 2 percent per year over 10 years in the mean of an indicator across a standard federal region can be achieved (ASA Committee on EMAP, 1992). As discussed later in this chapter, the pertinent scales for ecological processes are considerably finer, however, and the usefulness of EMAP will depend on its ability to detect these smaller scale changes. EMAP should consider design augmentations to increase the probability of detecting smaller scale ecological signals. Some possibilities include:

• Increased revisitation rates at a subset of sample sites.
• Inclusion of a set of nonrandomly selected sentinel-sites with intensive data-collection, such as the Long-Term Ecological Research network.

• Stratified random sampling by ecoregion with data-quality objectives specified for strata.

The Assessment Framework addresses the issue of enhancing the probability of detecting effects by calling for selection of indicators that are linked to specific environmental values. This has been termed ''stressor-cognitive" indicator selection by EMAP staff. In a sense the EMAP indicator development strategy leans towards including some aspects of a prospective assessment, or stressor-oriented, approach as well. In some of the resource groups, e.g., estuaries, this approach has been strongly implemented, with some of the indicators chosen to detect the most probable and common stresses (organic loading and toxic contamination). EMAP should consider further developing this hybridization of effects-oriented and stressor-oriented monitoring approaches. By focusing on indicators that are sensitive to the most likely known stressors for each resource group, an increase in the probability of detecting meaningful ecological changes may be achievable within any given sampling program.

There remain issues of the EMAP design and assessment approach that have not been adequately addressed such as: the ability to detect changes at an appropriate scale, whether the sampling return period is adequate to detect cyclical events, and the efficacy of specific "stressor cognitive" indicators. EMAP should continue to evaluate features of the design in the light of its ability to detect important ecological change in a meaningful, timely, and useful manner.

A fundamental premise of EMAP is that the status of large and complex ecological systems can be assessed and monitored using a limited set of indicators. Choosing appropriate indicators has been a major focus of EMAP activity since the program began. Despite the obvious centrality of indicator development to EMAP,

the completion of a comprehensive indicator strategy document has been slow in coming. An early version of a strategy document (Olsen, 1992) was withdrawn following a major program reorganization. A new indicator strategy document was developed more recently and was received by the committee in the spring of 1994 (Barber, 1994).

A strong reliance on biological measurements is highly appropriate for a monitoring program with EMAP's goals. Because environmental managers increasingly emphasize issues of biological integrity and integrated approaches to watershed and ecosystem management, environmental monitoring programs can no longer rely solely or primarily on measuring physical and chemical conditions of ecosystem quality.

In contrast to chemical indicators, which tend to reflect short-term or instantaneous conditions, biological indicators integrate conditions over time. This attribute is especially important for a monitoring program in which resource units are sampled at most once per year. In addition, advances in the development of multispecies biological indices using multivariate statistical methods offer some hope of defining and quantifying certain aspects of ecosystem status, and perhaps will allow useful quantification of the still nebulous concept of appropriate biological diversity, and the even more indeterminate concepts of ecological health and biological integrity.

Exciting advances offer opportunities for development of innovative, molecular-level measurements of ecosystem functioning and responses to stress. Examples include the use of genetic markers (gene probes) to detect the presence of certain organisms or types of organisms, molecular indicators to detect the exposure of organisms to classes of toxic compounds, and molecular measures of biological functions. Moreover, EPA needs information about biota in the nation's ecosystems that goes beyond questions of biological diversity and structural aspects of biological integrity. For example, there is a long-term need to test biota for levels of toxic contaminants, not only because of human health issues but also because of concerns about the health of animals that accumulate contaminants from lower levels of the food-web (e.g., DDT and birds of prey).

On the other hand, there are several serious difficulties that must be overcome before EMAP is able to place more reliance on biological indicators in its long-term monitoring and assessment strategy. Many of these problems have been recognized by EMAP scientists in various documents discussing selection of indicators for specific resource groups (see Appendix A) and in the recent indicator strategy document. The following are among the most important of these difficulties. First, virtually every characteristic of biological integrity one can think of has an extremely wide range of "appropriate values," depending on the nature of the ecosystem being considered. There are few, if any, absolute biological characteristics that identify an ecosystem as healthy, pristine, or undegraded. Beginning in 1991, the EMAP-Estuaries pilot study used an index that is compared to a local reference system (NRC, 1994a). This approach could be extended to other parts of EMAP. The extent to which a system is degraded is decided by comparison with similar systems. Second, biological variables do not respond in simple or linear ways to stress, creating special statistical difficulties in developing indicators for them. Third, many community and ecosystem-level measures of ecosystem function are quite insensitive to stress. Fourth, from a practical perspective, taxonomic identifications can be tedious, time-consuming, and expensive, thus perhaps limiting the number of samples that can be measured.

Finally, the most crucial problem is that for most ecosystems there are no quantifiable biological indicators of ecosystem health, biological integrity, or of several other societal values associated with ecosystems. A major research and development effort will be needed before workable, reliable, and cost-effective measures are available. To date, EMAP has not developed such an aggressive and comprehensive research program for development of biological indicators, and this is perhaps the most important research need facing EMAP.

Given the scope of the problem, EMAP should initiate a major, focused research program on indicator development. This is a formidable scientific challenge. If it can be successfully met, it will need the involvement of many scientists for a period of at least 5 to 10 years. It should include a combination of research by EMAP scientists and external research that involves open

announcements of funding availability with peer reviewed grants. The difficulty and importance of this research requires that EPA attract the highest quality researchers in the environmental sciences to this program. Some of the needed research and development can be accomplished in association with existing pilot monitoring activities, but fundamental research also will be needed that cannot be addressed by sampling at the spatial and temporal scales of the pilot programs. As part of this indicator development program, scientists will need to study the relationship of the indicators being considered to assessment end points and the statistical properties and power of potential indicators. The research program should be directed not only at development of biological indicators appropriate for the various resource groups but also at applying new advances to develop new measures of ecosystem function as well as ecosystem structure.

EMAP has developed a probability-based sampling design to address certain questions regarding the status and trends of ecological resource populations of interest in the United States. EMAP focuses on regional surveys rather than a "sentinel- site" approach; an EMAP objective is to select samples that are spatially well distributed so that the results will apply to fairly large areas. EMAP is willing to forego intensive, site-specific monitoring data in return for reliable data on regional changes and trends in regional population statistics (Messer et al., 1991). According to Foley (1994), assessment of temporal trends using sentinel-site-based monitoring will be left to "other ORD [EPA's Office of Research and Development], federal, and academic programs, because according to EPA, site-based information alone is insufficient for detecting `meaningful trends at scales relevant to policy decisions'."

The EMAP hexagonal, grid design does make it possible to sample at varying spatial densities such that the base grid is a subset of higher density grids (Stevens, 1993). Sites will be revisited every four-years. EMAP claims that the 4-year interval is consistent with the time scale of trends that EMAP must de-

tect. "Trends that result in immediate extreme changes will be reflected in annual estimates, by other monitoring programs, or by casual observation. However, faint trends require some time before the cumulative change is detectable, and as great a population coverage as possible is needed in order to isolate subpopulations that may respond differently than others" (Stevens, 1993, p. 20).

A "large-scale, regional estimates only" approach is not compatible with one that claims to use the same spatial grid density to isolate sensitive subpopulations. By enhancing the grid density, such subpopulations might be identified and monitored. But they would have to be identified in advance by some other program or programs outside the EMAP sampling frameworks, because monitoring an enhanced grid at a regional or national scale would be prohibitively expensive. For instance, the selection of indicators of biological condition for the various resource groups may be driven more by what can be detected in four-years' time than by their ecological relevance. Regarding the issues of detection of temporal or spatial processes outside the current EMAP sampling frameworks, EMAP claims that other federal programs (e.g., LTER, LMER) and state and regional programs including the Regional Environmental Monitoring Assessment Program (REMAP) are really the programs designed to investigate those issues. In that case, the coordination between EMAP and other more site-specific programs should be greatly enhanced.

Initially, EMAP had planned to augment the interpenetrating, 4-year-cycle design by sampling some sites annually; the contribution of this added component is described by Urquhart et al. (1993). Annual sampling during the initial years of the survey was mentioned specifically for lakes and streams as a way to increase the power of the EMAP design to detect trends (Larsen et al., 1993). As a result of further simulation studies, this augmentation sampling plan apparently was dropped and no lakes will be revisited annually for more than two consecutive years. This has the consequence of generating no site-specific information for longer than two years; the EMAP claim is that more information is to be gained by sampling more sites than by extensive repeat sampling at a single site (Appendix A).

The amount of statistical work assessing the effectiveness of the overall EMAP sampling design varies greatly among the different resource groups. For example, much work has gone into assessing the effectiveness of the EMAP-Surface Waters sampling design for lakes. The inclusion probabilities for the sampling design have been adjusted to yield an adequate sample of large lakes. Power analyses, sample size calculations, least significant trends, and investigation of variance components have been done for certain chemical and physical responses (Urquhart et al., 1993; Stehman and Overton, 1994). The EMAP lakes component has benefited from having teams of statisticians working specifically on assessing the effectiveness of EMAP for that particular resource. Similar work should be conducted in the other resource groups.

EMAP-Estuaries implemented a stratified sampling design in pilot demonstration projects to ensure adequate representation of small estuaries and tidal rivers, large tidal rivers, and large estuaries. EMAP for streams will not use the grid sample, but a probability sample where streams are treated as a discrete resource. EMAP-Surface Waters focuses on the population of stream miles rather than stream reaches; thus streams are sampled in proportion to their lengths. EMAP needs to make clear how such a sampling scheme will be compatible with a watershed approach to sampling. For example, it is not evident how stream sampling will be coordinated with the sampling of surrounding forest, or if the Forest Health Monitoring program is following the EMAP grid.

With respect to agroecosystems, the North Carolina pilot study compared the sampling design of the National Agricultural Statistical Survey (NASS, a well-established program) to the EMAP sampling design. The conclusion was that the EMAP sampling design is no more cost-effective than the NASS sampling design. This experiment is being repeated in the Nebraska agroecosystem pilot, but no reports have been released. Ultimately, the EMAP agroecosystem sampling design is expected to coordinate with the NASS design by augmenting the sampling grid and data requirements of NASS (further details in Chapter 4).

The statistical power to detect spatial or temporal differences (or even to answer specific questions regarding status) raises concerns because of economic and other practical constraints on

the number of samples imposed on EMAP. For example, to obtain the acidification status of lakes in the Northeast, the original EMAP grid had to be enhanced beyond the base design. The sampling approach varies among the different resource groups; for this reason power analyses should be applied to each resource group to analyze the effectiveness of the sample design.

EMAP will use cumulative distribution functions to display information in the annual statistical summaries. This approach does not focus on a single parameter, like a mean or median, but instead allows a compact display of an entire probability distribution for an indicator of interest. Changes in inherent variability, or in the tails of the distribution (e.g., the 90th or 10th percentiles), can be diagnosed using cumulative distribution functions, which will also have confidence bands around them. With increased numbers of samples, the level of confidence will not change, but the width of the confidence bands will become smaller. The power analyses that the committee has seen thus far deal with changes in the mean, and future power analysis research assessing the effectiveness of EMAP designs should pay attention to more extreme quantiles, like the 10th or 90th percentiles. This is especially important in light of the statement that confidence bounds obtained using the Horvitz-Thompson variance estimators (part of the statistical underpinnings of the EMAP grid design) are based on normal approximations. These normal approximations may be inadequate, even for moderate sample sizes, for estimating confidence bounds at the tails of the distribution (Lesser and Overton, 1994). For environmental monitoring objectives, the tails of the distribution may be where much of the interest lies for certain indicators.

EMAP now seems to have an extensive cooperative statistical research program, involving statistics departments in at least eight universities, and with at least twelve principal investigators. It is to EMAP's credit that many papers regarding the statistical underpinnings of this complex effort are now available in peer reviewed literature.

Statistical analyses have varied according to the needs of the various resource groups. Nonetheless, every sampling site will generate multivariate data when sampled for various indicators. EMAP has considered the development of indices as a way of condensing multivariate data into single numbers for easier understanding and easier analyses. Rather than relying exclusively upon an index with unknown statistical properties however, it is possible to use the vector of original responses and apply multivariate statistical techniques for analysis, or exploratory techniques involving visualization of multi-dimensional data (e.g., Becker et al., 1987; Cleveland and McGill, 1988). Nonparametric multivariate techniques to ascertain differences among groups of data points and to detect trends in multivariate data also exist (e.g., Zimmerman et al., 1985; Saila, 1993). In response to a recommendation from this committee that nonlinear types of trends be investigated (including threshold effects and step functions) (NRC, 1994a), nonlinear trends are being included in the statistical research (Foley, 1994). There is value in using a multipronged data analysis approach, as one can have more faith when similar conclusions are derived from independent statistical methods.

In analyzing its ability to detect temporal trends, EMAP has used as its criterion for success the ability to detect a monotonic trend of 20 percent over 10 years. EMAP should consider in detail and in depth the conditions under which this criterion is likely to be useful, and those conditions under which it might not be meaningful. Are there some systems or some indicators in which a change in value of 20 percent over 10 years is so rare that it will likely never occur, or in which one might be extremely concerned about a much smaller change? Included in this analysis should be consideration of systems or indicators for which a change in the variance rather than location parameters (e.g., mean, median) is important. Measures of central tendency are insensitive to some types of changes in living systems. In studies of human and animal behavior for example, changes in distribu-

tion of reactions in the face of stress can be striking even while the mean or median remain more or less unchanged. While information on variability is contained in cumulative distribution functions, the power analyses thus far have concentrated on trends in the mean.

For example, a monitoring program that could detect only a 20 percent or greater change in mean annual surface temperatures in the United States, or in tropospheric ozone, or in human age-specific mortality rates, would not be sensitive enough to be useful to policy makers. Similarly, a 20 percent change in the visibility of lake waters over 10 years on a regional basis is much larger than one might reasonably expect to occur, and policies will need to be made before such a change can be detected. On the other hand, a 20 percent change in dissolved oxygen in estuaries or harbors appears to be small enough to be possible and important enough to be worth detecting.

EMAP will collect data on a large number of grid points. The status and trends of condition of ecological resources can be reported on these or any other level. Some aggregation of data is required to yield the means and confidence intervals desired. EMAP intends to use Standard Federal Regions (SFRs) as the primary scale for summarizing data and inferring the status and trends of the nation's ecological resources.

Standard Federal Regions have a few advantages for data analysis and reporting. The major advantage is that EPA is administratively organized according to these regions. Summarizing EMAP data accordingly will facilitate its use by EPA field personnel as well as administrative personnel in Washington. In some cases, summarizing data by SFRs may help reporting to congressional delegations, particularly when congressional delegations themselves are organized accordingly. For example, New England is in SFR 1 and there may be times when EMAP wishes to report to the New England congressional delegation on the status and

trends of ecological resources in that region. However, not all congressional delegations are organized by SFRs.

In contrast, there are a number of serious ecological disadvantages and additional administrative difficulties in summarizing data by SFRs. First, these regions do not correspond to scales of relevant ecological processes. Ecological processes tend to occur at scales typical of first- or second-order watersheds, insect outbreaks, fires, and the like. This means areas of tens to thousands of hectares (typically best represented by maps at scales of 1:1,000 to 1:20,000). Therefore, summarizing data at the scale of SFRs—hundreds of thousands of square kilometers—will often obscure problems that arise at the ecosystem level. Second, the data-quality objectives set by EMAP SFRs will not be achieved at the smaller scales relevant to important ecological changes. Third, the boundaries of SFRs are not congruent with the boundaries of ecological units such as communities, ecoregions, or biomes. Ecological boundaries are generally determined by biological responses to geomorphic or climatic boundaries, not political boundaries. Also, SFRs are often not ecologically homogeneous; some western regions, for example, each include prairie, desert, montane forests, and alpine tundra.

An additional difficulty is that other federal agencies, the U.S. Forest Service, Soil Conservation Service, U.S. Fish and Wildlife Service, National Biological Service, and the National Park Service, are not organized by Standard Federal Regions as is EPA. Therefore, summarizing data in this way will impair information transfer to other land management agencies.

Some of the problems of reporting data on SFRs can be illustrated by the EMAP Forests data summaries. The 1992 Forest Health Monitoring Statistical Summary reaches the following conclusions:

• SFR 1 (CT, ME, MA, NH, RI, VT) and 2 (NJ, PR, VI) combined have more dead trees (on a per-area basis) than either SFR 3 (DE, DC, MD, PA, VA) or 4 (AL, FL, GA, KY, MS, NC, SC, TN).
• Species density was used as a measure of species diversity of trees and saplings in SFR 1 and 2 combined, SFR 3, and SFR 4. Two species per unit area were used as a preliminary suboptimal threshold.

• SFR 4 had a significantly higher proportion of plots with suboptimal tree species density than SFR 3. SFR 1 and 2 combined and SFR 3 did not differ significantly for these proportions.
• SFR 1 and 2 combined had a significantly higher proportion of plots with suboptimal sapling species density than SFR 3 or 4.

The regions being compared are made up of widely different biomes and ecosystems, and so the comparisons are not informative. The potential reasons for the first finding are legion: Perhaps trees decay faster in the South, SFR 4; Perhaps forests in SFR 1 and 2 are, on average, older than those in SFR 3 or 4; Perhaps there is a higher pollution level in SFRs 1 and 2 than in SFR 3 or 4. This finding will be reason for alarm to a few, an interesting conundrum to many, and an example of ''federal fog" to others. Such generalized conclusions are not good for EMAP nor will they engender much support for environmental monitoring.

The second and third findings are also problematic. Southern pine forests, which predominate in SFR 4, are naturally comprised of few species, but it is doubtful that they are in suboptimal ecological condition when compared to more species-rich high-graded woodlots, which predominate in parts of SFRs 1, 2, and 3. Likewise, climax forests in portions of SFR 1 and 2, which are in potentially optimal ecological condition, often have understories dominated by a single species—sugar maple. Broad generalizations such as findings 2 and 3 above are misleading, and by presenting easy targets for criticism, they will make it more difficult for EMAP to reach its objectives. The fourth finding is also problematic because optimal sapling density depends on species composition of specific stands, the ratio of ingrowth from seedling to sapling classes, mortality over all size classes, and the management objectives to be realized over a given planning horizon. Optimal sapling density is meaningless at regional scales.

The committee recognizes that the EMAP sampling design is flexible enough that data can be summarized in many different ways. However, the program's data-quality objectives pertain only to the primary scale for summarizing and reporting data, the SFRs. It is important that EMAP choose ecologically meaningful units as the primary scale on which to base data-quality objec-

tives and for summarizing and reporting data. Ecologically meaningful units, such as Bailey's or Omernik's ecoregions, should be the primary objects of statistical analysis and data reporting. Summarizing data at this level will facilitate the development of meaningful hypotheses regarding causes and effects. Summarizing data by ecoregions also will allow users to investigate the geographic pattern of responses to stresses in a way that lends itself to further investigations. For example, is a particular ecoregion responding to a regional or global stress such as acid deposition or global warming in the same way throughout its range? If not, why not? Such questions are meaningless if the data are primarily summarized by SFR. Ecological data that are focused on finer scales could be summarized for SFRs secondarily, and the data would then meet or exceed EMAP's data-quality objectives.

The Regional Environmental Monitoring and Assessment Program (REMAP) operates at scales and with boundaries more appropriate to ecological processes. REMAP was not a part of the original EMAP concept, but it has become a potentially significant contribution. It uses EMAP indicators and the EMAP probability-based sampling scheme at an intensified sampling density to address local problems. EMAP should reconsider the scale at which the national program collects and reports data.

As the first national scale, multiresource monitoring program, EMAP represents a potentially significant addition to the myriad environmental monitoring programs run by other agencies. Two aspects of EMAP make it unique: the probability-based sampling design, and the inclusion of all of the biomes in the coterminous United States. In initial EMAP literature, this second aspect, inclusiveness of all resources, was the major justification for a new environmental monitoring program. The committee believes that integration among EMAP resource groups is crucial for the following reasons.

• To a large extent, major trends within resource types may only be explained by interactions among resources. For example, most pollution problems occur across resource groups defined by EMAP. An explanation of a forest's condition may require the assessment of regional, urban or agricultural practices as they influence nitrogen gas emissions. Virtually every national environmental issue to date has involved such connections. Integration among resource groups explains trends in ecological status that are controlled by spatially explicit source-sink relationships.
• A large, national, cross-resource monitoring program could lead to important new advances in our knowledge of interactions among resources.

EMAP has given several indications that it plans to carry out integration among resource programs in the future. The committee has been informed that the resource groups are developing plans to use common methods. The fact that EMAP has or is developing a central sampling design, indicator development strategy, and information management system also suggests that EMAP is planning for future integration. Additionally, the presence of EMAP Center as a central administration provides support for integration. Perhaps most promising, the new Integration and Assessment Program suggests that major new advances can be made with the integration of the program.

The committee recognizes that a large, complex program like EMAP must continue to develop and evolve, and that this evolution is a process of iteration between the conceptual, integrative elements (top-down directives), and the empirical, on-the-ground, resource group elements (bottom-up guidance). The top-down directives of the program appear to have weakened considerably since this review began. In 1990, EMAP representatives stated that the Landscape Characterization program and the Indicator

Development Strategy were the foundations of the integration program for EMAP. Since that time, the Landscape Characterization program has been changed substantially, such that now it appears to be focused entirely on land cover mapping. The original elements that provided integration appear to be gone. These elements included guidance to resource groups on appropriate sampling intensity and locations, and analysis of indicators across resources. Finally, the technical support for cross-resource integration—the information system—still has not been fully developed. In sum, most of the positive aspects of integration listed above are not yet being implemented.

The groups are now sampling in separate locations resources that would appear to have very important connections because of their hydrologic linkages. The design would be much enhanced by a cooperative sampling scheme among resource groups in which lakes and streams were sampled in watersheds whose terrestrial systems (forests, agroecosystems, or arid systems) also were being sampled. A stratified random system such as this

would not compromise EMAP's ability to make regional scale generalizations, or to base those generalizations on probability-based samples. The data sets would be considerably stronger because the spatial covariance of the data sets could be used to test hypotheses related to cause and effect relationships. Several decades of measurements may be required to test these relationships on temporal data. In addition, the development of integration and assessment questions that crossresource groups could focus the program and significantly extend its value nationwide.

As EMAP now stands, there are relatively few financial and human resources directed specifically at integration. Such resources could be directed in various ways, but several important needs must be met. Programs directed at integration must have access to the information management system, and must have computer and software resources to generate and test generalizations.

One approach would support a team of individuals that focused on developing and addressing the integration and assessment questions. The team would either have one physical location or would be coordinated among resource groups by a central office. Key members of this group would include participants of the Landscape Characterization, Landscape Ecology, and Indicator Development groups. The new Integration and Assessment Program is a positive step in this direction, but there is no description to date of the activities of this program.

A second approach would be to provide the funding for Request for Proposals to be extended to the scientific community. This approach could be implemented in a number of ways, either by targeting specific assessment questions, or by allowing the scientific community to develop the more general questions of most interest to EMAP. An intriguing possibility would include EMAP support (both financial and data support) of the new National Science Foundation Center for Ecological Analysis and Synthesis. The center is intended to serve as a resource for the

ecological community at large, and accessibility to EMAP data and technology would likely provide a great incentive for cross-resource analysis, assessment of ecology-policy links, and analysis of long-term and large-scale ecological status and trends.

EMAP has improved its internal coordination over the past several years. The creation of EMAP Center has resulted in a concentration of technical personnel who provide support to the resource groups. During the reviews of the Surface Water, Forest, and Estuary programs, and during meetings with the NRC committee, EMAP staff commented favorably on coordination between EMAP Center and the resource groups. The recent publication of a newsletter facilitates communication among all components of EMAP as well as those outside of EMAP. None-theless, improvement is needed.

Several features of EMAP inhibit internal coordination. EMAP personnel are dispersed in laboratories and offices across the country. Personnel in each resource group are typically clustered together, but they are separate from other resource groups and from EMAP Center. In addition, the autonomy of resource groups appears nearly absolute. The combination of geographic separation and administrative autonomy makes internal coordination difficult at best. The committee recommends that EPA consider the advantages of assigning all EMAP administrative personnel to a common location.

Report review within EMAP appears to be cumbersome. Reports delivered to the committee for review are typically based on information gathered two or more years previously. During the early stages of a new program, it is especially important to learn from experience and to modify the program based on the lessons learned. It is not clear that experience is used in a timely and efficient manner to improve subsequent work. Possibly, the slowness of report reviews is at least partially responsible for such inefficiency. EPA should consider means by which report generation and review can be speeded up. Centralization of personnel, reports in peer reviewed journals, use of external review-

ers, reliance on scientific criteria for reviews, and reduced sensitivity to being wrong and to the prevailing political philosophy are potential means to shorten the time for report review.

The rate of turnover in personnel appears high in comparison with other scientific programs and activities. EMAP has had three directors, three associate directors are leaving, and several resource and support groups have had multiple leaders. Some of this turnover is inevitable as positions are filled by contract employees with statutory limits on service. For example, the director and the coordinator of the Agroecosystem Resource Group serve under term-limited Interagency Personnel Agreements. Such turnover further disrupts coordination among EPA personnel. The committee questions whether contract employees can be as effective as direct employees in facilitating coordination within EPA. Individual relationships are the basis of collegiality and collaboration. Relationships between EPA employees and term-limited contract employees are frequently disrupted, inhibiting coordination. All federal agencies have complex hiring requirements, but EPA must address these issues. EPA should commit the senior-level positions required to assure continuity within the management of this important program and recruit qualified people to fill them.

Through its REMAP program, EMAP has demonstrated excellent coordination with its regional offices. Representatives from regional offices report that they have found the EMAP design and staff useful in assessing issues within the regions. EMAP should be commended for its efforts in finding the means to interact with its regional offices for mutual benefit.

EPA senior management should facilitate close working relationships between EMAP and appropriate offices and divisions of EPA. For example, EMAP should continue its efforts to develop close working relationships with the Water Office of EPA to capture the benefits of EPA's past experience in collecting data on surface waters. In addition, EMAP should coordinate closely with other research programs in the Office of Research and Develop-

ment. EMAP's partnership with the Exploratory Research Program in administration of the solicitation for proposals for indicator development has been productive. Continued reliance on the experience of such programs leverages EMAP's resources and brings complementary expertise to the program.

EMAP is attempting to carry out a number of approaches that are new to ecology, including using indicators of ecological status or health, developing and applying indicators in many types of ecosystems, and making measurements over an entire continent. All of these novel approaches require the best possible judgment from the most experienced researchers. Is EMAP receiving the external scientific review it needs to carry out its mission?

All parts of EMAP have brought in external reviewers at various stages of planning and implementation. For example, the indicator development program held extensive workshops of EPA contractors and independent scientists to come up with ideas about possible indicators. Panels of experts have advised the various resource groups, and the EMAP reports have received extensive review by up to 100 reviewers, many of them from outside of EPA. One recent activity, an EMAP-initiated funding of projects to develop new and better ecological indicators, used a National Science Foundation panel for scientific review and ranking of the proposed projects.

One successful use of external reviewers was carried out at the early stages of EMAP-Estuaries. At the request of EMAP, the Estuarine Research Federation (ERF) set up an expert panel that was briefed by EMAP and produced reports based upon three meetings held a year apart. The ERF panel and EMAP scientists met at the end of each year to review progress and discuss problems. Participants report that the advice from the ERF panel was taken seriously. Several years later, the estuaries panel of this NRC committee found that the ERF reports dealt very well with the strengths and weaknesses of the EMAP program.

EMAP has been willing to use external reviewers more than any other agency program. Nonetheless, the review process

could be improved. First, it is important to use external reviewers as advisors as well as reviewers. The advice of these external reviewers should be used in designing parts of the program and its overall structure, as well as in reviewing finished or partly finished activities.

Second, at least in some cases, the reviewers could have been chosen better. For example, however skilled and knowledgeable EMAP contractors and employees might be, inasmuch as they have helped to design or implement the program, they are not independent of it, and therefore they should not be used as reviewers. Also, there is value to having reviewers who are active in other, related fields and whose funding and scientific publications indicate that they are leaders in developing the scientific research and technologies that EMAP needs.

Third, EMAP seems to have convened a new review panel for each task. Given the complexity of the EMAP program, it is a major task to bring each panel up to the proper level of understanding. Permanent or long-term scientific review panels for each of the eight resource groups were a part of the EMAP plan, but have not yet been set up. In addition to the need for good external review of these various resource groups and funding initiatives, there is also a need for overview of the entire program that is not filled by reviews of each separate part.

The current external review structure of EMAP should be modified so that its core is a permanent panel, with rotating membership, to provide continuity. This panel would advise both at the level of resource groups, such as the arid land or estuary resource level, and at the level of the entire EMAP program. The advice will be taken more seriously if good relationships are established between members of the panel and the directors of the various sections of EMAP. Above all, best scientists have to be recruited as advisors. Advice about membership should be sought from scientific societies, as was done with the Estuarine Research Federation, and from the program managers at the National Science Foundation, the Department of Agriculture, and the Department of Energy who organize competitive research programs in ecology. A structure for permanent advisory panels was proposed by EMAP personnel but has not been implemented.

EMAP is a large program that will require an appropriate administrative setting to be successful. It is important to reflect on the extent to which EPA has provided an appropriate administrative environment for EMAP.

Characteristics of the ideal administrative setting that would enhance EMAP's chances of success are described below. Of course, no agency has all these characteristics. Indeed, some of the characteristics are somewhat incompatible with each other. Conversely, some of the characteristics are related to each other—the first two, for example.

• EMAP's parent agency should be nonregulatory. A monitoring program housed within a regulatory agency will face problems of internal conflict of interest changing priorities, and data confidentiality. This may lead to a higher rate of denial of access to private lands. In addition, the need for short-term information for regulation might compromise the ability of the agency to commit resources to long-term monitoring programs.
• The agency needs to make EMAP one of its highest priorities internally and in its presentations to Congress. Large fluctuations in funding will seriously damage a program such as EMAP.
• The agency should have a strong administrative and scientific team capable of providing the initiative and scientific leadership required for such a large and highly visible program. The agency should make the commitment of long-term as well as rotating positions for key leadership and scientific advisory personnel.
• The agency should have strong familiarity with each of the resource types being monitored.
• The agency should have the capability to carry out strong research programs affiliated with EMAP to answer detailed questions raised by EMAP data.

• The agency should have a strong scientific reputation, making it easier to attract top scientists to EMAP.
• The agency should be in close communication with agencies that will administer policy derived from data collected.

EPA clearly has several of these characteristics. EPA scientists conceived the concept of EMAP and were successful in initiating and implementing its predecessor, the National Surface Waters Survey. Certainly, EPA will have some regulatory responsibility in enforcing policy derived from EMAP data, and lines of communication between monitoring and regulatory personnel should be strong within a single agency. EPA in its regional offices has started the promising Regional Environmental Monitoring Assessment Program to address specific questions using an EMAP-like approach. Close ties between REMAP and EMAP personnel will be mutually beneficial.

On the other hand, EPA does not have some of the ideal characteristics. For some of this there may be little EPA can do. For example, EPA is a regulatory agency. That regulatory role was mandated by Congress. Although EPA cannot resolve all the conflicts associated with its regulatory role, it can take steps to reduce their impact, as other agencies have done, by separating research and monitoring functions as much as possible from regulatory ones. There are several other areas where EPA could act to improve the administrative setting of EMAP. As much as possible, funding for EMAP needs to be long-term and predictable. Fluctuations or uncertainty in funding levels will be seriously damaging to a monitoring program this large and ambitious. Within the constraints imposed by the congressional funding process, EPA must act to ensure the institutional commitment to this long-term monitoring effort. EPA should find some mechanism to allow flexibility in timely hiring of qualified personnel at all levels of EMAP. It appears that difficulty in hiring and keeping personnel in key positions has hindered EMAP's progress. While some of the difficulties in hiring personnel are beyond EPA's control, high enough priority may not have been given within the agency to filling key positions, such as the Indicator Coordinator. Finding ways to attract and keep top scientists in EMAP will

strengthen the program overall and lead to an enhanced scientific reputation.