|
Items in cart [0] |
Questions? Call 888-624-8373 |
PAPERBACK PDF BOOK Free Resources |
||||||||||||||||
|
||||||||||||||||
|
|
|||||||||||||||
| ||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 111
4
Common Issues
THE COMMITTEE'S EVALUATION of the 11 ecological studies that were in-
cluded in the Navy's ELF ecological monitoring program revealed several
issues that were common to many or all of the studies. Those common issues
are discussed here. This chapter was not intended to discuss each study in the
context of each common issue. Specific studies are discussed below for illus-
trative purposes. Overall conclusions and recommendations concerning the
common issues are presented in Chapter 5.
USE OF EXPOSURE DATA BY
ECOLOGICAL MONITORING TEAMS
All ecological monitoring teams made use of the division between treat-
ment and control sites. All pairs of sites satisfied the criteria for 76-Hz expo-
sure except the aquatic study sites, for which the ratio of electric fields in the
earth at treatment sites to those at the control sites did not quite satisfy the
criteria (see Chapter 2~. That was rectified by inclusion of supplementary sites
closer to the antenna in 1990. About 10% of the study sites did not satisfy the
criterion for 60-Hz exposure, but in all cases the 60-Hz exposures were very
low. From an ELF-EMF exposure point of view, the pairing of sites into
treatment and control was satisfactory. IlTRI was successful in characterizing
the ELF EMFs for this purpose.
1 1 1
OCR for page 112
2 EVALUATION OF ELF ECOLOGICAL MONITORING PROGRAM
In some studies, it was important to know whether the transmitter was
on or off during critical "exposure" periods. Those were the studies that used
response variables potentially sensitive to ELF-EMF exposure for only short
periods. For example, three of the four major completed studies of small
vertebrates considered short-term phenomena: embryonic development (4 days
or less), homing (4-5 hours), and maximal metabolic rate (minutes to perhaps
24 hours). The fours study, assessing fecundity, took place over a period of
weeks to 2 months, depending on the year; the investigators' division into
treatment and control sites might not have corresponded to the actual expo-
sures, which were most likely time averages of antenna activity over that
period.
Without verifying that the transmitter was on during the experiments, it
is not possible to know whether a treatment site was exposed to ELF-EMFs
from the antenna. And without that knowledge, there could be miscIassifica-
tion of subjects from the control category into the treatment category. How-
ever, there is little evidence that most of the investigators were aware of or
considered this factor in evaluating the results of their experiments. Chapter
5 discusses possible reanalysis of the data collected through some studies.
According to the monitoring-program reports reviewed by the committee,
only the upland-flora team tried to use any of the other data provided by
UTRI, although some teams requested additional ELF-EMF `data. Investiga-
tors from the earthworm and soil arthropods study requested and received
information on electric field vs. soil depth but never used it. Extensive mea-
surements of the electric field in the earth were made at the Martel's Lake site
(overhead antenna treatment site for upland-flora and litter decomposition and
microflora studies) but never used. The authors of the pollinating insects
study indicated that each hutch received a different exposure, but there is no
indication of an attempt to use these data in the analysis. Some of the teams
might have decided that there was no difference in ecological aspects between
treatment and control sites and hence that there was no need for further analy-
sis with specific ELF-EMF exposure data. Sometimes, however, only the
consideration of additional exposure hypotheses will identify an effect. Such
extensive exposure data were available, and they should have been used. The
use of ELF-EMF exposure data in two specific studies is discussed here.
ELF-EMF CHARACTERIZATIONS AT WETLANDS SITE
In 1986 and 1987, the wetlands researchers conducted a series of
OCR for page 113
COMMON ISSUES
~3
stoma/al-resistance measurements on wetlands plants. The measurements were
related by multiple-regression analysis to the independent variables: two envi-
ronmental variables and the magnetic field and electric field in the earth result-
ing from the antenna system under full-power conditions as measured by
TTTRI. The purpose of this analysis was to determine whether there was a
statistically significant relationship between stomata! resistance and any or all
of the independent variables.
The researchers noted that the antenna on or off condition was neither
predictable nor observable. They correctly noted that this was a potentially
confounding factor in the analysis of short-term responses because measure-
ments might have been taken at a treatment site when ELF-EMF exposure was
not occurring. They did not pursue the analysis, because they found inconsis-
tent results under a variety of both similar and substantially different exposure
conditions.
USE OF MAGNETIC-FIEED INTENSITY
AS A "DOSE" AT UPLAND-FLORA SITE
In the upland-flora study, the researchers designated a 76-Hz magnetic
field as an indicator of dose to each tree. According to one of their papers
(Reed et al. 1993), the indicator was based on "average exposure to magnetic
flux density during that particular growing season." It does not appear that
the actual measure used in the study is consistent with this definition.
The measured fields across the treatment area (during transmitter opera-
tion at full power) varied from about 5-10 mG (see Haradem et al. 1994, p.
D-7, for location of points 4T2-6,7,S, 12,13,26,34 on the hardwood stand and
p. D-30 for the historical measurements at these points). However, the
upland-flora report shows magnetic-field measurements of about I-9 mG and
a specific "effect" on tree growth at about 2 mG. The researchers would have
been expected to define the growing period and use antenna on and off time
to derive an "average" field during this time. According to the researchers,
the growing period for hardwood trees was about April through September.
The antenna onetime statistics for each antenna configuration allow calculation
of the average magnetic field over the growing period. These are shown in
Table 4-~; the statistics behind the calculations can be found in Appendix ~ of
the final engineering report (Haradem et al. 1994~.
It is clear from Table 4-! that no trees were exposed to magnetic fields
within the range of 0.3 and 3.0 mG. Thus, the method described here is not
OCR for page 114
~ ~ 4 EVALUATION OF ELF ECOLOGICAL MONITORING PROGRAM
TABLE 4-1 Range of Average Magnetic Fields During the Growing Period
Over the Martels Lake (Overhead Antenna) Treatment Hardwood Stand
by Year
Effective Magnetic Field as Range of Magnetic Field
a Percentage of Full-Power for Any Tree within the
Year Magnetic Field Treatment Site, mG
1986 0.05 0.0025-0.005
1987 0.3 0.015-0.03
1988 2.1 0.1-0.21
1989 59.0 3.0-5.9
1990 92.0 4.6-9.2
1991 63.0 3.2-6.3
1992 84.0 4.2-8.4
1993 93.0 4.7-9.3
the method actually used by the authors to determine the exposure. Instead,
the authors used spot measurements made during operation of the transmitter,
which were provided by I]:TRI. Consider the points 4T2-6 (closest to the
antenna) and 4T2-7 (farthest from the antenna) on the hardwood-stand treat-
ment site. The measurements provided by UTR} and the number of hours that
each point was exposed at the measured levels during the growing season are
shown in Table 4-2, derived from tables in the final engineering report
(Haradem et al. 1994~.
Note that exposures that correspond to field measurements between about
1.2 and 2.6 mG are within the range shown in Table 4-2 only for 1991. That
is important because the claimed effect occurs at about 2.0 mG. Furthermore,
during that year, the exposure was more often in the range of 5.4-10.3 mG
than it was in the range of 1.6-3.0 mG.
The following observations can be made about this study. The indicator
of dose reported in publication of the upland-flora work was not that actually
used by the authors. They reported that the indicator of dose was defined as
"average exposure to magnetic flux density during that particular growing
season." The indicator actually used was based on spot measurements while
at least one antenna was on. The authors did not provide a clear rationale for
the indicator of dose that they used. Specifically, they were not clear about
OCR for page 115
COMMON ISSUES
~5
TABLE 4-2 Spot Measurements of Magnetic Field at Hardwood-Stand
Treatment Site and Number of Growing-Season Hours at These Levels
Total Time Exposed
Measured Measured During Growing
Magnetic Field at Magnetic Field at Season (4,392 Total
Year Point 4T2-6, mG Point 4T2-7, mG Possible Hours), h
1986 0.73 0.37 24.8
1987 1.16 0.59 142.8
1988 5.0 2.6 162.2
1989 10.3 5.4 2,390
1990 11.0 5.8 3,795
1991 3.0 1.6 1,462
1992 10.3 5.4 3,701
1993 10.3 5.5 4,073
Notes: In 1986, there were 24.8 h of operation at levels indicated and 17.0
h of operation In which measured levels were 0.44 and 0.22 mG. It is not clear why
values In Me table were used, rather than those reported here or zero, which was by
far the most-common exposure level during growing season.
In 1991, there were 1,462 h of operation at levels indicated and 2,336 h of
operation In which measured levels were 10.3 and 5.4 mG. It is not clear why values
In the table were used, inasmuch as those reported here were more common during
growing season.
why they used particular values of magnetic field rather than others when
more than one was reported during a given year. The measurement values
between about I.2 and 2.6 mG all come from a single year (1991~. The
authors' conclusion that there is an effect on tree growth at about 2.0 mG is
not warranted unless they define their field measurement more carefully,
provide a clear rationale for it, and find consistent results from more than one
growing season.
CONCEUSIONS REGARDING USE OF EXPOSURE DATA
Exposure data were inadequately or inappropriately used in a number of
studies. The studies Hat used short-term response measures should have used,
OCR for page 116
~ ~ 6 EVALUATION OF ELF ECOLOGICAL MONITORING PROGRAM
but did not use, transmitter on and off times to determine exposure, in addi-
tion to the I]:TR} division of sites into treatment and control. With a few
exceptions (the upland-flora study and the wetlands study to a lesser degree),
the ELF-EMF data provided by TTTR} were used in site selection but not in
any way to attempt to establish an exposure-response relationship. Those data
should have been used more than they were. In the one case of a search for
an exposure-response relationship, the data were used in a way that was not
clearly related to actual exposures over the duration of the growing period.
Thus, without further justification, any conclusions about low-level magnetic-
field effects on tree growth are not warranted.
UTR! was successful in generating data sets on exposure to ELF EMFs
for each study at the two antenna facilities. As far as the committee knows,
the data sets were made available to researchers, and additional data were
generated at particular study locations on request. It is not clear, however,
whether UTR] followed up the generation of the ELF-EMF exposure data with
assistance to each monitoring study in using the information. And when the
researchers began to state their findings in annual reports or at the annual
meeting, it is not apparent whether TTTR] understood the extent of use of the
exposure information by each study. The committee's evaluation of the sepa-
rate studies indicates that use of the exposure information was often inade-
quate, inaccurate, or inappropriate. That should have become obvious to
UTRT and external reviewers when they read or listened to reports from each
study each year. The committee wonders why more guidance was not given
to each study investigator in using the exposure information to ensure that the
response results reported in the final reports of each study were based on
appropriate use of the exposure data. ITTR} should at least have required that
an EMF-exposure expert work closely with each study until the study leader
understood the types of data available, their variability (considering the vaga-
ries in antenna operation), and how they might be applied to gauge responses
of the selected ecological or biologic variables.
STUDY-SITE SELECTION
The original request for proposals (RFP) for research regarding the
effects of ELF EMFs on biologic systems emphasized selection of study sites
so that appropriate levels of ELF EMFs existed at control sites versus treat-
ment sites. Considerable attention was also paid to the importance of match-
ing sites so that any major differences uncovered would indeed reflect re
OCR for page 117
COMMON ISSUES
~ ~ 7
sponses to ELF-EMF exposure, as opposed to uncontrolled effects due to
contrasts in soil composition, chemistry, or vegetation. It would have been
useful at the outset to have an integrated research plan in mind to guide site
selection. The absence of such a plan meant that less than full value was
obtained from the research as a result of segregated selection of possible eco-
system effects. in addition, the lack of common exposure levels at sites lim-
ited the possibility of integrating results. (See the section later in this chapter
on lack of integration.)
THE PRACTICAE PROBLEM OF SITE SEEECTION
Given the unavoidable heterogeneity of soils and vegetation typical of the
Michigan Upper Peninsula, most researchers did as good a job as possible in
matching sites. For instance, the earthworm and soil-arthropod sites were well
matched with respect to soils and arthropods, but not with respect to earth-
worms. A better match with respect to earthworms could probably not have
been attained. In the aquatic study, the researchers sought an upstream control
site but could not find one, because none that was of the same stream order
and physical attributes as the treatment site existed. in general, so many condi-
tions had to be met for site selection that perfect matches were impossible.
ADJUSTING THE RESEARCH PLAN TO
PROBLEMS WITH SITE SEEECTION
Once the site-selection process was undertaken, it quickly became obvi-
ous that ideal matches were going to be difficult to find. Numerous research-
ers attested to that in their response to written questions from the committee.
Two modifications of research design should have been considered upon en-
countering the practical obstacles to perfect site matching. First, for a small
subset of critical variables, it would have been valuable to pursue spatially
extensive comparisons of multiple sites (control versus treatment), thereby
gaining the inferential power of multiple independent samples. The more-
extensive design would require less effort at each site, although there would
have to be a tradeoff in reduced number of variables measured. Second, for
the intensive paired comparisons of sites, more critical thought might have
been given to what processes, organisms, and observations would be most
valuable, given some site mismatches. For instance, in the soil arthropods and
OCR for page 118
~ ~ ~ EVALUATION OF ELF ECOLOGICAL MONITORING PROGRAM
earthworms study, the work on the dominant earthworm at the treatment site
alone could never yield results that could clearly be attributed to ELF-EMF
exposure, because there was no way of separating ELF-EMF effects from
other factors without a control site at which the same earthworm was studied.
CONCLUSIONS REGARDING SITE SELECTION
Researchers faced difficult problems in selecting sites because of hetero-
geneity in the environment. Additional money and time for the sampling of
more sites and some rethinking of experimental design and statistical inference
would have helped to address some of these problems. For example, the
program would have been improved by studying fewer variables at more and
larger sites and by eliminating studies with poorly matched treatment and
control sites. However, as with any environmental impact assessment, it
would be unrealistic to expect the Navy's monitoring program to conform
fully to an ideal experimental design.
PSEUDOREPLICATION
In many of the studies, the main effect of interest, namely the effect due
to the presence of ELF EMFs generated by the antennas, was pseudoreplicated
(that is, not truly replicated, as described by Huribert 1984) in that there was
only one site for each level of exposure. Even when more than one site was
available, treatments were not randomly interspersed so that background ef-
fects of soils, climate, etc., were equally (or at least randomly) distributed
among all treatments and controls. Therefore, the experimental data provide
an estimate of variance of responses studied within each site, but not the vari-
ance due to treatments across sites. The effects of the antenna on response
variables are therefore confounded with the background effects of the different
soils, climate, etc., on each site, and the two cannot easily be separated. This
problem arises in the litter decomposition and microflora studies, the aquatic
ecosystem studies, the upland-flora studies, and others. The wetland studies
and the bird community studies avoided this problem by having replicate sites
within different treatments and replicate plots or transects within each site. In
the latter studies, the variance of the response variable of interest could be
separated into two components: variance within each site and variance due to
treatment or its absence. In the pseudoreplicated studies, that was not possi
OCR for page 119
COMMON ISSUES
~9
ble, because there was only one site per treatment, so the variance recorded
can, strictly speaking, be attributed to only within-site, not between-treatment,
effects.
Sometimes pseudoreplication is necessary for logistical reasons. When
pseudoreplication is unavoidable, the generalization of treatment effects to
other sites might be justified (with caution) if it can be demonstrated that the
sites chosen for each single application of the treatment are not substantially
different from each other at the outset and are at or very near the modal values
of other environmental factors thought to affect the response variable of inter-
est. That type of pretreatment survey was performed only for the upland-flora
studies. In contrast, the litter decomposition studies are seriously compro-
mised because there were large differences in decay rates between sites before
the antennas were in operation. The antecedent site effects obscured the po-
tential detection of treatment effects.
The acceptance of all furler conclusions must proceed with those caveats
in mind. The caveats are generally not clearly stated anywhere in the
monitoring-program reports. The danger with pseudoreplication is in commit-
ting a type IT error accepting the null hypothesis (no effect) when it is, in
fact, false. Such an error could arise, for example, if the variability within
sites or between matched sites is within the range of responses imposed by the
treatment. That should not be taken to imply that the effect of the antenna is
small, as implied in some reports. Rather, it is not possible to separate the
effect of the antenna from effects of other environmental factors without repli-
cation both within sites and across sites. In many cases, it is impossible to
calculate the probability of a type II error because it depends on an independ-
ent estimate of differences between treatment and control, which requires
replication of sites, not simply of plots within sites. The putative treatment
effect is confounded with the site effect, so the differences between treatment
and control cannot be attributed solely to the antenna, inasmuch as they are
not independent of pre-existing effects and confounding site effects that contin-
ued during the experiment.
The extensive use of ANOVA and other linear models (including regres-
sion techniques) in the ELF study requires some strong assumptions about the
distribution of the data, namely, that effects are linear in the scales chosen,
that variances are constant, that error terms are independent, and that residuals
are normally distributed. Those assumptions were not usually tested by the
ELF researchers, with few exceptions such as the study of the effects of the
antenna on bird populations. Furthermore, when the assumptions were consid-
ered, the investigators seem to have misunderstood them and to have applied
other statistical analyses that might not have been the most appropriate. An
OCR for page 120
~ 20 EVALUATION OF ELF ECOLOGICAL MONITORING PROGRAM
example of the latter problem is the rejection of exponential-decay models in
the litter-decomposition experiments in favor of covariance models that are
more difficult to interpret. A further, troublesome aspect of the analyses is
the nearly complete absence of any quantitative discussion of the effects of
statistical bias, which could well dominate the role of random variation. The
lack of quantification of statistical bias is exacerbated by the pseudoreplication
or even lack of replication in many of the experiments.
SPECIES SELECTION
The Navy's original plan for an ecological monitoring program recom-
mended that species (or related species) be studied that are reported to be
sensitive to EMFs; are important ecologically, aesthetically, or economically;
and can be reasonably monitored. These recommendations were largely met
in these studies, as discussed below.
STUDY SPECIES
The diversity of species studied was considerable. Studies included
species in most major taxonomic groups, including vascular plants (trees and
shrubs), algae, slime molds, amebas, fungi, small mammals, birds, arthro-
pods, and various decomposers. However, studies on small-mammal popula-
tions and on development of vertebrates (birds and mammals) had design
problems, so information on this group is unreliable. Major groups of organ-
isms that were not included were some nonvascular plants (e.g., moss), rep-
tiles, and amphibians. In the wetlands study, a moss population was found to
increase significantly at the treatment site, but this apparent response was not
pursued, because moss was not a target species; this is unfortunate because the
finding might be an indicator that moss is especially sensitive to ELF-EMF
effects.
The species studied included types of organisms that had been reported
to be sensitive to EMFs in previous laboratory or field studies, including slime
molds, vascular plants, earthworms, birds, and bees. This coverage was very
good. Little information exists on the EMF sensitivity of most of the particu-
lar species in the site, so species related to those exhibiting an effect were
usually studied. For example, native bees, rather than honeybees, were stud-
ied because honeybees cannot survive the winter in this area.
Other species were usually well justified on the basis of their potential
OCR for page 121
COMMON ISSUES
121
ecological importance. No study used economic or aesthetic importance. The
most common criterion of ecological importance was abundance, which is
reasonable because abundant species often exert a large effect on ecosystem
processes and on other species that depend on them for food and performing
valuable ecological functions. For example, the most common tree and shrub
species were studied. The ecological importance of other species was based
on their potential functional importance to the ecosystem. For example,
periphyton species are valuable as food for many other species and as bioas-
says of water quality. Decomposers are crucial to nutrient cycling. Strepto-
mycete populations are associated with decomposition and nutrient cycling.
Most study species could be adequately monitored because they were
abundant, although some studies were limited by small sample sizes. No
studies focused on rare species, although some might inadvertently have been
included in aggregate variables such as bird censuses. In the wetlands study,
rare sedge species were dropped when it was recognized that sampling might
harm them.
The omission of rare species is problematic. Some rare species, such as
predators and keystone species, can exert major effects on communities.
Potentially endangered species were not studied. It is not known whether rare
populations at the edge of their range are more or less sensitive to bioassays
of additional stresses than abundant species. it is important to note that
studying rare species would have been useful as studvin~ more-common sne-
cies in an attempt to find any type of effect.
, O
The ability to measure response variables determined the choice of some
species. For example, in the wetlands study, several species were dropped
when it became clear that stomata! resistance could not be easily monitored.
in a few studies, species were chosen to provide interesting comparisons
and generalizations. For example, in the upland litter-decomposition studies,
the leaves of fast- and sIow-growing tree species were compared. In the
wetlands study, trees, shrubs, and herbs were compared. However, the lim-
ited number of species from each group precluded generalizations about life
forms. More such comparisons would have been valuable in these studies.
CONCLUSIONS REGARDING SPECIES SELECTION
OveralT, species selection was commendable. The species studied in-
cluded a broad range of organisms with potentially different responses. That
is important for detecting potential ELF-EMF effects on ecological systems
that contain a wide diversity of organisms. Representatives of most taxonomic
OCR for page 127
COMMON ISSUES
127
toring program are uninformative at best. At worst, a potentially important
effect might be missed and replaced with an unwarranted sense of confidence
that the null hypothesis of no effect has been substantiated.
The committee noted lack of adequate statistical power as a problem that
arose in more than one study. in the small-vertebrates study, an early de-
crease in sample size led to a decrease in power to 70%, and a large number
of variables were eventually analyzed with statistical power of less than 30% .
The power of statistical tests in the soil arthropods and earthworms study was
low because of the sampling scheme (few pitfall traps), large differences
between sites, and large variability in the data. It was impossible to estimate
accurately the statistical power in several studies, including Michigan and
Wisconsin birds and litter decomposition and microflora, because of pseudo-
replication and unclear exposure relationships. Reanalysis might improve a
number of these studies (see Chapter 51; others are unsalvageable because of
flaws in their design or execution. Some studies, such as the pollinating-
insects study, did a good job of calculating and discussing the minimal detect-
able differences and the power of statistical tests.
To quantify the likelihood of detection, or power, one needs to describe
three aspects of a study: the experimental design and sample size, the rules
by which an effect will be declared statistically significant (as distinct from
biologically significant), and the magnitude of effect that will be assumed to
result from the experimental intervention.
For example, consider the following hypothetical scenario chosen for its
relevance to the ELF ecological monitoring program:
· The goal is to compare 20 pairs of nesting birds in a control plot
with 20 pairs of birds at one of the treatment sites. The variable to be exam-
ined is the number of surviving hatchlings per nesting pair at some specified
time.
The variable (number of surviving hatchlings) will be assumed to
follow a Poisson distribution. The statistical rules are (~) the null hypothesis
of no difference in the means of the numbers of surviving hatchlings and (2)
rejection of the null hypothesis according to a two-tailed test at the 95 % signif-
icance level.
.
The statistical goal is to be able to detect a 20% or greater change
in mean number of survivors, under the assumption (justified from prior data)
that the mean control number is 3 surviving hatchlings per pair. (Typically,
the magnitude of the change to be detected is dictated by considerations of
biologic significance.)
OCR for page 128
28 EVALUATION OF ELF ECOLOGICAL MONITORING PROGRAM
For each case of exposition, we will also assume that the hatchlings are
independent of each other. Although this assumption is not correct, it eases
the presentation without compromising the illustration with respect to statistical
power. That scenario provides sufficient information to compute the statistical
power of such an experiment. Statistical power corresponds to the probability
that a significant effect will be observed if the mean number of survivors at
the treatment group differs from that at the control site by 20% or more,
provided that all other assumptions have been satisfied.
Tt is vital to consider confidence intervals, that is, to consider the results
of such experiments not as providing only two mean values with an associated
p value for their difference, but as providing the differences to be expected if
the experiment were repeated many times. For example, assume that the
results of the above experiment are that the mean number of hatchling survi-
vors in the control group differs from the mean number in the treatment group
by 22%, with 95% confidence limits of S% and 150%. If the experiment
were repeated many times, one would expect the mean difference between
treatment and control groups to be in the range of 8-150% in 95% of the
repetitions. Because the confidence interval does not include zero, the null
hypothesis is rejected at the 0.05 level of significance. The width of the
confidence interval implies that the effect could be uncomfortably close to the
null (~%) or could be quite large (150%~; that is, the experimental design and
sample sizes have led to very imprecise estimates.
What does the outcome of the experiment tell us if the power of the
design is 0.9? Tt indicates that if there is indeed an effect of 20% or greater,
90% of the time the confidence interval will not include the null hypothesis.
Tt provides us with some reasonable bounds on the uncertainty. An alternative
experimental design with a power of 0.3 or less (like those reported in a num-
ber of experiments in the ELF ecological monitoring program) tells us that
70% or more of the time the confidence interval could include the null hypoth-
esis. Such an experiment offers no bounds on the uncertainty, and it can
reasonably be questioned whether such an experiment should have been per-
formed at all. In this context, it is useful to note that it is common to require
that results be expressed in terms of confidence intervals.
RESPONSE TO REVIEWS AND CRITIQUES
Research teams associated with the ecological monitoring program re-
ceived annual comments from reviewers beginning in 1982. In its evaluation
OCR for page 129
COMMON ISSUES
129
of individual studies, the committee discovered that there was a great deal of
variation in how the researchers responded to the reviews of their annual
reports and presentations. Some review comments appeared to have been
taken seriously and to have led to modification of research designs or report
presentations; other comments appear to have been taken lightly or ignored.
The committee did not attempt to determine, in a systematic manner, the
extent to which the researchers considered the review comments to be appro-
priate. However, the committee found several instances in which the peer-
reviewer comments raised valid and important questions. Some of these issues
have been addressed in this report.
In a number of cases, reviewer comments were responded to in a satis-
factory manner. The initial study on Michigan and Wisconsin birds was
canceled in response to negative reviewer comments, and a new proposal was
accepted. The wetlands researchers modified their study design to accommo-
date reviewer comments and provided explanations if changes were not made.
Upland-flora researchers took peer review seriously and addressed the con-
cerns of their reviewers.
Several research teams responded only partially to reviewer comments.
The research team for the litter-decomposition study made minor changes in
experimental technique that were suggested by reviewers. However, more
difficult problems involving theoretical issues were only partially considered,
if at all. The researchers appeared not to understand some issues fully. In the
pollinating-insects study, most reviewer suggestions were heeded and contrib-
uted greatly to the quality of the research. However, suggestions on using a
BACI analysis with covariates and on increasing replication were not followed.
Small-vertebrates researchers were generally responsive to peer reviewers and
made several improvements in the study and in the clarity of the final report.
However, reviewer concerns regarding low statistical power and lack of data
archiving were not addressed.
Every year, TTTR] received annual reports on all the studies and held a
meeting at which each study was presented or discussed. ITTR} organized
outside reviewers to review the annual reports and to comment on the progress
of each study. Some of the external reviewers' comments were very critical.
Such criticism should have been a clarion call to TTTRI that something was
seriously wrong with the design or progress of the criticized study. It is not
apparent to the committee that IITRI followed up on reviewer critiques of
studies; followup seems to have been left to individual investigators. Conse-
quently, research design, analytic techniques, or interpretation that needed
improvement or correction according to external reviewers were often left
unattended. That lack of attention to reviewers' comments should have con
OCR for page 130
~ 30 EVALUATION OF ELF ECOLOGICAL MONITORING PROGRAM
cerned TITR} management. TTTR] should have established a regular internal
review process to ensure that each study adequately addressed external criti-
cism, even to the point of having external reviewers comment on responses to
their criticism. There are also instances (e.g., data archiving) where critiques
and suggestions of the peer reviewers were addressed to ITTR} managers
directly. These critiques appeared to have been ignored by ITTRI.
APPROPRIATENESS OF INTERPRETATION
Scientists are well acquainted with the potential for bias in conclusions
based on a given set of data. Preference for a particular outcome could result
in an interpretation favoring that outcome, and this could occur even in cases
where the persons making the judgment believe that they have done so without
bias. (Such a tendency toward bias could also be present in the subjects of
experiments involving humans. Those considerations have given rise to so-
called double blind experimental protocols, in which neither the subject nor
the experimenter has knowledge that will allow such potential bias.)
In several studies of the Navy's ecological monitoring program, modest
but significant differences were observed between data collected at treatment
sites and data from control sites. Researchers conducting the studies con-
cluded that five of these potential effects were due to factors other than the
ELF antenna. Without attempting to judge whether any of those interpreta-
tions suggested a predisposition to a particular outcome, it is important to
consider whether the conclusions were established with a credible scientific
basis. In the course of the committee's review and discussion of the research-
ers' final reports, concerns arose about the scientific credibility of some of the
conclusions.
Differences between treatment sites and control sites that were dismissed
by researchers and by ITTRI as not being clearly related to ELF exposure were
the increase in bee overwintering mortality, the reduction in leaves per bee
nest cell, accelerated litter decomposition, earlier eye-opening in mice, and
depressed earthworm reproductive rates. The committee believes that some
of these observed differences were dismissed too readily as alleged artifacts
of environmental variations or experiment design. In the pollinating insects
study, the final conclusion that ELF-EMF effects are absent or minimal is not
fully justified by the data presented. Similar concerns were expressed by
earlier reviewers, who concluded that ELF-EMF effects were demonstrated.
In the litter-decomposition study, the provisional conclusion of negligible
effects of the antenna appears to be based on a combination of small differ
OCR for page 131
COMMON ISSUES
131
ences in mean values and the large variation in the data, not on the validity of
the results. Even a 5-10% change in leaf-litter decomposition rates can have
a major impact on soil characteristics (McCIaugherty et al. 1985~. Although
dismissal of the five possible small effects described above might be correct,
the committee suggests in Chapter 5 that the results of some studies be reana-
{yzed.
Another concern has to do with the issue of multiple comparisons and the
observation of statistically significant results due to chance alone. For exam-
ple, when researchers perform 100 tests at the 5% level of significance, one
expects to find, on the average, five positive values (that is, to reject the null
hypothesis) because of chance alone. In the very large number of statistical
tests performed in the monitoring program, it would be surprising if no statis-
tically significant findings were reported. Therefore, the issue is not whether
significant results emerged from time to time, but whether the number of such
events was larger then expected. That issue was not examined systematically
as part of the ecological monitoring program.
DIFFERENT METHODS FOR
SIMILAR ORGANISMS
The broad range of studies in the program often resulted in examining
the possible effects of the antennas on similar organisms or processes, but with
somewhat different protocols. For example, litterbag studies of decomposition
were performed in uplands, wetlands, and streams, but these studies were
performed by different teams, often from different universities or institutions,
and were initiated at different times. Under those circumstances, it is under-
standable that different methods might be used. The upland litterbag studies
used leaf litter from dominant tree species, and bags were sampled at about
monthly intervals. In the wetland studies, decomposition experiments began
with cellulose strips and switched to leaf litter from Ledum groenZandicum, a
common shrub. However, the samples in the wetland decomposition studies
were taken only annually. Annual sampling precluded fitting different decay
models to the wetland decomposition data, as was possible in the upland stud-
ies. It is not possible, therefore, to analyze all the decomposition data with
a common statistical or mathematical model.
The use of different methods does not necessarily negate conclusions
from any one study.
_
Indeed, there might be sound scientific or logistical
reasons for using different methods in different situations. For example,
achieving greater precision to meet the objectives of one study might entail
OCR for page 132
~ 32 EVALUATION OF ELF ECOLOGICAL MONITORING PROGRAM
different methods from those in another study. But the use of different meth-
ods prevented researchers from making valid cross-site comparisons and there-
by impeded the realization of the full potential of integration across sites and
organisms. For example, in~the upland-forest decomposition studies, litter
from various species was placed at each site as an "index" material. In the
wetland-decomposition studies, cotton strips were used as index materials.
Litter in the upland studies was sampled monthly for mass-Ioss estimates, but
in the wetland studies it was sampled annually. There does not appear to have
been much discussion of coordinating methods before the experiments began.
Therefore, the inability to integrate properly across sites and organisms be-
cause of the use of different methods is an unfortunate after-effect of the
execution of the work, rather than a tradeoff that was intentionally made for
the sake of greater precision within any one study. The collection of studies
as a whole would have benefitted from the specification of a small number of
hypotheses at the earlier stages of study design.
LACK OF INTEGRATION AMONG STUDIES
AND SYNTHESIS OF INFORMATION
As noted in Chapter I, the original RFP for studies of the effects of ELF
EMFs on biologic systems in the region of the ELF communications system
antennas was developed by ITTRT on the basis of a monitoring program outline
from the Navy, previous research, information from state agencies and the
U.S. Forest Service, and comments on the Navy's draft environmental-impact
statement. Recommendations from the National Research Council report
(NRC 1977) on potential effects of ELF EMFs also influenced the RFP. The
RFP requested proposals addressing the responsiveness of select groups of
organisms to the environment created by the EMFs produced by the ELF
antennas. The list included such groups as mammals, birds, invertebrates,
plants, slime molds, and amebas. There was only a slight suggestion in the
RFP that studies of effects were to be in an ecosystem context; rather, the
emphasis seemed to be on a population approach.
SELECTION OF PROJECTS
In the process of selecting the ecological studies, there does not appear
to have been an attempt to fund research teams that were sufficiently close
together to encourage interaction. That could have been the result of selection
OCR for page 133
OCR for page 134
~ 34 EVALUATION OF ELF ECOLOGICAL MONITORING PROGRAM
adjacent to streams with the responses of aquatic attributes or compared re-
sponses of invertebrates within different but integrated study areas. Many
other examples could be given.
CONCLUSIONS REGARDING INTEGRATION OF
STUDIES AND SYNTHESIS OF INFORMATION
In the processes used for selection of studies and study sites, there ap-
pears to have been little recognition of the possibility that a response of one
attribute of an ecosystem (such as an organism or process) could influence
other ecosystem components and that information about related responses
might reinforce or undermine conclusions about responses taken one at a time.
Instead, possible responses were considered as isolated events, that is, outside
an ecosystem or integrated context. Such a perspective might have caused the
managers of the ecological monitoring program to establish study site selection
requirements based only on exposure levels and research impacts on limited
populations of sensitive species. The committee notes that it would be infeasi-
ble to synthesize the data on response variables because they represent diverse
aspects of ecosystems and because in most cases the measured responses were
insignificant. In addition, looking for these interactions and developing a
synthesis document might take too long and thus be outside the purview of the
ecological monitoring program. The committee also notes that there might be
some value in use of varied approaches, given that so little is known about the
effects of ELF EMFs on ecosystem components and processes. Nevertheless,
recognition of interactions among ecosystem components and encouragement
of integration among studies with full development and application of appro-
priate statistical approaches should have been guiding principles in the early
research design. That would have given the research community the opportu-
nity to synthesize the extensive information generated over many years of
study, even outside the funding of TTTRI. Both the advisory committee and
the monitoring program's management team were remiss in not including
recommendations, and perhaps requirements, for integration and synthesis in
the RFP.
An additional problem with the lack of integration and synthesis is that
Were was virtually no opportunity to follow up on the most-positive research
findings in lieu of continuing with less-promising research. For instance, the
finding of a statistically significant increase in chIorophyIl-a in response to
ELF EMFs in streams was never followed up with the obvious laboratory
OCR for page 135
COMMON ISSUES
135
studies, because of lack of funds, even though the researchers recognized it as
a potentially important finding that needed laboratory confirmation. Con-
versely, earthworm comparisons between sites that differed vastly in their
initial earthworm fauna were pursued, even though findings were doomed to
ambiguity (because of poor comparability of sites with respect to the earth-
worms). The funding and contract basis of these studies locked the program
into an inflexible pattern of support for research without matching the funding
to a continuous review of the results. If there had been consistent integration
and comparison of findings for the different studies, the overall research effort
could have been improved for the same amount of money. With better inte-
gration, there could have been more pursuit of promising results, the hallmark
of good research.
By failing to integrate the studies of different species and ecosystem
processes, this large-scale effort largely surrendered the possibility of detecting
small changes in interactions of components and gave up the major advantage
of such large-scale research. Given the absence of synthetic overview, this
might just as well have been many isolated studies of one variable at a time.
The project thus missed an excellent opportunity to conduct pioneering
ecosystem-level research that spanned physiologic, population, and ecosystem
responses. All ingredients for such research were in place- but no one put the
ingredients together.
DATA ARCHIVING
The final and annual reports do not contain information on archiving of
data generated by the Navy's ecological monitoring program. To understand
the current state of data archiving, the committee spoke with several research-
ers directly. The results of these telephone conversations indicate that each
group of researchers used its own protocol for archiving data and that the
quality, durability, and accessibility of these protocols differed dramatically.
For the wetlands study, data do not exist in electronic form but are available
in the original notebooks and in the annual and final reports published (F.
Stearns, formerly of University of Wisconsin-Milwaukee, personal commun.,
1996~. For the soil-ameba study, biologic data are recorded in notebooks, and
environmental data are available on computer printouts. Although many of
these data were analyzed in electronic format, the software used was old and
the data are probably not retrievable at this point (R.N. Band, Michigan State
University, personal commun., 19961. All data from the pollinating-insects
OCR for page 136
~ 36 EVALUATION OF ELF ECOLOGICAL MONITORING PROGRAM
study are contained in a relational database, RBase. However, the data would
not be useable with the limited documentation available, and anyone who
wanted to use them would need to speak to the researchers to have the vari-
ables and peculiarities of the database explained (K. Strickler, University of
Idaho, personal commun., 1996~. Data from the study of bird populations in
Michigan were archived with a relational database, Paradox. With some
documentation, others would be able to use the files. These data were archived
in a standardized format used at the researchers' institute If. Hanowski, Uni-
versity of Minnesota, personal commun., 1996~.
The small-vertebrate study is worth noting in this context because the
large, ambitious study yielded voluminous data. It is important to separate the
value of the study as a purely scientific investigation from its value as a tar-
geted effort to ascertain whether the Navy's ELF communications system had
deleterious effects on the neighboring ecosystems. As a purely scientific
investigation, it has yielded mostly high-quality data with incomplete statistical
analyses. Some of the flaws identified in the evaluation of the study in Chap-
ter 3 could, in theory, be remedied by analyses that used a framework both
physiologically appropriate and statistically sound. Such analyses are not
likely, in part because of decisions on the part of the IlTR} management team
regarding data archiving. The management team failed to observe elementary
practices of data management that would have yielded a documented archive
of data suitable for re-examination. The latter is an egregious failure, inas-
much as a peer reviewer of the entire monitoring program, made the following
pointed comments before the 1988 contract-renewal process (letter dated May
5, 1987~:
All participants should be sending YOU hard copies and floppy disks
(IBM compatible) right now. All data sets should have excellent docu
mentation and you should have copies of it. ~ was a little disturbed that
some participants take a lackadaisical view of their data sets.... Each
investigator should consider data management as an obligation. For
IBM] it is an essential." [italics added]
Apparently, the recommendation was not followed, and documented archiving
of data was not undertaken.
The Navy ELF ecological monitoring program supported 11 {ong-term
studies. Those studies generated extensive data sets on biologic response
variables and environmental conditions. Most of the studies continued for 5
years or more, and each produced annual reports of progress and information
OCR for page 137
COMMON ISSUES
137
gathered. From a review of the overall program, it appears as though all that
was expected from each study was an annual report, presentation or attendance
at an annual meeting of monitoring-program investigators, and a final report
that discussed responses of selected variables and drew conclusions. To meet
those expectations, the investigators in each study must have developed exten-
sive sets of information and organized them in a fashion that allowed analysis.
Obviously, the data sets include working field notes, tables in electronic and
possibly paper form, and results of analyses. As far as the committee is
aware, those data sets were kept by individual researchers and not transferred
to TITRT. No protocol seems to have been established for the request for
proposal (RFP) or later for formatting, documenting, or reporting the data.
After millions of dollars had been spent on a monitoring program that could
be used for further understanding of the monitored ecosystems, the resulting
information appears not to be readily available; if available, it is not in a
uniform, user friendly format; and there is no common location to which an
outsider can address requests for information. Responsibility for the lack of
archiving and of planning for long-term availability of the monitoring informa-
tion appears to rest with ITTRI's management of the program. All aspects of
archiving of monitoring data should have been designed as part of the RFP,
and researchers in all the monitoring studies should have been required to
submit their data sets, in the appropriate format, at regular intervals or at least
at the completion of each study.
Representative terms from entire chapter:
ecological monitoring
