National Academies Press: OpenBook
« Previous: 6 Opportunities for Methodologic Advances in Data Analysis
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×

7
Review of the Gray Literature from State Reports

The committee's first report (NRC, 1991) reviewed published studies on the possible associations of exposure to hazardous wastes and human health. This chapter continues that assessment by reviewing selected studies on this subject from the gray literature, that is, studies that are available to the public but not published in the indexed scientific and technical literature. This chapter explains the process that the committee used to identify selected studies from the gray literature for assessment; presents a general review and assessment of those studies using criteria developed by the committee evaluates in depth some reports that specifically examined reproductive hazards of exposures to hazardous wastes and other materials and notes briefly some recent unpublished reports on human health and the environment that come from central Europe, where environmental contamination appears to be considerable. Despite some potentially serious problems, the gray literature can be an important source of information about environmental epidemiology.

Introduction

Many studies of health effects that may be associated with hazardous wastes are available on request but are never published. They fall into the category of ''gray literature," which has been defined as literature that is not "white" (available and cataloged), and that is not "black" (not available, unknown, or not obtainable); gray-literature reports are usually produced in small quantities, intended for limited audiences, and not widely known (Schmidmaier, 1986). In the field of environmental epidemiology,

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×

these gray-literature studies may include such items as state health-department reports, doctoral and master's theses, and reports produced by special-interest groups. In addition, the committee is aware that, with the dramatic changes in government in the former Union of Soviet Socialist Republics, large amounts of information on environmental health in that region are becoming available. Most of these reports are in the gray literature and even in that form have not appeared in the English-language literature.

Review of Gray Literature

Why is the gray literature gray? Many studies with substantial implications for environmental epidemiology are never published. A part of the problem is that in the United States and some other countries, the assessment of environmental health effects is often conducted in a context of litigation or potential regulatory action. This climate exerts strong pressures on any scientific assessment that may affect the selection of specific topics for study, protocol design, and methods of analysis, as well as public availability (e.g., sealing of records as a part of a negotiated settlement).

Earlier, the committee discussed some of the limitations that may confront public-health agencies in performing environmental-epidemiologic studies, including limited expertise, limited resources, lack of concern, secrecy of data, political pressure to conduct studies despite inadequate knowledge about the exposures or diseases in question, and the inherent limitations of uncontrolled "natural" experiments in which the populations are too small, the latent period too short, or the exposures or outcomes too poorly defined to yield useful results. In addition, there may be an issue of "publication bias" in which well-designed and well-conducted studies are not published, because of a lack of interest by journals or because of a bias against publishing negative findings in some instances and positive findings in others, especially when the "exposure'' is not of great current interest among other scientists. Agencies may have little motivation for assembling studies into a format suitable for publication as staff move from fighting one fire to another. To understand better the question of why the gray literature is not published, the committee undertook to obtain a collection of such studies that had been produced by state health departments and others. Defined criteria were used to assess the quality, strengths, and weaknesses of each study and to estimate whether each report would be publishable in the peer-reviewed literature.

NGA-ATSDR State Environmental Health Information Clearinghouse

The National Governors' Association (NGA) under contract with the Agency for Toxic Substances and Disease Registry (ATSDR) initiated a

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×

pilot project in May 1990 to set up a clearinghouse for state environmental-health studies. NGA established an advisory body to oversee the project, and members of this committee attended the initial meeting in July 1990. NGA and ATSDR decided at this meeting to obtain state-generated reports. The findings of the NGA pilot study were as follows:

  • The State Environmental Health Information Clearinghouse was well received and very useful.
  • The collection of clearinghouse information requires active solicitation.
  • Operation of the clearinghouse by ATSDR would be mutually beneficial for the agency and the states.

Clearinghouse holdings were accessed to supplement the committee's database of studies. Because the clearinghouse was in its formative stages at the time of the committee's review, it could not serve as a primary source of studies for review, and it was necessary for the committee to request studies from the states directly as described below.

Obtaining State Studies

While deciding which states to contact, the committee considered more than 20 state health departments. The committee formally contacted officials in states that were considered to have the most-appropriate data. These were California, Connecticut, Florida, Iowa, Massachusetts, Michigan, Minnesota, New Jersey, New York, Ohio, Texas, and Wisconsin in 1990 and again in 1991. Studies from additional states were obtained informally. Officials in the 18 states were asked to provide examples of studies meeting the following criteria.

  • Health outcomes. The study investigates and reports on health end points, including biologic markers, physiologic alterations, or patterns of morbidity and mortality.
  • Exposure assessments. The study is intended to identify risks of adverse health effects associated with specific exposures from hazardous-waste sites; these assessments are of potential use in epidemiologic investigation.
  • Population studied. The study investigates health outcomes in a community possibly exposed to pollutants from hazardous-waste sites, rather than among workers.
  • Biologic plausibility. The study investigates the association of biologically plausible (to state officials) health outcomes with potential exposure to hazardous wastes. (The committee is aware that concerned communities may request the investigation of associations that are not
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
  • biologically plausible, such as cancer clusters where the latency period is too long to link exposure to outcome.)
  • Explicit hypothesis or study question investigated. The study was initiated to investigate a possible association between specified exposure(s) from a hazardous-waste site and specified health outcomes(s).
  • Methodology. The study includes a comparison group, such as persons exposed at lower levels, an unexposed control group, or a baseline population or registry comparison.
  • Contributions to environmental epidemiology. The study may contribute to the understanding of the health effects associated with hazardous wastes or may refine existing methods to introduce a new method for the epidemiologic investigation of hazardous-waste sites. Because many state-sponsored studies involve the investigation of cancer clusters, those approached were asked to use this criterion to screen such studies.
  • Stage of completion. Preliminary or interim reports and descriptions of work in progress (for which the data collection and preliminary analysis had been completed) that met the above criteria were regarded as pertinent by the committee.

The limitations of our review must be noted. We necessarily focused on state reports, to the exclusion of other kinds of gray literature that may be quite different in important ways. Further, since only certain states participated and since the states themselves selected which reports to send, the reports reviewed by the committee would in no way represent a cross-section of all studies conducted by those states, nor would they represent studies conducted by all states. As noted above, studies submitted by the states and meeting the criteria above were supplemented with studies found in the NGA clearinghouse and those that were submitted informally. In general, we suspect that the reports reviewed here may be of better average quality than those not reviewed.

Initial Evaluation

The committee evaluated the studies from the standpoint of characterizing the nature and quality of gray-literature reports. We did not attempt to evaluate results or conduct a meta-analysis of findings. Once the material was received, we selected a subset for further evaluation and asked the following questions: Is it an epidemiologic study? Is a community or residential population involved? Was there some kind of peer-review process or other evidence of quality assurance or accuracy checks? Did the investigators try to collect exposure data? Was the study done since 1980?

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×

Epidemiologic Criteria for the Evaluation of Studies from the Gray Literature

To establish the role of the gray literature in environmental-health policy, the committee examined the literature submitted in much the same way that it would examine any other epidemiologic studies to assess quality, public-health significance, value to other researchers, and whether in the committee's judgment the (possibly revised) study would be publishable in a peer-reviewed journal. The following factors were examined.

Study Design

There is a relation between study power, sample size, prevalence of exposure, and expected rates of a given outcome in the study and control population. In general, studies of larger numbers of persons over longer periods with a big change in level of risk are more likely to yield positive results than are those involving smaller populations, shorter periods, and small changes in risk. The sample size to achieve a given study power is also related to whether exposure outcome is measured as a dichotomous or continuous variable, the variability in distribution of the exposure, the effects of confounders and errors on the measure of exposure, and the statistical methods used.

Exposure Assessment

Exposure measures are an important factor in the quality of an environmental-epidemiologic study. In some cases, exposure measurements already available can be used, but in other cases, special exposure assessments are required to fully assess possible effects on community residents. In still other cases, exposure measurements are not available and cannot be obtained, so some surrogate for exposure must be used. The validity and accuracy of these surrogate measures must be assessed. The time and duration of the exposure measurements, the methodologies used, and the relevance of these measures to the outcomes being evaluated are also important. Obtaining information for study participants who have had several different levels of exposure can provide an opportunity to look at important dose-response relations.

Health End Points

The health end points being evaluated should be appropriate with respect to the suspected exposure(s). Findings from a study examining a wide variety of end points may be less valuable than findings of a study

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×

focusing on end points known to be associated with a particular chemical exposure both because the data and the analysis can be more-precisely targeted and because the problem of multiple comparisons is reduced. The method of ascertaining the end points is important and includes the accuracy of the source and validation of reported end points by other means. For example, the validity of self-reported medical diagnoses is improved if they are verified with medical records.

Study Results

Results of a study are stronger if the magnitude of the association between exposure and health outcome is greater; however, this must be balanced with the public-health significance of the outcome; i.e., a highly elevated relative risk of a minor and rare health problem may be less important than a less-elevated risk of a common and more-serious condition. The demonstration of a dose-response relation significantly increases the strength of a study. To determine whether a result is likely to reflect a true underlying relation, epidemiologists evaluate the congruence or consistency of findings with those of other, related studies and other scientific evidence.

Likelihood of Acceptance by a Peer-Reviewed Journal

The committee made a subjective evaluation of whether the studies would be definitely accepted, possibly accepted after substantial revision, or probably never accepted by a peer-reviewed journal. We determined potential reasons for rejection and made an assessment of the major weaknesses, if any, of the studies.

Findings

Twenty-nine studies from the states of Colorado, Connecticut, Florida, Idaho, Louisiana, Maine, Massachusetts, Michigan, New Hampshire, New Jersey, North Carolina, Ohio, Tennessee, Texas, West Virginia, and Wisconsin were reviewed. California and New York were excluded because of the makeup of the committee.

Kinds of Designs

The most commonly conducted study was a comparison of disease incidence or prevalence in a study population with that in the general population. These studies were usually undertaken because of a reported clustering of some health outcome, such as leukemia, or because of com-

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×

munity concerns about a source of exposure, such as a hazardous-waste site. Such self-selection tends to create a problem with response bias, in that persons who believe they are at risk might selectively recall exposures or outcomes better than those without such a concern. Some cross-sectional studies sought evidence on exposure, usually with a questionnaire accompanied by biologic monitoring (e.g., lead).

Ecologic correlational analyses were also undertaken in some instances. Some of these studies included data at the county level and compared disease rates in terms of available measures of exposure, such as drinking-water monitoring data. A few states conducted case-control studies to see whether individuals with and without some health condition had different environmental exposures.

Number of Subjects and Statistical Power

Studies mounted in response to community concerns often had low statistical power because of the small numbers of persons exposed. It was not clear whether estimates of statistical power had much influence on decisions about whether to conduct a study. Some studies were limited to the small numbers of people in the "concerned" or "exposed" community; others were expanded to a wider population to obtain a larger sample (or to use readily compiled data) but at the expense of including a greater proportion of people unlikely to be exposed. Investigators conducting the studies generally seemed aware of this problem and often provided lengthy explanations of the resulting limitations of the study design.

Exposure Assessment

In 25 of the 29 surveys reviewed for this chapter, the main exposure variable was a dichotomous estimate of exposure, usually whether or not the person lived in a defined area near a hazardous-waste site or industrial facility. Measurements from environmental sampling were usually sparse and only indirectly incorporated into the study design; e.g., only a few wells in a neighborhood were sampled.

In a few studies, biologic monitoring was conducted, generally accompanied by a questionnaire assessment of exposure to the substance of concern. The questionnaires often provided valuable information on exposures in a community. However, the biologic monitoring in the studies usually did not include any direct assessment of health end points.

The committee found the general lack of exposure assessment to be a significant weakness in these reports from the gray literature, and the characteristic most in need of improvement.

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Adequacy of Measure of Health End Points

The range of health end points addressed in these studies was as broad as the spectrum of medical practice. They went from vaguely conceived complaints of neurobehavioral dysfunction to well-characterized syndromes of pulmonary hypersensitivity. A limitation of many of the studies was the reliance on available health records for ascertaining cases. State data can be efficient sources of information on health outcomes that can be nearly completely identified and aggregated by geographic area, but only a few health conditions can be adequately addressed in this way, e.g., cancer. While existing state data may provide a relatively inexpensive approach for responding to a community's health concerns, the limitations of these data sources must be recognized. These include availability only for relatively large geographic areas (unless special tabulations can be obtained) and lack of information on confounding factors. Few of these gray literature studies used clinical testing or medical examinations to assess health end points.

Study Results and Significance

A few studies demonstrated significant public-health problems that might not be appreciated by a review of the "white" literature alone, such as elevated levels of mercury in Chippewa Indians who consumed contaminated fish. While isolated positive findings must be viewed with skepticism, some are likely to reflect real risk. The results of this study and of 3 other biologic-monitoring studies examined provide information on human exposure from specific sources of environmental pollution. This information is very helpful for the evaluation of the human health significance of these pollution sources and for general scientific understanding of these types of exposure.

Approximately one-fourth of the studies report on methodologic approaches that might be useful for scientists considering studies of similar situations. For example, medical examination of residents living near hazardous-waste sites might be useful for investigators planning similar studies, even though the results from a particular study were inconclusive. As long as their existence is known to potential users, these studies need not be available in the peer-reviewed literature to help in evaluating a similar situation elsewhere; however, some centralized availability would be helpful.

Likelihood of Acceptability for Publication

Few of the state-sponsored studies would be acceptable for publication without substantial revision. However, in the judgment of the com-

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×

mittee, at least one-third of these studies would be publishable if the authors invested the significant effort needed to extend the analyses and rewrite the reports. Other studies would never be publishable, because of the small numbers of persons, inadequate information on exposure, and potential confounders. This is not to say that the studies were of no value; many studies in the gray literature clearly have objectives other than the advancement of scientific knowledge, such as to allay public worries or to show official concern.

State Studies of Reproductive End Points

To focus attention on the problems and prospects for environmental epidemiology, the committee considered evidence from the gray literature on the possible association of environmental pollution with adverse reproductive outcomes. As a previous National Research Council report noted, the causes of the great majority of poor outcomes in human reproduction are not known (NRC, 1989). Moreover, increases in several of these adverse reproductive outcomes over time have been reported. These reports cannot easily be evaluated, however, because the completeness of the data for many reproductive outcomes, such as miscarriage and congenital anomalies, may have changed over time and is still not uniform in all regions of this country. The committee has found important data gaps in the field of reproductive epidemiology, and this review highlights the research opportunities.

Here we review 4 reports from state agencies that have evaluated the association between reproductive effects and environmental exposures (CBDMP, 1989; White et al., 1989; Johanson, 1991; MDPH, 1983). Three of the 4 studies reported an increased rate of the reproductive effect studied. These investigations illustrate 4 common difficulties in studying reproductive effects. First, exposure was seldom measured directly, so relations between exposure and outcome must be inferred rather than determined. Two, some of the studies lacked adequate control groups, so effects of potential confounders could not be taken into account. Three, not all potential reproductive end points were assessed. Fourth, the studies often had low statistical power to detect an effect of even substantial size. Following the summary of the studies, these difficulties are discussed in more detail.

California Birth Defects Monitoring Program: Investigations of Suspected Clusters of Birth Defects by County, September 1, 1989, California Department of Health Services

This report examines 113 investigations of possible clusters of various

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×

categories of birth defects reported to the California Birth Defects Monitoring Program (CBDMP). The possible clusters were recorded between April 1981 and September 1989. Of the populations with these 113 reported clusters, 8 were found to have statistically significantly elevated risks. Birth defects were identified and evaluated appropriately with an active surveillance system based on a review of medical records through the first year of life. For cluster studies, exposure was not determined for individual cases, and place of residence was the proxy for exposure. For case-control studies, exposures were determined by interviews. Five of the clusters had been or were being investigated with case-control methods. Two were not followed up, because the rate of birth defects of interest went down in subsequent years.

One cluster of brain tumors was referred to the California Tumor Registry. In a separate study of 8 counties where there was concern about an elevated rate of limb reductions in the offspring of agricultural workers, the CBDMP found an excess in the general population. The program did not have sufficient resources to examine rates in offspring of agricultural workers separately. The numbers of cases are not given for the studies; thus, the power cannot be determined. Because the outcomes of interest are rare and small populations were covered by most of the investigations, the committee assumes that these 113 investigations may have generally had low power.

Michigan Department of Public Health: Evaluation of Congenital Malformation Rates for Midland and Other Selected Michigan Counties Compared Nationally and Statewide, May 4, 1983

The Michigan Department of Public Health examined birth defect rates in the Midland, Mich., area, the site of a substantial chemical industry, for the years 1970-1980 in response to concerns from the community about environmental and occupational exposures to certain chemicals. Information on birth defects was obtained from the Birth Defects Monitoring Program (BDMP) of the Centers for Disease Control (CDC) and from birth and fetal-death records at the Michigan Department of Public Health. This is a descriptive study that serves as a screen to identify areas for future research. It is divided into 2 parts: the birth-defect rates for Midland County, based on the CDC BDMP databases and the birth-defect rates for the state of Michigan, based on state birth and fetal-death records.

In the first part of the study, the investigators used the BDMP database to determine the number of observed cases, the number expected from rates in the general population, and the ratio of the observed to expected numbers. They used county of birth as a surrogate for exposure.

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×

Investigators found that of the 37 congenital-malformation rates followed in the database from 1970 to 1980, only one rate, hip dislocation without central nervous system (CNS) complications, was found to be statistically significantly higher in Midland County than in the United States. They believed that the study of 37 different rates could produce 1 or 2 elevated rates because of chance. They concluded also that the rate of hip dislocation without CNS complications was not expected to be related to environmental or occupational agents. Furthermore, the increased rate was not peculiar to this county but was significantly high for 5 other counties in Michigan during the same period.

The authors discussed 4 important problems in their use of the CDC BDMP database. First, there were 3 changes in the coding scheme over the 11-year period examined: from 1970 to 1973, the data were classified with the first edition of the International Classification of Diseases, Adopted Code for Hospitals (HICDA I); from 1974 to 1978, the data were classified with the second edition; (HICDA II); and from 1979 onward, the data were classified with the International Classification of Diseases, Adapted, Clinical Modification (ICD-A-CM). Code categories were added, deleted, and modified by these changes, so consistency in the overall database was reduced. However, they did not report on whether the specific outcome found to be elevated was affected by the changes. The second principal problem with the CDC BDMP database is underestimation of the number of infants with major anomalies, because the data were derived from hospital medical-discharge data, which do not capture anomalies noted only at outpatient visits. The third problem is that, during the period studied, the facilities participating in the system changed. For example, a hospital may have participated for only 5 of the 10 years under study. Finally, the place of residence of the parents is not listed. Other data show substantial in-migration of women delivering babies; only about 68% of live births in Midland County are to residents of that county. Out-migration is a smaller problem; 92% of live births and fetal deaths with a selected anomaly born to Midland County residents are delivered in Midland County.

The second part of the study evaluates the data from the birth and fetal-death records from the Michigan Department of Public Health. The quality of data is uncertain because the only sources of data used are birth certificates and fetal death certificates. Thus, the anomalies would have to be apparent at birth and recorded as the immediate cause of death or as a significant medical condition contributing to death. Reporting is often incomplete and, when compared with the CDC BDMP database, some anomalies in the state of Michigan appear to occur 25% less often than those in Midland County. In this example, the birth and death records in Michigan include documentation for only 25% of the cases. Two of the

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×

anomalies, oral cleft and clubfoot, are reported as consistently in the Michigan database as in the BDMP database from 1970 to 1980.

There were 13,689 births to Midland County residents in 1970-1980. Of these, 136 were recorded in the BDMP database as having one or more of the selected anomalies. Grouping all congenital abnormalities from the Michigan birth and death records produced significantly higher rates, i.e., an observed/expected rate ratio of 1.71 for the years 1970-1975 and of 1.37 for the years 1976-1981. Investigators found 4 anomalies (cleft lip with or without cleft palate, cleft palate without cleft lip, hypospadias, and hip dislocation without CNS defects) to have significantly higher rates in Midland County than in the state of Michigan for the years 1970-1975. Furthermore, only the rates for hip dislocation without CNS defects were significantly elevated from 1970 to 1981. Investigators grouped cleft lip and cleft palate into one category, oral cleft, and examined the rates for this group. They found a ''run" of successive years of higher-rates-than-normal of oral cleft and commented that other counties experienced similar runs. Also, Midland County's rates of oral cleft fluctuated from highest to lowest from 1976 to 1981. Investigators concluded that a case-control study is needed to assess factors that might contribute to the higher rates of oral cleft for Midland County.

Grouping of possibly related abnormalities—for example, all those in a single organ, such as the heart—could raise the power of a study to detect a significant relation. Such biologically meaningful grouping has at least 2 advantages. First, it reduces the problems of imprecise and overlapping case definitions for specific abnormalities, such that different observers may diagnose and record the same defect differently. Second, such grouping recognizes the possibility that different abnormalities of one organ could stem from similar causes, although specific expressions of the defect could differ. On the other hand, if these abnormalities stem from different causes, pooling data risks a dilution of effect. Therefore, both "lumping" and "splitting" strategies need to be explored and considered in studies of birth defects.

Additional studies of possible environmental causes of reproductive abnormalities are warranted, given the data in this report. In addition, the development of more-sophisticated analyses of existing data should be explored, including techniques to group diagnostic entities in a meaningful manner.

Louisiana Department of Health and Hospitals: Final Report of St. Gabriel Miscarriage Investigation, East Bank of Iberville Parish, Louisiana, September 27, 1989

Residents of St. Gabriel, LA, were concerned about an elevated miscarriage rate and a possible link to air contaminants. Researchers from

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×

the Tulane University School of Public Health and Tropical Medicine (White et al., 1989) conducted a descriptive study for the Louisiana Department of Health and Hospitals in response to these concerns from the community. They collected information from 354 women who were 18-50 years old at the time of the study, had conceived between April 1, 1982, and April 1, 1987, and had miscarried or delivered between May 1, 1982, and December 31, 1987. During protocol development, the investigators decided not to select a comparison community. They based this decision on problems with recall and selection bias and other factors such as inbreeding, that might affect the rates of miscarriages in small community comparison groups but would be difficult to assess. They instead decided to compare the rates to a range of rates of miscarriages that were previously reported in well-designed and documented studies. A priori, they decided that miscarriage rates below 15% would be considered normal; rates from 15% to 24% would be cause for examining known risk factors; and rates greater than 24% would be cause for further investigation. The investigators used proximity to industrial sites during pregnancy as the measure of exposure. Stillbirth rates were compared with the rates for all of Louisiana.

The sample was obtained from volunteers through extensive community outreach and from nonvolunteers through a review of hospital records, emergency-room logs, and vital records. The study of volunteers was by telephone. The investigators determined that, of 877 telephone numbers in the area, one would expect about 232 eligible participants (women 18-50 years old who had been pregnant), but they found only 119 eligible, or 51% of the number expected. The researchers attributed the low number of observed eligible women to problems with recruitment, and they used hospital records to find the rest of the expected eligible women. No information was given on biases that may have been introduced by this low rate of response. If there was a problem in finding women, then there may have been a problem in determining miscarriages.

Reported miscarriages were divided into those that were documented (by medical records) and undocumented. The 354 eligible women had 372 live births, 7 stillbirths, and 69 miscarriages (54 documented and 15 undocumented). Researchers found a documented miscarriage rate of 12.7% ± 1.6% and a documented plus undocumented miscarriage rate of 15.7% ± 1.7%. After age adjustment, they found a miscarriage rate among whites of 17.4% ± 2.9% and among blacks of 13.2% ± 2.1%. The researchers commented that white women in the study might be more likely to have their miscarriages recorded because they were more likely to seek medical attention and to release their medical records. Whatever the reason, without a suitable control group it is impossible to tell whether these rates are elevated. First, ascertainment may have been either more or less

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×

complete than in other studies. Second, the rates in a small area may differ from rates in other studies because of differences in ethnic group, economic status, social class, or other factors. There are also considerations of random variation in small samples. The stillbirth rates were based on very small numbers and not statistically different from those for the rest of the state. However, while rates for white women (6.6/1000) were similar to rates for white women in the whole state (6.4/1000), rates for black women (22.9/1000) were nearly double the state rate for black women (12.1/1000), p = 0.08.

To assess exposure, the researchers classified the 400 pregnancies that had mailing addresses according to their distance from chemical plants and other industrial sites in the area. Subjects were divided into 3 categories: less than 0.5 mile, 0.5-1 mile, and over 1 mile. The investigators found no statistically significant relation between miscarriage rates and proximity to industrial sites. They also divided the pregnancies into 2 calendar-year-of-conception groups, using dates of delivery and gestational ages. They did not find a difference in miscarriage rate between conceptions in 1984-1985 and those in 1982-1983.

The researchers concluded that miscarriage rates were not elevated and that further studies of the rates were not warranted.

University of Texas Health Science Center at Houston, School of Public Health: W. Johanson, An Analysis of a Health Effects Survey Conducted by Residents Living Near a Toxic-Waste Site, December 1991 Thesis for Master of Public Health Degree, The University of Texas Health Science Center at Houston, School of Public Health

For years, concern had been expressed that exposures from waste-disposal facilities near Brio, Tex., endangered the health of nearby residents. A citizen-generated survey compiled evidence of environmental pollution and suggestions of a broad array of health problems, including an increased rate of birth defects in children conceived during the period of waste-disposal operations. This survey was conducted from January to September 1990, using volunteers to canvass the neighborhood and to distribute questionnaires to households. One person in each home was asked to complete and return the questionnaire. The quality of responses varied widely. Some respondents were very thorough and included detailed information on visits to doctors and dates. Others included only vague descriptions of health problems, no physician names, or no dates, including no birth dates. No information was collected to examine participation rates in various areas of the community, so there was an unquantifiable potential for response bias. Reported symptoms and diag-

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×

noses were not verified independently. A team from the University of Texas School of Public Health (UTSPH), led by a physician-epidemiologist, conducted an independent evaluation of these same data from Brio, Tex. The investigators divided the population into 3 zones of potential exposure based on proximity to the waste site and wind patterns in the area. Zone 1 was defined as adjacent to the waste site, Zone 2 was 1,460-3,000 ft (0.4-0.9 km) and downwind from the site, and Zone 3 was 2,100-4,100 ft (0.6-1.2 km) from the site and away from the prevailing winds.

The UTSPH team evaluated 652 household-response forms for various health effects. Respondents reported 121 pregnancies, of which 25 (20.7%) ended in spontaneous abortion. They also evaluated the rates of reported birth defects and used data obtained from the CDC Congenital Malformations Surveillance Report (CDC, 1988) to estimate expected rates. Among the 96 live births, 18 (19%) were reported to have had congenital abnormalities. The investigators attempted to correct for recall bias by using a conservative risk estimate. They assumed that the number of cases recorded in 1990 for about one-third of the area was the annual incidence for the entire population that lived in the area over the period 1983-1989, while also assuming that the medical end points were valid as reported. With this assumption, they found a lower-bound relative risk of 2.4 for congenital birth defects and 3.8 for major CNS malformations. In 181 women in the study 19-50 years old, there were 126 pregnancies, for a fertility rate of 0.7 births per woman per year. This seems very high, even in the absence of a control group, and may indicate a serious problem in the data.

There may have been biases in the ascertainment of cases and confirmation of reported congenital malformations. First, prior to the health survey, extensive media coverage about the site included anecdotal reports about adverse health effects. This may have biased interview responses. Secondly, the survey was conducted by volunteers, and the response rate was low, leading to the possible biases. While there was some attempt to standardize the questioning procedure, there was no recording of home visits and outcomes or of attendance at training sessions. Volunteers may have been especially diligent in seeking positive responses, which would enhance the potential for recall bias. Third, the form was to be filled out by the interviewee rather than the interviewer, which could cause differences in interpretation of the questions and hence increase uncertainty in the answers. Lastly, no attempt was made to confirm the diagnoses by contacting physicians. Thus, there is no independent verification of reported cases. Other factors also inhibited the development of an independent assessment of this problem, including the protracted nature of the dispute, the inability to gather independent information, and the difficulty of obtaining validated measures of exposure.

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×

Analysis of Studies Reviewed

As with many public-health studies, none had a direct measure of the exposure that might have caused the observed outcome. The investigations addressed exposure assessment in different ways. The investigators in Texas used a surrogate measure of exposure that depended on distance to the toxic-waste site and the prevailing winds. In the Louisiana study, the researchers classified pregnancies by distance to industrial sites. In the Michigan and California studies, the place of residence was the surrogate for exposure, and rates were compared with background rates.

Another problem is the lack of non-exposed or less-exposed persons as controls. Comparing regional rates of outcomes with national rates can give misleading results because the national averages do not take into account regional characteristics (e.g., race and socioeconomic status) that may differ from characteristics of the general population. Many communities near hazardous-waste sites consist of minority groups, are economically disadvantaged, or both (Commission for Racial Justice, 1987; Bullard, 1990; EPA, 1992a,b). In the Michigan study, investigators found a higher rate of oral clefts in Midland County than in the state of Michigan. They commented that this would be expected because there was a higher proportion of whites in Midland County than in the state and because higher rates of oral cleft are reported in whites than in blacks. Similarly, the Louisiana study found different miscarriage rates among whites and blacks, indicating that race is a confounder. The Texas study also lacked a control group. For the California studies, availability of a population-based registry of birth defects allowed comparisons to be made among similar areas with identical means of case ascertainment and verification.

A third difficulty in examining potential reproductive hazards comes from the many reproductive and developmental health effects that could be studied (Mattison et al., 1989, 1990). Examination of multiple end points is necessary to avoid possibly missing some important effects. None of the investigations looked for changes in fertility rates.

One way to improve the power of a study is to group related defects. In the early stages of investigating possible links between outcome and exposure, one must consider all plausible outcomes to capture those that are most likely to occur from the exposure (Axelson and Söderkvist, 1991). However, a major problem is that grouping may be biologically implausible for certain birth defects. On the other hand, if the effects examined are from exposure to a hazardous-waste site, where there may be exposure to multiple agents, it may be appropriate to group all defects, since different agents may cause different adverse outcomes.

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×

Reports from Other Countries

In accordance with its effort to examine the relevant literature on environmental epidemiology, the committee surveyed some preliminary findings that have been developed on environmental health in some countries other than the United States, focusing on the literature from China and eastern Europe. Given the difficulties in translation and access to information, the committee is not able to vouch for the quality of data from these countries. Instead, we have relied on reports developed by expert consultants to the World Bank, the Pan American Health Organization, and other international organizations. These studies do not appear in the peer-reviewed literature but are available from the sponsoring organizations. They represent the work of visiting teams of well-qualified researchers who have generated important analyses using accepted techniques. As is often the case, commonly available information tends to focus chiefly on patterns of environmental contamination. However, in several instances, analyses of geographic variations in disease have also been produced.

Examination of patterns of disease within countries can yield important clues about environmental causes of disease. Urban living tends to be correlated with greater smoking and alcohol drinking, as well as access to health services—factors that clearly influence health outcomes and their recording. A marked urban gradient is evident for many diseases, including lung cancer. In contrast, for outcomes that are dependent on lack of health services, such as maternal mortality, rates tend to be higher in rural areas (Bertaud and Young, 1991).

In some Chinese cities, dramatic environmental inequity exists, despite relatively homogeneous income levels and universal health services. In Tianjin in 1988, female lung-cancer mortality was 9 per 100,000, the highest in China; and in the more-urban sections of this area, the rate averaged 30 per 100,000, ranging from 10 to 88. Approximately 13% of the population lives in districts where the yearly rate of female lung cancer is higher than 40 per 100,000. This is more than 12 times the national average (Bertaud and Young, 1991). Smoking is unlikely to be a major determinant of these differences, as about 12% of females in China were smokers (Junshi et al., 1990), and smoking patterns are not likely to vary greatly within small areas.

The causes of these elevated rates of lung cancer in a largely nonsmoking population need to be sought but are likely to reside in environmental factors that are not yet identified. In some areas of China, researchers have linked elevated rates of lung cancer to the use of smoky domestic fuels and cooking oils (Mumford et al., 1987). Researchers from China and the US National Cancer Institute have recently confirmed that

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×

the elevated risk of lung cancer in nonsmoking women can be linked to a variety of exposures to household smoke (McIntosh, 1992).

Studies from Rumania show a similar result, with a marked urban gradient for many diseases. Given the highly centralized nature of the former government, health statistics are believed to be reliable. Lung cancer in women exhibits interesting variations. Women in some areas of the country have rates of lung cancer that are 5 times the rates in other areas. In general, women have not smoked much, although those in cities smoke more than those in rural areas. Overall rates of cancer in Rumania are lower than those of developed countries.

According to one survey by the ministry of health of the former Soviet Union, people living in 103 cities breathe air that carries at least 5 times the allowed limits of many pollutants (Feshbach and Friendly, 1992). Studies of the impact of this pollution are just now emerging. Some reports from cross-sectional surveys of 68 cities where air pollution is regularly measured have found levels at least 10 times above permissible maximums, with rates of illness more than double those of the national average (Kondrusev, 1990). Almost three-fourths of the nation's surface water is polluted, and one-fourth is untreated.

References

Axelson, O., and P. Söderkvist. 1991. Characteristics of disease and some exposure considerations. Appl. Occup. Environ. Hyg. 6:428-435.


Bertaud, A., and M. Young. 1991. Geographical Pattern of Environmental Health in Tianjin, China: The Development of an Environmental Health Data Base. INURD Working Paper 13, February, 1991. [Washington, DC]: Urban Development Division Sector Policy and Research, The World Bank .

Bullard, R.D. 1990. Dumping in Dixie: Race, Class, and Environmental Quality. Boulder, CO: Westview Press.


CBDMP (California Birth Defects Monitoring Program). 1989. Investigations of Suspected Clusters of Birth Defects by County. September 1, 1989. [California Department of Health Services.]

CDC Centers for Disease Control. 1988. Congenital Malformations Surveillance Report, January 1982-December 1985.

Commission for Racial Justice, United Church of Christ. 1987. Toxic-Wastes and Race in the United States: A National Report on the Racial and SocioEconomic Characteristics of Communities with Hazardous Waste Sites. [New York]: Public Data Access.


EPA (U.S. Environmental Protection Agency). 1992a. Environmental Protection: Has It Been Fair? EPA Journal 18(March/April):entire issue.

EPA (U.S. Environmental Protection Agency). 1992b. Environmental Equity: Reducing Risk For All Communities. EPA230-R-92-008 (vol. 1), EPA230-R-92-008a (vol. 2). Washington, DC: Policy, Planning, and Evaluation, US Environmental Protection Agency.


Feshbach, M.O., and A., Jr. Friendly. 1992. Ecocide in the USSR: Health and Nature Under Siege. New York: Basic Books.


Johanson, W. 1991. An Analysis of a Health Effects Survey Conducted by Residents Living

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×

Near a Toxic-Waste Site. Thesis for Master of Public Health. Houston: The University of Texas Health Science Center at Houston School of Public Health .

Junshi, C., T.C. Campbell, L. Junyao, and R. Peto. 1990. Diet, Life-style, and Mortality in China: A Study of the Characteristics of 65 Chinese Counties. New York: Oxford University Press.

Kondrusev, A. 1990. Poison without illusions: who will guarantee the right to health? From interview with N. Gogolin, Pravda, November 14, 1990, 2d ed., p.6. Pp. 50-52 in Joint Publications Research Service, Soviet Union, Political Affairs. JPRS-UPA-90-066, December 4.


Mattison, D.R., J.W. Hanson, D.M Kochhar, and K.S. Rao. 1989. Criteria for identifying and listing substances known to cause developmental toxicity under California's proposition 65. Reprod. Toxicol. 3:3-12.

Mattison, D.R., P.K. Working, W.F. Blazak, C.L. Hughes, J.M. Killinger, D.L. Olive, and K.S. Rao. 1990. Criteria for identifying and listing substances known to cause reproductive toxicity under California's proposition 65. Reprod. Toxicol. 4:163-175.

McIntosh, H. 1992. Far East's cancer mortality patterns shift, reflecting cultural changes. J. Natl. Cancer Inst. 84:1069-1071.

MDPH (Michigan Department of Public Health). 1983. Evaluation of Congenital Malformation Rates for Midland and Other Selected Michigan Counties Compared Nationally and Statewide, 1970-1981. 33pp.

Mumford, J.L., X.Z. He, R.S. Chapman, S.R. Cao, D. B. Harris, X.M. Lit, Y.L. Xian, W.Z. Jiang, C.W. Xu, J.C. Chuang, W.E. Wilson, and M. Cooke. 1987. Lung cancer and indoor air pollution in Xuan Wei, China. Science 235:217-220.


NRC (National Research Council). 1989. Biologic Markers in Reproductive Toxicology. Washington, DC: National Academy Press.

NRC (National Research Council). 1991. Environmental Epidemiology. Vol. 1. Public Health and Hazardous Wastes. Washington, DC: National Academy Press.


Schmidmaier, D. 1986. Ask no questions and you'll be told no lies: or how we can remove people's fear of "gray literature." Libri 36:98-112.


White, L.E., F.J. Mather, J.R. Clarkson. 1989. Final Report, St. Gabriel Miscarriage Investigation, East Bank of Iberville Parish, Louisiana. September 27, 1989. Prepared by Tulane University School of Public Health and Tropical Medicine, New Orleans, Louisiana, for Louisiana Department of Health and Hospitals, in cooperation with Division of Birth Defects and Developmental Disabilities, Center for Environmental Health and Injury Control, Centers for Disease Control, Atlanta, GA., and Agency for Toxic Substances and Disease Registry, Public Health Service, Atlanta, GA. 55 pp.

Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 154
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 155
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 156
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 157
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 158
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 159
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 160
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 161
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 162
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 163
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 164
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 165
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 166
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 167
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 168
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 169
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 170
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 171
Suggested Citation:"7 Review of the Gray Literature from State Reports." National Research Council. 1997. Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology. Washington, DC: The National Academies Press. doi: 10.17226/5804.
×
Page 172
Next: 8 Major Conclusions and Recommendations »
Environmental Epidemiology, Volume 2: Use of the Gray Literature and Other Data in Environmental Epidemiology Get This Book
×
Buy Hardback | $55.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Determining the health risks to humans of exposure to toxic substances in the environment is made difficult by problems such as measuring the degree to which people have been exposed and determining causation—whether observed health effects are due to exposure to a suspected toxicant. Building on the well-received first volume, Environmental Epidemiology: Hazardous Wastes and Public Health, this second volume continues the examination of ways to address these difficulties. It describes effective epidemiological methods for analyzing data and focuses on errors that may occur in the course of analyses.

The book also investigates the utility of the gray literature in helping to identify the often elusive causative agent behind reported health effects. Although gray literature studies are often based on a study group that is quite small, use inadequate measures of exposure, and are not published, many of the reports from about 20 states that were examined by the committee were judged to be publishable with some additional work. The committee makes recommendations to improve the utility of the gray literature by enhancing quality and availability.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!