National Academies Press: OpenBook
« Previous: Peer-Review Practices at EPA
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

4

Strengthening Science at EPA

THIS chapter summarizes and integrates our committee's most important findings and recommendations concerning research management and peer-review practices and presents related recommendations concerning scientific leadership at EPA. These findings and recommendations are based on the committee's consideration of the matters discussed and documents cited in the preceding chapters of this report and the work and reports of the Committee on Research Opportunities and Priorities for EPA – our companion committee in this study. Members of our committee made many site visits to EPA's laboratories, centers, and headquarters and regional program offices, as listed in the first chapter of this report. In addition, several members of our committee have gained considerable knowledge of EPA research-management and peer-review practices through participation in previous and ongoing reviews of various scientific activities in EPA under the auspices of the NRC, the SAB, the BOSC, and other organizations.

Based on all these factors, the committee concludes that the following broad themes and the recommendations associated with them, as presented below, are of paramount importance to the goal of strengthening the scientific performance of ORD and the agency overall:

  • scientific leadership and talent;

  • research continuity and balance;

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
  • research partnerships and outreach;

  • research accountability; and

  • scientific peer review.

SCIENTIFIC LEADERSHIP AND TALENT

In the 30 years since EPA was created, the agency's scientific practices and performance have been criticized many times in reports from the NRC, EPA's SAB, the General Accounting Office, and many other organizations; in congressional oversight and judicial proceedings; and in countless criticisms and lawsuits from stakeholders with interests in particular EPA regulatory decisions. In one such report, Safeguarding the Future: Credible Science, Credible Decisions (EPA 1992), a panel of academicians, including two members of our committee, concluded, “Currently, EPA science is of uneven quality, and the Agency's policies and regulations are frequently perceived as lacking a strong scientific foundation.” While acknowledging that EPA had a number of knowledgeable scientists on its staff, the panel reported that the science base at EPA was not perceived to be strong by the university community, and that many EPA scientists at all levels throughout the agency believed that EPA did not use their scientific knowledge and resources effectively. The panel further observed “A perception exists that regulations based on unsound science have led to unneeded economic and social burdens, and that unsound science has sometimes led to decisions that expose people and ecosystems to avoidable risks.” The panel commented that EPA had not always ensured that contrasting, reputable scientific views were well-explored and well-documented from the beginning to the end of the regulatory process. It pointed out that the agency was often perceived to have a conflict of interest because it needed science to support its regulatory activities, and it described a widely held perception by people both outside and inside the agency, that EPA science was “adjusted” by EPA scientists or decision-makers, consciously or unconsciously, to fit policy.

As discussed in many places throughout this report, EPA has made significant improvements in some of its scientific practices since that panel issued its report in 1992. However, the committee concludes that there is a continuing basis for many of the scientific concerns raised in

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

that panel's report and others, such as the 1999 Resources for the Future report Science at EPA: Information in the Regulatory Process (Powell 1999). We base this conclusion on the extensive experience of the members of our committee in assessing EPA's scientific practices and performance, including the matters discussed and documents cited in this report and other independent investigations of EPA science in which members of our committee have participated.

The creation of a new position of science advisor to the administrator of EPA was one of the principal recommendations of the 1992 report. It envisioned that the essential function of the science advisor would be to ensure that EPA policy decisions are informed by a clear understanding of relevant science. The panel recommended that the new science advisor advise the EPA administrator, implement a peer-review and quality-assurance program for all EPA science-based products; be a key player when EPA makes a policy decision, ensuring that the science and uncertainties relevant to a policy or regulatory issue are considered; play a key role in evaluating the professional activities of EPA scientists; reach out to the broader scientific community for information; and maintain an appropriate relationship with EPA 's SAB. The panel suggested that the role of the new science advisor might be somewhat analogous to the role of the general counsel, who will not approve a document destined for an external audience until it is judged legally defensible.

The 1992 panel considered two models: the science advisor could be either a senior scientist on the EPA administrator's staff, chosen by each administrator, or the assistant administrator in charge of EPA's research office (or a deputy to that official). In assessing those options, the panel observed that a science advisor on the administrator 's staff would be more likely to have a relationship of confidence with the administrator but might be somewhat removed from scientists throughout the agency and unable to direct resources to address scientific issues. In comparison, the assistant administrator for EPA's research program would command considerable resources but in some cases might be involved in a conflict of responsibilities as the administrator 's science advisor (e.g., when ORD's work products or program budget were being evaluated). And a deputy to the assistant administrator for ORD might suffer the additional handicap of not being sufficiently senior to exercise a strong role in the inner counsels of the administrator.

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

In response to the 1992 report, Administrator Reilly appointed a science advisor on his immediate staff. Although the individual selected was highly qualified and was able to secure some accomplishments in his brief tenure, that individual was never given the authority that would be required to perform the agency-wide role envisioned by the 1992 panel concerning the improvement of EPA's scientific credibility. The science-advisor position did not survive long after the 1993 change in administration and is currently vacant.

Throughout EPA's history, no official below the level of the administrator has had overall responsibility or authority for the scientific and technical foundations of agency decisions, and administrators of EPA have typically been trained in law, not science. The agency's most senior science official has traditionally been the assistant administrator for research and development, but that official has never had agencywide responsibility or authority for overseeing the scientific and technical basis for regulatory and policy decision-making, and EPA's regulatory offices are not required to follow scientific advice from ORD. That is a formula for weak scientific performance in the agency and poor scientific credibility outside the agency.

In our interim report (NRC 1995b), this committee recommended “that the assistant administrator for research and development be designated as EPA's chief scientific and technical officer, responsible not only for ORD, but also for coordinating and overseeing agency-wide scientific policy, peer review, and quality assurance, as well as EPA's outreach to the broader domestic and international scientific community for scientific knowledge relevant to the agency's mission. ” Shortly thereafter, in partial response to that recommendation, the deputy administrator of EPA asked the head of ORD to coordinate the agency's scientific-planning and peer-review activities.

Although the 1995 designation appears to have been a small step in the right direction, our committee judges it to be insufficient. First, the head of ORD was not given real authority for agency-wide scientific policy. Second, although the agency subsequently achieved some commendable progress through its interoffice Science Policy Council and ORD-led efforts to begin developing an agency-wide inventory of scientific activities and a “Strategic Framework for EPA Science, ” all those efforts, relying on consensus and voluntary cooperation of the agency's regulatory and regional offices in the absence of central

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

science-policy authority, have had slow and limited success. The heads of EPA's regulatory and regional offices are of equal rank to the head of ORD and are generally not required to follow ORD' s guidance regarding scientific activities or science policy. Third, the ability of the head of ORD to coordinate agency-wide peer-review and quality-assurance practices was diminished in 1999 with the reassignment of some peer-review functions from ORD to the agency's newly created Office of Environmental Information.

Furthermore, based on our observations of these developments in the five years since our interim report, the committee has become convinced that our 1995 recommendation to designate the head of ORD as EPA 's chief scientific and technical officer also was insufficient. The committee now concludes that it underestimated in 1995 the level of authority needed to achieve the necessary degree of cooperation and coordination of scientific activities and policy in the regulatory and regional offices. In addition, the committee has become more aware of the enormous amount of scientific activity occurring in EPA's regulatory and regional offices, and it concludes that no single individual could reasonably be expected to direct a world-class research program in ORD while also trying to improve scientific practices and performance throughout the rest of the agency. These jobs are inherently different. Moreover, assigning agency-wide scientific authority to the assistant administrator for ORD might produce a conflict of responsibilities, because many decisions about science in the regulatory programs could affect ORD's budget or favor ORD 's research over research done elsewhere.

Based on many of the issues discussed in this report and the extensive experience of the members of our committee with scientific practices in EPA, the committee concludes that EPA needs an appropriately qualified scientific official at a sufficiently high level to carry both the authority and responsibility for agency-wide scientific performance. No official below the level of deputy administrator could perform this role. The requisite operating authority with accountability for agencywide scientific performance cannot be met by assigning the scientific gate-keeper function to any assistant administrator, including that of ORD, regardless of the qualifications or abilities of the individual holding that position. It is unrealistic to expect that an official at that level could effectively coordinate and oversee the scientific and technical

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

programs and work products of other EPA offices and regions, especially while also carrying out the important duties of directing a first-rate research program. EPA needs a top science official with the authority and responsibility to obtain and use the best possible science in support of the agency's mission and to identify the scientific uncertainties and conflicting evidence relevant to the agency's regulatory and policy decisions.

The importance of science in EPA decision-making should be no less than that afforded to legal considerations. Just as the advice of the agency's general counsel is relied upon by the administrator to determine whether a proposed action is “legal,” an appropriately qualified and adequately empowered scientific official is needed to attest to the administrator and the nation that the proposed action is “scientific” – that it is consistent, or at least not inconsistent, with available scientific knowledge – and that the agency has done a proper job of ascertaining and applying that knowledge and recognizing and characterizing the relevant uncertainties. Achieving these goals will require a level of accountability for EPA's scientific performance that cannot reasonably be expected from an administrator who is not trained in science, a staff advisor to the administrator without line authority, or an assistant administrator for research and development who has no authority over the use of scientific information by other offices of the agency.

Therefore, to enhance the effective and appropriate use of science in EPA and the agency's scientific credibility, the committee recommends the establishment of a new position at EPA: deputy administrator for science and technology.

This position would require authorization from Congress, appointment by the President, and confirmation by the Senate. The current position of deputy administrator might become deputy administrator for policy and management. The new deputy administrator for science and technology would have the following principal responsibilities:

  • Ensure that the most important scientific issues facing EPA are identified and defined, including those embedded in major policy or regulatory proposals.

  • Develop and oversee an agency-wide strategy to acquire and disseminate the necessary scientific information either through

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

intramural efforts or through extramural programs involving academia, other government agencies, and the private sector in this country and abroad.

  • Ensure that the complex scientific outreach and communication needs of the agency are met, including the need to reach throughout the agency for credible science in support of the regulatory offices, regions, and agency-wide policy deliberations, as well as the need to reach out to the broader domestic and international scientific community for scientific knowledge that is relevant to an agency policy or regulatory issue.

  • Coordinate and oversee scientific quality-assurance and peer-review activities throughout the agency, including activities in support of the regulatory and regional offices.

  • Develop processes to ensure that appropriate scientific information is utilized in decision making at all levels in the agency.

  • Ensure and, in effect, certify to the administrator and the nation that the scientific and technical information used in each EPA regulatory decision and policy is valid, appropriately characterized in terms of scientific uncertainty and cross-media issues, and appropriately applied.

The deputy administrator for science and technology would perform all of the roles envisioned by the Safeguarding the Future panel (EPA 1992). This official would coordinate and oversee the agency's Office of Research and Development, the newly-created Office of Environmental Information, Science Advisory Board, Science Policy Council, and scientific and technical activities in the agency's regulatory program and regional offices.

The individual appointed to this position would need to have an outstanding background, including research experience, experience in public forums, and the respect of scientific peers, administrative peers, and legislators.

The creation of this position would send a strong message that Congress and the administration are committed to strengthening science at EPA.

The position of next importance to EPA's scientific performance and reputation is the assistant administrator for ORD. Over the years, there has been occasional debate on whether the assistant administrator for

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

ORD should continue to have a political appointment or a longer-term statutory appointment. It is sometimes argued that a political appointment ensures that the head of ORD will have power in budget negotiations similar to that of the politically appointed heads of EPA's regulatory and regional offices. However, although the political aspect of the assistant administrator's job often receives considerable attention, the most important aspects of the job are not political. Besides running ORD, the assistant administrator is responsible for representing EPA's research program to the world's scientific community, interpreting the agency's research and development program to the public, and maintaining a national and international watch on major environmental science and engineering developments.

Under the present political-appointment model, the leadership of ORD changes at least as often as the administration changes. Historically, the typical tenure of ORD assistant administrators has been only about 2 or 3 years. Frequent changes in the leadership of ORD have been disruptive and have led to devastating effects on the continuity of programs and the morale of ORD scientists and staff. Over the years, the assistant administrator for ORD has typically been one of the last senior EPA officers appointed in a new administration, and although ORD has had some very capable assistant administrators, there have been a few cases in which little weight seems to have been given to the candidate's scientific or managerial qualifications.

Therefore, to foster greater continuity in the management of EPA's research program, the committee recommends that the position of assistant administrator for ORD be converted to a statutory term appointment of 6 years.

The assistant administrator for ORD should have an advanced degree in an appropriate scientific or technological discipline, a substantial record of scholarly achievement, and administrative experience that includes successful management of a substantial research program. The position should be defined to make it attractive to an eminent scientist or engineer who is willing to remain in the position for a sufficiently long period of time to bring stability to the direction and leadership of ORD. A statutory term appointment would make the position more like those of the leaders of NIH or NSF. Congressional action would be required to convert the position to a 6-year term.

Any organization that aspires to excellence must have effective lead-

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

ership at many levels, not just at the top. Criteria for the selection and advancement of managers at all levels in ORD laboratories and centers should favor research managers who not only are capable managers and supervisors, but also are scientifically and technically accomplished. They should understand the work and have the respect of their research staff while having the ability to select, inspire, lead, and otherwise encourage other scientists and engineers in their endeavors. They should also be strong advocates and defenders of the continuity and core capabilities required for the conduct of a good research program. EPA's ability to recruit, develop, and retain such leaders depends on many factors, including the agency's commitment and success in reducing bureaucratic impediments and finding ways to increase the flexibility afforded to research managers to fulfill their responsibilities.

The committee recommends that ORD make a concerted effort to give its research managers a high degree of flexibility and accountability. They should be empowered to make decisions at the lowest appropriate management level consistent with EPA policy and ORD's strategic goals and budget priorities.

ORD and EPA should make a special effort to resist a tendency commonly seen in large institutions to impose cumbersome bureaucratic procedures in response to management concerns. Bureaucratic paranoia, fault-finding, and fear of possible fault-finding can paralyze and demoralize an organization. Excessively bureaucratic procedures are antithetical to a creative research program with a high standard of quality, efficiency, and teamwork. ORD should frequently examine itself to identify and eliminate excessive bureaucratic safeguards, administrative hurdles, redundant requirements for approvals at multiple levels of management, and other bureaucratic impediments.

Research leaders at all levels in ORD should strive to minimize bureaucratic impediments, provide timely responses to requests from other organizations and from staff scientists and engineers, and place high priority on finding ways to increase flexibility in getting research done. ORD managers throughout the organization should be given the authority and resources to make decisions at the lowest appropriate level of management, provided that such decisions are compatible with EPA policies and ORD's strategic goals and budget priorities. Decisions that might fit within this category include research problem se-

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

lection; acquisition of most equipment and supplies, personnel assignment; staff attendance at scientific meetings; inviting and supporting visiting scientists; and granting permission and support for ORD scientists to work for a time in another laboratory in this country or abroad.

To achieve scientific and technical excellence, ORD, EPA, or any other institution must also attract, retain, and properly support first-rate, dedicated professionals on its staff. Our committee is aware of many outstanding scientists and engineers in ORD and other parts of the agency. As discussed in Chapter 2, the committee's site visits were especially encouraging in that regard, and the committee concurs with the judgments expressed by ORD's BOSC (EPABOSC 1998a-e) concerning the substantial number of highly capable and productive staff members in ORD's laboratories and centers.

The ORD work force is aging. More than 47% of ORD's employees are 50 years old or older, and more than 550 ORD employees will be eligible to retire within the next 5 years. Periodic EPA hiring freezes, combined with intense scientific and technical job-market competition from the private sector and academic institutions, are making it extremely difficult for ORD to recruit the new talent needed to sustain and enhance its research work force.

The committee commends ORD for the excellent fellowships program it has developed. This program has brought a stream of fresh scientific and technical talent into EPA's research program and is helping to train future research leaders in environmental science, engineering, and other disciplines.

The committee recommends that ORD continue to place high priority on its graduate fellowship and postdoctoral programs.

The 1992 Safeguarding the Future report recommended that ORD recruit and support on a long-term basis four to six senior research scientists and engineers with world-class reputations in areas vital to EPA's long-term strategy and direction. The panel envisioned that these eminent scientists and engineers would serve as examples and mentors for all scientists in EPA and would bring access to networks of world-class scientists to benefit the agency. The panel recommended that EPA's SAB be asked to form a search committee.

As discussed in Chapter 2, ORD responded to this recommendation by obtaining and using special authority to recruit and promote research scientists and engineers to senior, nonmanagerial positions — so—

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

called “ST” (scientific and technical) positions at the Senior Executive Service level – using merit-review panels that include outside experts to evaluate candidates. ORD now has eight such individuals in its laboratories.

Our committee fully supports ORD's use of the “ST” program and urges that it be continued. However, recognizing today 's intense job-market competition with industry and academic institutions for top research talent, the committee concludes that even greater measures are warranted and practicable to attract and retain outstanding research scientists and engineers in the ORD laboratories.

In research, perhaps even more than in other fields, pre-eminent leadership sets the standard and tone for the rest of the work force. A single world-class investigator can generate ideas and enthusiasm to elevate a research program dramatically. In research, leadership is not synonymous with management or administration.

The committee recommends that ORD create the equivalent of endowed academic research chairs in the national laboratories.

This might be accomplished by awarding prestigious 5-year renewable grants or distinguished fellowships to distinguished academic scientists to work at ORD laboratories; the committee suggests that it is not necessary for all such researches to be regular federal employees. Alternatively, ORD could seek authority to create and fill positions similar to NIH Title 42 senior research appointments, perhaps in connection with internal grants to ensure sustained research support for these distinguished investigators. The SAB or the BOSC should be asked to assist ORD in developing this program and selecting the candidates.

Using such mechanisms, ORD should recruit at least one additional distinguished investigator per laboratory in areas of research that are rapidly advancing and important to ORD's future. These individuals should be given maximum intellectual freedom to pursue potentially productive lines of research that are consistent with laboratory missions and ORD priorities. The distinguished investigators should be expected to serve as mentors and role models for other ORD research scientists but should have no managerial responsibility beyond their own research teams.

The committee supports ORD's use of individual development plans for each professional staff member. The plans, negotiated between an

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

employee and supervisor, address career paths, training, and rotational goals. The committee also supports the work of ORD's Human Resources Council to help managers at all levels address staff support and morale issues, including efforts to reduce bureaucracy, improve two-way communication between management and staff, provide career-development opportunities, and address infrastructure problems.

RESEARCH CONTINUITY AND BALANCE

Frequent changes in goals, priorities, practices, structure, or funding can disrupt any organization, but they are especially damaging to a research organization, which has special requirements for continuity in the development and maintenance of scientific and engineering talent, experience, and infrastructure if it is to be productive. Maintaining the requisite degree of stability in ORD has been a continuing challenge due to many expansions and other changes in EPA's legislative mandates and priorities, directives from Congress and different administrations, pressures from regulated parties and other interest groups, lawsuits and court decisions, inadequate budgets to meet competing demands, and changes in the leadership of ORD and EPA.

ORD has changed its research goals, priorities, and practices often and abruptly in the past. Greater stability, continuity, and predictability are among the most important needs in the ORD program. Becoming competent in a particular line of research requires a large investment of time, effort, and often dollars that often cannot be readily shifted to some new area of concern. The limited financial and human resources of ORD should be managed with a steady hand and a clear and persistent vision of how to invest its resources to maximize the long-term gains in scientific understanding resulting from its work.

Our committee's interim report (NRC 1995b) endorsed the general scope and direction of ORD's 1995 reorganization, in which ORD adopted as a principal organizing concept the reduction of uncertainty in risk assessment and risk management, initiated a new strategic research planning process, consolidated its laboratories and centers, and expanded and strengthened its research grants, centers, fellowships, and peer-review programs. Our interim report called these the most important changes in the history of EPA's research program. In the

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

long run, the courses ORD set for itself in 1995 should have a stabilizing effect on the research program, and our committee continues to support them. In many respects those changes are still a work in progress, and they need more time to mature.

Therefore, in addition to converting the position of assistant administrator for ORD to a statutory term appointment of 6 years, as recommended in the previous section of this chapter, the committee recommends that ORD continue steadily on the courses it set in the 1995 reorganization.

In the past few years, ORD has explored a multiyear approach for research planning to foster continuity and strategic integration of some research efforts. ORD has developed multiyear plans for its particulate-matter research program, as recommended by another NRC committee (NRC 1998a), and also for research on endocrine disruptors, drinking water, environmental monitoring, global climate change, and pollution prevention. Multiyear plans for additional areas of research are under development. The plans are developed by research teams from ORD laboratories and centers and are peer reviewed. Our committee expects that ORD's recent efforts in multiyear planning will contribute greatly to research program continuity and the achievement of strategic goals, and ORD merits commendation for these initiatives.

The committee recommends that ORD continue and expand its multiyear research planning approaches in both problem-driven and core research areas.

As recommended by our companion committee (NRC 1997), ORD also needs to maintain a balance between problem-driven research and technical support for the agency's regulatory programs, and the core research to better understand and anticipate environmental risks. These roles are not unrelated or incompatible; they are mutually reinforcing. Core research is the indispensable wellspring that prepares and enables ORD to provide better problem-specific research and technical assistance to the agency and the nation.

ORD should meet the continuing challenge to lead the agency by means of research while continuing to assist its client regulators who have variable levels of understanding or appreciation of science but a strong say in ORD's budget and priorities. Activities in support of regulatory programs often have a narrow focus and often compete to pre-empt long-term research programs. They tend to consume the re-

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

sources of a research organization disruptively and disproportionately. Regulatory strategies, which are typically prescriptive and specific, tend to freeze concepts and methods in time, and the more closely that ORD is tied to the regulatory programs, the greater the risk that ORD will to some extent be working on outdated problems or with outdated approaches.

On the other hand, using ORD for problem-driven research and technical assistance to regulatory and regional offices has important benefits. Perhaps the greatest dividends are the resulting improvements in the scientific aspects of the agency's regulatory decisions and the maintenance of an in-house scientific core group experienced in dealing with environmental risks and programs. An experienced scientific core group can be of great value in meeting emergency requirements for technical expertise. ORD's technical assistance provides the regulatory offices with competent scientific help, and it enables ORD's research scientists to keep abreast of regulatory and policy developments elsewhere in EPA.

The committee concurs with the recommendations of the 1997 report of its companion committee — the Committee on Research Opportunities and Priorities for EPA — that ORD should maintain approximately an even balance between core research and problem-driven research.

RESEARCH PARTNERSHIPS AND OUTREACH

From time to time the question arises whether EPA should have its own research program or rely on research results developed elsewhere. Advocates of having the research conducted elsewhere often cite past criticisms of the agency's research program and point to excellent research programs of other agencies, such as NIH, NSF, and the Department of Energy, that collectively, and in some cases individually, dwarf that of EPA. And, of course, the academic community and the private sector conduct much of the research relevant to EPA's mission.

In the 1992 report Safeguarding the Future: Credible Science, Credible Decisions, a panel of four senior academicians, including two members of our committee, concluded that EPA needs its own strong science base to provide the background required for effective environmental

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

protection programs. Similarly, Building a Foundation for Sound Environmental Decisions, the 1997 report of the NRC's Committee on Research Opportunities and Priorities for EPA – our companion committee in this study – concluded that EPA needs a strong in-house research program.

Our committee agrees that a vigorous research program should be maintained in EPA. Moving the research program out of the agency would most likely weaken, not strengthen, the scientific foundation of EPA' s decisions and actions. Although some abstract concept of scientific “quality” might be improved by reducing some kinds of ORD technical work that are unlikely to advance research frontiers, such work is often critically necessary to EPA's pursuit of its mission and statutory responsibilities. Overall, eliminating ORD or moving its functions out of EPA would be destructive, and the level of damage would increase with passing time as EPA became increasingly unable to pursue, apply, or even understand new research knowledge. An EPA devoid of a research program would not be likely to attract substantial scientific talent, and an EPA without scientific talent would be ineffective and potentially harmful to the nation.

However, even with a much larger budget, ORD could not meet all the vast and constantly expanding needs of EPA and the nation for scientific and technical knowledge to guide environmental protection efforts. ORD has had a first-rate research program in some important areas, such as aquatic toxicology and human inhalation toxicology, but it is not possible for ORD to be a leader across the full range of scientific knowledge required by EPA. EPA should recognize the limits of its research capabilities. It should be a leader in some areas of research, involved in other selected areas, and well-informed in all areas relevant to the agency's mission. As discussed in Chapter 2, ORD has been criticized at times for failing to obtain and apply the results of research performed elsewhere.

The committee recommends that ORD place greater emphasis on maintaining awareness of research conducted by other organizations. ORD should develop and implement a pro-active, structured, and visible strategy for stimulating, acquiring, and applying the results of research conducted or sponsored by other federal and state agencies, universities, and industry, both in this country and abroad.

The strategy should include increased sabbatical assignments for

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

ORD researchers to gain experience in other scientific organizations, and more visiting appointments of scientists and engineers from universities, other government agencies, and private organizations to work in ORD laboratories and centers.

ORD's expanded and strengthened competitive research grants and centers programs have greatly increased the number and activities of talented academic researchers across the nation engaged in research relevant to EPA's mission. The committee commends ORD for its excellent performance in developing and implementing these programs, as well as for the partnerships it has built with other agencies and funding organizations in joint grant solicitations. However, as discussed in Chapter 2, questions have been raised about the adequacy of the numbers and skill mix of the staff of ORD's National Center for Environmental Research who administer the research grants, centers, and fellowships programs. Since 1995, ORD has increased the research-grant funds administered by the center by about 400% without substantially increasing the center's staff. This has placed high demands on the staff who are responsible for the review process and the monitoring and dissemination of grantees' research products. The committee recognizes that the staff who manage EPA's STAR grants program must perform additional roles beyond those of NSF grants officers, because of the mission-relevance and technology-transfer requirements of the STAR program.

The committee recommends that the numbers and skill mix of the staff of ORD's National Center for Environmental Research be reassessed to ensure they are consistent with the needs of the current program of research grants, centers, and fellowships.

The committee encourages strengthening the interactions between STAR grantees and research scientists and engineers in the ORD laboratories. At present, there are insufficient mechanisms for facilitating such interactions. One possible mechanism might be to ask grant applicants to identify in their proposals how their research might be enhanced by interactions with EPA scientists and how their research might complement or supplement ongoing or planned research in the ORD laboratories. Reviewers of the proposals, as well as ORD scientists and the SAB, could also be asked for suggestions.

The committee recommends that the National Center for Environmental Research, in concert with ORD's national laboratories, de-

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

velop additional mechanisms to promote and facilitate research interactionsamong STAR grantees and ORD research staff.

Safeguarding the Future: Credible Science, Credible Decisions (EPA 1992) concluded that the academic community, Congress, other federal agencies, industry, the public, and even many within EPA are generally unfamiliar with the work of EPA scientists. The 1992 panel emphasized that many officials involved in funding EPA science were uncertain about what science products EPA had produced, and whether the quality and quantity of its products were commensurate with the dollars expended. It noted that EPA's policy and regulatory work receives a great deal of public attention, but the agency's science typically receives a similar degree of attention only when the scientific basis for a decision is questioned. The panel concluded that EPA should strive to make more widely known the short-term and long-term scientific goals and achievements of its research laboratories, contractors, and grantees. It urged the agency to develop and implement a coherent communications, outreach, and education plan to publicize the activities and accomplishments of EPA scientists.

Even within the agency, many regulatory and regional program officials throughout EPA's history have been largely unaware and even dubious of any important benefits from ORD's research program, and consequently they have not been supportive of ORD's budget. Recently, the GAO reported that one of EPA's regulatory program offices so acutely needed information on ORD's work (relevant to its program), well beyond the progress reports that ORD was providing, that the regulatory office found it necessary to pay for the development of a system to track ORD's projects (GAO 1999). That kind of situation is not healthful for ORD or the agency. ORD's ongoing efforts to disseminate its research products and inform others about them have, with some exceptions, been meager and unimaginative.

The committee recommends that EPA substantially increase its efforts to disseminate actively ORD's research products and ongoing projects, to explain their significance, and to assist others inside and outside the agency in applying them.

Publication of original research articles is critically important, but it is not sufficient. EPA should publish more individual research-topic-area summaries and comprehensive annual summaries of the results of its in-house and extramural research and technical support activities.

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

The summaries should be planned and tailored for specific audiences and should emphasize potential applications by other EPA offices, state agencies, industry, and others.

In addition, our committee concurs with the 1998 recommendation of ORD's BOSC that the National Center for Environmental Assessment should revise its mission to focus more on being an advisor, catalyst, and resource for the risk assessments performed by the rest of the agency, rather than trying itself to do individual risk assessments with its own limited resources. The center should focus on being a research organization dedicated to advancing the state of practice in risk assessment. It should reduce its role as a performer of individual risk assessments that could be done by EPA's regulatory offices.

RESEARCH ACCOUNTABILITY

In commenting on ORD's fiscal year 2000 budget, the SAB indicated that the lack of transparency in the decision-making process used by ORD to set research priorities made it difficult to evaluate the adequacy of the proposed budget. The ORD strategic plan describes general processes and criteria by which decisions are made on research priorities and funding allocations, but only in very broad terms.

During the committee's site visits and interviews, the staff of some EPA regulatory program offices expressed the belief that they have little influence on ORD 's research priorities through the Research Coordination Council or any other mechanism. They felt that they needed a stronger voice in the setting of ORD's priorities, and that ORD should be held more accountable to the agency's other offices for performing agreed-upon tasks.

In addition to the work performed in its in-house laboratories, ORD has a number of available mechanisms for extramural funding, including interagency agreements, contracts, grants, cooperative agreements, and fellowships. Each of these mechanisms has its own advantages and limitations. ORD's strategic plan (EPA 1996a, 1997a) lists several factors that are typically considered in making decisions about who will perform each research activity in ORD laboratories or outside the agency. These factors include the nature of the work, who has the appropriate expertise, how urgently the results are needed, the degree to

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

which the work must be specified or can be made flexible, the available in-house capacity, the potential value of involving multiple institutions, and the opportunities for funding leverage. The process by which ORD decides whether a project or task is to be performed by inhouse staff or through one or more of these extramural mechanisms is of crucial importance to the quality of the work and the cost-effective management of resources. Unfortunately, this decision-making process has not been sufficiently open or visible for persons outside ORD or EPA to reconstruct or assess how the decisions were made.

The committee recommends that ORD substantially improve the documentation and transparency of its decision-making processes for setting research and technical-assistance priorities, making intramural and extramural assignments, and allocating funds.

ORD should continue to be responsive to the agency's regulatory offices for the problem-driven and technical-assistance components of its program, and the agency's regulatory offices should continue to have a strong voice in decisions about the ORD plans and budget elements devoted to those components. For the core-research portion of its program, however, ORD should have greater freedom to set the agenda, without the need for specific concurrence of regulatory program offices that are focused on statutory requirements and regulatory goals. In the agency planning process, ORD should continue to consider the views and needs of the program offices in developing both components of its program, but it should maintain an adequate degree of independence in planning a core-research program that will successfully perform the leadership and anticipatory-research role that such a program can bring to the agency.

ORD's research should not be the only scientific studies held accountable in EPA. A great deal of research-like activity, including many activities in scientific and technical data-gathering, analysis, and interpretation, are being conducted or funded by EPA offices outside ORD. Much of this work is not labeled “research.” The other offices of EPA do not have the kind of authorization that ORD has to conduct research per se, and full disclosure might risk the loss of control of such activities by the regulatory offices. Historically, many of the scientific studies and analysis performed or funded outside ORD have not been fully coordinated across the agency or included in ORD's research-planning and peer-review programs. Our committee is by no

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

means opposed to scientific studies and analyses being conducted in parts of the agency outside ORD, but such activities require transparency, quality assurance, and accountability, just as ORD's program does.

ORD, with the help of others throughout the agency, has begun to develop an inventory of science projects and programs across EPA. Our committee commends EPA for this important step. We recommend that the preliminary inventory be expanded to include information well beyond its current scope, such as goals and objectives of the projects, milestones, schedules, principal investigators and project managers, and allocations of staff and financial resources.

The committee recommends that the administrator direct the deputy administrator for science and technology to expand upon the agency 's recently initiated science inventory by conducting, documenting, and publishing a more comprehensive and detailed inventory of all scientific activities conducted by agency units outside ORD. The results of the inventory should be used to ensure that such activities are properly coordinated through the agency-wide science-planning and budgeting process and are appropriately peer reviewed.

The inventory should include information well beyond its current scope — information such as project goals, milestones, schedules, principal investigators and project managers, and allocations of staff and financial resources. The SAB should be engaged in overseeing this inventory effort.

SCIENTIFIC PEER REVIEW

The committee congratulates EPA and its Science Policy Council for the excellent progress it has made in strengthening and expanding the agency's peer-review practices. The agency's 1998 peer-review handbook, discussed in Chapter 3, is a valuable resource and guidance document. In most respects, the handbook is consistent with the recommendations of a previous NRC report on peer-review practices (NRC 1998b).

EPA's SAB has expressed concern about potential conflicts of interest on the part of peer-review leaders – individuals assigned to manage reviews of agency work products – because current agency policy al-

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

lows the same individual to be a project manager for the development of a particular work product and the peer-review leader for the same work product (EPASAB 1999). Obviously, such a manager might have, or be perceived to have, a special interest in the outcome of the review and might therefore be unable to ensure the essential degree of independence. The SAB contrasted this policy with the agency' s data-quality-assurance practices, in which a staff officer is empowered to stop activity if there is a quality-assurance problem. It recommended that peer-review leaders be similarly empowered to stop a work product from moving forward if a peer review has not been properly completed.

EPA has made excellent progress in expanding and strengthening its peer-review practices, but the agency should find a way to ensure a greater degree of independence in the management of its peer reviews. The committee acknowledges that the agency should have adequate flexibility to accommodate statutory and court deadlines and resource limitations. Nevertheless, independence is essential to the proper and credible functioning of the peer-review process, and EPA's current policies fail to ensure adequate independence. Our committee shares the SAB 's concern about the potential conflicts of interest of EPA peer-review leaders and decision-makers. Despite good intentions, and even if the current policy works well much of the time, it is inevitable that some of these individuals, under great pressure to meet a deadline or implement a regulatory policy, are tempted to compromise the integrity of the peer-review process for some work products by making convenient or improper decisions on the form of peer review, the selection of reviewers, the specification of charges to the reviewers, or the responses to reviewers' comments.

The committee recommends that EPA change its peer-review policy to more strictly separate the management of the development of a work product from the management of the peer review of that work product, thereby ensuring greater independence of peer reviews from the control of program managers, or the potential appearance of control by program managers, throughout the agency.

The committee believes that the decision-maker and peer-review leader for a work product should never be the same person, and that wherever practicable, the peer-review leader should not report to the same organizational unit as the decision-maker. Although statutory

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×

and judicial deadlines might make it necessary that a program-office decision-maker retain the authority to proceed with an action on a provisional basis despite objections or concerns from a peer-review leader, with the final decision to be made by the EPA administrator, the independent decisions and any objections of a peer-review leader should be preserved and made a part of the agency decision package and public record for a work product. If such an independent assessment produces criticism of the adequacy or outcome of a peer review, EPA 's policy should be to ensure that the criticism is clearly noted, divulged, and explained.

The committee also recommends that the Science Policy Council's reviews of the agency's peer-review handbook and of experiences with its implementation include an explicit focus on promoting appropriate forms and levels of review for different types of work products and on reducing unnecessarily complex or inefficient requirements. The Science Policy Council should not necessarily wait the 5-year interval specified in the peer-review handbook; it should make changes as needed. The agency cannot afford to allow unnecessary or inefficient requirements to continue so long. The Science Policy Council's review should be ongoing. We also recommend that the Science Policy Council review a true random sample of peer-reviewed work products, examining the decisions made in structuring the review, the responses to review, and the cost, quality, timeliness, and impact of the review.

Finally, the committee wishes to emphasize that peer review must become accepted throughout EPA as a part of the agency's culture – a tool for improving quality – not merely a bureaucratic requirement. Measures such as periodic dissemination of the impacts and benefits of completed reviews might help to foster this cultural change in the agency.

Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 125
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 126
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 127
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 128
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 129
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 130
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 131
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 132
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 133
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 134
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 135
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 136
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 137
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 138
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 139
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 140
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 141
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 142
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 143
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 144
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 145
Suggested Citation:"Strengthening Science at EPA." National Research Council. 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academies Press. doi: 10.17226/9882.
×
Page 146
Next: References »
Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices Get This Book
×
Buy Paperback | $74.00 Buy Ebook | $59.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In the three decades since the U.S. Environmental Protection Agency (EPA) was created, the agency's scientific and technical practices and credibility have been independently assessed many times in reports from the National Research Council (NRC), EPA Science Advisory Board, General Accounting Office, and many other organizations; in congressional oversight and judicial proceedings; and in countless criticisms and lawsuits from stakeholders with interests in particular EPA regulatory decisions. As a previous independent panel put it in the 1992 report Safeguarding the Future: Credible Science, Credible Decisions, EPA's policy and regulatory work receives a great deal of public attention, but the agency's scientific performance typically receives a similar degree of attention only when the scientific basis for a decision is questioned. Thus, strong scientific performance is important not only to enable EPA to make informed and effective decisions, but also to gain credibility and public support for the environmental protection efforts of EPA and the nation.

This report is the fourth and final one in a series prepared by two independent expert committees convened by the NRC in response to a request from Congress and to subsequent, related requests from EPA. The Committee on Research Opportunities and Priorities for EPA—the companion committee in this study—was charged to provide an overview of significant emerging environmental issues, identify and prioritize research themes most relevant to understanding and resolving those issues, and consider the role of EPA's research program in the context of research being conducted or supported by other organizations. That committee published an interim report in 1996 and a final report, Building a Foundation for Sound Environmental Decisions, in 1997. The Committee on Research and Peer Review in EPA was charged to evaluate research management and scientific peer-review practices in the agency. The committee published an interim report in 1995 and this final report.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!