Report Purpose and Scope
The U.S. Army Corps of Engineers developed much of the nation’s waterway transportation, flood damage reduction, and coastal infrastructure, and it continues to play an important role in the operation and maintenance of these systems. As the nation’s watersheds have become more heavily developed and as social preferences have changed, the Corps has become involved in ecosystem restoration projects and is faced with the challenge of adjusting operations of existing projects in highly controlled watershed and river systems.
Before submitting proposed water projects to Congress for approval, the Corps conducts feasibility studies that assess the economic, engineering, and environmental dimensions of potential projects. These studies are guided largely by a federal water resources project planning document, the Principles and Guidelines (WRC, 1983), along with several other Corps engineering regulations and engineering circulars. The Corps’ water resources project planning studies and its water projects have never been free of controversy, and the Corps’ planning techniques and decision-making procedures have been challenged for decades (Maass, 1951; Reisner, 1986). Several dimensions of Corps planning studies have come under scrutiny, raising the question of how the credibility of and the analyses within these studies might be strengthened by subjecting these studies to some degree of independent, external review.
An example of a Corps of Engineers planning study that has generated national-level interest and controversy is the Corps’ feasibility study on the Upper Mississippi River-Illinois Waterway (UMR-IWW). In this study, which was begun in the late 1980s, the Corps is examining the economic feasibility of extending several locks on the lower portion of the Upper Mississippi River (Box 1–1). During this study (which continues at this
BOX 1–1 The Upper Mississippi River-Illinois Waterway Draft Feasibility Study
In the late 1980s, the Corps of Engineers began a study of the economic feasibility of extending several locks on the lower portion of the Upper Mississippi River. Towboats had been encountering congestion in this portion of the navigation system and were thus experiencing costly delays.
The study posed several analytical challenges to the Corps. The Corps conducted the study in a systems framework that considered the entire Upper Mississippi River-Illinois Waterway system, making it one of the agency’s more sophisticated planning studies. The Corps developed an economics model to forecast flows of waterway traffic through the system for different levels of grain supply and demand. The Corps also conducted and contracted for extensive environmental analyses to support the economics portion of the study. Moreover, the study was conducted in a highly charged political atmosphere, with passionately held viewpoints on many aspects of the Upper Mississippi River system. Farmers and towboat operators contend that the locks must be extended in order for them to be competitive in a global commodities market. Environmental groups contend that additional towboat traffic and the local environmental impacts of lock extensions will cause unacceptable damages to an already stressed ecosystem. Some taxpayer advocate groups question the price tag of approximately $1.1 billion for the extensions. The study, which by the year 2000 had reached over $50 million in cost, brought increased scrutiny to the Corps of Engineers and its planning and review procedures.
writing), controversies arose over key assumptions within the study and in regard to the study’s analytical credibility. As a result, the Department of the Army requested that the National Academies1 provide an independent review of the Corps’ draft feasibility study, focusing on the study’s economic analysis. The committee appointed by the Academies’ National Research Council (NRC) recommended several ways in which the feasibility study might be improved (NRC, 2001).
In Section 216 of the Water Resources Development Act of 2000 (WRDA 2000), Congress directed the National Academy of Sciences (“The National
Academies”) to “study the practicality and efficacy of the independent peer review2 of feasibility reports” (Appendix A). It further directed that the Academy study “the cost, time requirements, and other considerations relating to the implementation of independent peer review” and the “objective criteria that may be used to determine the most effective application of independent peer review to feasibility reports for each type of water resources project.”
This report from the Panel on Peer Review is one of the reports from the panels convened in response to the request in Section 216 of WRDA 2000 (this report’s Foreword and Preface list the other study panels). The panel began its review of the Corps’ review procedures in Fall 2001 and completed its report in July 2002. The panel was requested to review the current procedures for the review of Corps’ feasibility studies and related technical documents and to provide recommendations for improving those processes. The panel was also asked to review and comment on review procedures as they existed during the tenure of the former Board of Engineers for Rivers and Harbors (BERH). Box 1–2 lists the full charge to the panel.
This report of the Panel on Peer Review contains recommendations on how the Corps of Engineers could improve its review procedures, including independent, expert review. Rather than focusing on the content of review (e.g., the underlying assumptions of economic analysis or flood forecasting methods), this panel focused on the process of review. This report thus provides guidance to the Corps on identifying studies that should be subjected to review, on how reviewers should be selected, and on the timing and administration of the review process.
PEERS AND PEER REVIEW
Within the scientific and academic communities, the term “peer” is often used in the context of “peer review,” a practice widely employed by scientific and scholarly journals and scientific research programs. In this context, review consists of “peers” reviewing draft manuscripts, proposals, and strategic plans. But as conflicts of interest may taint this process, independent reviewers are almost always selected for reviews within the scientific and academic communities. For example, scientific journal editors almost never send a draft manuscript to a colleague in the author’s home institution, and reviewers of scientific research programs usually must be from a different institution. In these contexts, the term “peer review” connotes independence from the agency
BOX 1–2 Charge to the Panel on Peer Review
The panel will review the Army Corps of Engineers’ peer review procedures for the Corps’ water resources project feasibility reports and related technical documents, and provide recommendations for improving those procedures.
The panel will review the history, criteria, and future options for an independent review process. The panel will review the Corps’ existing technical review process conducted at the Corps’ district offices, Corps Headquarters policy review of draft and final authorization reports, and reviews by the Assistant Secretary of the Army and the Office of Management and Budget. The panel will also consider reviews during the feasibility study process by stakeholders and other agencies. The panel will review the previous concurrent Washington-level review process, in which Corps Headquarters, the former Board of Engineers for Rivers and Harbors (BERH), and the Office of the Assistant Secretary of the Army for Civil Works concurrently reviewed projects. The panel will also review the singular BERH process that was conducted before the late 1980s.
In formulating its findings and recommendations, the panel will consider cost, time requirements, and other appropriate considerations in formulating future peer review options for the Corps. The panel will consider the timing of peer review in the Corps planning process and will provide advice on implementing recommendations from peer review into feasibility reports.
and from the individuals whose work is being reviewed. The term “independent peer view” is commonly used to define these types of scientific and academic reviews, and this panel was requested to assess the prospects for an “independent review process” within the Corps (see Box 1–2).
In addition to independent review, this panel was also asked to review the Corps’ existing technical review process. The existing process, and any reviews that might include Corps of Engineers professional staff as reviewers, are not truly independent. As this panel was requested to comment on both independent review and the existing review process, the panel chose not to use the terms “peer” or “peer review,” and instead simply refers to both procedures as “review.” The panel distinguishes between review that is internal and review that is external. The report’s focus is on review of Corps planning studies, with careful attention given to the need for independent, external reviews by panels of well-qualified and impartial experts for large, complex, expensive, and controversial projects. Internal reviews should be conducted by panelists
appointed by the Corps, and some of these panelists may be Corps of Engineers professional staff.
In external reviews, panelists should be independent of the organization and personnel conducting the study. External (independent) reviews must be conducted by experts not employed by the Corps of Engineers, who are selected by a group outside of the Corps, and who are free of conflicts of interest. These external reviews are equivalent to academic and scientific “independent peer review.”
THE NEED FOR AND VALUE OF REVIEW
Reviews of Corps of Engineers planning studies will improve not only the studies’ technical dimensions and quality, but will also add credibility to the arguments offered and the conclusions drawn, which is important for highly visible and controversial projects. Reviews by themselves, however, cannot ensure high quality planning studies; realizing the benefits of review requires that results be used as inputs to decisions by an organization’s policy makers (Chubin and Hackett, 1990).
The Corps often faces a great deal scrutiny of its water resources project planning studies for several reasons: a willingness by the public to question federal agencies and to seek participation in their decision making procedures; improvements in scientific understanding of environmental impacts of large water projects; the desire of some members of the public to reduce expenditures of federal tax dollars; and a proliferation of engineering and scientific analysts outside the Corps. The Corps also operates under a complex set of mandates that reflect diverse interests and often contradictory views of water resources management. Moreover, the Corps operates at the behest of a Congress with diverse views of appropriate roles for the Corps, with diverse views on the appropriate balance between environmental conservation and resource development, and with competing visions of the desirable future state of the nation’s river and water resources systems.
Corps planning studies are often controversial when those studies are based on assumptions about forecasts of key economic or environmental parameters. Moreover, many Corps projects have a design life of a century or more. This presents the dilemma of the necessity for long-term forecasts, knowing that the accuracy of those forecasts diminishes as one moves further into the future.
Projections of factors such as flood damages avoided, net economic benefits generated, levels of waterway traffic demand, or future environmental values provided may be open to question. These projections are typically based not upon fundamental engineering principles and methods—areas in which many people agree that the Corps is competent—but rather upon water
resources economic and environmental issues that transcend traditional engineering considerations. These issues include estimates of supply and demand values for goods and services in global markets, valuation of environmental outcomes, risk and uncertainty analyses, and models of nonlinear ecosystem dynamics. Assumptions regarding future benefits from Corps projects have been criticized by some as being overly optimistic. This panel did not investigate these matters, but Corps projects come under a high degree of scrutiny when they hinge upon some unknown future level of economic or ecosystem services or when they are likely to cause significant environmental impacts. These factors have been central in Corps studies on the Upper Mississippi River-Illinois Waterway system, the Florida Everglades, and the Missouri River reservoir system. The Corps has come under fire in these studies, with many critics calling for a greater degree of formal review by independent experts.
External reviews of Corps planning studies, no matter how useful, should not be expected to resolve fundamental disagreements and controversies. Reviews should focus on environmental, engineering, and economic assumptions, data, methods, and models. Indeed, independent review is a necessary component of comprehensive water resources planning, as it ideally creates a process that is transparent and accountable. But such reviews should not be expected to resolve tensions regarding proposed water projects or operations alternatives. Those differences must ultimately be resolved by the Administration and by Congress in the authorization and appropriations process. Reviewers should assist the Corps in making decisions, but they should not be asked to make decisions themselves. Indeed, reviewers engaged in the external review processes described in this report should be identified for their professional expertise and should not be “stakeholders” at all.
SELECTING REVIEWERS FOR CORPS PLANNING STUDIES
In contrast to review of a scientific research and development (R&D) program, the Corps must decide on the fitness of proposals for the construction of civil works projects. In addition to technical considerations, civil works today include input and participation from local sponsors and are subject to authorization and appropriation from Congress. Accordingly, the Corps’ review procedures necessarily differ from the review procedures tailored specifically to R&D programs within other federal agencies, such as the U.S. Department of Energy (DOE) or the U.S. Environmental Protection Agency (EPA).
This report makes a distinction between external (independent) and internal review. Corps planning studies span a range of project proposals from traditional water resources engineering projects (e.g., rehabilitation of an aging lock chamber) to sophisticated environmental restoration projects (e.g., the Florida Everglades). This panel concluded that many Corps planning studies do not necessarily require external, independent review, and that, in many instances, Corps of Engineers professional staff should participate in reviews of technically focused planning studies. As planning studies become more sophisticated and expand beyond traditional engineering analyses, however, independent expert opinion would increase the review’s comprehensiveness and credibility.
Figure 1–1 depicts the different pools from which experts with varying degrees of independence might be drawn to participate in reviews of Corps of Engineers studies. Moving left to right, the groups represented in the three circles in Figure 1–1 are (1) Corps of Engineers professional staff, (2) experts on a given water resources topic (the Florida Everglades is used as an example), and (3) independent (i.e., not employed by the Corps) water resources experts, as well as other analysts with relevant expertise (e.g., a scholar or scientist who may not specialize in water resources issues, but whose expertise may be useful in providing a balanced and comprehensive review).
Moving left to right in Figure 1–1, the circles and their intersections portray five classes of analysts in terms of internal and external reviewers: Class 1, Corps of Engineers staff without expertise on a given topic (the Florida Everglades in this example); Class 2, Corps of Engineers’ staff with expertise in the Everglades; Class 3, Everglades experts not employed by the Corps, but who have conflicts of interest (e.g., an expert employed by the state of Florida, by an environmental advocacy group, or by the Florida sugar industry); Class 4, independent (non-Corps) analysts with Everglades experience; and Class 5, independent analysts without Everglades experience.
Internal reviews could be conducted by Corps personnel or with a balance of Corps staff and external experts. External and independent reviews would be conducted only with Class 4 and Class 5 experts as described above (it is worth noting that there may be few available “Class 4” experts and they may be difficult to identify and appoint). Furthermore, and just as important, those Class 4 and Class 5 experts should be appointed by a group outside of the Corps in order to constitute fully independent review. A third review option for the Corps would be to appoint experts from across this spectrum of reviewers, although the Corps would usually be best served by avoiding reviewers with conflicts of interest (Class 3 reviewers in Figure 1–1).
Before examining ways in which the current process might be improved, it is important to first understand the history of the review of planning studies within the Corps of Engineers, a history that explains much about the current status and approaches to review applied by the Corps. This history, described in Chapter 2, reveals that the Corps has utilized several mechanisms to ensure that the study proposed to Congress represents the Corps’ best technical advice and complies with its statutory and regulatory mandates. However, there are countervailing pressures on the Corps, including the desire of congressional members and local sponsors for favorable reviews, and a belief by some that review is costly and does not add value to projects. Furthermore, external review has not been a significant part of the Corps’ history.
The Corps’ water resources project planning procedures, reviewed in a 1999 National Research Council committee report (NRC, 1999), are summarized in Chapter 3. Chapter 4 reviews elements that are critical to the success of review. In Chapter 5, review alternatives available to the Corps are discussed, and the implications of each alternative are examined.
Chapter 6 summarizes recommendations that the Corps’ complex planning studies be subjected to external review and that a new Administrative Group for Project Review be created to coordinate review procedures. Chapter 6 also summarizes recommendations for the creation of a Review Advisory Board which would provide periodic independent advice regarding review procedures for the Corps’ planning studies. Chapter 7 summarizes the panel’s findings and recommendations.