Independent Review Principles and Considerations
The Corps of Engineers has a long history of subjecting parts of its planning studies to review. Implementing this report’s recommendations for more comprehensive project review may challenge the Corps, however, as the processes for review of Corps planning studies have in the past been largely limited to internal reviews by Corps staff. This chapter identifies general principles regarding independent review that the Corps can draw upon to improve the credibility and effectiveness of its review processes.
Programs and methods for review exist in a variety of organizations, including academia, government, and the private sector. Organizational structures and processes for review differ greatly between organizations owing to differences in missions, outputs, and cultures. No single administrative structure or set of processes will be ideal for all organizations, and the Corps will surely adjust its procedures for internal and external review as it gains experience with independent review. Nonetheless, there are some general review principles and processes and several related considerations the Corps should weigh before it implements new review procedures.
COMPONENTS OF IDEAL INDEPENDENT REVIEW
Figure 4–1 depicts the components of an ideal independent review process. An agency might consider an array of inputs in structuring project review, including stakeholder opinion, agency goals and constraints, and federal objectives. Based on these inputs, the agency should define the general scope and goals of review. The agency then contracts with an
independent organization, which selects and establishes an independent review panel (Chapter 5 discusses the features of independent organizations to oversee external reviews of Corps planning studies). The agency should provide to the panel information necessary for conducting the review. In addition to receiving viewpoints of the sponsoring agency, the review panel should receive input from relevant stakeholders. The panel’s conclusions are provided in a final report. The agency then responds to the independent review panel’s final report. The agency’s response to the conclusions completes the review cycle. Other factors and questions should also be considered in establishing and revising review processes. For example, will reviews be conducted to determine if projects are consistent with an organization’s strategic objectives? Are projects being executed in a manner that meets the organization’s goals?1
ESTABLISHING A REVIEW PROCESS
Independence and Reviewers
Many individuals, organizations, and study committees have discussed how independent review might be defined. If a study or project is to be independently reviewed, the process for reviewer selection, and the reviewers themselves, must be as independent of the Corps as possible. That is, the process for selection of reviewers should be conducted by an organization outside the Corps, the reviewers should not be employees of the Corps, and reviewers should be free of conflicts of interest.
The twin issues of independence and credibility prompt several questions the Corps will want to consider: Can reviewers be affiliated (e.g., as a consultant or former employee2) with the organization for which the review is being conducted (the Corps)? Can federal agency officials beyond the Corps be allowed to serve as reviewers? Who will select reviewers? The organization itself, or an outside body?
The affiliations and experiences of the reviewers strongly affect re-
view credibility. All potential reviewers carry professional and personal biases, and it is important that these biases be disclosed when reviewers are considered and selected. It should be determined which biases—if any—will disqualify prospective reviewers. An organization should also develop criteria for determining if review panels are properly balanced, both in terms of professional expertise as well as in points of view on the study or project at hand.
Structuring the Review Process
Review panels might carry out their duties in numerous ways. Reviews are often conducted in the traditional style of face-to-face panel discussion led by a panel chair. These meetings often extend over a one-to three-day period, and over the course of a study or project, several such meetings may be held. There are, however, other ways in which reviews might be conducted. Review panels might conduct their work sequentially, with pre-meeting assignments followed by discussions in subgroups, followed by reports and plenary discussion by the entire panel. A review panel could employ a professional facilitator, leaving the chair free to fully participate in the discussions. Panels might operate in the open or (consistent with applicable laws) behind closed doors, or both. Panels might meet once or dozens of times. Panels can be standing or ad hoc.
Review does not necessarily require panels to meet. There may be instances in which meetings are not feasible because of time, resource, or other constraints, and there are many alternatives to face-to-face meetings. For example, federal agencies commonly use “mail” or “ad hoc” reviews in which draft reports are mailed to expert reviewers. Mail reviews are much less expensive, as there are no travel costs, but they may be far less effective, as reviewers are not able to engage in face-to-face discussion. There may even be instances when a single expert, rather than a panel, is used to review an issue or report. Reviews can employ multiple review levels, in which a parent panel coordinates the review activities of smaller panels or task forces that are engaged in specific review activities. Different review panels could be employed at different stages of a study. Telephone calls have been used as a review mechanism, and videoconferencing is increasingly employed today. In revising its review procedures, the Corps should be aware of the range of review options, and it may wish to experiment with some of them as its review process matures and improves.
Maturing of the Review Process
Initiating and developing effective review procedures in large organizations often requires a considerable amount of time, as well as flexibility within the organization, so that lessons from the successes and failures with the procedures can be used to improve review. Ideally, an organization will develop a written and transparent set of procedures over time and will modify this set of procedures as necessary on a continuing basis.
The maturity of the process can be measured by how formal and by how widely and clearly understood the process is within the organization and to the reviewers. For example, is the process oral or documented? Is it clearly understood by the organization being reviewed? Are roles and responsibilities clearly defined and understood so that each individual knows what is expected? How closely is the process followed in practice? Are the criteria that define a successful project documented and understood by the project owners and the reviewers? As an organization gains experience with review procedures, the procedures will generally become more widely understood and more standardized throughout the organization.
Organizational and Social Issues
A strong commitment from an organization’s leadership is an important factor in ensuring effective reviews (Kostoff, 1997). An organization’s top decision makers must decide upon their level of commitment to high-quality review. They must also consider the types of motivations and incentives they will provide to their staff and reviewers to conduct thorough—and critical—review.
Another consideration is the audience of the review. For example, reviews of Corps of Engineers planning studies will usually be addressed to the Chief of Engineers (or to a Corps District Engineer in cases where reviews are done early in the planning study), but other organizations and government agencies are also likely to have interests in a review’s results. Other stakeholders that will often be interested in the outcomes of reviews of Corps planning studies include environmental groups, hydropower producers and distributors, navigation interests, and state natural resources agencies.
SELECTING THE TYPE AND LEVEL OF REVIEW
Decisions regarding the type and level of review are important, as reviews regarded as inadequate by higher-level administrators, by internal or external critics, or by affected stakeholders may delay or block a project, while reviews that are too stringent waste resources. Criteria for selecting the appropriate level of review should balance the risks and consequences of inadequate review against the resources required for more complex and stringent levels of review. Larger, more complex projects, projects in which uncertainties are significant, and projects in which consequences of failure may be severe generally should receive the most stringent degree of review. To enhance credibility, these reviews should be independent of the organization that has developed plans for the project.
A qualitative assessment regarding the appropriate level of review required can be made by mapping the project onto a two-dimensional space where one axis represents “project risks” and the other represents “project magnitude” (Figure 4–2).
Project risks include uncertainties about project outcomes and the potential for societal controversy. Imperfect analytical procedures or inadequacies in data or knowledge may lead to uncertainties in forecasts of project costs and in environmental, social, and engineering outcomes. Corps planning studies are often controversial when they are based on projections of key economic or environmental parameters, such as future levels of waterway traffic or future environmental outcomes. There is also potential for controversy when there are conflicting agency missions or societal interests. Consequences of failure may include undesirable and potentially severe effects that cannot be mitigated or reversed, such as losses of life and property. Other considerations include possible cost overruns and funding and budget shortfalls.
Project magnitude includes both the project itself and considerations such as costs (monetary and nonmonetary), importance, and complexity. “Importance” includes monetary and nonmonetary benefits, project scale (local, regional, national, or international), and cumulative and long-term effects. “Complexity” includes multiple social, political, environmental, biological, hydrologic, and engineering aspects, as well as the degree of novelty and extent of prior experience.
Figure 4–2 illustrates that several issues must be considered in decisions regarding the appropriate level of review. The arrow shows that increasing “project magnitude” and “project risks” warrant an increasing degree of independence of review, with an increased depth and complexity of review and an increased scope and diversity of the expertise of the
reviewers. Reviews of such projects will cost the most and require the most time.
Figure 4–3 illustrates the process for selecting the level of review independence with two examples of current Corps of Engineers projects. The Comprehensive Everglades Restoration Project will affect a large portion of southern Florida and the adjacent marine ecosystem. Many of this restoration project’s outcomes in this large, complex ecosystem defy prediction, and its key environmental outcomes are thus largely uncertain. Combined with a volatile political context with many well-informed and passionate stakeholders, and with the project’s current estimated cost (roughly $8 billion), the project clearly ranks very high on the “magnitude” axis and relatively high on the “risk” axis (the project is being conducted under an adaptive management paradigm, which implies that even though environmental and related outcomes are uncertain, lessons will be learned through the restoration process, and management
strategies will be continually adjusted, thereby reducing somewhat the prospects for irreversibility). The range of magnitude-risk of the Everglades project is represented as a broad oval that extends from the medium portion of the risk scale well into the high end. The largest portion of the oval falls within the space where external review is recommended.
In contrast to the Everglades restoration project, the St. John’s Bayou and New Madrid Floodway Project on the Mississippi River floodplain in the state of Missouri is less costly, is on a smaller scale, and has design features (levee and drainage system) that are less complex. As there are significant uncertainties and complexities surrounding the project, it would rank in the middle, rather than at the bottom, of the risk scale.
Several conceptual questions surround the creation of an effective review process. They include choosing appropriate projects for review, evaluating the appropriate level of independence of review, determining the audience, selecting the expertise, number, and independence of reviewers, and defining the scope of review. These questions are especially important in determining the specifics of reviews of Corps water resources planning and construction projects owing to the variety and intensity of stakeholder perspectives, the need to consider economic and environmental consequences along with scientific and technical issues, and the highly charged political environment that surrounds many projects. In the case of highly visible and controversial projects, the two most important considerations in selecting reviewers are the credentials of the reviewers (which includes affiliations as well as expertise) and the group that selects the reviewers. Public perception may well have greater influence than public understanding in determining the fate of a project. It is often the case, however, that a minority of stakeholders reflect that “public” perception. The review process thus needs to be structured such that good science, sound engineering, and public welfare are the most important factors that determine a project’s fate.
Determining the appropriate level of reviews will constitute important decisions for the Corps. With projects that are more complex, costly, and controversial, a review’s credibility will be based in large part on the independence of the reviewers and of the review process from the Corps and from the project being reviewed.