Peer review is an essential component of engineering practice and other scientific and technical undertakings. Peer reviews are conducted to ensure that activities are technically adequate, competently performed, and properly documented; to validate assumptions, calculations, and extrapolations; and to assess alternative interpretations, methodologies, acceptance criteria, and other aspects of the work products and the documentation that support them. Effective peer reviews are conducted in an environment of mutual respect, recognizing the contributions of all participants. Their primary objective is to help the project team achieve its goals. Reviews also contribute to quality assurance, risk management, and overall improvement of the management process.
The U.S. Department of Energy (DOE) conducts different types of peer reviews at the different stages of a project, including reviews to assess risks and other factors related to design, safety, cost estimates, value engineering, and project management. Independent project reviews (IPRs) are conducted by federal staff not directly affiliated with the project or program and management and operations (M&O) contractors. External independent reviews (EIRs) are overseen by the Office of Engineering and Construction Management and conducted by contractors external to the department. EIRs are the primary focus of this report. However, the committee found that, in many cases, IPRs are explicitly used as preparation for or as preliminary reviews prior to EIRs. Thus, because IPRs are integral to the review process in DOE, they are also discussed because they might have an effect on EIRs.
In October 2000, DOE issued Order 413.3, Program and Project Management for the Acquisition of Capital Assets (DOE, 2000). The order established a series of five critical decisions (CDs), or major milestones, that require senior management review and approval to ensure that a project satisfies applicable mission, design, security, and safety requirements:
CD-0 Approve mission need
CD-1 Approve alternative selection and cost range
CD-2 Approve performance baseline
CD-3 Approve start of construction
CD-4 Approve start of operations or project closeout
Each CD milestone marks an increase in the commitment of DOE resources and requires successful completion of the preceding phase. The order also required a variety of internal and external peer reviews prior to CD approvals, including EIRs and IPRs.
That order made DOE’s Office of Engineering and Construction Management (OECM) responsible for overseeing EIRs for all projects with total project costs (TPCs) greater than $5 million before CD-2. For projects with TPCs greater than $400 million (major systems, MS), EIRs are also required before CD-3.
Order 413.3 assigned responsibility for conducting IPRs to the program secretarial offices (PSOs) and gave the PSOs the option of determining when IPRs should be conducted.
A revised order (O 413.3A), issued on July 28, 2006 (DOE, 2006), raises the threshold for the TPC of projects that require an EIR before CD-2 to $100 million and the TPC of major systems that require an EIR before CD-3 to $750 million. The revised order requires the PSOs to conduct IPRs (as opposed to EIRs overseen by the OECM) prior to CD-2 for projects with a TPC less than $100 million.
The DOE has a portfolio of construction and environmental remediation projects valued at more than $205 billion. Some projects are highly complex, one-of-a-kind undertakings. The success of these projects depends on the effective management of risk during the acquisition, conceptual planning, design, and construction processes. Congress continues to support the OECM’s oversight of project management practices at the DOE and the use of EIRs to validate project performance baselines. Secretary of Energy Samuel W. Bodman has reiterated the importance of a disciplined project management process and adherence to established processes. Budgetary pressures also make it necessary for the OECM to ensure that EIRs are effectively planned and efficiently executed and that they provide added value to individual projects and the department.
STATEMENT OF TASK
Accordingly, DOE asked the National Research Council (NRC) to appoint an expert committee to advise the department on how EIRs can be tailored to a mix of project types to ensure that:
Essential information about project scope, cost, feasibility, and risk is provided at the optimum point in DOE’s critical decision process;
DOE’s resources (time, staff, funding) are focused on those projects with the greatest risk; and
EIRs are cost effective.
The Committee on Assessing the Results of External Independent Reviews for U.S. Department of Energy Projects was also asked to identify lessons learned from its EIR analysis that could be used to refine departmental guidance and policies to further support DOE’s goal of continuous improvement in project management.
There appears to be universal agreement among DOE staff, M&O contractors, and EIR contractors that projects benefit from the effort expended in preparing for EIRs and IPRs. This does not imply that there is concurrence on all aspects of the timing, cost, and comprehensiveness of specific reviews, but the concept of reviewing projects is considered to have value and to benefit the individual projects and DOE. This benefit increases as the size, complexity, and inherent risks of the project increase.
DOE policies designate EIRs as the primary process for validating project cost and schedule baselines prior to submitting funding requests to Congress. The committee (see Appendix A for biographies of committee members) examined a sample of 26 EIR reports (Appendix B) for 19 projects chosen to be representative of the 150 EIRs conducted between 2000 and 2005. The committee found them to be an effective record of the reviews and an effective vehicle for communicating the review teams’ findings and recommendations. At a June 2006 workshop conducted by the committee (Appendix C) DOE project directors and program managers reported additional benefits such as professional development, management process improvement, transfer of lessons learned from other sites and projects, and improved communication between DOE staff and contractor personnel.
If EIRs are intended only to validate project baselines, then the current review planning procedures, the practice of employing outside contractors responding directly to the OECM, and the use of a standardized set of key review elements might be appropriate. However, the value and cost-effectiveness of EIRs would be enhanced if they were (1) planned more carefully with the broader involvement of all stakeholders, (2) tailored in a more flexible manner using a collaborative process, and (3) integrated into the complete portfolio of peer reviews that are used to monitor and support DOE
projects. In addition, EIR reports can be a source of information to improve project management processes and provide valuable lessons for other projects throughout the DOE.
Finding 1: DOE employs a variety of peer reviews to assess risk and other factors that bear on design, safety, cost estimates, value engineering, and project management. Because the responsibility for organizing the reviews is dispersed throughout the department, the planning of project reviews is not well coordinated.
Finding 2: The value of a peer review (IPR or EIR) depends, in large part, on the experience and expertise of the review team, its capacity to understand the requirements of the project, its independence, and its ability to make objective assessments and recommendations. Assembling teams that have both expertise and independence can be difficult, especially for the complex, one-of-a-kind projects sometimes undertaken by DOE.
Finding 3: Total project cost is the leading criterion in guiding the planning and scheduling of reviews. Additional criteria may be equally appropriate.
Finding 4: Peer reviews (IPRs or EIRs) conducted prior to approval of mission need (CD-0) and again prior to approval of the alternative selection and cost range (CD-1) could play a valuable role in risk management and quality assurance during front-end project planning. Some projects could probably benefit from annual reviews for quality assurance purposes when the time between critical decision milestones exceeds 1 year.
Finding 5: Problems in communication stemming from the prereview planning process have occurred when an EIR team arrived at a significant finding after the on-site exit briefing but did not relay the finding to DOE staff until delivery of the final report.
Finding 6: The purpose of tailoring is to refine the focus of the EIR to suit the particular characteristics of a project. DOE policies emphasize the need for tailoring, but the committee observed little evidence of effective tailoring.
Finding 7: A clearly defined, systematic process for capturing data from peer reviews along with a system to catalogue, store, and disseminate this information is needed and, if implemented, would greatly enhance the benefits derived from peer reviews.
Finding 8: EIRs and other peer reviews are an intrinsic part of project management and are an integral part of project costs. The current policy of funding EIRs from program direction funds places them in competition with other program activities.
Recommendation 1: EIRs should be integrated into a comprehensive peer review process. To accomplish this,
A comprehensive plan to coordinate the planning for all project peer reviews should be developed by the integrated project team (IPT), with input from the OECM and the PSOs. Plans for peer reviews should be included in the project execution plan and the quality assurance program.
The IPT, the PSOs, and the OECM should determine the types of and schedules for these reviews using the guidelines shown in Table S.1. It is important to note that the committee’s recommendations for reviews to be conducted by default do not necessarily parallel Order 413.3A.
In addition to the TPC, the criteria for determining the appropriate type, form, and timing of a review should include project complexity; severity and types of inherent risks; project management maturity; and the qualifications and experience of the project teams. These criteria should be developed jointly by the OECM and the PSOs.
EIRs should be planned and executed jointly by the OECM, the PSOs, and the IPTs. The office responsible for validating the performance baseline (CD-2) should take the lead for reviews that occur prior to CD-2.
For projects with TPCs less than $100 million, an EIR should be required to validate the performance baseline (CD-2) unless project complexity, inherent risks, project management capabilities, or other factors warrant conducting an IPR. The determination of the most appropriate type of review should be made by the PSO.
For some projects, a review (IPR or EIR) should be conducted prior to CD-0 and then again prior to CD-1 if warranted by the project complexity, inherent risks, project management capabilities, or other factors. After CD-2, when the elapsed time between critical decision milestones exceeds 1 year, an annual IPR should be conducted for quality assurance purposes.
Recommendation 2: EIRs should assess whether the particular project management team—i.e., project manager, project team, federal project director, and IPT—has the ability to execute the project successfully. This assessment should evaluate the qualifications and experience of the project management team, the project’s organizational structure, and the communication and decision-making processes in place.
Recommendation 3: EIR review teams should be selected to provide the experience and expertise necessary to evaluate a project’s baseline cost, schedule, scope, inherent risks, and the capabilities of the project management team. The acquisition executive could add DOE contractor personnel who are not project advocates to EIR teams when outside technical expertise is not available. In the long term, the distinction between the EIR and IPR review teams can be lessened and review team members from within DOE and external to it can be selected as needed to address the critical issues presented by each project.
Recommendation 4: The OECM, the PSOs, and the IPT should collaborate on the tailoring of reviews. They should determine which key review elements should be emphasized and which ones should be streamlined based on their assessment of project risks, complexity, and cost.
Recommendation 5: Communication among all EIR participants should be improved from initial planning of the reviews through taking corrective actions:
If significant findings are arrived at subsequent to the formal on-site exit briefing, the EIR team should notify the IPT immediately and not wait until delivery of the final report.
Postreview assessments of the quality of the review process for EIRs should be conducted and led by the OECM with input from the IPT and the PSO. The purpose is to improve future reviews and provide input to the annual contractor performance appraisal.
TABLE S.1 Recommended Guidelines for Planning Peer Reviews
Recommendation 6: The DOE should document lessons learned from peer reviews and disseminate them throughout the department.
The OECM should establish a process to document and disseminate the lessons learned from all peer reviews across all program offices. The process should be developed through a coordinated effort with the full participation of the PSOs.
EIRs should clearly identify best practices observed in order to facilitate the lessons learned process.
Recommendation 7: DOE should view peer reviews as an intrinsic component of project management.
The costs for EIRs and IPRs should be covered by project funding—that is, be part of the preliminary engineering design funds prior to CD-2 and a part of project funds when these funds have become available to the PSO.
Prereview planning decisions about the most appropriate type of review (EIR or IPR) should be based on full-cost accounting of either type of review.
DOE (Department of Energy). 2000. Program and Project Management for the Acquisition of Capital Assets (Order O 413.3). Washington, D.C.: Department of Energy.
DOE. 2006. Program and Project Management for the Acquisition of Capital Assets (Order O 413.3A). Washington, D.C.: Department of Energy.