1
Introduction

Peer review is an essential component of the practice of engineering and other scientific and technical undertakings. A peer review is a documented critical assessment of a technical work product conducted by a qualified individual or group that is independent of those who performed the work but individually or collectively equivalent in technical expertise to them (NRC, 1997). Peer reviews are conducted to ensure that activities are technically adequate, competently performed, and properly documented; to validate assumptions, calculations, and extrapolations; and to assess alternative interpretations, methodologies, acceptance criteria, and other aspects of the work products and the documentation that supports them.

The U.S. Army Corps of Engineers, the Department of the Interior, and the National Aeronautics and Space Administration routinely use peer review for large engineering projects. The American Society of Civil Engineers (ASCE) promotes peer review as an additional factor that can increase confidence in the design of an engineering project and assure its quality (ASCE, 2004). ASCE describes peer review as an activity that should be used (1) when a project is critical to the public health, safety, and welfare; (2) where reliability of performance under emergency conditions is critical; (3) when the project uses innovative materials or techniques; (4) when it lacks redundancy in design; or (5) where the project’s construction sequencing is unique. ASCE recommends that reviews address a defined scope, as set forth by the initiating party, and that project peer review take place throughout the design process. Such reviews should be performed by independent teams or individuals not associated with the original design team.

Although it is rare that an individual or a team can identify all the flaws in its own work, there is often resistance to outside criticism (Weigers, 2006). Peer review can overcome this resistance by operating in an environment of mutual respect and recognizing the contribution and accomplishments of all participants. The review assesses the project team’s capacity to complete the project as planned. Its primary objective is to help the project team to achieve its goals, not to be an oversight or audit function.

Various forms of peer review can be undertaken at any phase of a project. Common to all is the need for them to be conducted by qualified and unbiased people who are independent of the project team. The people can come from within the organization or outside it as long as they are sufficiently detached from funding considerations or other interests to ensure that their review is objective (NRC, 1998a). The project team, senior management, or other key stakeholders can initiate reviews.

The Department of Energy (DOE) conducts different types of peer reviews at different stages of a project. Independent project reviews (IPRs) are conducted by federal staff not directly affiliated with the project or program and management and operations (M&O) contractors. External independent reviews (EIRs) are overseen by the Office of Engineering and Construction Management (OECM) and conducted by contractors external to the department. EIRs are the primary focus of this report. However, the committee found that, in many cases, IPRs are explicitly used as preparation for or as preliminary reviews prior to EIRs. Thus, because IPRs are integral to the review process in DOE, they are also discussed where they might have an effect on EIRs.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 7
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects 1 Introduction Peer review is an essential component of the practice of engineering and other scientific and technical undertakings. A peer review is a documented critical assessment of a technical work product conducted by a qualified individual or group that is independent of those who performed the work but individually or collectively equivalent in technical expertise to them (NRC, 1997). Peer reviews are conducted to ensure that activities are technically adequate, competently performed, and properly documented; to validate assumptions, calculations, and extrapolations; and to assess alternative interpretations, methodologies, acceptance criteria, and other aspects of the work products and the documentation that supports them. The U.S. Army Corps of Engineers, the Department of the Interior, and the National Aeronautics and Space Administration routinely use peer review for large engineering projects. The American Society of Civil Engineers (ASCE) promotes peer review as an additional factor that can increase confidence in the design of an engineering project and assure its quality (ASCE, 2004). ASCE describes peer review as an activity that should be used (1) when a project is critical to the public health, safety, and welfare; (2) where reliability of performance under emergency conditions is critical; (3) when the project uses innovative materials or techniques; (4) when it lacks redundancy in design; or (5) where the project’s construction sequencing is unique. ASCE recommends that reviews address a defined scope, as set forth by the initiating party, and that project peer review take place throughout the design process. Such reviews should be performed by independent teams or individuals not associated with the original design team. Although it is rare that an individual or a team can identify all the flaws in its own work, there is often resistance to outside criticism (Weigers, 2006). Peer review can overcome this resistance by operating in an environment of mutual respect and recognizing the contribution and accomplishments of all participants. The review assesses the project team’s capacity to complete the project as planned. Its primary objective is to help the project team to achieve its goals, not to be an oversight or audit function. Various forms of peer review can be undertaken at any phase of a project. Common to all is the need for them to be conducted by qualified and unbiased people who are independent of the project team. The people can come from within the organization or outside it as long as they are sufficiently detached from funding considerations or other interests to ensure that their review is objective (NRC, 1998a). The project team, senior management, or other key stakeholders can initiate reviews. The Department of Energy (DOE) conducts different types of peer reviews at different stages of a project. Independent project reviews (IPRs) are conducted by federal staff not directly affiliated with the project or program and management and operations (M&O) contractors. External independent reviews (EIRs) are overseen by the Office of Engineering and Construction Management (OECM) and conducted by contractors external to the department. EIRs are the primary focus of this report. However, the committee found that, in many cases, IPRs are explicitly used as preparation for or as preliminary reviews prior to EIRs. Thus, because IPRs are integral to the review process in DOE, they are also discussed where they might have an effect on EIRs.

OCR for page 7
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects BACKGROUND OF EXTERNAL INDEPENDENT REVIEWS AT THE DEPARTMENT OF ENERGY DOE has a portfolio of construction and environmental remediation projects valued at more than $205 billion. Some are highly complex, one-of-a-kind projects. Individual projects often cost hundreds of millions of dollars and some exceed $1 billion. The success of these projects depends on the effective management of risk and other factors during acquisition, conceptual planning, design, and construction. Because these projects involve large sums of public money and are critical to the national defense, the advancement of science, and protection of the environment, Congress needs to ensure that they are planned and managed effectively. In 1997, recognizing that DOE projects frequently exceeded their projected costs and schedules, Congress initiated a review of all construction projects started in FY 1998 and construction projects in the conceptual design phase (U.S. Congress, 1997). These reviews evaluated the quality of the technical scope of work, cost estimates, schedules, and supporting data, and critiqued the validity of the proposed costs, scopes of work, and schedules. Congress also directed DOE to arrange for an independent assessment of its overall project management structure and process for identifying, managing, designing, and constructing facilities. DOE asked the National Research Council (NRC) to undertake these studies. The resulting series of NRC studies recommended both internal and external independent project reviews to verify the mission need for DOE projects, their conceptual designs, the selection of alternatives, and project cost and schedule baselines. The recommendations applied to all projects of more than $20 million total estimated costs (TEC),1 to less costly projects meeting certain criteria for complexity and risk. The reports recommended a graded (or tailored) process for smaller, routine projects (NRC, 1998b, 1999), identified areas where DOE fell short of best practices for project management, and recommended actions to correct these shortfalls. Congress directed DOE to respond to the NRC recommendations, conduct external independent reviews (EIRs) before obligating construction funds in FY 2000, and formalize a process to ensure that EIRs are implemented (U.S. Congress, 1999). In response, DOE established the Office of Engineering and Construction Management (OECM) within the Office of the Chief Financial Officer to drive changes in DOE’s project management system and establish a strong project management capability (DOE, 1999). The OECM serves as DOE’s principal point of contact for project management. It develops policy, requirements, and guidance for the acquisition of capital assets; assists in the planning, programming, budgeting, and execution process for the acquisition of capital assets in coordination with the PSOs and project management support offices (PMSOs); and supports the Office of the Secretary and other senior managers in the critical decision (CD) process and in oversight of the acquisition management office. The OECM is also responsible for establishing, maintaining, and executing an independent review capability and validating the performance baseline for all capital asset projects with a TPC of more than $5 million before projects are included in the DOE annual budget request that is submitted to Congress. In 2000, Congress directed DOE to identify and document the process to be used to determine which projects require an EIR and at which points in a project the EIRs should be conducted (U.S. Congress, 2000). Congress also continued to support the use of EIRs for all new line-item capital projects. In October 2000, DOE issued Order 413.3 Program and Project Management for the 1 DOE defines total estimated cost (TEC) as the gross cost of the project, including the cost of land and land rights; engineering, design, and inspection costs; direct and indirect construction costs; and the cost of initial equipment necessary to place the plant or installation in operation. The TEC includes contingencies and can be funded as operating or construction expenses. Since 2000, project engineering and design expenses have not been included in congressionally approved TEC. DOE policy documents reference the total project cost (TPC), which consists of all the costs included in the TEC plus the preconstruction costs, such as conceptual design and research and development, as well as the costs associated with the preoperational phase, such as training and startup costs (DOE, 2000a).

OCR for page 7
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects Acquisition of Capital Assets (DOE, 2000a), which established a management process organized by phases and critical decisions. The phases represent a logical maturing of broadly stated mission needs into well-defined technical, system, safety, and quality requirements. A series of five critical decisions (CDs) was also established that required senior management to review and approve a project and associated management actions before moving to the next phase of development: CD-0 Approve mission need CD-1 Approve alternative selection and cost range CD-2 Approve performance baseline CD-3 Approve start of construction CD-4 Approve start of operations or project closeout The five CDs are major milestones approved by the Secretarial Acquisition Executive or the Acquisition Executive, who must ensure that a project meets applicable mission, design, security, and safety requirements. Each CD marks an incremental commitment of resources by DOE and requires successful completion of the preceding phase or CD. There is no set period of time between CD milestones because projects vary in their complexity, with some taking more time to develop than others. Collectively, the CDs are intended to affirm the following: There is a need that cannot be met through nonmaterial means; The selected alternative and approach is the right solution; A definitive cost, scope, and schedule baseline has been developed; The project is ready for implementation; and The project is ready for turnover or transition to operations (DOE, 2003a). The CD process incorporates a variety of internal and external peer reviews. Table 1.1 is based on DOE policies for project reviews extant during the course of this study. Revisions to these criteria were approved on July 28, 2006, and are discussed in Chapter 3. The IPRs are managed by the program secretarial offices (PSOs) and the EIRs are managed by the OECM with the support of the PSO and the project teams. Prior to CD-0, there is an option to conduct an IPR focused on the initial planning for the project and development of the mission need statement. Prior to CD-1, an optional IPR may be conducted to focus on the analysis supporting the selection of the preferred alternative to ensure that the system functions and requirements are defined and to confirm the preliminary cost and schedule range. EIRs are required prior to CD-2 for all projects with a TPC >$5 million to support the OECM’s validation of the performance baseline and provide reasonable assurance that the project can be successfully executed. EIRs are also conducted prior to CD-3 for all major system projects to assess their readiness for construction or implementation and to confirm the completeness and accuracy of the performance baselines. Order 413.3 directs that the CD and EIR processes be tailored to accommodate the project delivery process and characteristics of the project. Referring to the performance baseline review, the order states that “tailoring can apply to any project, but has increased applicability to projects that are considered to be routine in nature … and have relatively low risk. The key requirement for tailoring the Performance Baseline Review is the ability of OECM to validate the Performance Baseline” (DOE, 2003a, p. 9-8).

OCR for page 7
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects TABLE 1.1 Types of Project Reviews Review Title Accomplished Prior to Required or Optional Responsible Organization Type of Review Mission need CD-0 Optional PSO/deputy administrator IPR Alternative selection and cost range CD-1 Optional PSO/deputy administrator IPR Performance baseline CD-2 Required OECM EIR Construction or execution readiness CD-3 Required for major systema (MS) project; optional for non-MS projects OECM for MS projects; PSO for non-MS projects EIR for MS projects; IPR for non-MS projects a Major systems are projects with a TEC >$400 million. SOURCE: DOE, 2003a. The order also delineates a set of key review elements to be covered but allows the level of detail and the effort involved in collecting data on these elements to be tailored to the size, complexity, and inherent risks of a project (DOE, 2003a). EIRs examined in this study included the following elements: Work breakdown structure, Resource loaded schedule (basis of cost and schedule estimates), Key project cost and schedule assumptions, Funding profile, Critical path, Risk management, Systems functions and requirements, Preliminary design review and comment disposition, Basis of preliminary design (review element is limited to major complex projects), Hazards analysis, Value management/engineering, Project controls/earned value management system, Project execution plan, Start-up test plan, Acquisition strategy, and Integrated project team. The OECM is responsible for selecting EIR consultants and planning EIR activities, although the PSO and the integrated project team (IPT)2 play an active role in the process. Prior to its on-site 2 Order 413.3 defines Integrated Project Teams as a team of professionals representing diverse disciplines with the specific knowledge, skills, and abilities necessary to support the successful execution of projects. Project directors, contracting officers, safety and quality, legal and personnel in technical disciplines compose the

OCR for page 7
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects briefings, the IPT provides relevant project documents to the EIR contractors. The onsite sessions include, in addition to the briefings, discussions among the EIR contractors and the IPT and culminate in a presentation by the contractors of their preliminary findings. This is followed by a written draft EIR that is checked over by the IPT for factual errors. After the EIR is finalized, the IPT prepares a corrective action plan. The EIR’s findings and recommendations and the IPT’s corrective action plan are used by the OECM to determine the validity of the project’s performance baseline and the sufficiency of project management actions needed to move the project to its next phase of development. The project management documentation, EIR report(s), and corrective action plans are used by the OECM to develop recommendations for senior management and acquisition executive actions as input to CD-2 and to help determine whether a project is ready for inclusion in DOE’s annual budget request. Between the release of O 413.3 in October 2000 through FY 2005, the OECM has completed more than 150 EIRs. A 2004 NRC report found that The procedures for the congressionally mandated EIRs have matured over time and the reviews are more comprehensive and have improved in quality. Consequently, there is increased appreciation within DOE of the value of EIRs, and DOE project directors interviewed by the committee now acknowledge that EIRs have provided useful information and have added considerable value to the project delivery process…. Although many DOE managers recognize the value of EIRs, the committee has heard the opinion voiced by some DOE managers and M&O [management and operation] contractors that external independent reviews provide no added value on projects costing $5 million to $20 million TPC…. The committee continues to recommend that reviews of all projects should be conducted in some form. The scope and content of the review should be tailored to the complexities and peculiarities of the project, especially for those projects between $5 million and $20 million. . . . Independent reviews of plans, assumptions, designs, estimates, and schedules are the accepted standard in industry. In fact, independent reviews are one of the means by which industrial firms become successful at project management and continue to stay that way (NRC, 2002b). The committee strongly advocates that DOE continue to recognize the value of EIRs for improving project performance and expand their application for documenting lessons learned (NRC, 2004, p. 34-35). Congress continues to support the OECM’s oversight of project management practices at DOE and the use of EIRs to validate project performance baselines. An August 2005 memorandum from Secretary of Energy Bodman to the heads of all DOE offices and programs (DOE, 2005) reiterated the importance of a disciplined project management process and adherence to the processes described in O 413.3 and the OECM’s Project Management for the Acquisition of Capital Assets (DOE, 2003a). Budgetary pressures mean that the OECM must ensure that the EIRs are effectively planned, efficiently executed, and provide added value to DOE and its individual projects. STATEMENT OF TASK As recommended in the language of Senate Report 108-115 [212], DOE asked the NRC to appoint an expert committee to advise DOE on how its EIRs can be tailored to a mix of project types to ensure that Essential information about project scope, cost, feasibility, and risk is provided at the optimum point in DOE’s CD process; membership of the typical Integrated Project Team” (DOE, 2003a, p. 3-7). The IPT is led by the federal project director.

OCR for page 7
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects DOE’s resources (time, staff, funding) are focused on projects with the greatest risk; and EIRs are cost effective. The committee was also asked to identify lessons learned from the EIR analysis that could be used to refine departmental guidance and policies to further support DOE’s goal of continuous improvement in project management. To accomplish these tasks, the NRC appointed 11 people from industry, government, and academia. Collectively the committee members have expertise and experience in project management; civil engineering; facilities design and construction; risk management; environmental engineering; project delivery systems; government procurement procedures; project scheduling; cost estimating and analysis; program evaluation; and peer review procedures. Three members of the committee participated in the 2000-2003 oversight and assessment of DOE’s project management. One member also served on the first assessment committee (NRC, 1999). Lloyd Duscha, principal investigator of Assessing the Need for Independent Project Reviews in the Department of Energy (NRC, 1998) and a member of subsequent oversight and assessment committees, served as a consultant to the committee that wrote this report. Biographical sketches are presented in Appendix A. STUDY PROCESS The committee met three times and reviewed more than 70 project-specific and policy documents of the DOE. The first meeting, convened February 27-28, 2006, included briefings on policies and procedures and specific issues of interest to the OECM and the PSOs. The committee, with input from OECM staff, identified 19 projects chosen to be representative of the 150 EIRs conducted between 2000 and 2005. Project performance records (baseline performance according to the February 2006 report to the DOE deputy secretary) were available for each project. The selected projects represent a range of size and cost (small, <$20 million; medium, $20 to $100 million; and large, >$100 million), the major PSOs (National Nuclear Security Administration [NNSA], Office of Science [SC], and Office of Environmental Management [EM]), and each of the EIR contractors. The project sample was not randomized in a statistical sense. The projects and project-specific documents are listed in Appendix B. The committee also reviewed EIR reports prepared by contractors, DOE corrective action plans, related project correspondence, and engaged in telephone conferences with selected project directors to gain further insight into the written record. The second meeting, convened June 7-9, 2006, included a workshop with DOE staff from headquarters and field offices of the OECM, the NNSA, the SC, and the EM; M&O contractors; and EIR contractors, who discussed the issues identified by the committee that influence the planning, execution, and outcome of EIRs. Workshop participants are listed in Appendix C, where the discussions are summarized as well. At its third meeting, held July 25-26, 2006, the committee synthesized and analyzed the information it had gathered and developed its findings and recommendations. The recommendations, which are based, in part, on lessons learned from its EIR analysis, represent refinements to current departmental guidance and policies. The committee’s report underwent a peer review according to NRC procedures to ensure that the conclusions, findings, and recommendations were supported by the information presented.

OCR for page 7
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects ORGANIZATION OF THE REPORT This report is organized in four chapters and three appendixes that contain supporting information. Chapter 1, “Introduction,” provides background information. Chapter 2, “Characteristics of Effective External Independent Reviews,” also discusses how the results of EIRs are used. The committee describes its expectations for effective review programs in light of its observations of DOE’s EIRs and identifies some lessons learned from this review. It covers the benefits of EIRs, the planning process, the independence and expertise of the review teams, the integration of the various types of DOE peer reviews, the costs of such reviews and the impact of where the funding comes from, and using the EIRs to improve overall project management. Chapter 3, “Criteria for Structuring Review Programs,” discusses the policies and procedures for conducting a review program that includes EIRs and other types of peer reviews. It addresses DOE’s review structure based on project costs and changes that had been implemented during the course of the study (as of July 28, 2006). Criteria for the timing, assignment of responsibility, and tailoring of reviews are also addressed. The committee presents guidelines for structuring a peer review program based on criteria such as project costs, inherent risks, scope, and schedule, and the capabilities of the project team. As promised by its title, Chapter 4 contains conclusions, findings, and recommendations. In it, the committee assesses DOE’s program for EIRs and related peer reviews, specifically recommending actions to improve the planning of reviews, project team readiness, benefits of reviews, lessons learned from them, coordination of EIRs and IPRs, and their tailoring and funding. REFERENCES ASCE (American Society of Civil Engineers). 2004. Project Peer Review Policy. Herndon, Va.: ASCE. Available at http://www.asce.org/pressroom/news/policy_details.cfm?hdlid=117. Accessed September 19, 2006. DOE (Department of Energy). 1999. Memorandum for all departments from T.J. Glauthier, Deputy Secretary. Subject: Project Management Reform Initiative. June 25, 1999. DOE. 2000a. Program and Project Management for the Acquisition of Capital Assets (Order O 413.3). Washington, D.C.: Department of Energy. DOE. 2000b. Report to the Committees on Appropriations of the U.S. Congress on the U.S. Department of Energy Implementation Procedures for the Use of External Independent Reviews and Project Engineering and Design Funds. Washington, D.C.: Department of Energy. DOE. 2003a. Project Management for the Acquisition of Capital Assets (Manual M 413.3.1). Washington, D.C.: Department of Energy. DOE. 2003b. External Independent Review Standard Operating Procedures. Washington, D.C.: Department of Energy. DOE. 2005. Memorandum for heads of departmental elements from Samuel W. Bodman, Secretary of Energy. Subject: Improving Project Management. August 10, 2005. DOE. 2006. Program and Project Management for the Acquisition of Capital Assets (Order O 413.3A). Washington, D.C.: Department of Energy. NRC (National Research Council). 1997. Peer Review in the Department of Energy-Office of Science and Technology, Interim Report. Washington, D.C.: National Academy Press. NRC. 1998a. Peer Review in Environmental Technology Development Programs. Washington, D.C.: National Academy Press. NRC. 1998b. Assessing the Need for Independent Project Reviews in the Department of Energy. Washington, D.C.: National Academy Press. NRC. 1999. Improving Project Management in the Department of Energy. Washington, D.C.: National Academy Press.

OCR for page 7
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects NRC. 2003. Progress in Improving Project Management at the Department of Energy, 2002 Assessment. Washington, D.C.: The National Academies Press. NRC. 2004. Progress in Improving Project Management at the Department of Energy, 2003 Assessment. Washington, D.C.: The National Academies Press. U.S. Congress. 1997. Committee of Conference on Energy and Water Development. HR 105-271. Washington, D.C.: Government Printing Office. U.S. Congress. 1999. House of Representatives, Energy and Water Appropriations Bill, 2000. HR 106-253. Washington, D.C.: Government Printing Office. U.S. Congress. 2000. Energy and Water Appropriations. HR 106-701. Washington, D.C.: Government Printing Office. U.S. Congress. 2003. Energy and Water Development Appropriations Bill. SR 108-115 [212]. Washington, D.C.: Government Printing Office. Wiegers, Karl E. 2006. Humanizing Peer Reviews. Happy Valley, Ore.: Process Impact. Available at http://www.processimpact.com/articles/humanizing_reviews.html. Accessed April 10, 2006.