Use of Project Reviews to Improve Execution
Until 1994, DOE and its predecessor agencies had a formal structured review process, defined in DOE Order 4700.1, which was replaced in 1995 by LCAM (DOE Order 430. 1). LCAM and the implementing LCAM Good Practices Guides (discussed in Chapter 2) gave program offices full discretion to decide how to manage projects, including how and when to conduct reviews. Since then, with two exceptions, the committee observed that reviews have been initiated on an ad hoc basis. The two exceptions include a semiformal program for project reviews begun about 15 years ago for ER (now Office of Science) projects. This review process has become more structured recently but is not a completely independent review process. The other structured project review process used by DOE is the procedure developed on the basis of the Phase I NRC report (NRC, 1998).
Independent Project Reviews
In Phase I of this study, the NRC outlined recommendations for project selection, the content of project reviews, and the capabilities of independent reviewers. Although that report focused on the projects of fiscal year 1998, the process could be used on a continuing basis. That report found that DOE had developed comprehensive guidelines for reviews during the design and construction phases of a project, but not for the preconceptual and conceptual phases (when cost and schedule benefits are the highest). The committee believes that vigorous reviews to establish valid project definition, including cost, schedule,
and scope of work baselines, in the preconceptual and conceptual phases are vital to successful project execution and to controlling costs.
Based on the recommendations of the Phase I study, DOE has contracted for independent reviews to assess the technical scope, cost estimates, schedules, and supporting data for DOE projects in accordance with congressional language. These independent reviews have highlighted the problems in DOE's management and execution of projects. The findings and recommendations of the initial independent reviews, which have been transmitted to Congress, document deficiencies similar to those found by this committee and others, indicating that difficulties with project management continue. For example, the Spent Nuclear Fuel Dry Storage Project at the Idaho National Engineering and Environmental Laboratory is a "privatization" project with a TPC of $105 million. The independent reviewer found many deficiencies in planning, execution, and contracting for this project. First there was no PEP. Second, DOE headquarters had not been informed that the project was likely to exceed its TPC. Third, there was no formal procedure for selecting the privatization contract and no record of the analysis underlying that decision (Lockwood-Greene Technologies Corporation, 1998).
In a review of the Stockpile Management Restructuring Initiative Project at DOE's Pantex facility, many weaknesses were found in budgeting and planning. The schedules for some of the subprojects were "extremely conservative." Cost estimates for many of the routine demolition and construction activities were "very high." Too many layers of management were involved in approving "small, straightforward projects." The reviewers found that 8 of the 12 subprojects lacked adequate justification for proceeding (questionable economic benefits, reduced operational flexibility, or failure to take account of changing circumstances) (Foster Wheeler Environmental Corporation, 1998a).
An independent review of the Atlas Project of Los Alamos National Laboratory (a project to improve simulation of nuclear weapons performance) found that the project was unlikely to be completed within the baseline cost estimate and recommended that a variety of steps be taken to improve cost estimation (Cadmus Group and Project Performance Corporation, 1999).
A review of the Stockpile Management Restructuring Initiative at the Kansas City Plant called it "a well run, well managed project" but identified a number of weaknesses in cost estimation, including a failure to allow for $3.2 million for the installation of start-up equipment. The review team recommended measures to improve risk assessment and cost-benefit analysis and to streamline review and approval procedures (Foster Wheeler Environmental Corporation, 1998b).
The Jupiter Corporation (1998a), reviewed the Pit Disassembly and Conversion Facility Project of the DOE Office of Fissile Material Disposition and found that the project managers had not taken into account the applicability of Nuclear Regulatory Commission licensing standards and other regulatory requirements, although Congress has required that DOE meet those standards. In addition, the project had failed to document the analysis supporting its choice of contract type.
According to the independent reviewer, the Nuclear Materials Storage Facility Renovation project of Los Alamos National Laboratory has a number of deficiencies in cost estimation, scheduling, and procurement. The project managers corrected many of these problems while the review was under way, which resulted in substantial improvements (CETROM, 1998a).
CETROM Consulting Engineering Corporation (CETROM, 1998b) reviewed the Rapid Reactivation Project (to ensure that the United States could increase its nuclear weapons manufacturing if necessary). The project was found to be generally successful, although the contingency allowances for schedules were too small.
An independent review of the Dual Axis Radiographic Hydrotest Facility project found that it was generally successful and well managed. The reviewers recommended however, better coordination of procurement documentation among the several national laboratories involved (Jupiter Corporation, 1998b).
An independent review of the Chemistry and Metallurgy Research Facility Upgrades project of Los Alamos National Laboratory (LANL) (which had been suspended in 1997 because of rapidly growing costs and scope and was partially resumed in 1998) found serious weaknesses in setting baselines. Reviewers found that the original cost, scope, and schedule baselines had not been changed to meet current DOE missions, and they recommended that new baselines be established. They also observed that management at DOE headquarters, disregarding stated policy, had inserted itself into the approval process for changes in the baseline (Jupiter Corporation, 1998c).
The review of the Nuclear Materials Safeguards and Security Upgrades Project at LANL found that although a detailed schedule risk analysis was performed as part of the Critical Decision Review process, other risks, such as construction performance and the effects of ongoing changes to the project management process at LANL, had not been taken into account. The reviewers suggested that the schedule risk analysis be updated to incorporate all identified risks and lessons learned (CETROM, 1998c).
DOE program offices, in association with FM, are currently conducting external reviews in accordance with the processes and criteria recommended in the NRC Phase I report. Briefings and draft reports provided to the committee show that program offices are also developing a process to sponsor their own internal project reviews modeled partly on the existing procedures now used by the Office of Science and partly on the criteria and guidelines recommended in the NRC Phase I report. These internally-run project reviews will be considered to be independent reviews (by DOE) as long as they are conducted outside of the project's line management and the reviewers are technically capable and have no conflicts of interest. Setting aside the question of whether an "internal independent project review" is an oxymoron, the critical factor is whether the project review will increase the likelihood that cost, schedule, and technical scope baselines will be met. The objective of an independent assessment should be to
evaluate a project from the standpoint of project management leading up to approval of the baseline (CD-2), including technical scope, cost estimates, schedules, and supporting data. The assessment should determine whether the baselines are accurate and whether the project, as formulated, is executable. A fundamental requirement is that reviewers not be advocates for the project. This requirement will be met for the external project reviews being administered by FM. However, independence could also be achieved by using trained, skilled, and knowledgeable DOE employees from outside the cognizant program and operations offices. The team would have to be balanced by outside members.
DOE should adopt a graduated approach, with the depth of detail determined by the relative importance and cost of the project. The NRC Phase I report, recommended that independent reviews be conducted for all projects with TECs of more than $20 million and projects or project activities with TECs of $5 million to $20 million if they meet the following criteria:
- DOE has little or no experience with the proposed delivery method (e.g., privatization of waste management).
- The technology is new or requires significant research and development to show that it will be workable at field scale.
- The project does not obviously or strongly support the mission objectives in DOE's Strategic Plan.
- The projects has had significant cost or schedule overruns or has a high potential for overruns.
- The projects managed by an operations office that has a history of project overruns, failures, or terminations.
- Characterization for the project is incomplete.
This committee supports the process and criteria set forth in the NRC Phase I report and the procedure developed to conduct the independent, nonadvocate reviews. However, the benefits of conducting reviews will not be fully realized until DOE adopts a process to ensure that recommendations are properly reviewed and action is taken. Furthermore, in no case should an internal review be considered to replace independent external reviews.
Finding. Independent project reviews are essential tools for assessing the quality of project management and transferring lessons learned from project to project.
Finding. External independent reviews of 26 major projects are under way to assess their technical scope, costs, and schedules. The reviews so far have documented notable deficiencies in project performance verifying the committee's conclusion that DOE's project management has not improved and that its problems are ongoing. However, DOE has yet to formalize and institutionalize a process to ensure that the recommendations from these reviews are implemented.
Finding. Various DOE program offices are also developing the capability of conducting internal independent project reviews.
Recommendation. DOE should formalize and institutionalize procedures for continuing independent, nonadvocate reviews, as recommended in the Phase I report of the National Research Council to ensure that the findings and recommendations of those reviews are implemented. DOE should ensure that the reviewers are truly independent and have no conflicts of interest.
Recommendation. All programs that have projects with total estimated costs of more than $20 million should conduct internal reviews, provided that the value of the reviews would be equal to or greater than the costs of conducting them. The decision to proceed with internal reviews should be made by program management, depending on past experience with similar projects, the estimated cost of the project, and the uncertainty associated with the project. Internal reviews are expensive and take up the time of valuable people, so they should not be undertaken lightly. However, under the present circumstances, the committee believes that more internal reviews would be justified. The project management office should manage these reviews for the director or assistant secretary of the cognizant program office using nonadvocate reviewers. The results of these reviews should be taken by the program office to the Energy Secretary's Acquisition Advisory Board (ESAAB), and used as the basis for deciding on whether to continue the project.
Other Forms of Project Review
Project Reviews of the Office of Science
DOE's Office of Science has a long-established system of peer reviews to corroborate the scientific and technical feasibility of its projects. Peer review teams are established during the preconceptual or conceptual phase and participate in formulating the project to meet preconceived needs. Periodic reviews are also conducted during the later phases of the project. In addition to addressing the technical and scientific aspects of a project, some team members are assigned to address cost estimates and construction schedules (albeit not as rigorously). These reviews cannot be called fully independent, however, because the teams tend to be composed of internal proponents and external members with indirect interests in the projects. Therefore, they may not give full consideration to alternatives for achieving the goals of a project or question the need for the project itself. These reviews are systematic, however, and they do produce audit trails of the decision-making process.
The committee believes that peer review panels should not include internal proponents or others with conflicts of interest. The panel should be made up of
individuals with broad interests and should be able to develop alternative scenarios for accomplishing the project goals and assess rigorously cost estimates and construction schedules. In no case should internal reviews be considered as replacements for independent external reviews.
Special Panels of Experts
If in-house expertise is lacking or if the viability of proposed solutions must be assessed, DOE has occasionally convened special panels of experts to provide technical advice on complex or unusual issues. The committee endorses this approach.
Environmental Management Quarterly Management Review
In addition to the review process specified under the LCAM Order, EM conducts its own quarterly performance reviews in conjunction with all operations offices. These extensive reviews cover many established milestones, line items, and projects in the areas of waste management, environmental restoration, science and technology, and nuclear material and facility stabilization. The committee recognizes the management and technical challenges facing DOE in the environmental management arena and is highly supportive of EM's efforts to conduct these reviews. Despite these efforts, however, projects have not performed at a level comparable to that of the private sector or other federal agencies.
Reviews by the Defense Nuclear Facilities Safety Board
The DNFSB was created in part to review the design, construction, operation, and decommissioning of DOE's defense nuclear facilities to ensure that activities at those facilities are conducted in such a way that the health and safety of the public, including the environment and workers, are protected. The DNFSB and its staff conducts reviews to identify potential health and safety concerns. The primary mechanism used by the DNFSB to promote action is the issuance of recommendations to the secretary of energy on specific areas of concern. The DNFSB has great latitude in selecting projects to review, in access to individual projects, and in recommending safety and health issues to be addressed. The DNFSB does not manage DOE's work, but it does influence DOE projects when the health and safety risks warrant intervention.
Independent Reviews Using Statistical Models
EM has engaged IPA to review its project management system twice since 1993. IPA has a large database of private and government contracts, used mainly for analyzing cost and schedule overruns for industrial, chemical process, and
other projects similar to DOE's environmental restoration and waste management projects. IPA found that EM project costs were much higher than those of industry (typically 48 percent higher) or other government agencies and had more schedule slippages (IPA, 1993). IPA has also undertaken analyses for individual DOE projects (see Appendix A), in which the project parameters are used in a statistically-fitted model to determine if the project is consistent with the norms from other projects. The findings from these studies showed that intensive project reviews are necessary at all stages, especially in the early stage of establishing a baseline.
Cadmus Group and Project Performance Corporation. 1999. External Independent Review: The Atlas Project. Defense Programs Project No. 96-D-103. Final report prepared for the Office of Field Management, U.S. Department of Energy.
CETROM (CETROM Consulting Engineering Corporation). 1998a. External Independent Review: Nuclear Materials Storage Facility Renovation. Defense Programs Project No. 97-D-122. Final report prepared for the Office of Field Management, U.S. Department of Energy.
CETROM. 1998b. External Independent Review: Rapid Reactivation Project. Defense Programs Project No. 99-D-122. Final report prepared for the Office of Field Management, U.S. Department of Energy.
CETROM. 1998c. External Independent Review: Nuclear Materials Safeguards and Security Upgrades. Defense Programs Project No. 97-D-132. Final report prepared for the Office of Field Management, U.S. Department of Energy.
Foster Wheeler Environmental Corporation. 1998a. External Independent Review: Stockpile Management Restructuring Initiative Project, DOE Pantex Facility. Defense Programs Project No. 99-D-128. Final report prepared for the Office of Field Management, U.S. Department of Energy.
Foster Wheeler Environmental Corporation. 1998b. Independent Assessment: Stockpile Management Restructuring Initiative, Kansas City Plant. Report prepared for the Office of Field Management, U.S. Department of Energy.
IPA (Independent Project Analysis). 1993. U.S. Department of Energy, Office of Environmental Restoration and Waste Management, Project Performance Study. Reston, Va.: Independent Project Analysis, Inc.
Jupiter Corporation. 1998a. External Independent Review: Pit Disassembly and Conversion Facility. Project No. 99-D-141. Final report prepared for the Office of Field Management, U.S. Department of Energy.
Jupiter Corporation. 1998b. External Independent Review: Dual Axis Radiographic Hydrotest Facility. Defense Programs Project No. 97-D-102. Final report prepared for the Office of Field Management, U.S. Department of Energy.
Jupiter Corporation. 1998c. External Independent Review: Chemistry and Metallurgy Research Facility Upgrades. Project No. 95-D-102. Final report prepared for the Office of Field Management, U.S. Department of Energy.
Lockwood-Greene Technologies Corporation. 1998. External Independent Review: Spent Nuclear Fuel Dry Storage Project, Defense Environmental Management Privatization Project No. 98-PVT-2. Readiness Review Report prepared for the Office of Field Management , U.S. Department of Energy.
NRC (National Research Council). 1998. Assessing the Need for Independent Project Reviews in the Department of Energy. Board on Infrastructure and the Constructed Environment, National Research Council. Washington, D.C.: National Academy Press.