Progress in Improving Project Management at the Department of Energy: 2001 Assessment (2001)

The committee finds that DOE has made substantial progress in the implementation of reviews and the resultant corrective action plans and in the formalization and institutionalization of the review process since the issuance of the Phase I report (NRC, 1998), the Phase II report (NRC, 1999), and the NRC letter report dated January 17, 2001 (NRC, 2001). The committee continues to recommend the use of formalized assessments of management, scope, cost, and schedule at appropriate stages—from determining project need to determining readiness for construction—as well as regular performance reviews. The committee also recognizes that the Energy and Water Subcommittee of the House Appropriations Committee continues to rely heavily on external independent reviews and has mandated that all line-item projects be reviewed before any new money is spent. Despite the added emphasis on reviews, the potential for cost overruns for some projects, such as the National Ignition Facility, the Isotope Production Facility, and the Neutrinos at the Main Injection Facility, continue to evoke concern. A June 2001 GAO report notes that “NIF still lacks an independent external review process” (GAO, 2001). New projects just getting under way are also of great concern to the committee. Timely reviews in the front-end project planning stage could be very helpful in getting new projects off to the right start.

The committee heard some DOE field representatives at various forums endorse the reviews and was encouraged by these endorsements. At the same time, other individuals said they had problems with the required scope and detail of the reviews, their number and frequency, and their application to small projects.

This may suggest that the procedures and general requirements should be more carefully tailored to the circumstances of the different projects to assure that the reviews are cost effective, as recommended in the Phase II report.

The committee reviewed three documents—the Office of Defense Programs (DP) Project Review Procedures, dated September 19, 2000 (DOE, 2000a); the Office of Environmental Management (EM) Internal Independent Review Handbook, dated August 2000 (DOE, 2000b); and the Office of Science (SC) Independent Review Handbook, dated January 2001 (DOE, 2001a)—which describe the respective PSO review procedures for internal independent reviews. The DP and EM documents are formatted differently, but both are fairly comprehensive and detailed and appear to satisfactorily cover the aspects expected of a review. Both focus on reviews preceding a particular decision point. A distinctive feature of the Environmental Management approach is the use of the EM project definition rating index (EM-PDRI), which assesses how well a project is defined in terms of a numerical score. EM based its index on rating factors developed by the CII. EM-PDRI scores are an important element in the decision to proceed to the next phase, but are only one of several elements in a go/no-go decision (DOE, 2001b).

Although their approaches differ, DP and EM are to be commended for aggressively addressing this issue. By contrast, the SC handbook is less formal and detailed, perhaps because a process for review of scientific and technical issues has long been institutionalized at SC. The SC reviews are not specifically oriented to the critical decision points defined in DOE O413.3. They serve to validate technical, cost, and schedule baselines before construction funds are requested. Most SC reviews are conducted semiannually to assess the status of ongoing projects. These characteristics of SC reviews were commented on in the Phase I and Phase II reports (NRC, 1998, 1999).

The committee also reviewed the Office of Engineering and Construction Management (OECM) draft Independent Review Procedure, dated June 1, 2001 (DOE, 2001c), which addresses both internal and external reviews department-wide. This document most closely resembles the handbook issued by DP. The committee finds the OECM document to be comprehensive and well presented. A department-wide manual to achieve consistency in process, nomenclature, and reporting is strongly recommended by the committee. Department-wide procedures would mean broader recognition for the lessons learned in reviews and would facilitate the ability of reviewers to participate in reviews in multiple organizations. Any distinct features of a particular program could be identified and articulated in the text or in appendixes.

The OECM document names OECM as responsible for executing the external independent review (EIR) and the independent cost estimate (ICE) and names the respective PSOs as responsible for executing the internal project review (IPR)

and the internal cost review (ICR). The Phase II report (NRC, 1999) recommended that internal reviews be managed centrally; however, the committee is satisfied that by following the proposed OECM procedures, internal reviews can be executed by the respective program offices with OECM oversight.

Criteria for members of teams carrying out internal independent reviews vary among the PMSOs and OECM. SC requires the reviews to be conducted by a nonproponent review team and defines “nonproponent” as reviewers having no current affiliation with the project being reviewed and not being from the responsible program office in SC, related contractors, or the related funding office. The SC team leader is always a DOE employee. EM stipulates that reviewers have no current affiliation with the project and no current assignment to an organization participating in the project. EM team leaders can have neither current nor prior involvement with the project, nor can they have current affiliation with a participating line program or field site. The members of the DP review teams must be selected mainly from outside DP and from federal agencies outside DOE. The OECM procedure follows the DP concept, which is most reliable for ensuring that reviewers are truly independent and have no conflict of interest. The committee supports this approach but recognizes that it could become burdensome and expensive for small and/or uncomplicated projects.

DOE O413.3 specifies an external independent review (EIR) of the performance baseline prior to CD-2 and an internal project review (IPR) of readiness for construction prior to CD-3 for all projects with a total project cost (TPC) exceeding $5 million. If the project is classified as a major system (above $400 million), an internal project review is required prior to CD-0 and an EIR prior to CD-3.

The committee is concerned that mandatory review for projects between $5 million and $20 million TPC may be consuming too many resources and diverting too much management attention relative to the value added. The Phase I report recommended lowering the floor from $20 million to $5 million total estimated cost (TEC) only if certain criteria prevail. A number of DOE field representatives have voiced objections to the time and manpower involved in supporting the reviews for minor projects. The committee understands that although O413.3 refers to the $5 million level, OECM intends to waive the requirement if a review would not be cost effective. In any case, there is nothing to preclude an acquisition executive or other authorized DOE executive from ordering a review for any stage of any project without requiring reviews for all projects greater than $5 million.

An area of concern during the Phase I and Phase II studies was the weakness of DOE’s level of effort in front-end project planning and its lack of documentation for such planning. As noted above, the only formal review requirement is an IPR prior to CD-0 for major systems. The committee reviewed the subject further with the three PSOs and found that DP and EM recognize the weakness and are devising a means to strengthen the process. The committee believes much could be gained by expanding the use of independent reviews in the front-end project planning phase. It would appear appropriate to require an IPR prior to CD-0 for projects less than $20 million and require an ICR prior to CD-1 for projects over $20 million.

An external independent review process for projects was initiated by DOE subsequent to the issuance of the Phase I report, which recommended a number of specific project reviews (NRC, 1998). As time was of the essence to meet congressional mandates, DOE awarded contracts to various engineering firms and other consultants to perform the reviews. In some cases, the haste probably contributed to ineffective reviews—there may have been inadequate scope definition and/or selection of contractors that were underqualified or not truly independent. Nevertheless, the reviews exposed many defects in project definition, and the cost and schedule estimates and reviews were, on balance, a valuable tool for project management and maintaining credibility with the Congress.

The inconsistency in content, quality, and format of the early review reports, plus the fact that some reports were not well received internally or externally, illustrates the need for a more consistent and centralized approach and control. The committee strongly supports the decision to assign the responsibility for EIRs to the OECM. OECM has taken action to define the scope of an EIR more specifically in order to obtain more useful and consistent results and to decrease the time and money required to conduct an EIR. The continuing improvement of EIR reports should help project managers increase the probability of project success and better assure Congress of the credibility of project performance reports.

OECM contracted for a statistical analysis of 65 EIRs transmitted to Congress through fiscal year 2000 (RCI, 2000). The data were collected and organized around the lines of inquiry of the EM-PDRI system. Of the projects reviewed, 94 percent identified issues related to management, planning, and controls; 84 percent identified issues related to cost; 83 percent identified issues related to scope and technical factors; 66 percent identified issues related to schedule; and 23 percent identified issues related to external factors. The issues identified in reviews strongly indicate that a large majority of projects were not

ready to proceed to the next stage without corrective actions. Figures of such magnitude clearly portray the value of reviews as well as the desirability of continuing the EIR program until positive improvement is demonstrated.

The committee evaluated a sample of internal review reports from DP, EM, and SC. It commends the reports from DP for their penetration of the issues and uniformity of presentation. These reports will provide an excellent audit trail for future reviews and critical decisions. The reports from EM were less detailed because the use of the EM-PDRI allowed the narrative to be condensed. There was no attempt to judge the relative value of the EM and DP approaches, except that the DP reports appear to represent a larger investment of time and money. It is noted that, on occasion, DP performed an internal review as preparation for defining the contract for an external review. These internal reviews prior to external reviews may be very valuable, but they should be recognized as part of the cost of the review program. The SC report reviewed the status of project technical performance and management activities and contained less analysis of management procedures.

OECM recently developed a computerized project review management system (PRMS) as a tool for improving the quality of project reviews. The system stores pertinent project cost and schedule data, tracks progress, stores reports, schedules reviews and follow-up actions, creates statement-of-work templates, and promotes consistent report formats. The system also contains a master list of questions (lines of inquiry) to guide the review. The questions were developed from the EM-PDRI and DP methodology. Findings and recommendations emanating from the reviews are transferred to a corrective action plan (CAP), which is monitored through completion. The committee endorses the automation of the project review process and the tracking of the ensuing corrective actions but recognizes that the system is evolving and will be adjusted as experience is gained.

Finding. The evidence available to the committee indicates that the EIR program continues to identify significant management issues in the projects reviewed and in DOE’s operation in general. Absent substantial evidence of improvement in DOE project management and project performance, the EIR program needs to be continued.

Recommendation. DOE, through OECM, should establish performance metrics for the EIR program that identify trends and opportunities for improving project management performance.

Recommendation. The EIR program should continue in its present form under OECM direction until there is clear evidence of improvement in DOE project management and project performance.

Finding. DOE would benefit from a department-wide procedure governing external and internal independent reviews. Consistent procedures would increase the pool of qualified reviewers, expedite the review and report process, and enable an automated system for tracking deficiencies and corrective actions.

Recommendation. DOE should expedite the issuance of the Independent Review Procedure drafted by OECM.

Finding. A more thorough review analysis for defining mission need and setting a preliminary baseline range during the front-end project planning phase would give the decision makers more useful information.

Recommendation. DOE should expand the use of IPRs for the CD-0 decision and should require an ICR prior to CD-1.

Finding. There is some concern that mandating formalized reviews for projects costing between $5 million and $20 million TPC may be dedicating manpower and money beyond the point of significant value added. It also may be distracting project personnel from doing the project and diverting DOE’s project management resources from larger, more complex projects.

Recommendation. DOE should reevaluate the benefits gained from mandating reviews for projects costing between $5 million and $20 million TPC. The OECM should establish guidelines to evaluate the cost-effectiveness of review. At a minimum, the $5 million threshold should be based on TEC and provide for significant tailoring of the review process.

Finding. The EM Project Definition Rating Index (EM-PDRI) analyzes the readiness of a project by rating it on a numerical basis. It allows making judgments based on a multitude of rating factors, including risk, but users will need training and experience with the index in order to achieve uniformity of application and confidence in the results.

Recommendation. DOE should explore the potential application to other programs of the PDRI approach adopted by EM.

DOE (U.S. Department of Energy). 2000a. Office of Defense Programs Project Review Procedures. Washington, D.C.: Department of Energy.

DOE. 2000b. Office of Environmental Management Internal Independent Review Handbook. Washington, D.C.: Department of Energy.

DOE. 2001a. Office of Science Independent Review Handbook. Washington, D.C.: Department of Energy.

DOE. 2001b. Office of Environmental Management Project Definition Rating Index. Washington, D.C.: Department of Energy.

DOE. 2001c. Office of Engineering and Construction Management Draft Independent Review Procedures. Washington, D.C.: Department of Energy.

GAO (General Accounting Office). 2001. Department of Energy: Follow-up Review of the National Ignition Facility (GAO-01–677R). Washington, D.C.: General Accounting Office.

NRC (National Research Council). 1998. Assessing the Need for Independent Project Reviews in the Department of Energy. Washington, D.C.: National Academy Press.

NRC. 1999. Improving Project Management in the Department of Energy. Washington, D.C.: National Academy Press.

NRC. 2001. Improved Project Management in the Department of Energy. Washington, D.C.: National Academy Press.

RCI (Resource Consultants, Inc.). 2000. External Independent Review Analysis. Report for Office of Engineering and Construction Management, Department of Energy, November 30.