Progress in Improving Project Management at the Department of Energy: 2001 Assessment (2001)

The Department of Energy (DOE) is engaged in numerous multimillion- and even multibillion-dollar projects that are one of a kind or first of a kind and require cutting-edge technology. The projects represent the diverse nature of DOE’s missions, which encompass energy systems, nuclear weapons stewardship, environmental restoration, and basic research. Few other government or private organizations are challenged by projects of a similar magnitude, diversity, and complexity. To complete these complex projects on schedule, on budget, and in scope, the DOE needs highly developed project management capabilities.

This report is an assessment of the status of project management in the Department of Energy as of mid-2001 and the progress DOE has made in this area since the National Research Council (NRC) report Improving Project Management in the Department of Energy (Phase II report) was published in June 1999 (NRC, 1999). The Phase II report findings and recommendations are reproduced as Appendix C. The findings presented in this report reiterate and expand on those given in the committee’s January 2001 interim letter report, Improved Project Management in the Department of Energy (NRC, 2001), reproduced here as Appendix D.

The Phase II report estimated that DOE projects costed taxpayers 50 percent more than comparable projects would cost if performed by the private sector or other government agencies, in large part because DOE did not use industry-standard best practices for project management. The Phase II report recommended, inter alia, that DOE develop policies, procedures, models, tools, techniques, and standards; train staff in their use; and require their use on DOE projects. It recommended further that DOE should develop and deploy a compre-

hensive project management system with clear definition of the specific roles and responsibilities of all parties associated with a project.

As noted in the interim letter report, the department has taken a number of positive steps since the Phase II report. On June 25, 1999, subsequent to the release of that report, the deputy secretary, as the DOE chief operating officer, issued a memorandum announcing a project management reform initiative. This memorandum directed a number of actions to be taken to improve project management capability. These included the formation of the Office of Engineering and Construction Management (OECM) in the office of the chief financial officer (CFO) and the formation and strengthening of project management support offices (PMSOs) in the three major program secretarial offices (PSOs). On January 3, 2000, the deputy secretary issued an interim instruction to serve as policy guidance on critical decisions by acquisition executives (AEs) and the Energy Systems Acquisition Advisory Board (ESAAB) and on the conduct of corporate-level performance reviews. On June 10, 2000, DOE issued Policy P413.1, which addresses project management accountability, the establishment of project management organizations, project management tools, and training of personnel. DOE Order O413.3 was issued October 13, 2000, to implement the DOE policy document. O413.3 covered department policies on project management; provision for project engineering and design (PED) funding for preconstruction planning; reestablishment of the ESAABs; and other matters related to the management and oversight of DOE projects. Finally, the Program and Project Management (PPM) manual and a companion volume, Project Management Practices (PMP), were released in draft form in October 2000.

The body of this report addresses certain specific areas that the committee believes needed greater definition and follow-up since the Phase II report. Not all the findings and recommendations in the Phase II report are repeated in the body of this report, although the committee continues to endorse them, so this report should be used in conjunction with the Phase II report.

The new secretary of energy, the deputy secretary, and the CFO were briefed on the recommendations of the Phase II report (NRC, 1999) and the 2001 letter report (NRC, 2001). All of these executives expressed their intention to gain control over DOE projects and to change the prevalent project management culture within DOE and its contractors through active management attention and oversight. Although DOE senior management has expressed the intent to achieve early results, it is too soon to see any effects from these management changes, as the new deputy secretary was confirmed only in June 2001. Accordingly, any policies and practices put in place by the new management team at DOE lie outside the scope of the present report, and their assessment must be deferred to

a later time. Nevertheless, the committee is optimistic that if these stated intentions are carried through, there can be positive change at DOE.

The committee believes the efforts of the deputy secretary, the CFO, OECM, and the PMSOs have unquestionably raised awareness of the importance of good project management within the department. Briefings by DOE officials indicate that new steps are being undertaken at all levels of the department. This is encouraging, and if the efforts continue, they can form the foundation of a coherent project management approach for the department. By and large, these initiatives are considered by the committee to be steps toward improving project performance. However, change has been inordinately slow, and the committee has found no evidence that DOE project management practice and performance in the field have actually improved.

The committee believes that the PMSOs in the Environmental Management (EM), National Nuclear Security Agency (NNSA), and Office of Science (SC) PSOs are having a positive influence on project management in their programs. The committee recognizes the need for tailoring project management programs to the different PSO mission requirements. However, some PSOs are improving faster than others, and progress could be accelerated by stronger central oversight and support using OECM as a unifying organization to assist the PMSOs and the PSO projects and to validate the results.

Department-wide, recognition of project management deficiencies and of the effectiveness of initiatives for improvement has been inconsistent. The PMSO in NNSA has been particularly active since formation of the agency. Excessive administrative impediments and time required to get the OECM up and running, along with the inadequate resources assigned to it, are symptoms of cultural resistance to change and the lack of a sense of urgency. The committee is aware that not all of the recommendations in the Phase II report could have been accomplished in the 2 years since it was released, but it does believe that much more progress could have been made. The committee recommends that OECM should be budgeted, staffed, and empowered to become the center of excellence in project management and the central manager for oversight and approval of all capital projects in DOE.

DOE continues to rely heavily on contractors, not only for the management of projects, which is to be expected, but also for project justification and definition of scope. DOE does not directly manage projects, unless the contractor is simply working on a time and materials basis, as do some management and operations (M&O) contractors. Although many projects are managed successfully, especially those identified as outstanding examples in the OECM October 2000 project management workshop awards, DOE is not in control of many of its projects and in some cases has virtually abdicated its role of owner in project oversight and management of contracts and contractors. As stated in the Phase II report, DOE needs to become proficient in the role of owner; and DOE project

managers should become knowledgeable owner’s representatives. The committee has seen little evidence that DOE is aware of the distinction between the owner’s and contractor’s role, let alone acting upon it.

Two of the overarching issues that particularly concern the committee are the lack of strategic planning and the lack of a system for process improvement. The first step toward process definition has been taken in O413.3, but much remains to be done. A consistent, documented process and procedures for efficient project planning, justification, and execution are essential. Treatment of each project on a unique, ad hoc basis continues and should be seen as unacceptable. The committee believes that each PSO should develop its own strategic plan and that project justification as required for approval of mission need (CD-0) should be based on an assessment of the congruity of project descriptions with these strategic plans. The committee recommends that DOE initiate a program for improving the project management process, following an established statistical process control procedure such as the well-known six-sigma process.

The committee has received no evidence that new projects are getting off to a better start than before the Phase II report was issued. The establishment of PED funding and the development of the performance baseline at 25 to 30 percent design completion are positive steps, but the committee’s review of new projects authorized in FY2000 and FY2001 uncovered little documentation of mission need, strategic requirements, scope justification, risk assessment, and the other early steps that are necessary for project success. Industry studies consistently correlate project performance with the quality of front-end planning (CII, 2000). The committee reiterates the Phase II finding that improvement in project performance will require improvement in the front-end planning process, and that any improvement in front-end planning will require positive, aggressive action by all responsible parties. It should be noted that all the projects reviewed by or presented to the committee were initiated prior to the activation of the OECM, the establishment of the PMSOs, and the issuance of O413.3. Therefore, this negative assessment does not mean that these initiatives are ineffective. What it does indicate, based on the evidence provided to the committee, is that the DOE process has not noticeably improved since the 1999 Phase II report and probably will not improve until the reforms have been implemented and have become effective.

The committee recommends that OECM assure that all program offices have a documented front-end planning process that meets the intent of O413.3, and that the information be used as input for Energy Systems Acquisition Advisory Boards (ESAABs) and ESAAB-equivalent readiness reviews. The outcomes of these reviews should be documented and used to assess project performance and

progress in improving project planning. Without immediate improvement in the planning knowledge and skills of personnel and more management emphasis on improving the planning process, projects will continue to have inadequate front-end planning. To overcome the lack of internal skilled project planners and the delays in training, and to bridge the gap until a training program takes effect, the committee recommends that DOE should engage a cadre of experienced project planners with a wide variety of planning capabilities and prior experience in the different project types, including high-risk projects. These individuals should be a part of the initial integrated project teams and should assist the project originators (as internal consultants) in getting front-end planning done correctly, even before CD-0.

Risk management is probably the most difficult aspect of project management, and for many DOE projects it is also the most critical. Discussions with DOE project managers have revealed that DOE has not yet established effective risk-management methodologies or systems, and managers lack the tools and training to adequately address risk management for projects with high levels of uncertainty. The acquisition risk management (ARM) pilot program by the DOE Contract Reform Office is beginning to address many risk management issues and has defined three iterative phases of risk management for DOE projects:

1. Risk identification,

2. Risk analysis and evaluation, and

3. Risk response.

The committee believes that all three phases of risk management are critical to effective project management at DOE. The committee also believes that DOE should undertake a department-wide assessment of risks for ongoing projects and adopt procedures for program-wide management of contingencies and management reserves.

The committee has found that there is no consistent system for evaluating the relative risks of projects with respect to scope, cost, or duration, which means that the deputy secretary, the CFO, and the PSO managers have no objective basis for knowing which projects are riskier (and therefore require more management attention) than others. Probability analysis has been used and misused as a tool for risk analysis. However, the failure to accurately identify the root causes of risk, including technical, environmental, and human factors, and the potential for common mode failure has led to underestimating risks on many DOE projects. The committee recommends that DOE should identify or develop personnel with the ability to perform qualitative and quantitative risk assessments and assign

them to work with personnel with in-depth understanding of a given project to undertake risk management. Independent assessors or reviewers who are not project proponents should separately evaluate internal project risk assessments for reasonableness of assumptions, estimates, and results. Risk mitigation and management plans should be prepared to deal with any significant risks identified, especially risks due to common modes or root causes.

The committee recommends that DOE should develop more expertise and improved tools for risk management. Nontraditional and innovative approaches, tools, and methods should be investigated for adaptation to DOE project conditions and use in DOE risk management, including those cited in this report and in the Phase II report (NRC, 1999, Appendix B), such as systems analysis, event trees, causal loop diagrams, system dynamics, stochastic simulation, and other approaches that have been tested and shown to be valuable on similar projects or in addressing similar challenges.

Effective oversight of project performance is dependent on systematic and realistic reporting of project performance data. Each PSO has its own active project reporting systems, and OECM has completed a specification and beta testing for a department-wide project analysis and reporting system (PARS). The committee believes that PARS should be designed so that it supports the data analysis needed by project managers to evaluate their projects’ performance as well as the oversight needs of the PSOs, OECM, the CFO, and the deputy secretary. The database should also provide the information needed for process improvement through statistical process control and benchmarking future projects.

An effective earned value management system (EVMS) is needed as the source of data for PARS and data analysis for project management and project oversight. To create an effective department-wide earned value management and reporting system, a consistent accrual accounting system is needed for all projects.

The committee believes that DOE management needs to be able to detect potentially adverse trends in project progress and distinguish them from mere random fluctuations in progress reporting. EVMS provides data that can be used to gain some very valuable insights into the health of a project and can predict the probable outcome. EVMS data can also allow useful insights into the conduct of the work, particularly when it is reported and analyzed to evaluate period-to-period trends. The committee recommends that DOE utilize EVMS data to calculate the incremental and cumulative cost performance index, schedule performance index, and contingency utilization index for each reporting period and that it produce process control charts to analyze and improve project performance.

The committee finds that DOE has made substantial progress in the implementation of reviews and resultant corrective action plans and in the formalization and institutionalization of the review process. It continues to recommend the use of formalized assessments of management, scope, cost, and schedule at appropriate stages—from determining need for the project to determining readiness for construction—as well as regular performance reviews.

The committee examined the internal review processes documented by the three PMSOs and noted some inconsistency in general approach and the degree of recent process improvement. It strongly endorses a department-wide manual to achieve consistency in process, nomenclature, and reporting. Department-wide procedures would allow the lessons learned in reviews to gain broader recognition and would facilitate the ability of reviewers to participate in reviews in multiple organizations. Any differences or distinct features pertinent to a particular program could be identified and articulated in the text or in appendixes.

The committee is concerned that mandatory review for projects between $5 million and $20 million total project cost may be consuming too many resources and diverting too much management attention relative to the value it adds. The committee understands that although O413.3 refers to the $5 million level, OECM intends to waive the requirement if it believes a review would not be cost effective.

The committee has reaffirmed the integral role that acquisition planning and acquisition and contracting techniques play in successful project management, as noted in both of its previous reports (NRC 1998, 1999). This role is particularly critical in the DOE environment, where some 90 percent of the department’s budget is expended by contract (GAO, 2001). The committee stresses the importance of developing and employing contracting methods, such as performance-based contracting (PBC), that ensure accountability, adequately address risk, and focus the government and the contractor on achieving the outcomes sought.

Successful PBC is based on defining existing conditions, specific requirements, and the desired results or outcomes, along with objective, meaningful, and measurable performance and quality standards. In addition, incentives are used to focus contractor efforts and to reward success. In a successful performance-based contract, expectations should be made clear, with agency and contractor teams working together in a business partnership to achieve well-defined and measurable results.

The integrated project team (IPT) concept included in DOE Order O413.3 is an essential element in implementing a performance-based approach. The committee strongly supports the use of IPTs and suggests a PBC methodology for

forming teams and developing performance metrics and incentives. The committee recommends intensive PBS training for IPTs and recommends that OECM should, in the near term, bring on board a cadre of experts skilled in performance-based contracting to provide technical assistance to IPTs responsible for new major system initiatives.

The committee recommends that tailoring contracting approaches to use fixed-price and performance-based methods where practicable will assist the department in getting the most cost-effective results and will also result in greater competition. In addition, the department should continue to explore other innovative commercial contracting approaches to meet its needs.

The OECM is to be commended on the significant progress in getting Policy P413.1 and Order O413.3 produced and published; however, the committee has noted shortcomings in these documents and in the draft documents Program and Project Management (PPM) manual and Project Management Practices (PMP). The committee observed that O413.3 has proven to be effective in defining and implementing a number of fundamental and beneficial changes for the department that will improve long-term project performance; however, there are several clarifications, improvements, and adjustments that, while not changing the basic policy, would improve it.

The draft PPM and PMP were overwhelming in their detail in some sections, which in some instances was superfluous and in others misleading; major issues, by contrast, received very little attention. Some important issues were missing entirely, such as team alignment and teamwork procedures for including stakeholder and public participation; project scope definition; and control of scope change. The committee recommends that the PPM and PMP texts should be tailored to specific DOE requirements. It should be made clear which parts of the text constitute DOE required procedures and which parts reflect general advice on good project management practices. OECM should assure that policies and required procedures add value by streamlining the process and improving project performance. Examples should be given, where possible, to illustrate the application of procedures and the necessary documentation. The examples should have adequate explanations and represent realistic project situations. Over time, a set of templates and case studies should be built up.

More important, and of much greater concern to the committee, is the pervasive slowness of change at DOE in response to these problems. There continues to be excessive dependence on written policies and procedures to effect change. Although O413.3 was useful for defining DOE policy, guidelines, directives, and orders from DOE headquarters, even if they are good, will not by themselves

effect change. If paper guidelines had been adequate, there would have been no need for this committee or for its predecessor. In the final analysis, the effectiveness of any policies and procedures depends on a commitment by DOE leadership to the continuous improvement of DOE project performance by proven project management methods and techniques, as defined in departmental policies and procedures.

In the more than 2 years since the 1999 NRC report recommended training for DOE project managers, no departmental training program is in place. Although an expensive study on human resources management is under way, no one has been trained under it, and the committee has been advised that training may not begin until 2003. The need for project management training by experienced project managers was apparent to DOE long before the Phase II report, and the lack of training is a major impediment to improved project management in the department.

In January 2001, the deputy secretary directed OECM to lead a 2-year effort to develop and implement the Project Management Career Development Program (PMCDP). To accomplish this goal, a task force was established that included representatives from PSO headquarters and field offices and experts from other federal agencies. The committee applauds the task force effort to create a program geared to developing the knowledge and skills needed by project managers to fulfill the missions of the agency. However, despite the fact that the final curriculum for project manager training will not be completed until next year, it is imperative that training not be neglected in the interim. The committee recommends that DOE should implement an immediate, accelerated training program to improve the knowledge, skills, and abilities of project managers to address recognized gaps while continuing the PMCDP planning effort. This should be accomplished by eliminating impediments and using current resources, as well as exploring creative and cost-effective nonclassroom alternatives. DOE management should budget the funds required to accomplish the projected training objectives and should persist in mandating the accomplishment of individual career development objectives.

As stated in the Phase II report (NRC, 1999) and the committee’s later letter report (NRC, 2001), effective and accountable project management should be a priority for DOE and its leaders at all levels. Through actions taken to date, DOE has begun to address some of the core issues; however, a number of issues have not been resolved. The committee addresses these issues in detail in the findings

and recommendations in this report. The committee will continue to look for long-term project management reforms, process improvement, and the resultant improvement in project performance.

CII (Construction Industry Institute). 2000. Benchmarking and Metric Report. Austin, Tex.: Construction Industry Institute.

GAO (General Accounting Office). 2001. Major Management Challenges and Program Risks: Department of Energy. GAO/GAO-01–246. Washington, D.C.: Government Printing Office.

NRC (National Research Council). 1998. Assessing the Need for Independent Project Reviews in the Department of Energy. Washington, D.C.: National Academy Press.

NRC. 1999. Improving Project Management in the Department of Energy. Washington, D.C.: National Academy Press.

NRC. 2001. Improved Project Management in the Department of Energy. Letter report, January. Washington, D.C.: National Academy Press.