Progress in Improving Project Management at the Department of Energy: 2001 Assessment (2001)

The National Research Council (NRC) Committee on Oversight and Assessment of U.S. Department of Energy (DOE) Project Management has completed its initial assessment of DOE’s progress in implementing the recommendations from the 1999 NRC report, Improving Project Management in the Department of Energy (the Phase II report), and related actions. The committee’s assessment is based on briefings by DOE staff and others involved with improving project management, a review of documents provided by DOE, and other relevant materials. The review and assessment were directed by the 106th Congressional Committee of Conference on Energy and Water Development (House Report 106– 336).

This letter report is submitted pursuant to an agreement between DOE and NRC for a report six months after initiation of the study. It transmits the committee’s assessment of DOE’s progress and recommends additional actions to further improve DOE project management capabilities. The letter consists of an overall summary, observations, findings, and recommendations relating to the general categories of recommendations in the Phase II report.

The committee intends to seek further input from DOE headquarters, field offices, and projects, as well as from current, former, and potential DOE contractors, in subsequent efforts to determine how well project management reforms are working and what additional steps may be necessary for DOE to achieve

excellence in project management. The committee appreciates the cooperation and support of the Office of Engineering and Construction Management (OECM), the project management support offices (PMSOs), and the other elements of DOE.

DOE has undertaken a number of initiatives to improve project management since the 1999 NRC report, Improving Project Management in the Department of Energy (the Phase II report), was published. In 1999, DOE established OECM and the PMSOs in three program secretarial offices (PSOs): the Office of Science, the Office of Defense Programs, and the Office of Environmental Management. The release of DOE Order O413.3, “Program and Project Management for the Acquisition of Capital Assets,” and the DOE-wide Program and Project Management 2000 Workshop, both in October 2000, were also notable steps in the right direction and indicative of greater interest and involvement on the part of the deputy secretary and the chief financial officer (CFO) in project management.

As stated in the Phase II report, effective and accountable project management should be a continuing priority for DOE and its leaders at all levels. Through actions taken to date, DOE has begun to address some of the core issues. However, a number of issues have not been resolved. The most important unresolved issues are: (1) the definition of the authority and scope of OECM; (2) the provision of adequate financial and staff resources to improve project management; (3) the development and implementation of contract performance-measurement systems; (4) the design and implementation of an information-management system that can track contracts and contractor performance and feed information back into key decisions; and (5) continued emphasis on close cooperation and trust within DOE and with its contractors that will be fundamental to the long-term effectiveness of project-management reforms.

Although the committee considers the organizational changes made so far as generally positive, they are only beginnings. In the 18 months since the Phase II report was published, DOE could not possibly have implemented all of the necessary project-management reforms or achieved a high level of excellence. Much more time and attention will be necessary to achieve the goals set out in the Phase II report, and the committee recognizes that, until reforms have taken effect throughout the organization, project-management failures can be anticipated. As stated in the Phase II report, there is no “quick fix” for DOE’s problems. Improving project management in DOE will require changes in organizational structures, documents, policies, and procedures, as well as substantial changes in the culture of the department. In order to be effective, these changes must be embraced at all levels of the organization, especially in field and project offices.

Based on information provided by DOE, the committee believes that OECM and the PMSOs do not have adequate resources to perform their many functions

effectively, particularly in light of the high costs, complexity, and urgency of DOE projects and the great need for improved project management. To ensure that the necessary changes and improvements are made, the committee strongly recommends that the authority of OECM and the PMSOs be strengthened and that the resources and personnel available to them be increased to support their responsibilities. By strengthening the roles of OECM and the PMSOs, DOE can establish a strong in-house center of excellence that will ensure the implementation of improved project-management procedures.

To strengthen and affirm DOE’s commitment to reforming its project management, the committee reiterates the recommendation in the Phase II report that OECM be the unifying organization for project management throughout the department. OECM should be at the level of assistant secretary and report directly to the deputy secretary of energy. This would promote consistency and commitment throughout the department and encourage a culture of excellence in project management.

Subsequent to the publication of the Phase II report, DOE established OECM within the office of the department’s CFO and the PMSOs in the three major PSOs. Their responsibilities were defined in DOE Order O413.3. The committee believes that these positive steps can lead to significant improvements in project performance. Nevertheless, this organizational structure differs significantly from the one recommended in the Phase II report, which endorsed a strong, central, project-management office reporting directly to the deputy secretary. Although the PMSOs are positive additions to the new project-management structure, the committee believes that OECM would have a greater positive impact if it were elevated to the level of assistant secretary and reported directly to the deputy secretary; this would establish a peer relationship among OECM and the PSOs while maintaining consistent professional leadership for the office. The committee also believes that DOE project management and OECM would be more effective if the following OECM responsibilities were included in Order O413.3:

• Specify project-reporting requirements.

• Define and implement a DOE project-management information reporting system.

• Review all projects and validate that they are in compliance with the DOE project policies and procedures, and initiate actions to correct noncompliant practices.

• Review and validate proposed variations in project-management procedures to ensure continued compliance with the established objectives.

• Initiate and maintain a database of project-management experiences for the department.

• Advise the deputy secretary of all matters related to projects and project management.

Competent project-management professionals are essential to successful projects. The committee affirms the recommendation in the Phase II report that the department institute an effective career-development program to facilitate the recruitment, development, and retention of competent, professional project managers. OECM has developed a plan to create a department-wide career-development program and has received funds to carry out the planning phase. The committee does not have sufficient information to assess progress in training and professional development; however, an effective, widely implemented career-development program will require sufficient resources and support for full implementation.

The committee recommends that OECM ensure that the career-development program provides DOE personnel with access to a variety of learning resources and training methods and that the curriculum addresses competency in team building, DOE policies, and general project-management tools and techniques. DOE should foster a climate of learning and cultural change by supporting project-management personnel in obtaining professional certification and participating in professional activities. DOE should also encourage its contractors to support similar career-development efforts.

The implementation of an effective, department-wide, career-development program will be critical to improving DOE’s project management. Therefore, the committee will continue to monitor the department’s efforts in this area closely.

The Phase II report recommended that DOE develop more effective project-management policies, procedures, models, tools, techniques, and standards; train staff in their use; and require their application for all DOE projects. The report also recommended that DOE develop a project-management system that includes a requirement for a standard project-management plan, including a statement of the project organization covering all participating parties and a description of the specific roles and responsibilities of each party.

To date, the efforts of the deputy secretary, OECM, the PSOs, and the PMSOs have unquestionably raised awareness of the importance of good project management. Briefings by representatives of the PMSOs on project- management procedures they have established reflect good coordination with OECM. If these activities are continued and extended, they could become the foundation of a coherent project-management approach for the entire department.

DOE Order O413.3 and drafts of Program and Project Management Manual (PPMM) and Program and Project Management Practices (PPMP) are evidence that a start has been made on improving project-management policies, procedures, models, tools, techniques, and standards. However, much remains to be done. Although the committee has not yet completed a comprehensive review of the PPMM and PPMP, a few general observations can be made at this time. (A more detailed review and assessment may be included in a future report.)

The PPMM and the PPMP are greatly improved over the previous DOE guidance documents, and the committee congratulates OECM and other contributors on their efforts. OECM has stated that they intend to revise and issue the documents as directives within the next year. If the documents are revised appropriately they could form a central framework for DOE’s project-management capability, as recommended in the Phase II report. However, these documents should focus on defining how DOE does business, as opposed to general project-management methodologies, which should be incorporated by reference to texts and handbooks in the field. The sections on risk analysis and contingency in particular should be rewritten to reflect how DOE wants these procedures to be carried out and to promote a consistent approach throughout the department.

The effectiveness of policies, procedures, and models is determined by how consistently they are understood and supported by the individuals who carry them out. The committee found some indications that the PPMM and PPMP have been accepted throughout the organization. However, it is not clear who is responsible for verifying implementation of the policies and procedures. Project-management documentation should clearly define DOE’s systems and processes, and expectations of senior management for project performance, as well as organizational and individual incentives for managers at all levels to pursue effective, accountable project management. The documents should clearly identify a staff position responsible for verifying policy implementation and quality assurance. The committee believes that this oversight would be an appropriate function for OECM.

Neither the PPMM nor the PPMP defines the terms program and project as they are used by DOE. In fact, the terms program and project are used interchangeably. Although this may not create an immediate problem, the application of the policy documents may require that the difference between programs and projects be clearly understood. A project is usually a specific set of tasks, with a beginning, a middle, and an end. A project also has a well defined scope, cost, and schedule. Thus, a project is likely to be a controllable effort, the progress and performance of which can be assessed using standardized methods. A program is usually a group of projects. The complex scope and extended duration of a program can be made manageable by subdividing the whole into definable, understandable, controllable units or projects. A program, however, is more than the sum of its projects because each program must respond to the specific mission and integrate projects into a working whole. For example, the risks and

contingencies for a program are not simply the sum of the risks and contingencies for the projects.

The Phase II report recommended that DOE develop and implement a comprehensive project reporting system. The committee notes that the PMSOs are reporting some project data and that OECM has established a general target for reporting practices. However, current reporting requirements, tools, and practices are still incomplete and inconsistent among projects and programs. The lack of standard tools and procedures has prevented the aggregation of project data that could be used to evaluate project performance at the program and departmental levels. A consistent, reliable project-reporting system will be critical to achieving excellence in project management. Therefore, OECM should develop specific, precise requirements for integrated project and program reporting. In addition, OECM should provide training and support services to the PMSOs for reporting and collecting project data. Insufficient progress has been made in the development of an effective reporting system for the committee to offer a more detailed assessment, but project reporting should be given a high priority.

The Phase II report recommended that DOE develop and implement a comprehensive change-management system. To date, DOE has defined a target process for change management in DOE Order O413.3, and the committee is eagerly awaiting the implementation of the proposed process and looking forward to an opportunity to assess its effects on projects. An effective change-management system is critical in the prevailing DOE cost-plus environment, as well as for fixed-price and lump-sum projects. Change-management processes, including reviews by the Energy Systems Acquisition Advisory Board (ESAAB) and the change-control boards (CCBs), will be evaluated when sufficient data on experiences with change-control practices in actual projects have been collected.

The Phase II report recommended that the DOE use an earned value management system (EVMS)¹ to track project performance. EVMS has been identified as a primary project-management procedure in DOE Order O413.3. Some individual projects have already reported earned value data in their quarterly reports,

and DOE has awarded honors to three projects that have used EVMS. The committee acknowledges progress in this area and reiterates the importance of an earned value approach for project management. A consistent, earned value management approach would provide DOE project managers, program managers, and senior managers with an objective means of evaluating the status of projects, predicting future progress, and responding effectively to actual project conditions. Significantly more support from senior management including training, technical resources, and encouragement, will be necessary for EVMS to be implemented and used by DOE managers at all levels. The committee encourages DOE to use EVMS to predict project-performance outcomes and to manage projects proactively, as well as to report project status accurately.

The Phase II report recommended that DOE obtain ISO 9000² certification, but no preparations for ISO 9000 certification have been initiated to date. However, OECM indicates that certification will be sought when measurable improvements have been made to DOE project-management processes and structures. Although DOE has not yet begun the formal process of obtaining ISO certification, DOE Order O413.3, the PPMM, and PPMP are appropriate beginnings for an ISO 9000 process. The committee recognizes that ISO 9000 certification by itself will not improve DOE’s project management. However, the process involved in preparing for and seeking ISO 9000 certification will have direct benefits. For example, DOE will be required to purge the outdated and inconsistent policies, procedures, and regulations that have accumulated over the years and to focus on the essential elements of successful project management. The committee will continue to assess DOE’s implementation plan for ISO 9000 certification.

Although OECM has described value engineering (VE)³ as a desirable practice, according to the DOE Inspector General,⁴ it has not been widely or consistently used. DOE Order O413.3 lists OMB Circular A-131, “Value Engineering,” as a reference and states that DOE is committed to using VE. The Contractor Requirements Document, Attachment 1 to DOE O413.3, states that a VE process

must be used, and VE is mentioned in the draft PPMM and PPMP. Because DOE Order O413.3 was issued only recently, compliance cannot yet be assessed. However, the committee believes that DOE Order O413.3 does not define a VE process and does not define a process for verifying the effective use of VE.

Although DOE Order O413.3 is a positive step toward the implementation of VE, it can not, by itself, effectively make VE an integral part of DOE project management. DOE project managers have to be trained in the use and interpretation of VE, and a certified VE specialist should be appointed to oversee and promote its application.

The committee is encouraged by an agreement between Congress and DOE to establish project baselines after 20 to 30 percent of a project design has been completed and the creation of a funding mechanism for project planning, engineering, and design (PED). As noted in the Phase II report, adequate PED funding, preproject planning, and project controls are all critical to successful projects. The committee encourages DOE to continue implementing procedures to establish project baselines at an appropriate level of design completion and to implement other measures to improve the accuracy and reliability of cost and schedule estimates.

DOE has developed a fairly detailed project-planning process as part of its capital budget cycle, which should promote effective planning of projects. Other tools, such as checklists, communications software and methods, planning reviews, third-party audits, economic modeling, setting of measurable objectives, and team building, can also help. The committee believes that objective evaluations of new technology, and information-flow and work-flow design should be made during the project-planning phase.

OECM, in conjunction with the PMSOs, has begun to develop some of these project-planning initiatives. OECM has already documented some planning procedures and should revise and expand the descriptions in the PPMM and PPMP. These documents should also reference appropriate, up-to-date sources of project-planning methodologies. The PMSOs should provide supporting policies and procedures tailored to the specific projects and needs of their programs, as well as oversight to ensure quality; OECM should validate project plans prior to critical decision points. Procedures should be established to ensure that projects are not unnecessarily delayed by poor plans and that time pressures do not lead to projects being approved without adequate planning. All members of the project team should review project plans and provide written commitments and concurrence on the project scope, cost, and schedule.

The primary responsibility for the planning phase of project development lies with DOE personnel. Contractor assistance should be sought as needed. Even when a planning process is in place, it is the responsibility of DOE management

to ensure that every project is planned effectively. This monitoring could be accomplished through process audits, performance benchmarking, and direct observation and interaction with project teams. Project-team members should be held accountable for project planning and subsequent performance, and projects in trouble must be identified early—not in the late execution phase. Senior management can ensure that effective project planning is being conducted in the following ways:

• asking questions at project review meetings

• providing resources to support process training and implementation

• ensuring strategic flexibility (including cost and schedule contingencies)

• maintaining discipline in sticking to the plan

• benchmarking results

The committee recognizes that it will take time before consistent preproject planning can be integrated into project management throughout the organization. Preproject planning will require both procedural and cultural changes. However, DOE management should make it known that effective preproject planning will be required for all projects, without exception. Training or proof of proficiency in preproject planning should be required of all project-team members prior to the start of new projects.

Language in the Energy and Water Development Appropriations Bill, 2001, indicates that Congress relies heavily on external independent reviews (EIRs) for objective project evaluations. As a result, the number of EIRs has increased perceptibly in the past two years. Although EIRs are, overall, useful to DOE and to DOE projects, EIRs that provide only general information are of limited value. Some reviews have even provided inaccurate and misleading conclusions³, raising questions about the competence and independence of the reviewers. Some deficiencies can probably be attributed to inadequate definitions of the scope of the reviews and a lack of understanding of the fundamental goals of the review.

The committee believes that the EIR program is important but that it requires some modification. In view of the emphasis on EIRs by Congress, DOE should ensure that this program is effective. OECM has reported taking some steps to establish procedures, goals, and expected results for EIRs. However, the docu-

mented policies and procedures have not been reviewed or evaluated by the committee.

OECM should develop quality standards for EIRs and monitor projects to ensure that the reviews are conducted properly. All concerns raised during project reviews should be well documented and satisfactorily addressed. The committee reiterates the Phase II recommendation that OECM ensure that reviewers are truly independent and have no conflicts of interest. DOE should formally evaluate reviewers and use the evaluations as references in the selection of future independent reviewers.

Congressional requirements also mandate that all line-item projects be reviewed before any new money is spent. The congressional requirements go even further than the recommendations for internal reviews in the Phase II report. DOE had a history of conducting internal reviews even prior to the Phase II report. The most formalized and intensive internal review process has been developed by the Office of Science, which recently released a draft Independent Review Handbook documenting its approach. The committee has not reviewed this document in sufficient detail to evaluate it at this time.

Although Congress has promoted internal independent reviews, it is not clear to what extent current DOE procedures have addressed congressional concerns. Although internal reviews are not currently managed centrally, as was recommended in the Phase II report, OECM has been involved in a support role. The congressional emphasis on internal reviews and their potential for ensuring project success warrant the development of procedures and guidelines for all of the PSOs, not just the Office of Science. The specific missions of the program offices may require different internal review procedures, but the fundamental goals and objectives of internal reviews should be identical. In the absence of department-wide control of the internal review process, the PMSOs should formalize a coordinated process to facilitate central oversight and the transfer of lessons learned among programs. OECM should also evaluate the effectiveness and economic justification for internal reviews of small projects.

OECM has taken a number of actions that could improve DOE acquisition and contracting processes. DOE Order O 413.3 enumerates the steps to be followed in preconcept planning, risk analysis, and the overall acquisition process. Many sections of the draft PPMM also address these issues. Although documents can provide useful guidance, success will be determined by how well these procedures are followed and the willingness of all participants in the contracting process to develop appropriate contracting types, terms, and conditions.

DOE has stated that the major vehicle for improving acquisition and contracting is the integrated project team (IPT), which is made up of key staff from the contracting office, the program and project offices, and OECM. Although OECM does not have direct project responsibilities, it should be an active participant in the IPT, which should address the risks and uncertainties that have plagued previous DOE contracting efforts.

Because the IPT bears much of the front-end responsibility for contracting new projects, team members must be well versed in using performance-based contracting (PBC), which is designed to focus all participants on the critical issues, such as risk, uncertainty, and accountability. IPT members should be skilled in using PBC techniques with appropriate performance standards and incentives and an effective system for assessing contractor progress. OECM should designate a senior staff member to become an authority on PBC. The committee notes that OECM has not yet focused on this area and recommends that DOE develop a formal process for identifying the need for training in PBC, as well as staff to be trained. The objective should be to create an acquisition workforce of both program and contracting personnel capable of using innovative contracting methods. The committee will continue to assess DOE acquisition and contracting reforms and the application of these reforms to projects and programs including management-and-integration (M&I) and management-and-operations (M&O) contractors.

In general, the committee commends DOE for taking positive steps toward improving its project-management capabilities. Much remains to be done, however, and the committee will continue to review and assess DOE’s progress. The committee appreciates this opportunity to be of service to DOE and looks forward to assisting the department in its continuing efforts to improve project performance.

KENNETH F.REINSCHMIDT, chair, Stone and Webster, Inc. (retired), Littleton, Massachusetts

DON JEFFREY BOSTOCK, Lockheed Martin Energy Systems (retired), Oak Ridge, Tennessee

DONALD A.BRAND, Pacific Gas and Electric Company (retired), Novato, California

ALLAN V.BURMAN, Jefferson Solutions, Washington, D.C.

LLOYD A.DUSCHA, U.S. Army Corps of Engineers (retired), Reston, Virginia

G.BRIAN ESTES, Consulting Engineer, Williamsburg, Virginia

DAVID N.FORD, Texas A&M University, College Station, Texas

G.EDWARD GIBSON, University of Texas, Austin

PAUL H.GILBERT, Parsons Brinckerhoff International, Inc., Seattle, Washington

PATRICIA W.INGRAHAM, Syracuse University, Syracuse, New York

THEODORE C.KENNEDY, BE&K, Inc., Birmingham, Alabama

MICHAEL A.PRICE, Project Management Institute, Newtown Square, Pennsylvania