National Academies Press: OpenBook
« Previous: 1 Introduction
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

2
Progress and Opportunities

INTRODUCTION

The 1999 NRC report entitled Improving Project Management in the Department of Energy provided guidelines for “lifting DOE’s project management to a level commensurate with other agencies and private industry” (NRC, 1999, p. 3). This chapter provides the committee’s observations of DOE’s accomplishments in implementing the recommendations in the 1999 report and in subsequent assessment reports from 2001 through 2003 (NRC, 1999, 2001a, 2001b, 2002a, 2002b, 2003). The committee has observed significant progress over the past 3 years in the improvement of project management at DOE, but action on most of the committee’s past recommendations is still incomplete, and the results department-wide are inconsistent. Progress has been demonstrated in the areas of administrative organization, departmental policies and manuals, management directives, a career development program, and changes in attitude of many DOE personnel. The committee also notes that, despite these actions, the need for additional improvement continues, and that significant commitment and additional effort will be needed to ensure consistent implementation of the improved policies and procedures.

DOE projects are often very large and technically challenging. Because these projects are so important and so costly, the committee believes that DOE’s goal should be to improve its project management to a level at least commensurate with that in other agencies and private industry and that DOE should strive to achieve a level of excellence commensurate with its accomplishments in science and defense technology. In its previous annual reports (NRC, 2001b, 2003), the

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

committee identified specific aspects of project management at DOE that need improvement in order to bring the department’s project management procedures and project performance to an acceptable level of competence. The three previous NRC reports (NRC, 1999, 2001b, 2003) include 76 findings and 94 recommendations that the committee believes are still applicable. (See Appendix D for a compilation of findings and recommendations.) The previous findings and recommendations as well as those in this report address 10 recurring objectives for the changes needed to improve project management at DOE. The recurring objectives include the following:

  • Develop policies and procedures to define the DOE method of managing projects;

  • Create a project management culture across the agency that supports the consistent implementation of policies and procedures;

  • Provide leadership that ensures disciplined planning and execution of projects as well as support for continuous process improvement;

  • Provide a project management champion at the highest level of the department to ensure that a focus on the importance of project management is established and maintained;

  • Develop competence in fulfilling the owner’s role in strategic planning, front-end project planning, risk management, and project execution;

  • Apply rigorous project reporting and controls that include earned value systems; link day-to-day management data to periodic reporting and forecast time and cost to complete; and maintain historical data with which to benchmark project performance;

  • Document processes and performance to support benchmarking and trend analysis;

  • Invest in human capital by providing training and career development to ensure an adequate supply of qualified, skilled project directors and support staff;

  • Continue, refine, and document a program of external and internal project reviews; and

  • Employ innovative approaches to capital acquisition and the use of performance-based contracting.

This chapter provides the committee’s assessment of DOE’s progress in achieving these objectives and actions needed to continue progress toward an appropriate level of excellence in project management. Although the committee has provided suggested approaches, it believes that DOE senior managers have the responsibility to identify and apply metrics to define the appropriate level of excellence and to drive continuous process improvement.

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

ORGANIZATION

Organizational changes in DOE in the past 3 years have established a number of administrative offices specifically to manage projects more effectively and to improve project outcomes. These include the Office of Engineering and Construction Management (OECM) and the Office of Program Analysis and Evaluation (PA&E) in the Office of Management and Budget Evaluation (OMBE) and the project management support offices (PMSOs) within the major programs. The committee notes also that the leadership and involvement of senior DOE managers are key factors in the success of project management improvement efforts.

Office of Engineering and Construction Management

The OECM was established to implement project management reforms initiated in June 1999 (DOE, 1999). DOE’s Program and Project Management Policy for the Planning, Programming, and Acquisition of Capital Assets (P 413.1) established OECM in the Office of the Chief Financial Officer, which was later reorganized as the Office of Management and Budget Evaluation (DOE, 2000a).

The OECM mission is to provide the department with consistent, reliable project management processes, to drive improvements in project management systems, to integrate sound acquisition and business practices, to support the professional development of project managers, and to facilitate senior managers’ oversight of the department’s projects (DOE, 2003d). In the past 3 years, OECM developed Program and Project Management for the Acquisition of Capital Assets (Order O 413.3) (DOE, 2000b); Project Management for the Acquisition of Capital Assets (Manual M 413.3-1) (DOE, 2003a); Project Management Practices (DOE, 2003f); the Project Management Career Development Program (PMCDP) (DOE, 2003b); and the Project Assessment and Reporting System (PARS) (DOE, 2003d).

OECM manages and coordinates external independent reviews (EIRs) of projects, reviews acquisition plans, and serves as the secretariat for the Energy Systems Acquisition Advisory Boards (ESAABs). The committee believes that OECM is providing a vital quality-assurance function by providing DOE senior management with the information and advice essential to determine if and when a project is ready to proceed to the next step, and if projects are appropriately planned and executed. The committee also believes that OECM is at a level in the organization that it can be effective in improving project management if it is adequately staffed, senior management is sufficiently committed and involved, and senior management uses OECM and other resources to positively influence project management discipline across DOE.

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

Office of Program Analysis and Evaluation

The Office of Program Analysis and Evaluation, like OECM, reports to the OMBE director. PA&E was established in 2001 to provide independent analytical advice to DOE acquisition executives regarding the planning, execution, evaluation, and measurement of the effectiveness of DOE missions and programs by developing the department-wide strategic management system (DOE, 2003c). PA&E has helped to improve project management by evaluating proposed projects in order to assess their alignment with the department’s mission and strategic plan. The committee believes that PA&E can provide a vital function by facilitating senior management’s determination that the department is funding the right projects with the appropriate priority and level of funding.

Project Management Support Offices

The majority of DOE projects are accomplished by three program organizations—the Office of Environmental Management (EM), the National Nuclear Security Administration (NNSA), and the Office of Science (SC). Each of these organizations has established a project management support office that provides guidance for project management procedures, supports internal project reviews (IPRs) and internal reporting, and coordinates project manager training and certification, ESAAB critical decision processes, and senior management reporting and oversight with OECM. The PMSOs, though generally understaffed in the committee’s judgment, have played an important role in advocating and implementing project management improvement within their respective program offices.

Senior Management

The committee believes that PMSOs, PA&E, and OECM can be effective in improving the management and execution of DOE projects only if they are used and supported by senior managers. To ensure that these offices are effective, the deputy secretary, undersecretaries, and assistant secretaries should do the following:

  • Actively use PA&E to provide objective analysis and advice on mission need and project cost-benefit justification at the early stages of incipient projects;

  • Actively use OECM to develop and promulgate requirements for management decisions, monitor the progress of projects, and provide objective analysis and advice concerning project management through all project reviews and critical decision points; and

  • Strengthen these functions by providing adequate resources, staffing, and training and by empowering OECM, PA&E, the PMSOs, and DOE project directors and project support staffs.

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

Project Management Leadership

The committee has been impressed by examples of management leadership at the highest levels of DOE in implementing improvements to project management practices. The committee believes that senior management at DOE, especially the deputy secretary, needs to be highly visible in promoting project management excellence to ensure that it will succeed and that improvements will be permanent.

The committee believes that sustainable competence in project management at DOE can only come through the efforts of senior management with the authority to ensure agency-wide compliance with the project management procedures. In the 2001 assessment, the committee emphasized the need for intensive senior management involvement. Key actions by senior managers—for example, involvement in reviews, ensuring that adequate resources are provided, and ensuring process discipline—are critical to long-term improvement (NRC, 2001b). The committee has observed instances of effective leadership, but the leadership has been neither consistent enough nor rigorous enough to ensure the continued improvement of project management. For example, the committee notes that the direct intervention of the undersecretary and deputy secretary was necessary to achieve consensus on the project management manual so that it could be issued. However, it took almost 3 years for DOE to resolve the internal differences and issue this manual, indicating that DOE management does not have a clear strategy on how to manage projects. This example amounts to short-term firefighting, and a more sustained, visible commitment by senior managers is needed in order to continue improving project management and to make these improvements a permanent part of DOE.

Industrial organizations that have created excellent internal project control organizations have typically assigned the responsibility for project management programs to managers at the level of vice president or senior vice president. These senior executives, and even corporate CEOs, find it essential for success to maintain direct cognizance over projects—even those costing as little as $5 million. In DOE, the deputy secretary, as the department’s COO and senior acquisition executive, is responsible for effective project management. The deputy secretary should be DOE’s champion of project management excellence. As noted in previous NRC reports, the committee believes that DOE needs a visible, active, senior-level manager to promote and defend efforts to improve project management capabilities and their consistent application throughout the department (NRC, 1999).

The committee also recognizes that DOE program organizations are given independent authority and accountability for managing projects. The assistant secretaries for SC and EM have been delegated the authority by the undersecretary to act as the acquisition executives for projects under $400 million in their respective programs. The NNSA administrator, as DOE undersecretary, also has

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

this same authority. A memorandum from Undersecretary Robert Card to the assistant secretaries notes that they are also accountable for the quality of their respective programs’ project management systems (see Appendix E). The committee believes that just as the deputy secretary should be a champion for project management throughout the department, the program heads should be strong, visible champions for project management in their programs.

Although it is essential that each program office have strong project management capabilities and support, the committee continues to recommend that it is necessary that there be a single entity responsible for project management policies, procedures, personnel career development and training, and project tracking and reporting in order to achieve consistent project management excellence across the agency.

A number of DOE project directors expressed the belief that DOE upper management would not support them if they rejected contractors’ submittals or decisions, and they were thus reluctant to challenge contractors. Orders, manuals, and guidelines are necessary but not sufficient—project directors need to know that they have someone to whom they can turn for professional guidance and support when making difficult decisions. To have the necessary impact to affect the DOE culture, this function should report directly to the deputy secretary.1

POLICIES AND PROCEDURES

Considerable effort has been devoted to developing project management policies and procedures over the past 3 years. An order entitled Program and Project Management for the Acquisition of Capital Assets (Order O 413.3); the manual Project Management for the Acquisition of Capital Assets (M 413.3-1); a guide entitled Project Management Practices (PMP); and the notice Value Engineering (N 413.2) have been developed (DOE, 2000b, 2002, 2003a, 2003d).

Program and Project Management Policies for the Planning, Programming, and Acquisition of Capital Assets

DOE Policy P 413.1 (DOE, 2000a) set the stage for revitalizing project management within DOE. It charged OECM with the responsibility of preparing project management procedures for the department. DOE Order O 413.3 (DOE, 2000b), issued in October 2000, defines the principles of DOE project manage-

1  

Editor’s note: The 2003 assessment is based on information reviewed by the committee through September 2003. In December 2003 the DOE deputy secretary appointed an associate deputy secretary with responsibilities for capital acquisition and project management, a positive step of which readers should be aware as they consider the committee’s comments and recommendations regarding the need for a strong and visible champion of project management issues in DOE.

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

ment and project oversight. It includes requirements for Integrated Project Teams (IPTs) and defines a critical decision process to assure the involvement of senior management as responsible decision makers from the inception to the completion of projects. In addition, the oversight process is enhanced by the ESAABs, which advise the acquisition executives at critical decision points.

Following the issuance of Order O 413.3, OECM conducted workshops to obtain feedback and build support among DOE project managers. The order was reinforced by memoranda issued by Francis Blake, deputy secretary of DOE, in September and November 2001, and subsequently by Bruce Carnes, director of OMBE, in February 2002. The committee applauds both the order and the efforts to implement it throughout the department. The committee understands that the order may need to be updated from time to time, but believes that the basic principles, procedures, and applicability should be kept intact.

Project Management for the Acquisition of Capital Assets Manual

Considerable effort went into preparing a manual for the implementation of Order O 413.3. The first draft was issued in October 2000, and revised drafts were issued in February, June, and August 2002. Each iteration responded to critical review feedback from DOE project managers (project directors), contractors, this committee, and others. The final document’s format and organization present the roles, responsibilities, limits of authority, and required project controls and review procedures with respect to project management for the acquisition of capital assets. Efforts to issue the manual were threatened by internal and external resistance, but contentious issues were finally resolved through strong leadership and direct intervention by the undersecretary and the persistent efforts of OECM and the PMSOs. Manual M 413.3-1 was finally published in March 2003 (DOE, 2003a). Further, the deputy secretary issued a memorandum directing each Program Secretarial Office (PSO) to hold implementation sessions at field offices to demonstrate the commitment of line management to the project management system outlined by the manual. The committee was pleased to see this direct involvement of top DOE leadership in efforts to improve project management. This episode shows that attention by senior management will continue to be necessary to achieve the cultural change necessary to institutionalize project management best practices throughout the department.

The overall structure and integration of M 413.3-1 as published have been considerably improved in comparison with previous versions. The document does an admirable job of describing the key issues that need to be addressed for project initiation—that is, approval of mission need (critical decision 0 [CD-0]) and approval of system requirements and alternatives (CD-1), including determining the estimated cost range. However, the document provides little direction or guidance with regard to the critical tasks that must be performed in the period between CD-1 and the approval of the project baseline (CD-2). This

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

important front-end planning phase of a project is mentioned only on a few pages (that is, on pages 2-5, 2-6, 6-1, 6-2; at CD-2 review on pages 9-6 and 9-7; and again briefly in Chapter 10). In contrast, the CD-0 and CD-1 deliverables merit an entire chapter each (Chapters 4 and 5, respectively).

Since the activities between CD-1 and CD-2 typically are used to define the detailed scope of the project (as opposed to the conceptual scope) and encompass critical functions such as site evaluation, process flow design, design parameters, coordination issues, safety, instrument and electrical diagrams, equipment scope, execution plans, and so on, the committee believes this to be a significant omission from the manual. As noted in the 2001 forum on preproject planning, these functions are critical to the front-end planning process in successful project management organizations, and they can significantly affect project performance (NRC, 2002b).

Although contractors will typically perform these activities on DOE projects with oversight by a federal project director, it is imperative that the department outline specific guidance on the detailed deliverables needed so as to ensure that the activities are done properly. In addition, federal staff needs to have sufficient management and technical expertise to assess the quality of the contractor’s project management products.

The committee believes that the lack of explicit guidance for project management oversight between CD-1 and CD-2 exemplifies the absence of a clear understanding of the owner’s role in the project planning process. The key findings in the external independent reviews for the Center for Nanophase Materials Sciences, the Oak Ridge Research Support Center, and the Ashtabula Closure Project illustrate the problems that are still prevalent on many projects (see Appendix F). Both the manual and Project Management Practices need to be expanded to discuss the procedures and deliverables required at this phase, and federal project directors need to be involved in this effort to incorporate best industry practices.

Project Management Practices

Project Management Practices (PMP) consists of guidelines issued as references for project directors. The PMP elaborates on the information contained in Order O 413.3 and Manual M 413.3-1 by providing supplementary instructions that, although not required, are recommended in order to improve DOE’s ability to manage projects (DOE, 2003d). The PMP is distributed on CD-ROM and on the World Wide Web in order to permit frequent updating. The committee endorses issuance of the practices as a useful tool for present and future DOE project directors.

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

Value Engineering

Deficiencies in DOE’s value engineering (VE) efforts were noted in the Phase II report (NRC, 1999) and the 2001 letter report (NRC, 2001a), and VE procedures were cited as being in need of additional documentation in the 2001 assessment (NRC, 2001b). DOE has made a considerable effort to make known to project directors the federal requirements to perform value engineering, primarily through a Web page listing of federal statutes, Office of Management and Budget (OMB) circulars, and DOE policies and procedures; this Web page is linked to the OECM Web site. DOE issued VE program implementation policies and procedures, including a VE policy notice (Notice N 413.2), in December 2002 (DOE, 2002). The notice was followed by an implementing memorandum in January 2003 and a VE acquisition letter (No. 2003-04) in August 2003 (DOE, 2003e) that provides direction and guidance on the application of VE for management and operations (M&O) contracts and other contracts for the performance of work at DOE sites and facilities. VE requirements are also addressed in Chapters 2 and 5 of Manual M 413.3-1. Training for VE is included in the Project Management Career Development Program (PMCDP) as an elective. Work is under way to include guidance on VE in the PMP and to develop a Web-based tutorial.

Policies are now in place, but the committee cannot determine if they are being implemented, because performance measures are not yet available for assessing actual progress. The committee is aware of $16 million in VE savings in NNSA for 2002 but has no information on savings in other programs. It appears that DOE continues to lag behind the Department of Defense (DoD) and other federal agencies in implementing established, government-wide VE requirements. DOE appears to be taking action to implement a VE program, but additional attention and resources are needed to make VE an integral part of project management.

DOE’s VE program exemplifies the committee’s concerns with DOE: VE is not new; it is not mysterious, exotic, or difficult; it is routinely practiced by other federal agencies; it is mandated by law; and it has been required by O 413.3 since 2000. Yet DOE has not effectively implemented VE. The apparent lack of commitment to implementing VE reduces the committee’s confidence in DOE’s long-term commitments to implementing policies and procedures and improving project management.

Owner’s Role in Project Management

The committee has emphasized the need for the federal employees involved in project management functions to assume the role of the owner’s representative (NRC, 1999, 2001b, 2003). In its 2002 assessment the committee made the following recommendation:

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

In order for DOE to be an effective owner of capital acquisition projects it should:

  • Consider capital projects critical to organizational success.

  • Require senior management involvement in project decision making, usually at the $5 million and higher level.

  • Have a detailed and well-recognized internal front-end planning process.

  • Capture metrics on planning effort and project performance.

  • Require owner involvement and leadership in front-end planning.

  • Ensure that projects support DOE’s mission and are consistent with DOE’s strategic plan. (NRC, 2003, p. 31)

As noted in other sections of this report, DOE’s performance is inconsistent in activities that are critical to the role of an effective owner.

The change of the title “federal project manager” to “federal project director” may help differentiate the roles and responsibilities of contractor personnel and help reduce confusion and focus federal employees on their appropriate roles (DOE, 2003a). However, Manual M 413.3-1 contains an inadequate definition of the roles and responsibilities of DOE personnel and contractors. The definition of roles and responsibilities should follow the requirements of O 413.3. It should cover the DOE chain of responsibilities from the acquisition executive to the project director, and the contractor’s chain of responsibilities from executives to managers in the field. The definitions should define responsibilities to develop, review, comment, approve, and execute at each step of the DOE capital acquisition process. These definitions should do more to address the authority that DOE project directors have in carrying out these responsibilities. In addition, each of the PSOs needs to develop a detailed “roles and responsibilities” document to reflect the different procedures used by their separate organizations. The committee notes that the process of developing this detailed information for the manual will help DOE identify opportunities to streamline the process.

DOE project directors in the field are asking for better definition of their roles, responsibilities, and authority. In particular, they want to know if anyone in DOE will support them if they make a decision that a contractor does not like.

Applicability of Policies and Procedures

Order O 413.3 applies to all budget line-item projects over $5 million total project cost (TPC). The committee believes that, with provisions for tailoring requirements to the complexity of a project, this is an appropriate level. Because Congress wants DOE to control all of its projects, the general applicability of project management policies and procedures should remain at this level. Waivers of designated sites or projects from the requirements of O 413.3 (e.g., the Advanced Reactor Hydrogen Co-Generation Project [U.S. Senate, 2003]) are viewed by the committee as dangerous precedents that, if repeated, will undermine DOE’s ability to establish a department-wide culture that can manage projects well.

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

Some personnel in DOE and M&O contractors in the field expressed their concerns to the committee that the O 413.3 and M 413.3-1 are overly prescriptive and that the threshold value of projects that must comply with the order is set too low. These persons expressed the opinion that compliance with the order would result in project delays caused by the involvement of senior management at the critical decision points. The committee considered these positions thoroughly and believes they lack both basis and merit. The committee saw no actual evidence of projects that were delayed during the CD-0 through CD-2 reviews. On the contrary, the committee observed many projects that were commenced under O 413.3 and that proceeded expeditiously and on schedule.

The committee noted in its previous reports the absence of adequate project justification and front-end planning, manifested in the lack of documentation such as acquisition plans, risk management plans, and project execution plans. If inadequate front-end planning documents have been returned for more justification and correction in order to support management decisions, the committee does not regard such action as a delay but as a desirable outcome.

The committee reiterates its view that doing adequate planning up front is an essential activity and should be included in the project schedule. Projects may be spending more time and effort on planning than they did before O 413.3 because, as the committee has noted in previous assessments, thorough project planning was not being performed. If a project schedule allots insufficient time for adequate planning, it is a fault in the schedule, not evidence of delay. On the basis of best practices in industry and in other government agencies, the committee expects that the time taken for adequate planning to support management decisions at CD-0 and CD-1 can be made up at later CD points, which will go more smoothly if CD-0 and CD-1 are done correctly.

If projects are actually delayed because the project justification and planning documents were inadequate to support DOE management decisions, the solution is not to eliminate management decision points but to improve the planning and the documentation. Review of project justification is an essential management quality-control point. The summary of EIR findings (see Appendix F) indicates that quality-control planning documents continue to be necessary. The committee believes that the solution is to improve quality, not to eliminate the quality-control function. If delays are indeed due to a lack of resources for OMBE and the ESAABs to perform reviews expeditiously, then more resources should be added to those activities. It is an appropriate exercise of the senior management function to delay some projects if they are considered to be marginal or perhaps unnecessary or of low priority, and to send them back for further analysis, justification, and documentation of mission need. In the past, GAO noted DOE’s history of executing projects that turned out to have little value (GAO, 1998). DOE has made progress in the area of documenting and justifying projects and should not backslide. The objective of a good project management process is to execute the right projects well, not to build the wrong projects faster.

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

The critical management decision process is an integral part of project development in any successful organization; it should be part of DOE’s. Management reviews are not delays—they are critical steps in a project’s development path. The committee notes that the requirements for project justification are, in the end, whatever DOE senior management needs in order to make informed decisions. If DOE senior management wishes to avoid project mistakes, it will require adequate project justification, especially at CD-0 and CD-1. The real issue here is not procedures in O 413.3, but whether DOE senior management is able to control DOE projects and to assure that they are aligned with DOE’s mission needs.

The committee has observed project documents with a full page of management sign-offs and recognizes that there are opportunities to streamline the review process by reducing the number of internal sign-offs required for ESAAB reviews. The committee recommends that the PSOs reengineer the ESAAB process to eliminate requirements for sign-offs that are not absolutely essential and do not add value to the project.

HUMAN CAPITAL AND PROFESSIONAL DEVELOPMENT

Project Management Career Development Program

The committee endorses the concept and contents of the Project Management Career Development Program, which is now Attachment 4 of Acquisition Career Development Program (Order O 361.1) (DOE, 2003b). The committee commends the deputy secretary for his support of the program and recommends that the department maintain its commitment to implementing the plan, which calls for education, certification, and demonstration of proven project management abilities on successively more challenging projects. However, the committee finds that after 3 years of development, the PMCDP has not been widely implemented. The committee believes that the reasons for this slow rate of progress are the lack of emphasis and low priority assigned to this program and that this slowness may reflect an underlying culture that resists change within the department. Holding back funding for personnel and training and an unwillingness to expend resources (monetary and otherwise) to support the project management team is completely inappropriate for an organization that has an ongoing capital acquisition program in excess of $40 billion.

Project Management Professional Certification

Although professional project manager certification does not guarantee success in itself, the committee continues to emphasize it as an indicator of the importance that DOE places (or does not place) on professional education, development, and credentials for project directors. Firms in the private sector and other federal agencies such as the National Aeronautics and Space Administration

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

(NASA) and DoD routinely certify project managers; indeed, some organizations in DOE have done it, and there is no reason why DOE should not do it throughout the department. At least one DOE organization, NNSA’s Office of Project Management and Systems Support (NA-54), has achieved 100 percent PMP certification, and others have established this as a goal, sending the message that at least part of DOE management considers project management to be a profession with professional standards. The efforts expended to become certified will enhance a project director’s capabilities and provide a goal for aspiring project directors to achieve. PMP certification should be considered one means of developing project management competency within DOE. The deputy secretary’s commitment to full certification of DOE project directors in all PSOs is a significant step forward.

Training

Project management training at DOE continues to progress very slowly, with relatively few courses being offered to personnel in the field. Uncertain funding commitments have contributed to this slow start and have jeopardized full implementation of the PMCDP. The committee believes that central funding for course development, tuition, and travel expenses is essential to fostering a viable training program and that elimination of centralized funding would be tantamount to eliminating the program. The committee understands that the central working capital fund for project director training and career development has been established through FY 2005. The committee commends DOE management for recognizing the value of this essential mission-critical function.

The committee is encouraged by the fact that some field units, such as Los Alamos National Laboratory (LANL), have taken the initiative to provide needed career development training and just-in-time training for project teams, and to establish internships to develop the project directors who will be needed on current as well as future projects. These frontline personnel appreciate the immediate needs for training and professional project management certification, despite the long period of time required to develop a DOE-wide program.

The committee sees particularly urgent needs for department-wide training in front-end planning, risk analysis, and project controls—all aspects of project management that greatly need improved consistency. To make the most efficient use of training resources and to maximize staff access to training opportunities, DOE should utilize alternatives to traditional classroom delivery of instructional materials.

Career Development

The committee is concerned that the PMCDP focuses only on current project directors and does not identify career paths for personnel aspiring to become

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

project directors, nor does it address training and development for project management support staff. The committee believes that a comprehensive career development program is key to improving the project management culture in DOE. The best-performing organizations consistently train and promote from within (Collins, 1994). This point has been taken to heart by other federal agencies with large project responsibilities and budget—for example, NASA, which has an Academy for Program and Project Leadership, and DoD, which established the Defense Acquisition University to educate federal project management personnel.

Project Management Workshops

The committee has recognized the value of project management workshops sponsored by OECM and the PMSOs in past reports (NRC, 2001a, 2001b, 2002b, 2003). Unfortunately, funding decisions resulted in cancellation of these workshops. The committee believes that these workshops serve a valuable function by enhancing the visibility of project management capabilities and providing opportunities for sharing lessons learned across projects and across programs. DOE project management culture is significantly enhanced by assembling a large group of personnel across the department to witness senior management’s commitment to project management by the presentation of awards for outstanding project skills and project performance. Therefore, the committee reiterates the recommendation that these workshops should be continued and expanded, not terminated.

Authority and Responsibility

The committee heard from project directors in the field that they lacked sufficient authority to discharge the responsibilities assigned by current project management policies. The project directors believe that they need to be designated as DOE contracting officer’s representatives (CORs) in order to have the authority to accept or reject a contractor’s project management documents and project performance reports. The committee supports appointing project directors as CORs and believes that this authority will strengthen their abilities to function effectively as owner’s representatives. The committee does not advocate the development of detailed job descriptions defining the authority of project directors, although it is agreed that this authority is currently ambiguous. The committee believes that the single most effective way to strengthen the authority of the DOE project directors would be to provide them with a single mentor, supporter, and champion for project management across the complex.

Management Resources

The committee reiterates its finding and deep concern that there are too few project directors and support staff personnel to properly discharge DOE’s owner-

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

ship responsibilities, considering the size and complexity of the programs. The committee believes that it is imperative for DOE to develop an adequate human resource base. The inclusion of personnel assigned to project management activities within the program direction account hinders the rational application of federal personnel resources to varying workloads. However, the most serious consequence of funding project management in the program direction account is that, as a result, DOE project management has been chronically and seriously underfunded and understaffed. Therefore, the committee recommends that project directors and project management functions be funded from project appropriations. The use of project funds for project management personnel (as is done within the Department of Defense) could be very effective in improving project management in DOE.

Cross-Utilization

The 2002 assessment report (NRC, 2003) recommended that project managers (project directors) and project management staff be considered a departmentwide resource rather than being confined to program offices. The committee has detected no significant acceptance of this principle, and it again recommends that senior managers support this approach to using scarce resources more effectively across the complex.

RECOGNIZED VALUE OF CHANGE

The 1999 NRC report noted that the DOE culture was not conducive to effective project management and execution and that fundamental changes in the organization’s culture were needed to improve performance (NRC, 1999). The following strategies were listed as possible means for creating a project management culture that would be more conducive to project management excellence than the situation extant in 1999:

The cultural change “levers” available to the secretary are the recommendations throughout this report, including the following:

  • Create a culture of excellence in project management and execution.

  • Establish the goal of becoming a leader in project management skills, methodology, technology, systems, and performance.

  • Promulgate clear directions on project management policy, stressing that completion of projects to scope, on time, and on budget is of the highest priority.

  • Provide clear definitions of responsibility, authority, and accountability for all personnel involved in projects. Prohibit interference from outside the chain of responsibility. Clarify DOE field office and contractor roles, responsibilities, authorities, and relationships.

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
  • Enhance preconstruction planning, so that scope definition, baselines, budgets, contingencies, and schedules are realistic, and everyone involved understands what will be done, and when. After budgets are fixed, design and construct the project to meet the budget.

  • Engage user managers early and require that users be committed to project scope, requirements, budget, and schedule.

  • Ensure that user/client decisions are made in a timely manner to avoid project delays.

  • Provide objective, standard methods for assessing project risks and uncertainties, and assign realistic budgets, schedules, and contingencies.

  • Give the assigned project manager authority to control the project budget and schedule (including contingencies).

  • Institute contracting methods that select contractors who are committed to the goals of the project and the organization. Develop contract management procedures that hold contractors accountable for performance without creating a counterproductive adversarial atmosphere.

  • Institute rigorous identification and control of changes, especially changes in scope. Make it clear that scope, budget, and schedule are inextricably linked and prohibit changes in scope that cannot be accommodated in the assigned budget.

  • Provide consistent, uniform methods for tracking projects (e.g., earned value analysis) and disseminate this information so that all parties understand the status of every project with respect to its established scope, budget, and schedule.

  • Provide a uniform financial accounting system for all projects.

  • Train and qualify project managers in the classroom and on sites.

  • Provide visible, recognized career paths for professional project managers.

  • Assign increasing responsibilities to successful project managers.

  • Create a climate of learning and openness to outside ideas, criticism, and standards through external project reviews, ISO 9000 certification, and participation in professional project and construction management organizations.

  • Measure performance by results and provide positive incentives for the successful completion of projects on time and on budget.

  • Provide a highly visible core competency in project management, an agent for cultural change, a role model, and a champion for project managers by establishing and supporting an office of project management that reports directly to the secretary. (NRC, 1999, pp. 75-76)

DOE has taken steps to implement many of these strategies to various degrees. Some, such as project manager training, are just beginning to be implemented and need time to demonstrate their effectiveness while others, such as providing a highly visible project management champion, have gone nowhere.

Nonetheless, the committee has observed signs of positive change in the DOE project management culture over the past 3 years. The development of a consistent set of project management principles and requirements and the recognition of project management as a discipline are providing a foundation on which

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

DOE management can build cultural change. Evidence that the highest levels of management are engaged in process improvement has been crucial in beginning this change and is also crucial to the institutionalization of an effective project management system.

The committee met with many DOE personnel in the field and observed a good deal of energy focused on project management improvement. Although pockets of denial and resistance remain, the committee was impressed with the enthusiasm expressed at several field organizations. The committee’s visits to the Hanford, Savannah River, Sandia, and Los Alamos complexes over the past year provided some encouraging signs of progress in improving project management competency among the federal employees at those sites. Particularly noteworthy was the growth in size from 4 to 20 project direction staff, in part through the use of interns, and the increased competency of the project direction staff at the Los Alamos Site Office over the past 2 years.

Examples of DOE personnel seeking and using lessons learned from completed and ongoing projects were evident in project briefings to the committee. At Hanford, a subcontractor had been engaged to incorporate operability and maintainability lessons learned from vitrification plants at Savannah River and West Valley into the Hanford waste treatment facility. Another, more general sign of positive change is the level of preparedness and knowledge of project details exhibited by project personnel at Hanford, Savannah River, Sandia, and Los Alamos.

PROGRAM AND PROJECT PLANNING

Strategic Planning

In its 2002 assessment report, the committee stressed the urgent need for the development of a strategic planning process to integrate program and project planning (NRC, 2003). The committee has observed significant progress in the development of these processes.

NNSA has recognized the situation and has taken active steps to reengineer the NNSA ESAAB process to integrate its portfolio management system with construction project execution and project outcomes. NNSA is commended for the development of its Future Years’ Nuclear Security Plan (FYNSP), Ten Year Comprehensive Site Plans (TYCSPs), Integrated Construction Program Plan (ICPP), and 5-year budget process. The NNSA administrator has expressed his full support for these improvements in the project selection and execution process.

EM has begun its accelerated, 10-year closure planning process (a comprehensive new approach to project planning and management derived from the top-to-bottom review), with the full support and encouragement of the assistant secretary and undersecretary. The new closure plan could have a significant impact on the time and costs of site cleanup and closure. EM is to be commended

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

for taking this comprehensive step toward gaining control of its site remediation projects for the benefit of the public, the environment, and taxpayers.

SC has begun to integrate its infrastructure needs beyond just the current budget period. These planning tools should continue to be developed and matured, and their effective use by management should improve project planning and project outcomes.

Front-End Project Planning

The committee emphasized the need to perform effective front-end planning in both the 2001 and 2002 assessment reports (NRC, 2001b, 2003). The committee is encouraged by the improvements seen in front-end planning, especially the definition of project need at CD-0 and the identification of system requirements and alternatives at CD-1, on projects reviewed during site visits in the past year. Many mission need statements and acquisition planning documents, as well as early risk identification and management plans, show an improved emphasis on front-end planning.

The committee’s assessment of the state of DOE front-end planning must also take into consideration the inconsistency of quality and incompleteness of project planning documents that the committee reviewed in the past year. The inconsistency of front-end planning is illustrated by the Glass Storage Facility at the Savannah River Site (SRS), despite the fact that it was essentially a replication of a previous project, which should have simplified the planning effort. A broad range of front-end planning issues is also highlighted in key findings of recent EIR reports (see Appendix F). As noted in the 2002 assessment report, consistency in the approach to and implementation of front-end planning is critical to overall project performance at DOE. Inconsistency is a symptom of problems with many issues, as outlined elsewhere in this report, including personnel issues, lack of training, limited accountability, and an immature project management culture.

Project Budgets

As recommended in the 1999 NRC report, DOE has implemented procedures to avoid establishing project baselines before substantial engineering has been completed, in order to support the preparation of credible estimates. Prior to the availability of engineering data at CD-2, an estimated cost range is used to quantify expected project costs. However, DOE field personnel told the committee that internal pressure still exists to use the low end of the estimated cost range as the baseline cost. If this culturally driven practice continues, it will nullify the intent of the improved planning practices; prevent effective risk management; be inconsistent with Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs (OMB Circular A-94); and demonstrate the

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

persistence of bad practices, such as that referred to by GAO’s 1998 criticism of DOE’s habit of presenting rough order-of-magnitude numbers as point estimates (GAO, 1998). Using the cost estimate range until sufficient engineering data are available to support credible cost estimates is critical to improving project management and DOE’s credibility with Congress.

RISK MANAGEMENT

The committee has observed some improvement in DOE’s risk management efforts, from the minimal and weak planning and mitigation observed in 2000, to documented planning for project risks and more active mitigation of risks during project execution. Improved risk management plans were reviewed by the committee at Hanford, Savannah River, Sandia, and Los Alamos. Risk mitigation at Savannah River has included risk monitoring and ongoing attention and documentation with tools such as risk registries. The Plutonium Packaging and Stabilization Project at Savannah River has integrated risk issues as a part of the project schedule. The committee continues to stress that improvements in risk management practices are essential to improving DOE project management (NRC, 2001b). This limited sample indicates a potentially significant change and demonstrates that good risk management practices do exist within DOE and could become widespread across the department if they were supported.

Although the committee found examples of improved risk management, the EIR summaries (see Appendix F) indicate that 8 of the 19 projects reviewed had key findings related to deficiencies in risk management or mitigation. Reviewing these and other projects during the past year, the committee believes that risk identification is not done consistently; active risk mitigation plans are not being developed and applied on many projects. The knowledge, skills, and abilities needed to perform and oversee risk management are inadequate, isolated, and not readily transferred to projects needing them because of the separate PSO structures and the reluctance to transfer human resources across the PSO boundaries. Risk management is an integral requirement of Order O 413.3 but has not been adequately addressed in Manual M 413.3-1 and in Project Management Principles.

DOE’s projects are often first-of-a-kind projects that require extraordinary effort and the use of best practices and skills to manage risks. The committee believes that no project should pass CD-1 or CD-2 without an effective risk mitigation plan, especially the complex first-of-a-kind projects. A project with no risk management plan or an incomplete plan is not in control. A consistent approach to risk identification, assessment, and mitigation would be a first step toward making risk management more useful and usable across the department. The committee believes that DOE should develop such guidance and require its implementation for all projects.

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

PROJECT CONTROLS

Project controls include earned value management systems (EVMSs); the Project Assessment and Reporting System (PARS); other reporting, change control, and change tracking systems; and other established management procedures. The ESAAB critical decision reviews and decision process are also considered to be project controls. The committee observed that most federal project directors now freely acknowledge the value of procedures such as the critical decision reviews and the related project justification and control activities. The committee believes that compliance with the requirements of Order O 413.3 corresponds to increased efforts to apply project controls for improved project performance. The committee also believes that DOE has made progress in some areas of project controls, but many others need additional improvement and more consistent application to be effective.

Project Reporting

The Project Assessment and Reporting System is a Web-based distributed database for collecting and analyzing current project earned value data. The PARS manual notes that “the purpose of the DOE project reporting system is to deliver project status and assessment information to DOE senior managers and key program stakeholders” (DOE, 2003g, p. 1). The committee notes that PARS potentially could become a valuable tool for project monitoring, reporting, and oversight, although substantial improvements are needed to make it effective. PARS is in its second generation, but it has not yet been adequately refined to accomplish its objective. Further refinement of definitions of data-entry fields is needed to ensure the collection of consistent data. This problem is particularly troubling because typical PARS earned value data are 3 or more months out of date and PARS does not display the future project plan for budgeted cost of work scheduled (BCWS). Data-entry requirements should include planned budgets and schedules (e.g., BCWS) through the completion of the project, as well as historical performance data.

OECM has issued an updated users manual for PARS, but it gives no guidance on how to use or interpret the data. More follow-up is needed to determine who is using PARS, why they use it, and how it might be improved to increase its value to more users. The committee believes that a robust project database would be of considerable value as a project management tool, and it encourages DOE to continue to improve PARS and to extend its capabilities.

The use of PARS for any project management function is currently limited by its deficiencies. It cannot be used to assess the performance of projects in real time because of the lateness of data reporting (generally at least three months behind and often much more), nor can it be used to assess historical performance because an historical record of baseline changes is not provided. Owing to the

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

failure to show baseline plans extended into the future (i.e., BCWS) and the lack of controls over changes to the baseline, evaluative indices of progress (cost performance index [CPI] and schedule performance index [SPI]) are so consistent that they are unbelievable to anyone with project management experience. Because of the incredible number of projects that are shown to be not approximately but exactly on schedule and on budget (i.e., 0.99 ≤ SPI ≤ 1.01 and 0.99 ≤ CPI ≤ 1.01), and considering other anomalies, the committee concludes that no confidence should be placed in PARS data and that PARS cannot be used to demonstrate whether or not DOE project management is effective or has even been improved. However, the committee believes that an effective project management culture would demand an effective project reporting system.

Earned Value Management System

The committee recommended in previous reports that DOE apply a rigorous earned value management system and that it use this system to analyze and improve project performance (NRC, 1999, 2001b). DOE Order O 413.3 requires the implementation of EVMS on all projects over $20 million TPC (DOE, 2000b). The committee saw evidence in project management awards documents and in briefings to the committee that EVMS is being or will be used on projects below $20 million TPC. The application of EVMS is necessary for all projects that are monitored through the PARS database. From an examination of PARS data, the committee is concerned that the quality of EVMS department-wide is inconsistent.

OECM has initiated activities to provide EVMS training to DOE project directors and has proposed contracting with the Defense Contract Management Agency to verify contractor application of EVMS standards specified in O 413.3. EVMS is a critical part of project controls, and at this time there is no way of knowing if the reported data are accurate and reliable. The lack of evidence of consistent work breakdown structures, resource-loaded schedules, and baseline performance plans for cost and schedule causes the committee to doubt that EVMS has been effectively implemented.

PERFORMANCE MEASURES

Assessment Metrics

DOE does not have a uniform set of objective measures for assessing the quality of project management. The lack of objective measures or even reliable historic project data makes it difficult to assess progress in improving project management. It also makes it difficult to build confidence within GAO, Congress, OMB, and the public in the department’s ability to manage the money it spends on its projects. Evidence continues to be anecdotal rather than objective, quantitative, and verifiable. The absence of objective performance measures

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

prevents the identification of best practices and impedes widespread improvement in project management throughout the agency.

DOE Undersecretary Robert Card delegated acquisition executive authority to the assistant secretary for environmental management and the director of the Office of Science with the provision that their respective project management systems must be approved within a year (see Appendix E). However, the committee is not aware of any established DOE criteria for evaluating project management success, performance, or maturity. The Project Management Institute is developing its Organizational Project Management Maturity Model (OPM3) to assess the capabilities of an organization’s procedures and personnel, but the model will not be ready until the beginning of 2004 at the earliest and would require some testing and validation before it could be used to evaluate DOE programs.

The committee has identified several lines of inquiry but does not have sufficient data to begin to discuss performance criteria. These lines of inquiry include the following:

  • The percentage of DOE project managers who are certified professional project managers;

  • The numbers of PMCDP courses offered and of personnel trained;

  • The number of project directors and support staff for the value of projects managed;

  • The functional quality of Integrated Project Teams;

  • The quality of project planning documents (acquisition plans, project execution plans, risk management plans, and so on) submitted for ESAAB review;

  • Management participation in ESAABs related to acquisition executive decisions;

  • Effective use of performance-based contracts;

  • Effective use of value engineering;

  • Effective use of project controls;

  • Trends in findings and comparison of independent project reviews and external independent reviews;

  • Ratings by the EM Project Definition Rating Index (EM PDRI); and

  • Comparison of actual project performance to original baselines for budget, schedule, and scope.

The committee believes that the approval of the program offices’ project management systems should be based on a valid, comprehensive assessment. If such an assessment is not available, the committee recommends that the delegation of acquisition executive authority to the PSOs be revoked or continue to be provisional until additional direction on performance metrics and a revised assessment schedule can be provided and assessed.

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

The committee noted above that current project performance data available in PARS are not useful for assessing DOE project management because of problems with inconsistent data and the lack of historical trend data. Despite the recommendation in the 1999 NRC report (NRC, 1999) that DOE should develop a reliable database of all of its projects over time—past, present, and forth-coming—in order to be able to assess progress in improving project management, there is none 4 years later, and the committee has seen little evidence of progress in this regard.

The Construction Industry Institute’s (CII’s) Project Definition Rating Index was described in the 2001 assessment report as one example of an approach to assessing the maturity of front-end planning (NRC, 2001b). EM’s adaptation of the PDRI continues to be used as a project review tool, but the committee has seen no evidence that the PDRI or any other approach has been used to calibrate DOE’s front-end planning procedures (DOE, 2001a).

In the absence of a dedicated retrospective database of past DOE projects for comparison purposes, the capture of prospective data through PARS might be a way to eventually build a project database. To make this database useful for project analysis, PARS should archive the original cost and schedule baselines and all modifications to these baselines—constant rebaselining of projects precludes the ability to make valid comparisons over time and across projects. Although current revised baselines should be used to make day-to-day project management decisions, the original approved baselines and all subsequent changes should be tracked in order to evaluate overall project management performance.

By maintaining all baseline data, project management performance can be objectively assessed by comparing the actual total project costs with original budget estimates, actual completion dates with original deadlines, and delivered scope and quality with original project specifications. The variance between the original baselines and the cost, schedule, and scope at completion is one indicator of performance. Process improvement can be assessed by analysis of the trend of variances for projects initiated over a period of time. Unfortunately, this approach will require years before it produces usable results, but it is essential to long-term performance, and it should start now.

Benchmarking

Benchmarking performance and management processes throughout a project’s life cycle and from different perspectives can provide a basis for a measure of improvement of project management procedures. Both internal and external benchmarking perspectives are useful and should be a regular part of DOE benchmarking procedures. Internal benchmarking can compare projects across programs and across time. External benchmarking can compare DOE projects with those of similar complexity, size, and other characteristics in other federal agencies and in industry.

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

The committee found little evidence of external benchmarking (other than attempts by NA-54 to use CII benchmarking data), nor did it find evidence of internal benchmarking. The committee recognizes the inherent difficulty in benchmarking first-of-a-kind and one-of-a-kind projects but reiterates its belief that a consistent, continuously applied benchmarking program would be an effective tool for improving project management. The committee noted in its 2001 assessment that the PARS database should provide accurate, up-to-date information for use in benchmarking (NRC, 2001b), but there is no evidence that PARS data are used for this purpose.

PROJECT REVIEWS

Prior to the 1998 NRC report, independent reviews in DOE were largely confined to those conducted by the Office of Science and mainly addressed the technical aspects of projects (NRC, 1998). Although these reviews neglected some aspects fundamental to sound project management, their contribution to executing technologically complex, one-of-a-kind projects is not to be minimized. OECM has issued a draft Independent Review Procedures, which outlines the departmental process and requirements (DOE, 2001b). The three major program offices (NNSA, EM, and SC) have each institutionalized and formalized their respective procedures in documents that cover the details and peculiarities pertinent to their programs (DOE, 2000c, 2000d, 2001a). These documents were reviewed in the 2001 assessment; the committee is not aware of any revisions since that time.

The procedures for the congressionally mandated EIRs have matured over time and the reviews are more comprehensive and have improved in quality. Consequently, there is increased appreciation within DOE of the value of EIRs, and DOE project directors interviewed by the committee now acknowledge that EIRs have provided useful information and have added considerable value to the project delivery process. Also, there has been more concentration on conducting rigorous reviews in the early planning stages. Because project cost and schedule are particularly difficult to evaluate in the early stages of a project, it is essential that well-qualified individuals or contractors perform these reviews. The committee has also observed value in DOE internal reviews, particularly those employing nonadvocate participants from across organizational lines.

Criteria for Reviews

Although many DOE managers recognize the value of EIRs, the committee has heard the opinion voiced by some DOE managers and M&O contractors that external independent reviews provide no added value on projects costing $5 million to $20 million TPC. One PSO manager advocates that EIRs should not be required for projects costing less than $100 million TPC. The committee does

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

not share this opinion. Review of the key EIR findings (see Appendix F) indicates persistent problems in planning and other areas across all Program Secretarial Offices for projects with less than $100 million TPC. On this basis, the committee concludes that there is no justification for reducing EIR requirements, at least until there is demonstrated improvement in project outcomes.

The committee continues to recommend that reviews of all projects should be conducted in some form. The scope and content of the review should be tailored to the complexities and peculiarities of the project, especially for those projects between $5 million and $20 million. By tailoring reviews to the complexity of projects, the expenditure of effort can be made compatible with the value added. Procedures already exist for waiving independent reviews for routine projects when this is justified, and there are procedures for tailoring reviews (DOE, 2001b).

Independent reviews of plans, assumptions, designs, estimates, and schedules are the accepted standard in industry. In fact, independent reviews are one of the means by which industrial firms become successful at project management and continue to stay that way (NRC, 2002b). The view that DOE should not have to do the things typically done by the most successful industrial owner firms does not withstand scrutiny. The committee strongly advocates that DOE continue to recognize the value of EIRs for improving project performance and expand their application for documenting lessons learned.

Capability of Independent Reviewers

DOE has awarded three Management, Organization and Business Improvements (MOBIs) contracts for the performance of EIRs department-wide. The committee recognizes the benefits of such contracts for this type of work but believes that DOE needs to ensure that the contractors assign qualified personnel to each task. It is incumbent on DOE to exercise the necessary oversight to ensure the quality of reviews and to ensure that reviewers have the necessary qualifications and experience to add value to a project.

Independent Cost Estimates

The committee reviewed independent cost estimates (ICEs) for a variety of projects. It found them to be essentially arithmetical checks of the extant estimate rather than critical analyses of the work breakdown structures, cost elements, risk assessments, and other factors that affect the accuracy and credibility of the estimates. The committee learned that the scope and definition of the cost estimate reviews have been changed to an independent cost review (ICR), which includes the needed critical analysis. If properly structured and performed, the ICRs may provide the rigorous analysis that the committee believes is lacking in the ICEs. However, there are no data for evaluating ICRs at this time.

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

ACQUISITION AND CONTRACTING

In each of its previous reports, the committee stressed the linkage between effective acquisition and contracting, and successful project management (NRC, 2001a, 2001b, 2002a, 2003). These reports identified and encouraged the use of a variety of acquisition best practices, many of which have been described in the new manual, M 413.3-1 (DOE, 2003a).

For example, the manual requires that IPTs develop sound acquisition strategies to assess both risks and potential alternative contracting approaches. The OECM staff then assesses the completeness of the acquisition execution plans (AEPs) before forwarding them for senior management approval as part of the ESAABs and the critical decision process. This process is an essential means for senior managers to assess progress at critical steps.

The committee was pleased to see an example of the effective use of the acquisition process, when NNSA recently sought to develop a quick-response approach for assisting the Russian Federation in closing down its three remaining plutonium reactors and replacing them with fossil fuel plants. NNSA worked effectively to develop a reasonable comprehensive strategy and risk mitigation plan, consistent with OECM guidance. Moreover, the deputy secretary showed his commitment to the process, maintaining continuing control through at least the CD-1 milestone.

Other acquisition-related improvements include the following:

  • EM’s development of a new contracting strategy relying on incentives for site closure. EM is now applying that strategy to the Rocky Flats site.

  • Increased use of performance objectives in determining fees for site contractors. DOE increased the proportion of contractor fees tied to performance objectives from 34 percent in FY 1996 to 70 percent in FY 2001 (GAO, 2003).

  • Increased use of competition among contractors to achieve best value for the department. Of 16 Federally Funded Research and Development Center (FFRDC) contracts, 6 have been competed, and other competitions are planned. Moreover, the percentage of major site contracts awarded competitively increased from 38 percent in FY 1996 to 56 percent in FY 2001 (GAO, 2003).

  • The effective use of alternative contracting techniques to meet DOE needs. An example is Los Alamos National Laboratory’s use of a design-build approach for one of its major construction projects.

These examples demonstrate good front-end planning, senior management involvement, the effective use of performance metrics and incentives, and flexibility in contracting approaches. The committee strongly supports continuing the emphasis on all of these techniques.

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

In its previous reports, the committee stressed the benefits of systematic training in the use of performance-based contracting methods and encouraged the department to collect data on the effectiveness of these techniques. It reiterates the need to follow through on these recommendations.

RECOMMENDATIONS

Conclusion

DOE has made significant progress in improving project management through organizational changes and the development of policies and procedures. These changes were completed in 2003 with the release of the manual Project Management for the Acquisition of Capital Assets (Manual M 413.3-1) and have just begun to be implemented. The committee believes that current policies and procedures need to be fully and consistently implemented, and that opportunities exist for additional changes that are needed in order to maintain the progress achieved to date and to continue the improvement process so as to bring DOE’s project management capabilities to an appropriate level of excellence. The committee believes that action on most of its past recommendations is still incomplete and that the results department-wide are inconsistent. The opportunities for continued improvement in the application of accepted project management practices are presented above in this chapter and are summarized below as committee recommendations.

Recommendations

  • DOE senior managers should actively use the Office of Program Analysis and Evaluation to provide objective analysis and advice on mission need and project cost-benefit justification at the early stages of incipient projects and should use the Office of Engineering and Construction Management to standardize requirements for management decisions, monitor the progress of projects, and provide objective analysis and advice. These offices, as well as the project management support offices and the DOE site offices, should be provided adequate resources, staffing, training, and moral support.

  • The DOE deputy secretary either should be the champion for project management improvement—to develop project management into a core competency of the department, to assure that the department maintains an adequate staff of qualified project directors to manage its portfolio of projects, and to assure that the disciplined execution of projects is a priority for managers at all levels—or should appoint someone to perform this role, reporting to the deputy secretary (see footnote on page 16).

  • Just as the deputy secretary should be a champion for project management

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

throughout the department, the program heads should be strong, visible champions for improved project management in their programs (see footnote on page16).

  • Current policies and procedures should be kept intact and revised only as necessary to improve the planning and execution of projects. The applicability of project management policies and procedures should remain at current levels. Requirements should be tailored to the size and complexity of projects, but exemptions for projects or sites should not be considered until such decisions can be supported by a record of excellence in project management and project performance is established.

  • Manual M 413.3-1 should define the roles and responsibilities required by Order O 413.3. It should cover the DOE chain of responsibilities from the acquisition executive to the project director, and the contractor chain of responsibilities from executives to managers in the field. The definitions should describe responsibilities to develop, review, comment, approve, and execute at each step of the DOE capital acquisition process. In addition, each of the Program Secretarial Offices needs to develop a detailed “roles and responsibilities” document to reflect the different procedures used by their separate organizations.

  • If the application of policies and procedures results in delays because resources to perform reviews expeditiously are lacking, then more resources should be provided. Steps should also be taken to streamline the critical decision review process by eliminating unnecessary sign-offs.

  • DOE should ensure that funding and implementation of the Project Management Career Development Program are priorities. The committee sees particularly urgent needs for department-wide training in front-end planning, risk analysis, and project controls. Training for project directors and project support staff should be centrally funded and should utilize alternatives to traditional classroom delivery of instructional materials.

  • DOE should adopt a department-wide strategy to develop a sustainable, qualified workforce for directing the projects required to achieve the department’s missions. DOE should work with Congress to allow funding for project direction to be included in project budgets.

  • DOE should provide contracting officer representative authority to project directors.

  • DOE should continue and expand department-wide and program-specific project management workshops to recognize project management achievements, reinforce the professional identity of project directors, and share lessons learned.

  • Strategic planning initiatives should be continued, expanded, and used by management to improve project selection, planning, and execution.

  • DOE should develop detailed procedures and guidance for identifying risks, planning strategies to address risks, and managing risks throughout

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

the life cycle of projects, and should require their implementation for all projects. Projects should not pass CD-1 or CD-2 without an effective risk mitigation plan.

  • Manual M 413.3-1 and Project Management Practices should be expanded to include detailed procedures and deliverables required between CD-1 and CD-2. Federal project directors should be involved in this development effort so as to incorporate the best industry practices in DOE procedures.

  • Steps should be taken to ensure the quality and consistent application of project controls. Planned project scope, budgets, and schedules should be maintained through completion of a project. Although current revised baselines should be used to make day-to-day project management decisions, the original approved baselines and all subsequent changes should be retained in order to evaluate overall project management performance.

  • Steps should be taken to ensure that the Project Assessment and Reporting System (PARS) data are timely, accurate, and consistent from project to project. Analytical tools and presentations should be enhanced. PARS should archive the original cost and schedule baselines and all modifications to these baselines.

  • DOE should develop consistent performance metrics for evaluating project management maturity department-wide. Both internal and external benchmarking should be a regular part of procedures for all phases of projects.

  • DOE should continue to recognize the value of external independent reviews for improving project performance and should expand their application for documenting lessons learned.

  • DOE should exercise the necessary oversight to ensure the quality of reviews and to ensure that reviewers have the necessary qualifications and experience to add value to the project.

  • DOE should follow previous recommendations to provide systematic training in the use of performance-based contracting methods and should collect data on the effectiveness of these techniques.

REFERENCES

Collins, James C. 1994. Built to Last. New York, N.Y.: HarperCollins.


DOE (Department of Energy). 1999. Memorandum for All Departments from T.J. Glauthier, Deputy Secretary; Subject: Project Management Reform Initiative. June 25.

DOE. 2000a. Program and Project Management Policy for the Planning, Programming, and Acquisition of Capital Assets (Policy P 413.1). Washington, D.C.: Department of Energy.

DOE. 2000b. Program and Project Management for the Acquisition of Capital Assets (Order O 413.3). Washington, D.C.: Department of Energy.

DOE. 2000c. Office of Defense Programs Project Review Procedures. Washington, D.C.: Department of Energy.

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

DOE. 2000d. Office of Environmental Management Internal Independent Review Handbook. Washington, D.C.: Department of Energy.

DOE. 2001a. Office of Science Independent Review Handbook. Washington, D.C.: Department of Energy.

DOE. 2001b. Office of Engineering and Construction Management Draft Independent Review Procedures. Washington, D.C.: Department of Energy.

DOE. 2002. Value Engineering (Notice N 413.2). Washington, D.C.: Department of Energy.

DOE. 2003a. Project Management for the Acquisition of Capital Assets (Manual M 413.3-1). Washington, D.C.: Department of Energy.

DOE. 2003b. Acquisition Career Development Program, Attachment 4 (Order O 361.1). Washington, D.C.: Department of Energy.

DOE. 2003c. Office of Management and Budget Evaluation. Available online at http://www.mbe.doe.gov/crOrg/me20.htm. Accessed September 9, 2003.

DOE. 2003d. Office of Engineering and Construction Management. Available online at http://oecm.energy.gov/. Accessed September 9, 2003.

DOE. 2003e. Value Engineering (Acquisition Letter 2003-4). Washington, D.C.: Department of Energy.

DOE. 2003f. Project Management Practices. Available online at http://oecm.energy.gov/. Accessed September 9, 2003.

DOE. 2003g. Project Assessment and Reporting System User Manual, version 3.0. Washington, D.C.: Department of Energy.


GAO (General Accounting Office). 1998. Nuclear Waste: Department of Energy’s Hanford Tank Waste Project: Schedule, Cost, and Management Issues (GAO/RCED-9913). Washington, D.C.: General Accounting Office.

GAO. 2003. Department of Energy: Status of Contract and Project Management Reforms (GAO-03-570T). Washington, D.C.: General Accounting Office.


NRC (National Research Council). 1998. Assessing the Need for Independent Project Reviews in the Department of Energy. Washington, D.C.: National Academy Press.

NRC. 1999. Improving Project Management in the Department of Energy. Washington, D.C.: National Academy Press.

NRC. 2001a. Improved Project Management in the Department of Energy. Letter report, January. Washington, D.C.: National Academy Press.

NRC. 2001b. Progress in Improving Project Management at the Department of Energy, 2001 Assessment. Washington, D.C.: National Academy Press.

NRC. 2002a. Progress in Improving Project Management at the Department of Energy, 2002 Interim Assessment. Letter report, May. Washington, D.C.: National Academy Press.

NRC. 2002b. Proceedings of Government/Industry Forum: The Owner’s Role in Project Management and Preproject Planning. Washington, D.C.:National Academy Press.

NRC. 2003. Progress in Improving Project Management at the Department of Energy, 2002 Assessment. Washington, D.C.: The National Academies Press.


U.S. Senate. 2003. The Energy Policy Act of 2003 Report of the Committee on Energy and Natural Resources, United States Senate, to Accompany S. 1005 (Report 108-43). Washington, D.C.: U.S. Government Printing Office.

Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 11
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 12
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 13
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 14
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 15
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 16
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 17
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 18
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 19
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 20
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 21
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 22
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 23
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 24
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 25
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 26
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 27
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 28
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 29
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 30
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 31
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 32
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 33
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 34
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 35
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 36
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 37
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 38
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 39
Suggested Citation:"2 Progress and Opportunities." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 40
Next: 3 Prognosis for Progress »
Progress in Improving Project Management at the Department of Energy: 2003 Assessment Get This Book
×
Buy Paperback | $45.00 Buy Ebook | $35.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In 1997, Congress, in the conference report, H.R. 105-271, to the FY1998 Energy and Water Development Appropriation Bill, directed the NRC to carry out a series of assessments of project management at the Department of Energy (DOE). This report, the 2003 Assessment, is the final one in that series. It presents an examination of DOE's progress in improving program management over the past three years including the Department's response to the recommendations of the previous assessments in this series. In addition to assessing DOE’s progress, the report also describes opportunities for further improvement and gives a prognosis for future developments.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!