Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 11
Progress in Improving Project Management at the Department of Energy: 2003 Assessment 2 Progress and Opportunities INTRODUCTION The 1999 NRC report entitled Improving Project Management in the Department of Energy provided guidelines for “lifting DOE’s project management to a level commensurate with other agencies and private industry” (NRC, 1999, p. 3). This chapter provides the committee’s observations of DOE’s accomplishments in implementing the recommendations in the 1999 report and in subsequent assessment reports from 2001 through 2003 (NRC, 1999, 2001a, 2001b, 2002a, 2002b, 2003). The committee has observed significant progress over the past 3 years in the improvement of project management at DOE, but action on most of the committee’s past recommendations is still incomplete, and the results department-wide are inconsistent. Progress has been demonstrated in the areas of administrative organization, departmental policies and manuals, management directives, a career development program, and changes in attitude of many DOE personnel. The committee also notes that, despite these actions, the need for additional improvement continues, and that significant commitment and additional effort will be needed to ensure consistent implementation of the improved policies and procedures. DOE projects are often very large and technically challenging. Because these projects are so important and so costly, the committee believes that DOE’s goal should be to improve its project management to a level at least commensurate with that in other agencies and private industry and that DOE should strive to achieve a level of excellence commensurate with its accomplishments in science and defense technology. In its previous annual reports (NRC, 2001b, 2003), the
OCR for page 12
Progress in Improving Project Management at the Department of Energy: 2003 Assessment committee identified specific aspects of project management at DOE that need improvement in order to bring the department’s project management procedures and project performance to an acceptable level of competence. The three previous NRC reports (NRC, 1999, 2001b, 2003) include 76 findings and 94 recommendations that the committee believes are still applicable. (See Appendix D for a compilation of findings and recommendations.) The previous findings and recommendations as well as those in this report address 10 recurring objectives for the changes needed to improve project management at DOE. The recurring objectives include the following: Develop policies and procedures to define the DOE method of managing projects; Create a project management culture across the agency that supports the consistent implementation of policies and procedures; Provide leadership that ensures disciplined planning and execution of projects as well as support for continuous process improvement; Provide a project management champion at the highest level of the department to ensure that a focus on the importance of project management is established and maintained; Develop competence in fulfilling the owner’s role in strategic planning, front-end project planning, risk management, and project execution; Apply rigorous project reporting and controls that include earned value systems; link day-to-day management data to periodic reporting and forecast time and cost to complete; and maintain historical data with which to benchmark project performance; Document processes and performance to support benchmarking and trend analysis; Invest in human capital by providing training and career development to ensure an adequate supply of qualified, skilled project directors and support staff; Continue, refine, and document a program of external and internal project reviews; and Employ innovative approaches to capital acquisition and the use of performance-based contracting. This chapter provides the committee’s assessment of DOE’s progress in achieving these objectives and actions needed to continue progress toward an appropriate level of excellence in project management. Although the committee has provided suggested approaches, it believes that DOE senior managers have the responsibility to identify and apply metrics to define the appropriate level of excellence and to drive continuous process improvement.
OCR for page 13
Progress in Improving Project Management at the Department of Energy: 2003 Assessment ORGANIZATION Organizational changes in DOE in the past 3 years have established a number of administrative offices specifically to manage projects more effectively and to improve project outcomes. These include the Office of Engineering and Construction Management (OECM) and the Office of Program Analysis and Evaluation (PA&E) in the Office of Management and Budget Evaluation (OMBE) and the project management support offices (PMSOs) within the major programs. The committee notes also that the leadership and involvement of senior DOE managers are key factors in the success of project management improvement efforts. Office of Engineering and Construction Management The OECM was established to implement project management reforms initiated in June 1999 (DOE, 1999). DOE’s Program and Project Management Policy for the Planning, Programming, and Acquisition of Capital Assets (P 413.1) established OECM in the Office of the Chief Financial Officer, which was later reorganized as the Office of Management and Budget Evaluation (DOE, 2000a). The OECM mission is to provide the department with consistent, reliable project management processes, to drive improvements in project management systems, to integrate sound acquisition and business practices, to support the professional development of project managers, and to facilitate senior managers’ oversight of the department’s projects (DOE, 2003d). In the past 3 years, OECM developed Program and Project Management for the Acquisition of Capital Assets (Order O 413.3) (DOE, 2000b); Project Management for the Acquisition of Capital Assets (Manual M 413.3-1) (DOE, 2003a); Project Management Practices (DOE, 2003f); the Project Management Career Development Program (PMCDP) (DOE, 2003b); and the Project Assessment and Reporting System (PARS) (DOE, 2003d). OECM manages and coordinates external independent reviews (EIRs) of projects, reviews acquisition plans, and serves as the secretariat for the Energy Systems Acquisition Advisory Boards (ESAABs). The committee believes that OECM is providing a vital quality-assurance function by providing DOE senior management with the information and advice essential to determine if and when a project is ready to proceed to the next step, and if projects are appropriately planned and executed. The committee also believes that OECM is at a level in the organization that it can be effective in improving project management if it is adequately staffed, senior management is sufficiently committed and involved, and senior management uses OECM and other resources to positively influence project management discipline across DOE.
OCR for page 14
Progress in Improving Project Management at the Department of Energy: 2003 Assessment Office of Program Analysis and Evaluation The Office of Program Analysis and Evaluation, like OECM, reports to the OMBE director. PA&E was established in 2001 to provide independent analytical advice to DOE acquisition executives regarding the planning, execution, evaluation, and measurement of the effectiveness of DOE missions and programs by developing the department-wide strategic management system (DOE, 2003c). PA&E has helped to improve project management by evaluating proposed projects in order to assess their alignment with the department’s mission and strategic plan. The committee believes that PA&E can provide a vital function by facilitating senior management’s determination that the department is funding the right projects with the appropriate priority and level of funding. Project Management Support Offices The majority of DOE projects are accomplished by three program organizations—the Office of Environmental Management (EM), the National Nuclear Security Administration (NNSA), and the Office of Science (SC). Each of these organizations has established a project management support office that provides guidance for project management procedures, supports internal project reviews (IPRs) and internal reporting, and coordinates project manager training and certification, ESAAB critical decision processes, and senior management reporting and oversight with OECM. The PMSOs, though generally understaffed in the committee’s judgment, have played an important role in advocating and implementing project management improvement within their respective program offices. Senior Management The committee believes that PMSOs, PA&E, and OECM can be effective in improving the management and execution of DOE projects only if they are used and supported by senior managers. To ensure that these offices are effective, the deputy secretary, undersecretaries, and assistant secretaries should do the following: Actively use PA&E to provide objective analysis and advice on mission need and project cost-benefit justification at the early stages of incipient projects; Actively use OECM to develop and promulgate requirements for management decisions, monitor the progress of projects, and provide objective analysis and advice concerning project management through all project reviews and critical decision points; and Strengthen these functions by providing adequate resources, staffing, and training and by empowering OECM, PA&E, the PMSOs, and DOE project directors and project support staffs.
OCR for page 15
Progress in Improving Project Management at the Department of Energy: 2003 Assessment Project Management Leadership The committee has been impressed by examples of management leadership at the highest levels of DOE in implementing improvements to project management practices. The committee believes that senior management at DOE, especially the deputy secretary, needs to be highly visible in promoting project management excellence to ensure that it will succeed and that improvements will be permanent. The committee believes that sustainable competence in project management at DOE can only come through the efforts of senior management with the authority to ensure agency-wide compliance with the project management procedures. In the 2001 assessment, the committee emphasized the need for intensive senior management involvement. Key actions by senior managers—for example, involvement in reviews, ensuring that adequate resources are provided, and ensuring process discipline—are critical to long-term improvement (NRC, 2001b). The committee has observed instances of effective leadership, but the leadership has been neither consistent enough nor rigorous enough to ensure the continued improvement of project management. For example, the committee notes that the direct intervention of the undersecretary and deputy secretary was necessary to achieve consensus on the project management manual so that it could be issued. However, it took almost 3 years for DOE to resolve the internal differences and issue this manual, indicating that DOE management does not have a clear strategy on how to manage projects. This example amounts to short-term firefighting, and a more sustained, visible commitment by senior managers is needed in order to continue improving project management and to make these improvements a permanent part of DOE. Industrial organizations that have created excellent internal project control organizations have typically assigned the responsibility for project management programs to managers at the level of vice president or senior vice president. These senior executives, and even corporate CEOs, find it essential for success to maintain direct cognizance over projects—even those costing as little as $5 million. In DOE, the deputy secretary, as the department’s COO and senior acquisition executive, is responsible for effective project management. The deputy secretary should be DOE’s champion of project management excellence. As noted in previous NRC reports, the committee believes that DOE needs a visible, active, senior-level manager to promote and defend efforts to improve project management capabilities and their consistent application throughout the department (NRC, 1999). The committee also recognizes that DOE program organizations are given independent authority and accountability for managing projects. The assistant secretaries for SC and EM have been delegated the authority by the undersecretary to act as the acquisition executives for projects under $400 million in their respective programs. The NNSA administrator, as DOE undersecretary, also has
OCR for page 16
Progress in Improving Project Management at the Department of Energy: 2003 Assessment this same authority. A memorandum from Undersecretary Robert Card to the assistant secretaries notes that they are also accountable for the quality of their respective programs’ project management systems (see Appendix E). The committee believes that just as the deputy secretary should be a champion for project management throughout the department, the program heads should be strong, visible champions for project management in their programs. Although it is essential that each program office have strong project management capabilities and support, the committee continues to recommend that it is necessary that there be a single entity responsible for project management policies, procedures, personnel career development and training, and project tracking and reporting in order to achieve consistent project management excellence across the agency. A number of DOE project directors expressed the belief that DOE upper management would not support them if they rejected contractors’ submittals or decisions, and they were thus reluctant to challenge contractors. Orders, manuals, and guidelines are necessary but not sufficient—project directors need to know that they have someone to whom they can turn for professional guidance and support when making difficult decisions. To have the necessary impact to affect the DOE culture, this function should report directly to the deputy secretary.1 POLICIES AND PROCEDURES Considerable effort has been devoted to developing project management policies and procedures over the past 3 years. An order entitled Program and Project Management for the Acquisition of Capital Assets (Order O 413.3); the manual Project Management for the Acquisition of Capital Assets (M 413.3-1); a guide entitled Project Management Practices (PMP); and the notice Value Engineering (N 413.2) have been developed (DOE, 2000b, 2002, 2003a, 2003d). Program and Project Management Policies for the Planning, Programming, and Acquisition of Capital Assets DOE Policy P 413.1 (DOE, 2000a) set the stage for revitalizing project management within DOE. It charged OECM with the responsibility of preparing project management procedures for the department. DOE Order O 413.3 (DOE, 2000b), issued in October 2000, defines the principles of DOE project manage- 1 Editor’s note: The 2003 assessment is based on information reviewed by the committee through September 2003. In December 2003 the DOE deputy secretary appointed an associate deputy secretary with responsibilities for capital acquisition and project management, a positive step of which readers should be aware as they consider the committee’s comments and recommendations regarding the need for a strong and visible champion of project management issues in DOE.
OCR for page 17
Progress in Improving Project Management at the Department of Energy: 2003 Assessment ment and project oversight. It includes requirements for Integrated Project Teams (IPTs) and defines a critical decision process to assure the involvement of senior management as responsible decision makers from the inception to the completion of projects. In addition, the oversight process is enhanced by the ESAABs, which advise the acquisition executives at critical decision points. Following the issuance of Order O 413.3, OECM conducted workshops to obtain feedback and build support among DOE project managers. The order was reinforced by memoranda issued by Francis Blake, deputy secretary of DOE, in September and November 2001, and subsequently by Bruce Carnes, director of OMBE, in February 2002. The committee applauds both the order and the efforts to implement it throughout the department. The committee understands that the order may need to be updated from time to time, but believes that the basic principles, procedures, and applicability should be kept intact. Project Management for the Acquisition of Capital Assets Manual Considerable effort went into preparing a manual for the implementation of Order O 413.3. The first draft was issued in October 2000, and revised drafts were issued in February, June, and August 2002. Each iteration responded to critical review feedback from DOE project managers (project directors), contractors, this committee, and others. The final document’s format and organization present the roles, responsibilities, limits of authority, and required project controls and review procedures with respect to project management for the acquisition of capital assets. Efforts to issue the manual were threatened by internal and external resistance, but contentious issues were finally resolved through strong leadership and direct intervention by the undersecretary and the persistent efforts of OECM and the PMSOs. Manual M 413.3-1 was finally published in March 2003 (DOE, 2003a). Further, the deputy secretary issued a memorandum directing each Program Secretarial Office (PSO) to hold implementation sessions at field offices to demonstrate the commitment of line management to the project management system outlined by the manual. The committee was pleased to see this direct involvement of top DOE leadership in efforts to improve project management. This episode shows that attention by senior management will continue to be necessary to achieve the cultural change necessary to institutionalize project management best practices throughout the department. The overall structure and integration of M 413.3-1 as published have been considerably improved in comparison with previous versions. The document does an admirable job of describing the key issues that need to be addressed for project initiation—that is, approval of mission need (critical decision 0 [CD-0]) and approval of system requirements and alternatives (CD-1), including determining the estimated cost range. However, the document provides little direction or guidance with regard to the critical tasks that must be performed in the period between CD-1 and the approval of the project baseline (CD-2). This
OCR for page 18
Progress in Improving Project Management at the Department of Energy: 2003 Assessment important front-end planning phase of a project is mentioned only on a few pages (that is, on pages 2-5, 2-6, 6-1, 6-2; at CD-2 review on pages 9-6 and 9-7; and again briefly in Chapter 10). In contrast, the CD-0 and CD-1 deliverables merit an entire chapter each (Chapters 4 and 5, respectively). Since the activities between CD-1 and CD-2 typically are used to define the detailed scope of the project (as opposed to the conceptual scope) and encompass critical functions such as site evaluation, process flow design, design parameters, coordination issues, safety, instrument and electrical diagrams, equipment scope, execution plans, and so on, the committee believes this to be a significant omission from the manual. As noted in the 2001 forum on preproject planning, these functions are critical to the front-end planning process in successful project management organizations, and they can significantly affect project performance (NRC, 2002b). Although contractors will typically perform these activities on DOE projects with oversight by a federal project director, it is imperative that the department outline specific guidance on the detailed deliverables needed so as to ensure that the activities are done properly. In addition, federal staff needs to have sufficient management and technical expertise to assess the quality of the contractor’s project management products. The committee believes that the lack of explicit guidance for project management oversight between CD-1 and CD-2 exemplifies the absence of a clear understanding of the owner’s role in the project planning process. The key findings in the external independent reviews for the Center for Nanophase Materials Sciences, the Oak Ridge Research Support Center, and the Ashtabula Closure Project illustrate the problems that are still prevalent on many projects (see Appendix F). Both the manual and Project Management Practices need to be expanded to discuss the procedures and deliverables required at this phase, and federal project directors need to be involved in this effort to incorporate best industry practices. Project Management Practices Project Management Practices (PMP) consists of guidelines issued as references for project directors. The PMP elaborates on the information contained in Order O 413.3 and Manual M 413.3-1 by providing supplementary instructions that, although not required, are recommended in order to improve DOE’s ability to manage projects (DOE, 2003d). The PMP is distributed on CD-ROM and on the World Wide Web in order to permit frequent updating. The committee endorses issuance of the practices as a useful tool for present and future DOE project directors.
OCR for page 19
Progress in Improving Project Management at the Department of Energy: 2003 Assessment Value Engineering Deficiencies in DOE’s value engineering (VE) efforts were noted in the Phase II report (NRC, 1999) and the 2001 letter report (NRC, 2001a), and VE procedures were cited as being in need of additional documentation in the 2001 assessment (NRC, 2001b). DOE has made a considerable effort to make known to project directors the federal requirements to perform value engineering, primarily through a Web page listing of federal statutes, Office of Management and Budget (OMB) circulars, and DOE policies and procedures; this Web page is linked to the OECM Web site. DOE issued VE program implementation policies and procedures, including a VE policy notice (Notice N 413.2), in December 2002 (DOE, 2002). The notice was followed by an implementing memorandum in January 2003 and a VE acquisition letter (No. 2003-04) in August 2003 (DOE, 2003e) that provides direction and guidance on the application of VE for management and operations (M&O) contracts and other contracts for the performance of work at DOE sites and facilities. VE requirements are also addressed in Chapters 2 and 5 of Manual M 413.3-1. Training for VE is included in the Project Management Career Development Program (PMCDP) as an elective. Work is under way to include guidance on VE in the PMP and to develop a Web-based tutorial. Policies are now in place, but the committee cannot determine if they are being implemented, because performance measures are not yet available for assessing actual progress. The committee is aware of $16 million in VE savings in NNSA for 2002 but has no information on savings in other programs. It appears that DOE continues to lag behind the Department of Defense (DoD) and other federal agencies in implementing established, government-wide VE requirements. DOE appears to be taking action to implement a VE program, but additional attention and resources are needed to make VE an integral part of project management. DOE’s VE program exemplifies the committee’s concerns with DOE: VE is not new; it is not mysterious, exotic, or difficult; it is routinely practiced by other federal agencies; it is mandated by law; and it has been required by O 413.3 since 2000. Yet DOE has not effectively implemented VE. The apparent lack of commitment to implementing VE reduces the committee’s confidence in DOE’s long-term commitments to implementing policies and procedures and improving project management. Owner’s Role in Project Management The committee has emphasized the need for the federal employees involved in project management functions to assume the role of the owner’s representative (NRC, 1999, 2001b, 2003). In its 2002 assessment the committee made the following recommendation:
OCR for page 20
Progress in Improving Project Management at the Department of Energy: 2003 Assessment In order for DOE to be an effective owner of capital acquisition projects it should: Consider capital projects critical to organizational success. Require senior management involvement in project decision making, usually at the $5 million and higher level. Have a detailed and well-recognized internal front-end planning process. Capture metrics on planning effort and project performance. Require owner involvement and leadership in front-end planning. Ensure that projects support DOE’s mission and are consistent with DOE’s strategic plan. (NRC, 2003, p. 31) As noted in other sections of this report, DOE’s performance is inconsistent in activities that are critical to the role of an effective owner. The change of the title “federal project manager” to “federal project director” may help differentiate the roles and responsibilities of contractor personnel and help reduce confusion and focus federal employees on their appropriate roles (DOE, 2003a). However, Manual M 413.3-1 contains an inadequate definition of the roles and responsibilities of DOE personnel and contractors. The definition of roles and responsibilities should follow the requirements of O 413.3. It should cover the DOE chain of responsibilities from the acquisition executive to the project director, and the contractor’s chain of responsibilities from executives to managers in the field. The definitions should define responsibilities to develop, review, comment, approve, and execute at each step of the DOE capital acquisition process. These definitions should do more to address the authority that DOE project directors have in carrying out these responsibilities. In addition, each of the PSOs needs to develop a detailed “roles and responsibilities” document to reflect the different procedures used by their separate organizations. The committee notes that the process of developing this detailed information for the manual will help DOE identify opportunities to streamline the process. DOE project directors in the field are asking for better definition of their roles, responsibilities, and authority. In particular, they want to know if anyone in DOE will support them if they make a decision that a contractor does not like. Applicability of Policies and Procedures Order O 413.3 applies to all budget line-item projects over $5 million total project cost (TPC). The committee believes that, with provisions for tailoring requirements to the complexity of a project, this is an appropriate level. Because Congress wants DOE to control all of its projects, the general applicability of project management policies and procedures should remain at this level. Waivers of designated sites or projects from the requirements of O 413.3 (e.g., the Advanced Reactor Hydrogen Co-Generation Project [U.S. Senate, 2003]) are viewed by the committee as dangerous precedents that, if repeated, will undermine DOE’s ability to establish a department-wide culture that can manage projects well.
OCR for page 21
Progress in Improving Project Management at the Department of Energy: 2003 Assessment Some personnel in DOE and M&O contractors in the field expressed their concerns to the committee that the O 413.3 and M 413.3-1 are overly prescriptive and that the threshold value of projects that must comply with the order is set too low. These persons expressed the opinion that compliance with the order would result in project delays caused by the involvement of senior management at the critical decision points. The committee considered these positions thoroughly and believes they lack both basis and merit. The committee saw no actual evidence of projects that were delayed during the CD-0 through CD-2 reviews. On the contrary, the committee observed many projects that were commenced under O 413.3 and that proceeded expeditiously and on schedule. The committee noted in its previous reports the absence of adequate project justification and front-end planning, manifested in the lack of documentation such as acquisition plans, risk management plans, and project execution plans. If inadequate front-end planning documents have been returned for more justification and correction in order to support management decisions, the committee does not regard such action as a delay but as a desirable outcome. The committee reiterates its view that doing adequate planning up front is an essential activity and should be included in the project schedule. Projects may be spending more time and effort on planning than they did before O 413.3 because, as the committee has noted in previous assessments, thorough project planning was not being performed. If a project schedule allots insufficient time for adequate planning, it is a fault in the schedule, not evidence of delay. On the basis of best practices in industry and in other government agencies, the committee expects that the time taken for adequate planning to support management decisions at CD-0 and CD-1 can be made up at later CD points, which will go more smoothly if CD-0 and CD-1 are done correctly. If projects are actually delayed because the project justification and planning documents were inadequate to support DOE management decisions, the solution is not to eliminate management decision points but to improve the planning and the documentation. Review of project justification is an essential management quality-control point. The summary of EIR findings (see Appendix F) indicates that quality-control planning documents continue to be necessary. The committee believes that the solution is to improve quality, not to eliminate the quality-control function. If delays are indeed due to a lack of resources for OMBE and the ESAABs to perform reviews expeditiously, then more resources should be added to those activities. It is an appropriate exercise of the senior management function to delay some projects if they are considered to be marginal or perhaps unnecessary or of low priority, and to send them back for further analysis, justification, and documentation of mission need. In the past, GAO noted DOE’s history of executing projects that turned out to have little value (GAO, 1998). DOE has made progress in the area of documenting and justifying projects and should not backslide. The objective of a good project management process is to execute the right projects well, not to build the wrong projects faster.
OCR for page 30
Progress in Improving Project Management at the Department of Energy: 2003 Assessment PROJECT CONTROLS Project controls include earned value management systems (EVMSs); the Project Assessment and Reporting System (PARS); other reporting, change control, and change tracking systems; and other established management procedures. The ESAAB critical decision reviews and decision process are also considered to be project controls. The committee observed that most federal project directors now freely acknowledge the value of procedures such as the critical decision reviews and the related project justification and control activities. The committee believes that compliance with the requirements of Order O 413.3 corresponds to increased efforts to apply project controls for improved project performance. The committee also believes that DOE has made progress in some areas of project controls, but many others need additional improvement and more consistent application to be effective. Project Reporting The Project Assessment and Reporting System is a Web-based distributed database for collecting and analyzing current project earned value data. The PARS manual notes that “the purpose of the DOE project reporting system is to deliver project status and assessment information to DOE senior managers and key program stakeholders” (DOE, 2003g, p. 1). The committee notes that PARS potentially could become a valuable tool for project monitoring, reporting, and oversight, although substantial improvements are needed to make it effective. PARS is in its second generation, but it has not yet been adequately refined to accomplish its objective. Further refinement of definitions of data-entry fields is needed to ensure the collection of consistent data. This problem is particularly troubling because typical PARS earned value data are 3 or more months out of date and PARS does not display the future project plan for budgeted cost of work scheduled (BCWS). Data-entry requirements should include planned budgets and schedules (e.g., BCWS) through the completion of the project, as well as historical performance data. OECM has issued an updated users manual for PARS, but it gives no guidance on how to use or interpret the data. More follow-up is needed to determine who is using PARS, why they use it, and how it might be improved to increase its value to more users. The committee believes that a robust project database would be of considerable value as a project management tool, and it encourages DOE to continue to improve PARS and to extend its capabilities. The use of PARS for any project management function is currently limited by its deficiencies. It cannot be used to assess the performance of projects in real time because of the lateness of data reporting (generally at least three months behind and often much more), nor can it be used to assess historical performance because an historical record of baseline changes is not provided. Owing to the
OCR for page 31
Progress in Improving Project Management at the Department of Energy: 2003 Assessment failure to show baseline plans extended into the future (i.e., BCWS) and the lack of controls over changes to the baseline, evaluative indices of progress (cost performance index [CPI] and schedule performance index [SPI]) are so consistent that they are unbelievable to anyone with project management experience. Because of the incredible number of projects that are shown to be not approximately but exactly on schedule and on budget (i.e., 0.99 ≤ SPI ≤ 1.01 and 0.99 ≤ CPI ≤ 1.01), and considering other anomalies, the committee concludes that no confidence should be placed in PARS data and that PARS cannot be used to demonstrate whether or not DOE project management is effective or has even been improved. However, the committee believes that an effective project management culture would demand an effective project reporting system. Earned Value Management System The committee recommended in previous reports that DOE apply a rigorous earned value management system and that it use this system to analyze and improve project performance (NRC, 1999, 2001b). DOE Order O 413.3 requires the implementation of EVMS on all projects over $20 million TPC (DOE, 2000b). The committee saw evidence in project management awards documents and in briefings to the committee that EVMS is being or will be used on projects below $20 million TPC. The application of EVMS is necessary for all projects that are monitored through the PARS database. From an examination of PARS data, the committee is concerned that the quality of EVMS department-wide is inconsistent. OECM has initiated activities to provide EVMS training to DOE project directors and has proposed contracting with the Defense Contract Management Agency to verify contractor application of EVMS standards specified in O 413.3. EVMS is a critical part of project controls, and at this time there is no way of knowing if the reported data are accurate and reliable. The lack of evidence of consistent work breakdown structures, resource-loaded schedules, and baseline performance plans for cost and schedule causes the committee to doubt that EVMS has been effectively implemented. PERFORMANCE MEASURES Assessment Metrics DOE does not have a uniform set of objective measures for assessing the quality of project management. The lack of objective measures or even reliable historic project data makes it difficult to assess progress in improving project management. It also makes it difficult to build confidence within GAO, Congress, OMB, and the public in the department’s ability to manage the money it spends on its projects. Evidence continues to be anecdotal rather than objective, quantitative, and verifiable. The absence of objective performance measures
OCR for page 32
Progress in Improving Project Management at the Department of Energy: 2003 Assessment prevents the identification of best practices and impedes widespread improvement in project management throughout the agency. DOE Undersecretary Robert Card delegated acquisition executive authority to the assistant secretary for environmental management and the director of the Office of Science with the provision that their respective project management systems must be approved within a year (see Appendix E). However, the committee is not aware of any established DOE criteria for evaluating project management success, performance, or maturity. The Project Management Institute is developing its Organizational Project Management Maturity Model (OPM3) to assess the capabilities of an organization’s procedures and personnel, but the model will not be ready until the beginning of 2004 at the earliest and would require some testing and validation before it could be used to evaluate DOE programs. The committee has identified several lines of inquiry but does not have sufficient data to begin to discuss performance criteria. These lines of inquiry include the following: The percentage of DOE project managers who are certified professional project managers; The numbers of PMCDP courses offered and of personnel trained; The number of project directors and support staff for the value of projects managed; The functional quality of Integrated Project Teams; The quality of project planning documents (acquisition plans, project execution plans, risk management plans, and so on) submitted for ESAAB review; Management participation in ESAABs related to acquisition executive decisions; Effective use of performance-based contracts; Effective use of value engineering; Effective use of project controls; Trends in findings and comparison of independent project reviews and external independent reviews; Ratings by the EM Project Definition Rating Index (EM PDRI); and Comparison of actual project performance to original baselines for budget, schedule, and scope. The committee believes that the approval of the program offices’ project management systems should be based on a valid, comprehensive assessment. If such an assessment is not available, the committee recommends that the delegation of acquisition executive authority to the PSOs be revoked or continue to be provisional until additional direction on performance metrics and a revised assessment schedule can be provided and assessed.
OCR for page 33
Progress in Improving Project Management at the Department of Energy: 2003 Assessment The committee noted above that current project performance data available in PARS are not useful for assessing DOE project management because of problems with inconsistent data and the lack of historical trend data. Despite the recommendation in the 1999 NRC report (NRC, 1999) that DOE should develop a reliable database of all of its projects over time—past, present, and forth-coming—in order to be able to assess progress in improving project management, there is none 4 years later, and the committee has seen little evidence of progress in this regard. The Construction Industry Institute’s (CII’s) Project Definition Rating Index was described in the 2001 assessment report as one example of an approach to assessing the maturity of front-end planning (NRC, 2001b). EM’s adaptation of the PDRI continues to be used as a project review tool, but the committee has seen no evidence that the PDRI or any other approach has been used to calibrate DOE’s front-end planning procedures (DOE, 2001a). In the absence of a dedicated retrospective database of past DOE projects for comparison purposes, the capture of prospective data through PARS might be a way to eventually build a project database. To make this database useful for project analysis, PARS should archive the original cost and schedule baselines and all modifications to these baselines—constant rebaselining of projects precludes the ability to make valid comparisons over time and across projects. Although current revised baselines should be used to make day-to-day project management decisions, the original approved baselines and all subsequent changes should be tracked in order to evaluate overall project management performance. By maintaining all baseline data, project management performance can be objectively assessed by comparing the actual total project costs with original budget estimates, actual completion dates with original deadlines, and delivered scope and quality with original project specifications. The variance between the original baselines and the cost, schedule, and scope at completion is one indicator of performance. Process improvement can be assessed by analysis of the trend of variances for projects initiated over a period of time. Unfortunately, this approach will require years before it produces usable results, but it is essential to long-term performance, and it should start now. Benchmarking Benchmarking performance and management processes throughout a project’s life cycle and from different perspectives can provide a basis for a measure of improvement of project management procedures. Both internal and external benchmarking perspectives are useful and should be a regular part of DOE benchmarking procedures. Internal benchmarking can compare projects across programs and across time. External benchmarking can compare DOE projects with those of similar complexity, size, and other characteristics in other federal agencies and in industry.
OCR for page 34
Progress in Improving Project Management at the Department of Energy: 2003 Assessment The committee found little evidence of external benchmarking (other than attempts by NA-54 to use CII benchmarking data), nor did it find evidence of internal benchmarking. The committee recognizes the inherent difficulty in benchmarking first-of-a-kind and one-of-a-kind projects but reiterates its belief that a consistent, continuously applied benchmarking program would be an effective tool for improving project management. The committee noted in its 2001 assessment that the PARS database should provide accurate, up-to-date information for use in benchmarking (NRC, 2001b), but there is no evidence that PARS data are used for this purpose. PROJECT REVIEWS Prior to the 1998 NRC report, independent reviews in DOE were largely confined to those conducted by the Office of Science and mainly addressed the technical aspects of projects (NRC, 1998). Although these reviews neglected some aspects fundamental to sound project management, their contribution to executing technologically complex, one-of-a-kind projects is not to be minimized. OECM has issued a draft Independent Review Procedures, which outlines the departmental process and requirements (DOE, 2001b). The three major program offices (NNSA, EM, and SC) have each institutionalized and formalized their respective procedures in documents that cover the details and peculiarities pertinent to their programs (DOE, 2000c, 2000d, 2001a). These documents were reviewed in the 2001 assessment; the committee is not aware of any revisions since that time. The procedures for the congressionally mandated EIRs have matured over time and the reviews are more comprehensive and have improved in quality. Consequently, there is increased appreciation within DOE of the value of EIRs, and DOE project directors interviewed by the committee now acknowledge that EIRs have provided useful information and have added considerable value to the project delivery process. Also, there has been more concentration on conducting rigorous reviews in the early planning stages. Because project cost and schedule are particularly difficult to evaluate in the early stages of a project, it is essential that well-qualified individuals or contractors perform these reviews. The committee has also observed value in DOE internal reviews, particularly those employing nonadvocate participants from across organizational lines. Criteria for Reviews Although many DOE managers recognize the value of EIRs, the committee has heard the opinion voiced by some DOE managers and M&O contractors that external independent reviews provide no added value on projects costing $5 million to $20 million TPC. One PSO manager advocates that EIRs should not be required for projects costing less than $100 million TPC. The committee does
OCR for page 35
Progress in Improving Project Management at the Department of Energy: 2003 Assessment not share this opinion. Review of the key EIR findings (see Appendix F) indicates persistent problems in planning and other areas across all Program Secretarial Offices for projects with less than $100 million TPC. On this basis, the committee concludes that there is no justification for reducing EIR requirements, at least until there is demonstrated improvement in project outcomes. The committee continues to recommend that reviews of all projects should be conducted in some form. The scope and content of the review should be tailored to the complexities and peculiarities of the project, especially for those projects between $5 million and $20 million. By tailoring reviews to the complexity of projects, the expenditure of effort can be made compatible with the value added. Procedures already exist for waiving independent reviews for routine projects when this is justified, and there are procedures for tailoring reviews (DOE, 2001b). Independent reviews of plans, assumptions, designs, estimates, and schedules are the accepted standard in industry. In fact, independent reviews are one of the means by which industrial firms become successful at project management and continue to stay that way (NRC, 2002b). The view that DOE should not have to do the things typically done by the most successful industrial owner firms does not withstand scrutiny. The committee strongly advocates that DOE continue to recognize the value of EIRs for improving project performance and expand their application for documenting lessons learned. Capability of Independent Reviewers DOE has awarded three Management, Organization and Business Improvements (MOBIs) contracts for the performance of EIRs department-wide. The committee recognizes the benefits of such contracts for this type of work but believes that DOE needs to ensure that the contractors assign qualified personnel to each task. It is incumbent on DOE to exercise the necessary oversight to ensure the quality of reviews and to ensure that reviewers have the necessary qualifications and experience to add value to a project. Independent Cost Estimates The committee reviewed independent cost estimates (ICEs) for a variety of projects. It found them to be essentially arithmetical checks of the extant estimate rather than critical analyses of the work breakdown structures, cost elements, risk assessments, and other factors that affect the accuracy and credibility of the estimates. The committee learned that the scope and definition of the cost estimate reviews have been changed to an independent cost review (ICR), which includes the needed critical analysis. If properly structured and performed, the ICRs may provide the rigorous analysis that the committee believes is lacking in the ICEs. However, there are no data for evaluating ICRs at this time.
OCR for page 36
Progress in Improving Project Management at the Department of Energy: 2003 Assessment ACQUISITION AND CONTRACTING In each of its previous reports, the committee stressed the linkage between effective acquisition and contracting, and successful project management (NRC, 2001a, 2001b, 2002a, 2003). These reports identified and encouraged the use of a variety of acquisition best practices, many of which have been described in the new manual, M 413.3-1 (DOE, 2003a). For example, the manual requires that IPTs develop sound acquisition strategies to assess both risks and potential alternative contracting approaches. The OECM staff then assesses the completeness of the acquisition execution plans (AEPs) before forwarding them for senior management approval as part of the ESAABs and the critical decision process. This process is an essential means for senior managers to assess progress at critical steps. The committee was pleased to see an example of the effective use of the acquisition process, when NNSA recently sought to develop a quick-response approach for assisting the Russian Federation in closing down its three remaining plutonium reactors and replacing them with fossil fuel plants. NNSA worked effectively to develop a reasonable comprehensive strategy and risk mitigation plan, consistent with OECM guidance. Moreover, the deputy secretary showed his commitment to the process, maintaining continuing control through at least the CD-1 milestone. Other acquisition-related improvements include the following: EM’s development of a new contracting strategy relying on incentives for site closure. EM is now applying that strategy to the Rocky Flats site. Increased use of performance objectives in determining fees for site contractors. DOE increased the proportion of contractor fees tied to performance objectives from 34 percent in FY 1996 to 70 percent in FY 2001 (GAO, 2003). Increased use of competition among contractors to achieve best value for the department. Of 16 Federally Funded Research and Development Center (FFRDC) contracts, 6 have been competed, and other competitions are planned. Moreover, the percentage of major site contracts awarded competitively increased from 38 percent in FY 1996 to 56 percent in FY 2001 (GAO, 2003). The effective use of alternative contracting techniques to meet DOE needs. An example is Los Alamos National Laboratory’s use of a design-build approach for one of its major construction projects. These examples demonstrate good front-end planning, senior management involvement, the effective use of performance metrics and incentives, and flexibility in contracting approaches. The committee strongly supports continuing the emphasis on all of these techniques.
OCR for page 37
Progress in Improving Project Management at the Department of Energy: 2003 Assessment In its previous reports, the committee stressed the benefits of systematic training in the use of performance-based contracting methods and encouraged the department to collect data on the effectiveness of these techniques. It reiterates the need to follow through on these recommendations. RECOMMENDATIONS Conclusion DOE has made significant progress in improving project management through organizational changes and the development of policies and procedures. These changes were completed in 2003 with the release of the manual Project Management for the Acquisition of Capital Assets (Manual M 413.3-1) and have just begun to be implemented. The committee believes that current policies and procedures need to be fully and consistently implemented, and that opportunities exist for additional changes that are needed in order to maintain the progress achieved to date and to continue the improvement process so as to bring DOE’s project management capabilities to an appropriate level of excellence. The committee believes that action on most of its past recommendations is still incomplete and that the results department-wide are inconsistent. The opportunities for continued improvement in the application of accepted project management practices are presented above in this chapter and are summarized below as committee recommendations. Recommendations DOE senior managers should actively use the Office of Program Analysis and Evaluation to provide objective analysis and advice on mission need and project cost-benefit justification at the early stages of incipient projects and should use the Office of Engineering and Construction Management to standardize requirements for management decisions, monitor the progress of projects, and provide objective analysis and advice. These offices, as well as the project management support offices and the DOE site offices, should be provided adequate resources, staffing, training, and moral support. The DOE deputy secretary either should be the champion for project management improvement—to develop project management into a core competency of the department, to assure that the department maintains an adequate staff of qualified project directors to manage its portfolio of projects, and to assure that the disciplined execution of projects is a priority for managers at all levels—or should appoint someone to perform this role, reporting to the deputy secretary (see footnote on page 16). Just as the deputy secretary should be a champion for project management
OCR for page 38
Progress in Improving Project Management at the Department of Energy: 2003 Assessment throughout the department, the program heads should be strong, visible champions for improved project management in their programs (see footnote on page16). Current policies and procedures should be kept intact and revised only as necessary to improve the planning and execution of projects. The applicability of project management policies and procedures should remain at current levels. Requirements should be tailored to the size and complexity of projects, but exemptions for projects or sites should not be considered until such decisions can be supported by a record of excellence in project management and project performance is established. Manual M 413.3-1 should define the roles and responsibilities required by Order O 413.3. It should cover the DOE chain of responsibilities from the acquisition executive to the project director, and the contractor chain of responsibilities from executives to managers in the field. The definitions should describe responsibilities to develop, review, comment, approve, and execute at each step of the DOE capital acquisition process. In addition, each of the Program Secretarial Offices needs to develop a detailed “roles and responsibilities” document to reflect the different procedures used by their separate organizations. If the application of policies and procedures results in delays because resources to perform reviews expeditiously are lacking, then more resources should be provided. Steps should also be taken to streamline the critical decision review process by eliminating unnecessary sign-offs. DOE should ensure that funding and implementation of the Project Management Career Development Program are priorities. The committee sees particularly urgent needs for department-wide training in front-end planning, risk analysis, and project controls. Training for project directors and project support staff should be centrally funded and should utilize alternatives to traditional classroom delivery of instructional materials. DOE should adopt a department-wide strategy to develop a sustainable, qualified workforce for directing the projects required to achieve the department’s missions. DOE should work with Congress to allow funding for project direction to be included in project budgets. DOE should provide contracting officer representative authority to project directors. DOE should continue and expand department-wide and program-specific project management workshops to recognize project management achievements, reinforce the professional identity of project directors, and share lessons learned. Strategic planning initiatives should be continued, expanded, and used by management to improve project selection, planning, and execution. DOE should develop detailed procedures and guidance for identifying risks, planning strategies to address risks, and managing risks throughout
OCR for page 39
Progress in Improving Project Management at the Department of Energy: 2003 Assessment the life cycle of projects, and should require their implementation for all projects. Projects should not pass CD-1 or CD-2 without an effective risk mitigation plan. Manual M 413.3-1 and Project Management Practices should be expanded to include detailed procedures and deliverables required between CD-1 and CD-2. Federal project directors should be involved in this development effort so as to incorporate the best industry practices in DOE procedures. Steps should be taken to ensure the quality and consistent application of project controls. Planned project scope, budgets, and schedules should be maintained through completion of a project. Although current revised baselines should be used to make day-to-day project management decisions, the original approved baselines and all subsequent changes should be retained in order to evaluate overall project management performance. Steps should be taken to ensure that the Project Assessment and Reporting System (PARS) data are timely, accurate, and consistent from project to project. Analytical tools and presentations should be enhanced. PARS should archive the original cost and schedule baselines and all modifications to these baselines. DOE should develop consistent performance metrics for evaluating project management maturity department-wide. Both internal and external benchmarking should be a regular part of procedures for all phases of projects. DOE should continue to recognize the value of external independent reviews for improving project performance and should expand their application for documenting lessons learned. DOE should exercise the necessary oversight to ensure the quality of reviews and to ensure that reviewers have the necessary qualifications and experience to add value to the project. DOE should follow previous recommendations to provide systematic training in the use of performance-based contracting methods and should collect data on the effectiveness of these techniques. REFERENCES Collins, James C. 1994. Built to Last. New York, N.Y.: HarperCollins. DOE (Department of Energy). 1999. Memorandum for All Departments from T.J. Glauthier, Deputy Secretary; Subject: Project Management Reform Initiative. June 25. DOE. 2000a. Program and Project Management Policy for the Planning, Programming, and Acquisition of Capital Assets (Policy P 413.1). Washington, D.C.: Department of Energy. DOE. 2000b. Program and Project Management for the Acquisition of Capital Assets (Order O 413.3). Washington, D.C.: Department of Energy. DOE. 2000c. Office of Defense Programs Project Review Procedures. Washington, D.C.: Department of Energy.
OCR for page 40
Progress in Improving Project Management at the Department of Energy: 2003 Assessment DOE. 2000d. Office of Environmental Management Internal Independent Review Handbook. Washington, D.C.: Department of Energy. DOE. 2001a. Office of Science Independent Review Handbook. Washington, D.C.: Department of Energy. DOE. 2001b. Office of Engineering and Construction Management Draft Independent Review Procedures. Washington, D.C.: Department of Energy. DOE. 2002. Value Engineering (Notice N 413.2). Washington, D.C.: Department of Energy. DOE. 2003a. Project Management for the Acquisition of Capital Assets (Manual M 413.3-1). Washington, D.C.: Department of Energy. DOE. 2003b. Acquisition Career Development Program, Attachment 4 (Order O 361.1). Washington, D.C.: Department of Energy. DOE. 2003c. Office of Management and Budget Evaluation. Available online at http://www.mbe.doe.gov/crOrg/me20.htm. Accessed September 9, 2003. DOE. 2003d. Office of Engineering and Construction Management. Available online at http://oecm.energy.gov/. Accessed September 9, 2003. DOE. 2003e. Value Engineering (Acquisition Letter 2003-4). Washington, D.C.: Department of Energy. DOE. 2003f. Project Management Practices. Available online at http://oecm.energy.gov/. Accessed September 9, 2003. DOE. 2003g. Project Assessment and Reporting System User Manual, version 3.0. Washington, D.C.: Department of Energy. GAO (General Accounting Office). 1998. Nuclear Waste: Department of Energy’s Hanford Tank Waste Project: Schedule, Cost, and Management Issues (GAO/RCED-9913). Washington, D.C.: General Accounting Office. GAO. 2003. Department of Energy: Status of Contract and Project Management Reforms (GAO-03-570T). Washington, D.C.: General Accounting Office. NRC (National Research Council). 1998. Assessing the Need for Independent Project Reviews in the Department of Energy. Washington, D.C.: National Academy Press. NRC. 1999. Improving Project Management in the Department of Energy. Washington, D.C.: National Academy Press. NRC. 2001a. Improved Project Management in the Department of Energy. Letter report, January. Washington, D.C.: National Academy Press. NRC. 2001b. Progress in Improving Project Management at the Department of Energy, 2001 Assessment. Washington, D.C.: National Academy Press. NRC. 2002a. Progress in Improving Project Management at the Department of Energy, 2002 Interim Assessment. Letter report, May. Washington, D.C.: National Academy Press. NRC. 2002b. Proceedings of Government/Industry Forum: The Owner’s Role in Project Management and Preproject Planning. Washington, D.C.:National Academy Press. NRC. 2003. Progress in Improving Project Management at the Department of Energy, 2002 Assessment. Washington, D.C.: The National Academies Press. U.S. Senate. 2003. The Energy Policy Act of 2003 Report of the Committee on Energy and Natural Resources, United States Senate, to Accompany S. 1005 (Report 108-43). Washington, D.C.: U.S. Government Printing Office.
Representative terms from entire chapter: