APPENDIX D
Compilation of Findings and Recommendations

The three previous NRC reports (NRC, 1999, 2001, 2003) include 76 findings and 94 recommendations that are compiled below. The previous findings and recommendations as well as those in this report address 10 recurring objectives for the changes needed to improve project management at DOE. Most of these changes relate to inadequate planning, inadequate risk management, and inadequate monitoring and follow-up. The recurring objectives include the following:

  • Develop policies and procedures to define the DOE method of managing projects;

  • Create a project management culture across the agency that supports the consistent implementation of policies and procedures;

  • Provide leadership that ensures disciplined planning and execution of projects as well as support for continuous process improvement;

  • Provide a project management champion at the highest level of the department to ensure that a focus on the importance of project management is established and maintained;

  • Develop competence in fulfilling the owner’s role in front-end project planning, risk management, and project execution;

  • Apply rigorous project reporting and controls that include earned value systems, link day-to-day management data to periodic reporting, forecast time and cost to complete, and maintain historical data with which to benchmark project performance;

  • Document processes and performance to support benchmarking and trend analysis;



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment APPENDIX D Compilation of Findings and Recommendations The three previous NRC reports (NRC, 1999, 2001, 2003) include 76 findings and 94 recommendations that are compiled below. The previous findings and recommendations as well as those in this report address 10 recurring objectives for the changes needed to improve project management at DOE. Most of these changes relate to inadequate planning, inadequate risk management, and inadequate monitoring and follow-up. The recurring objectives include the following: Develop policies and procedures to define the DOE method of managing projects; Create a project management culture across the agency that supports the consistent implementation of policies and procedures; Provide leadership that ensures disciplined planning and execution of projects as well as support for continuous process improvement; Provide a project management champion at the highest level of the department to ensure that a focus on the importance of project management is established and maintained; Develop competence in fulfilling the owner’s role in front-end project planning, risk management, and project execution; Apply rigorous project reporting and controls that include earned value systems, link day-to-day management data to periodic reporting, forecast time and cost to complete, and maintain historical data with which to benchmark project performance; Document processes and performance to support benchmarking and trend analysis;

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment Invest in human capital by providing training and career development to ensure an adequate supply of qualified, skilled project directors; Continue, refine, and document a program of external and internal project reviews; and Employ innovative approaches to capital acquisition and the use of performance-based contracting. Although the committee has provided suggested approaches, it believes that DOE senior managers have the responsibility to identify and apply metrics to define the appropriate level of excellence and to drive continuous process improvement. POLICIES, PROCEDURES, DOCUMENTATION, AND REPORTING 1999 Finding. DOE does not have adequate policies and procedures for managing projects. No single authority is responsible for enforcing or ensuring that project management tools are used. Finding. DOE has developed comprehensive practice guidelines for the design and construction phases of projects but has not developed comparable guidelines for the early conceptual and preconceptual phases, when the potential for substantial savings is high. Finding. Many DOE projects do not have comprehensive project management plans to define project organization, lines of authority, and the responsibilities of all parties. Finding. DOE does not effectively use value engineering to achieve project savings, even though federal agencies are required to do so. Finding. DOE project documentation is not up to the standards of the private sector and other government agencies. Finding. DOE does not have a consistent system for controlling changes in project baselines. Finding. DOE does not effectively use available tools, such as earned value management, to track the progress of projects with respect to budget and schedule. Finding. ISO 9000 provides a certification process by which an organization can measure itself against its stated goals, but DOE has not obtained certification.

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment The certification process would help DOE remake the entrenched operating procedures and standards that have accumulated over the past 50 years. Recommendation. As a part of its project management system, DOE should issue fundamental policies, procedures, models, tools, techniques, and standards; train project staff in their use; and require their use on DOE projects. DOE should develop and support the use of a comprehensive project management system that includes a requirement for a comprehensive project management plan document with a standard format that includes a statement of the project organization covering all participating parties and a description of the specific roles and responsibilities of each party. Recommendation. DOE should update the project performance studies to document progress in these areas and extend the benchmarking baseline to include all major DOE construction projects. The study results should then be used to improve project procurement and management practices. Recommendation. DOE should mandate a reporting system that provides the necessary data for each level of management to track and communicate the cost, schedule, and scope of a project. Recommendation. DOE should establish a system for managing change that provides traceability and visibility for all baseline changes. Change control requirements should apply to the contractor, the field elements, and headquarters. Recommendation. DOE should establish minimum requirements for a cost-effective earned-value performance measurement system that integrates information on the work scope (technical baseline), cost, and schedule of each project. These requirements should be included in the request for proposals. Recommendation. DOE, as an organization, should obtain and maintain ISO 9000 certification for all of its project management activities. To accomplish this, DOE should name one office and one individual to be responsible for acquiring and maintaining ISO 9000 certification for the whole department and should require that consultants and contractors involved in the engineering, design, and construction of projects also be ISO 9000 certified. Recommendation. DOE should establish an organization-wide value-engineering program to analyze the functions of systems, equipment, facilities, services, and supplies for determining and maintaining essential functions at the lowest life-cycle cost consistent with required levels of performance, reliability, availability, quality, and safety. Value engineering should be done early in most projects, and project managers should take the resulting recommendations under serious consideration.

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment 2001 Finding. The recommendations in the Phase II report to develop and publish a set of policies and procedures for management of DOE projects appear to have been addressed to some extent by the draft [Program and Project Management manual] PPM and the draft [Project Management Practices] PMP; however, the committee finds that there is a need for additional detail and clarity and elimination of discontinuities, gaps, overlaps, and repetitions. The committee recognizes that OECM is addressing these issues as it develops the next iterations and commends OECM for its leadership role. Recommendation. The PPM and PMP text should be tailored to specific DOE requirements. It should be clear which parts of the text constitute DOE required procedures and which parts reflect general advice on good project management practices. Recommendation. OECM should assure that policies and required procedures add value by streamlining the process and improving project performance. Policies and procedures that do not demonstrably add value should be revised or eliminated. Recommendation. The PPM and PMP should have parallel structures. A complete index and a glossary of terms should be provided for both documents. Recommendation. Examples should be given where they will illustrate the application of procedures and the necessary documentation. Examples should have adequate explanations and represent realistic project situations. Over time, a set of templates and case studies should be built up. Recommendation. OECM should be provided the resources needed to publish improved, revised versions of the PMP and PPM as soon as possible. OECM should be given the authority to authorize case-by-case exceptions when appropriate to ensure that common sense and cost-effectiveness prevail in the retrofitting of procedures to ongoing projects. 2002 Finding: DOE and DOE contractor personnel expressed some concern that requirements in DOE Order O 413.3 and drafts of the Program and Project Management (PPM) manual and Project Management Practices (PMP) result in excessive and unnecessary effort and cost for projects of less than $20 million; they believed these requirements should not apply to environmental projects and

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment that front-end planning documentation and review requirements are excessive. The committee does not agree with their views but it does believe that requirements should be tailored to the complexity of the project. Recommendation: DOE should resist efforts that reduce requirements for frontend planning and the critical decision-review process. This resistance is necessary to ensure that the process is uniform and that projects selected for execution are consistent with DOE’s strategic plan. The requirements should apply to all projects over $5 million and be tailored to the complexity of the project. Finding: The committee has observed examples of both effective and ineffective project management practices at DOE. Order O 413.3 is intended to create a consistent department-wide definition of what is required of DOE project managers. OECM is revising the PPM to better define the project management practices to be used to achieve the objectives of O 413.3. The committee believes that the order is beginning to increase the level of consistency throughout the department. It also believes that a document to define the minimum actions required to implement O 413.3 is necessary. This document needs to carry commensurate authority and be coordinated to develop a consistent DOE approach to project management and project oversight. Recommendation: OECM should accelerate development of the PPM and should issue the current draft immediately to guide interim compliance with O 413.3. The order and the manual should be separate but coordinated documents to create a consistent DOE approach to project management. The order should continue to define what is required and remain relatively unchanged over time. The manual should continue to be a separate document to specify minimum requirements for compliance to O 413.3. PROJECT PLANNING AND CONTROLS 1999 Finding. DOE preconstruction planning is inadequate and ineffective, even though preconstruction planning is one of the most important factors in achieving project success. Finding. DOE often sets project baselines too early, usually at the 2- to 3-percent design stage, sometimes even lower. (An agreement between Congress and DOE’s chief financial officer for establishing baselines at the 20- to 30-percent design stage is scheduled to be implemented in fiscal year 2001.)

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment Finding. DOE often sets project contingencies too low because they are often based on the total estimated cost of a project rather than on the risk of performing the project. Finding. DOE does not always use proven techniques for assessing risks of major projects in terms of costs, schedules, and scopes. Recommendation. DOE should require that strategic plans, integrated project plans, integrated regulatory plans, and detailed project execution plans be completed prior to the establishment of project baselines. To ensure facility user and program involvement in the preconstruction planning process, DOE should require written commitments to project requirements from the ultimate users. Recommendation. DOE should significantly increase the percentage of design completed prior to establishing baselines. Depending on the complexity of the project, the point at which project baselines are established should be between the completion of conceptual design and the completion of the preliminary design, which should fall between 10 and 30 percent of total design. The committee supports continuing efforts by Congress and the DOE to develop project baselines at a point of adequate definition beginning with fiscal year 2001. Recommendation. Baseline validation should be assigned specifically to the project management office recommended in this report. The Military Construction Program of the U.S. Department of Defense, which requests planning and design funds for all projects in the preliminary design stage on the basis of total program size, is a potential model for DOE. Recommendation. DOE should establish contingency levels for each project based on acceptable risk, degree of uncertainty, and confidence levels for meeting baseline requirements. The authority and responsibility for managing contingencies should be assigned to the project manager responsible for doing the work. In the process of evaluating potential projects, DOE should apply risk assessment and probabilistic estimating techniques, as required by the Office of Management and Budget. STRATEGIC PLANNING 2001 Finding. There are no PSO strategic plans defining long-range goals and objectives or mission needs, and documentation of project justification is almost entirely lacking or inadequate, so that it is impossible to assess whether the right

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment projects are being done. Cost-benefit analysis and performance measurements required by GPRA cannot be performed effectively without effective strategic plans. Recommendation. The PSOs should develop budget-based rolling 5-year strategic plans that identify the mission goals and objectives of the program, the projects necessary to achieve them, and the benefits to be expected from these projects. FRONT-END PLANNING 2001 Finding. The DOE process for project initiation, planning, justification, and execution continues to need substantial improvement. A top-to-bottom process that recognizes best practices in both government and industry, as well as the unique and specific requirements of DOE programs and projects, is essential. Recommendation. The DOE secretarial acquisition executive should sponsor a process improvement program, and OECM should be named the program champion in DOE. Finding. Compliance with the front-end planning requirements in O 413.3 has been inconsistent among PSOs and among individual projects. Recommendation. OECM should assure that all program offices have a documented front-end planning process that meets the intent of O 413.3, and that the information used as input for Energy Systems Acquisition Advisory Boards (ESAABs) and ESAAB-equivalent readiness reviews, as well as the outcomes of these reviews, is documented and used to assess project performance and progress in improving project planning. Recommendation. The PMSOs should consider developing tailored checklists such as the EM-PDRI [Office of Environmental Management Project Definition Rating Index] as in-process planning tools, train project personnel to use them, and analyze their effectiveness for projects throughout the DOE complex. Effective and consistent front-end planning should be made mandatory for all projects. Finding. Tools such as checklists, communications software/methods, planning reviews, third-party audits, economic modeling, objective setting, and team building, if used correctly, can contribute to effective front-end planning. Performance of technical evaluation during planning is essential for projects involving new

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment technology, complex site conditions, and complex project-flow requirements. Consistent documentation and planning structure would increase the effectiveness of front-end planning in the department. Recommendation. OECM should clarify, expand, and revise the front-end planning procedures in the Program and Project Management manual and Project Management Practices. DOE should use standard industry procedures where applicable; however, the PMSOs should provide supporting policies and procedures tailored to the specific projects and needs of each program. The PMSOs and OECM should assure the adequacy of front-end project planning prior to each critical decision, to assure that projects are not unnecessarily delayed by poor plans and that time constraints do not cause projects to be approved without adequate planning. Recommendation. The deputy secretary and the designated program acquisition executives should strengthen their interest and support, thereby confirming that truly effective front-end planning will be required without exception. OECM and the PMSOs should pay close attention to documentation of front-end planning decisions. Finding. DOE has established a process to significantly increase the accuracy and reliability of project baselines. Recommendation. OECM should actively participate in the process and monitor the performance of projects baselined under this new process to document its impact and opportunities for improvement. Finding. Overall, insufficient attention from DOE management is being given to the front-end planning process; however, the committee observed that management was acting in isolated cases and to varying degrees within the program offices. Recommendation. DOE senior management should emphasize the importance of thorough and complete front-end planning (including written documentation). ESAABs and ESAAB-equivalent reviews should be used to enhance the quality of front-end project planning and assure that the project team is pursuing the right project—that is, that the project has adequate justification and will satisfy a wellconceived need. Finding. Front-end planning improvement requires metrics for trend analysis. The committee was not able to obtain this information for specific projects because DOE does not have enough data for front-end planning trend analysis.

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment Recommendation. OECM should begin benchmarking project practices and performance metrics to identify areas in need of improvement and establish a baseline for future evaluation. This benchmarking effort should be systematic, quantitative, and analytical, and it should compare practices in industry and in other government agencies. It should capture both front-end planning and performance metrics, including actual performance versus forecast. Finding. A training program addressing front-end planning and other project management practices is being developed. The completion date of this effort was reported to the committee to be December 2002, with training to start soon afterward. Without immediate improvement in the planning knowledge and skills of personnel and more management emphasis on improving the planning process, projects will continue to have inadequate front-end planning. Recommendation. OECM should do more than develop policies and procedures—it should become fully engaged in process improvement beginning with front-end planning. To overcome the lack within the department of skilled project planners and the delays in training, and to bridge the gap until a training program takes effect, DOE should establish a cadre of experienced project planners within OECM; they should have a wide variety of planning capabilities and prior experience in different project types, including high-risk projects. These individuals should be a part of the initial integrated project teams and should assist the project originators (as internal consultants) in getting front-end planning done correctly, including planning prior to CD-0. This cadre of internal consultants should champion the DOE front-end planning process, providing just-in-time training for front-end planning to project teams. DOE should benchmark its management of project planning personnel and application of their expertise with that of private sector companies that have successfully undertaken similar activities. In this way, DOE may be able to jump-start an immediate improvement in planning capability. Recommendation. DOE should eliminate impediments to initiating training for front-end project planning prior to December 2002. Training should begin as soon as possible. RISK MANAGEMENT 2001 Finding. With rare exceptions, there are no risk models for ongoing DOE projects, and back-fitting risk assessment to ongoing legacy projects does not seem to be part of the acquisition risk management (ARM) study. There is no consistent

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment system for evaluating the relative risks of projects with respect to scope, cost, or duration, so the deputy secretary, the chief financial officer, and the PSO managers have no objective basis for knowing which projects are riskier (and therefore require more management attention) than others. Recommendation. DOE should develop the ability to perform quantitative risk assessments. These assessments should be carried out by DOE personnel with experience in such analyses working with persons who have an in-depth understanding of a given project. Internal project risk assessments should be separately evaluated by independent assessors or reviewers who are not project proponents for reasonableness of assumptions, estimates, and results. Risk mitigation and management plans should be prepared that can deal with significant risks identified. Recommendation. DOE project management personnel should be trained in risk assessment methodology. This training should cover not only risk analysis methodology and techniques, but also the managerial responsibilities related to interpretation of risk assessments and mitigation and management of risks. Recommendation. Risk analyses should explicitly consider the interdependence of the various activities due to common modes (root causes), or document why there is no dependence. Finding. DOE has not implemented statistical models (the “objective” analysis cited in OMB Circular No. A-94, Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs) because it has no usable database of past and current projects. Recommendation. DOE should develop an internal database of data on its own projects and on projects of other owners. A system should be established to capture data on current and future projects. Data on comparable projects performed by other federal agencies and by industry should be obtained and included. The current development of the Project Analysis and Reporting System (PARS) (discussed in Chapter 5) could be a step toward this goal, and the committee plans to follow this work with interest. Although its early stage of development prevents assessing its effectiveness at this time, the level of participation by projects, accuracy of data, completeness of data, and avoidance of duplication should be addressed by OECM. The architecture of this data system should be specifically designed to provide support for the analysis of risks for ongoing and future projects. Finding. By and large, DOE’s practices in risk assessment and risk management have not significantly improved since the Phase II report. The committee reviewed

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment some project risk assessment studies but did not see an example of a risk assessment or risk mitigation plan that it finds acceptable. The discussion in the draft PPM is merely an outline, and the material in the draft PMP is not useful as a guide for practicing risk management. Conversely, the current ongoing acquisition risk management (ARM) pilot study at three DOE sites and by the Contract Reform and Privatization Office and the EM Division Steering Group/Working Group, due for completion by December 2001, is a positive move and shows promise. The committee intends to follow this study with interest as it evolves. Recommendation. The current acquisition risk management (ARM) pilot study should be continued and expanded beyond budget risks to cover the issues addressed in the Phase II report and in this report, such as schedule, scope, quality, and performance risks. Finding. DOE’s deficiencies in risk analysis lead to inadequate risk mitigation planning and execution. Plans often address symptoms but not causes. Execution is typically reactive or nonexistent. To be useful during project implementation, this planning should, at a minimum, do the following: Characterize the root causes of major risks that were identified and quantified in earlier portions of the risk management process. Identify alternative mitigation strategies, methods, and tools for each major risk. Evaluate risk interaction effects. Identify and assign priorities to mitigation alternatives. Select and commit required resources to specific risk mitigation alternatives. Communicate planning results to all project participants for implementation. Recommendation. DOE should develop and implement risk mitigation planning processes and standards. Project risk assessment and management should be carried out throughout the project life cycle and should be part of the documentation for each critical decision point. Risk mitigation plans should be reviewed, critiqued, returned for additional work if needed, and approved by an independent organization such as the ESAABs at each critical decision point and prior to project approval for design or construction funding. Recommendation. Until DOE project managers can be adequately trained in risk management, OECM should establish a cadre of experienced risk assessment personnel, who can be detailed or seconded to projects in the very early stages, to provide risk assessment expertise from the beginning of projects and incorporate risk management into the initial project management plan. Finding. DOE needs to take a flexible approach in managing risk because of the high levels of uncertainty. To be effective in risk management, flexibility should

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment Finding. The existing contract for training offers a means to deliver consistent content throughout the department; however, it reduces the range of options for training. Recommendation. DOE should modify or replace the current contract to allow greater flexibility in accessing courses pertinent to the project management skills utilized by industry and other federal agencies. DOE should develop new courses consistent with the new knowledge, skills, and abilities requirements identified by the findings of the gap analysis. HUMAN CAPITAL 2002 Finding. There is reason to believe, based on the reported numbers of DOE project management personnel and the volume of DOE projects, that DOE is understaffed in the area of project managers and essential project management support staff. The committee concludes that there may not be enough DOE project management personnel to discharge their responsibilities as the owner’s representatives. This apparent deficiency may lead to a situation in which [management and organization] M&O and [management and integrations] M&I contractors, by design or default, are performing the roles and functions that should be the prerogative of the owner’s representatives. This inappropriate devolution of some of the department’s project management responsibilities to contractors may be creating a conflict of interest. Recommendation. DOE project management should be staffed to the level needed to ensure that the government’s interests are protected. DOE should assess whether it has enough project management personnel to properly discharge its ownership role or whether DOE understaffing in project management is permitting contractors to take on responsibilities and functions that should be reserved for the government’s representatives. To do this, DOE will have to define the roles and responsibilities of federal project managers and then assess the number of project managers needed to carry out these responsibilities. The roles and responsibilities of the contractors’ project managers vis-à-vis the federal project managers should also be clarified. Recommendation. DOE should develop a vision for what project management in the department should become, and then hire, train, and promote personnel specifically to staff and fulfill this vision.

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment Recommendation. Concurrent with the DOE staffing assessment, DOE should also assess the project management staffing of its (M&O and M&I) contractors in terms of both quantity and quality (knowledge, background, and experience). It would be desirable to know if contractors, perhaps because of the declining competition for DOE projects, are not assigning their best managers to DOE projects. Recommendation. DOE should estimate its future requirements for project management and other project support personnel and develop a plan to address recruitment, turnover, and retention in the future. Hiring personnel with experience in preproject planning, cost estimating, risk management, EVMS, team facilitation, and other critical skills can be a means of meeting some of those needs in the near term. Finding. The committee perceives a need for improved utilization of existing and incoming project management personnel. This need can be fulfilled through training and career development and by facilitating the movement of personnel across organizational lines. Executing the PMCDP as a DOE-wide program will go a long way toward overcoming present training deficiencies. However, a long-term commitment to funding implementation of the PMCDP is critical. Recommendation. The projected annual tuition expenditure for training and development of $1.5 million is considered adequate for the immediate concentrated need. Every effort should be made to allocate this amount centrally based on a DOE-wide decision, especially in the first few years, to assure implementation of the PMCDP throughout the organization. In the interim, the DOE field and project offices should continue to meet immediate needs with their own training programs. Recommendation. In a previous report, an NRC committee recommended that DOE should “develop and maintain a cadre of professional certified project managers who would be assigned to manage DOE projects for all program offices” (NRC, 1999, p. 77). Since it is clear that DOE does not intend to implement this recommendation, the committee recommends that DOE treat qualified project management personnel as a shared resource and facilitate their movement to assignments across the organization as the needs arise. OECM, in conjunction with the operation of the PMCDP, should maintain an inventory of all project managers throughout the DOE complex, along with their experience and capabilities, and make this inventory available to all DOE programs as they staff their projects.

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment AWARDS 2002 Finding. DOE has executed several recent projects successfully and on or ahead of budget and schedule, as indicated by its 2002 project management award program. Finding. While all projects considered for 2002 awards were initiated prior to the publication of DOE Order O 413.3, the principles and procedures required by the order and outlined in this and prior committee reports were important factors in successful completion. Finding. Lessons learned from briefings by award recipients have application to project personnel who did not attend the Project Management Workshop. Recommendation. Copies of briefings by the 2002 award recipients should be distributed to all field offices that have project personnel. Finding. [The National Nuclear Security Administration] NNSA and EM were the only program offices that participated in the 2002 project management awards program. Other DOE offices that execute projects had no nominations. Recommendation. DOE should determine why the other program components did not participate in the awards program. DOE should encourage full participation in the future. Finding. The Project Management Workshop is a step forward in recognizing exemplary projects and project managers and in building a sense of professionalism among project personnel. Recommendation. DOE should continue and even expand this workshop in future years. PROJECT REVIEWS 1999 Finding. Independent project reviews are essential tools for assessing the quality of project management and transferring lessons learned from project to project. Finding. External independent reviews of 26 major projects are under way to assess their technical scope, costs, and schedules. The reviews so far have docu-

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment mented notable deficiencies in project performance verifying the committee’s conclusion that DOE’s project management has not improved and that its problems are ongoing. However, DOE has yet to formalize and institutionalize a process to ensure that the recommendations from these reviews are implemented. Finding. Various DOE program offices are also developing the capability of conducting internal independent project reviews. Recommendation. DOE should formalize and institutionalize procedures for continuing independent, nonadvocate reviews, as recommended in the Phase I report of the National Research Council to ensure that the findings and recommendations of those reviews are implemented. DOE should ensure that reviewers are truly independent and have no conflicts of interest. Recommendation. All programs that have projects with total estimated costs of more than $20 million should conduct internal reviews, provided that the value of the reviews would be equal to or greater than the costs of conducting them. Deciding if an internal review is justified for a given project should be the joint responsibility of program management and the project management organization. The decision should be based on past experience with similar projects, the estimated cost of the project, and the uncertainty associated with the project. Internal reviews are expensive and take up the time of valuable people, so they should not be undertaken lightly. However, under the present circumstances, the committee believes that more internal reviews would be justified. The project management organization should manage these reviews for the director or assistant secretary of the cognizant program office. The results of these reviews should be taken by the program office to the Energy Systems Acquisition Advisory Board (ESAAB) and used as a basis for the decision whether to continue the project. 2001 Finding. The evidence available to the committee indicates that the [external independent review] EIR program continues to identify significant management issues in the projects reviewed and in DOE’s operation in general. Absent substantial evidence of improvement in DOE project management and project performance, the EIR program needs to be continued. Recommendation. DOE, through OECM, should establish performance metrics for the EIR program that identify trends and opportunities for improving project management performance. Recommendation. The EIR program should continue in its present form under

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment OECM direction until there is clear evidence of improvement in DOE project management and project performance. Finding. DOE would benefit from a department-wide procedure governing external and internal independent reviews. Consistent procedures would increase the pool of qualified reviewers, expedite the review and report process, and enable an automated system for tracking deficiencies and corrective actions. Recommendation. DOE should expedite the issuance of the Independent Review Procedure drafted by OECM. Finding. A more thorough review analysis for defining mission need and setting a preliminary baseline range during the front-end project planning phase would give the decision makers more useful information. Recommendation. DOE should expand the use of [internal project reviews] IPRs for the CD-0 decision and should require an [independent cost review] ICR prior to CD-1. Finding. There is some concern that mandating formalized reviews for projects costing between $5 million and $20 million TPC may be dedicating manpower and money beyond the point of significant value added. It also may be distracting project personnel from doing the project and diverting DOE’s project management resources from larger, more complex projects. Recommendation. DOE should reevaluate the benefits gained from mandating reviews for projects costing between $5 million and $20 million TPC. The OECM should establish guidelines to evaluate the cost-effectiveness of review. At a minimum, the $5 million threshold should be based on TEC and provide for significant tailoring of the review process. Finding. The EM Project Definition Rating Index (EM-PDRI) analyzes the readiness of a project by rating it on a numerical basis. It allows making judgments based on a multitude of rating factors, including risk, but users will need training and experience with the index in order to achieve uniformity of application and confidence in the results. Recommendation. DOE should explore the potential application to other programs of the PDRI approach adopted by EM.

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment ACQUISITION AND CONTRACTING 1999 Finding. Traditional DOE contracting mechanisms, such as cost-plus-award-fee and management and operations (M&O) arrangements, are not always optimal for DOE’s complex mission. These approaches are being replaced with more effective approaches based on objective performance incentives, but change has been slow. Finding. DOE’s long history of hiring contractors to manage and operate its sites on the basis of cost-plus-award-fee contracts has created a culture in which neither DOE nor its contractors is sufficiently accountable for cost and schedule performance. Finding. DOE does not use effective performance-based incentives and does not have standard methods for measuring project performance. Finding. DOE does not effectively match project requirements and contracting methods. Mismatching often results in cost and schedule overruns. Finding. The numbers of bidders on major DOE contracts has been declining and in some cases have not elicited truly competitive bids. This may indicate that projects are not being appropriately defined and packaged and that the disincentives to bid often outweigh the incentives. Recommendation. DOE should strengthen its commitment to contract reform focusing on the assessment and quantification of project uncertainties, the selection of the appropriate contract type and scope for each job, and increased use of performance-based incentive fees rather than award fees to meet defined project cost and schedule goals. A comprehensive risk analysis should be conducted before deciding whether to issue fixed-price contracts for work that involves a high level of uncertainty (such as new technology or incomplete characterization). Specific contract scopes and terms should be negotiated to define both DOE and contractor responsibilities to prevent cost overruns. Clear, written roles, authorities, and responsibilities should be established for DOE headquarters, field elements, contractors, and subcontractors for each contract. Guidelines should be provided for the appropriate times in the project for the selection of contractors. Recommendation. DOE should develop written guidelines for structuring and administering performance-based contracts. The guidelines should address, but need not be limited to, the following topics: the development of the statement of work; the allocation of risks to whomever would be most effective at controlling

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment the risks (either DOE or the contractor); the development of performance measures and incentives; the selection of the contracting mechanism; the selection of the contractor; the administration of the contract; and the implications of federal and DOE acquisition regulations. DOE should train its employees in the roles and responsibilities of a performance-based culture and then hold both employees and contractors accountable for meeting these requirements. Recommendation. DOE should provide financial rewards for outstanding contractor performance to attract bids from the best contractors. A DOE-wide policy should be developed that provides fiscal rewards for contractors who meet or exceed schedule, cost, and scope performance targets. Contractor fees should be based on contractor performance. Recommendation. DOE employees and contractor employees essential to projects should be trained in acquisition and contract reform. The training of source selection officials and members of source evaluation boards should be expedited; a minimum level of training should be a prerequisite. 2001 Finding. The extent of training and use of performance-based contracting (PBC) in DOE contracting efforts is unclear. There is no DOE-wide database that shows the extent of use of PBC or the number of staff trained in PBC techniques. Recommendation. The committee reaffirms the recommendations made in previous reports (NRC, 1999, 2001) on using PBC and encourages OECM to play a lead role in supporting this practice. OECM should work closely with the Office of Procurement and Assistance Management to see that PBC training is provided as part of the career development process for project management personnel and just-in-time training for the [integrated project team] IPT. In the near term, OECM should bring on board a cadre of experts, skilled in performance-based contracting, to provide technical assistance to IPTs responsible for new major system initiatives. Finding. The draft Program and Project Management (PPM) manual and draft Project Management Practices (PMP) developed by the OECM fail to address PBC adequately. Recommendation. The detailed descriptions of PBC alternatives and their application to DOE projects should be included in the revised PMP and PPM. Finding. There have been continuing efforts on the part of DOE to move toward a more effective use of PBC methods and to support these efforts.

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment Recommendation. Contract approaches should be tailored to use fixed-price and performance-based methods where practicable to assist the DOE to get the most cost-effective results and to stimulate competition. In addition, the department should continue to explore other innovative commercial contracting approaches to meet its needs. 2002 Finding. The committee believes that the August 2002 draft PPM provides a good framework for addressing acquisition strategy issues and offers a useful model for DOE project managers to follow in preparing and planning their efforts. However, it finds that the acquisition strategy documents being reviewed by OECM are of mixed quality and believes that this indicates a need for more training and development of additional reference documents. The iterative process of review and correction will also improve the overall quality of planning documents over time. The most recent draft PPM now provides clear and consistent guidance on what needs to be addressed in each draft acquisition strategy and, as such, should significantly increase the quality of the documents submitted for review. Recommendation. The committee recommends that senior management continue to require project teams to focus on up-front acquisition planning and that it continue to use the approval process for ensuring compliance and consistency. DOE management should return documents that do not meet management expectations and should follow up by asking why these inadequacies were not fixed at lower levels. Project teams should be trained in developing effective acquisition strategies. DOE leadership should also continue to focus on competition to obtain a range of innovative approaches from a variety of contractors to meet its management, operating, and development needs. PERFORMANCE-BASED CONTRACTING 2002 Finding: For large cost-reimbursement contracts, many factors compromise the ability of the government to use purely objective measures for assessing performance. Moreover, federal agencies are comfortable using a more traditional cost reimbursement award fee approach, in which the award fee is at the discretion of the federal project manager. Recommendation: A significant amount of up-front planning by the IPT is needed to specify outcomes and identify those aspects of an overall project for

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment which a contractor can effectively be held accountable. The committee reiterates its recommendation that training in performance-based contracting methods be provided to IPT members. In addition, DOE should collect best practices information on the use of performance-based contracting in DOE contracts and identify those activities most suitable for use of these metrics. Finding. DOE is reassessing its cleanup efforts, giving them a new focus on cost-effective and rapid closure of sites, and setting up incentives that can best achieve that goal. Recommendation. DOE should reassess its use of incentives in existing contracts to ensure that they focus on closure and that interim goals are effective in driving this overall objective. ORGANIZATIONAL STRUCTURE, RESPONSIBILITY, AND ACCOUNTABILITY 1999 Finding. DOE’s organizational structure makes it much more difficult to carry out projects than in comparable private and public sector organizations. Successful corporations and agencies responsible for major projects arrange their organizations to provide focused and consistent management attention to projects. Finding. Too many people in DOE act as if they were project managers for the same project, and too many organizations and individuals outside the official project organizations and lines of accountability can affect project performance. Finding. Compliance with DOE’s policy requiring the establishment of performance agreements and self-assessments from the field has been limited and slow. Recommendation. To improve its project management performance, DOE should establish an office of project management on a level equal to or higher than the level of the offices of assistant secretaries. Department-wide project management functions should be assigned to the project management office, and the director of this office should have the authority and the resources to set and enforce reporting requirements for all projects. Other responsibilities, such as property and asset management, should be assigned to existing DOE headquarters offices. To be successful, the office of project management must have the full and continuing support of the secretary, the under secretary, the deputy secretary, and of all of the program offices and field offices as a top-down management initiative.

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment 2001 Finding. DOE continues to rely excessively on contractors for project justification and definition of scope. There are some large projects in which DOE is not effectively executing its role of owner with respect to the oversight and management of contracts and contractors. Recommendation. DOE should develop its position as an effective owner of projects and should assure that federal project managers are trained and qualified owner’s representatives, capable of dealing effectively with contractors. Finding. The combination of the OECM and the PMSOs in the three major PSOs addresses many of the issues raised in the Phase II report but not all. This organizational structure is probably workable, but it does not fully address the departmentwide issues of consistency, discipline, and excellence in project management that the Phase II committee felt were essential. Recommendation. The roles and responsibilities of the OECM should be strengthened, as set forth in the Phase II report, and the OECM should be budgeted, staffed, and empowered to become the center of excellence in project management and the coordinator for project manager training and development and for oversight and approval of all capital projects in DOE. 2002 Findings. The forum held in November 2001 provided examples of the points made in the committee’s previous reports about how industry fulfills its role as owner in planning and managing projects. In subsequent meetings with DOE and DOE contractor personnel the committee saw evidence of increased emphasis on front-end planning and a clearer understanding of DOE’s role as owner. Recent policy memoranda that emphasize acquisition planning are encouraging. Recommendation. The committee believes that in order for DOE to be an effective owner of capital acquisition projects it should: Consider capital projects critical to organizational success. Require senior management involvement in project decision making, usually at the $5 million dollar and higher level. Have a detailed and well-recognized internal front-end planning process. Capture metrics on planning effort and project performance. Require owner involvement and leadership in front-end planning. Ensure that projects support DOE’s mission and are consistent with DOE’s strategic plan.

OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2003 Assessment Recommendation. DOE should periodically benchmark its performance in project planning and control processes and compare it with the performance of industry leaders to ensure that it is consistently utilizing the best practices. Recommendation. Senior managers in each program secretarial organization (PSO) in DOE should develop a complete definition of the roles and responsibilities of project managers. Recommendation. Senior managers should continue to emphasize the importance of improving the project management processes and procedures to assure long-term improvement throughout the organization. PROJECT MANAGEMENT CULTURE 2002 Finding. DOE personnel and contractors generally support the need for a comprehensive project management system but prefer a system with fewer requirements for upper management oversight. Recommendation. DOE should resist efforts to reduce up-front planning requirements and to lower the level of authority at which critical decisions are approved. DOE should apply persistent pressure to ensure that the right projects are picked for execution and that they are planned and executed according to established policies and procedures. Procedures should continue to include a process for tailoring requirements to the size and complexity of projects. Recommendation. DOE should assess its culture and subcultures and develop strategies to bring about organization alignment on core project management principles at all levels of the organization. REFERENCES NRC (National Research Council). 1999. Improving Project Management in the Department of Energy . Washington, D.C.: National Academy Press. NRC. 2001. Progress in Improving Project Management at the Department of Energy, 2001 Assessment. Washington, D.C.: National Academy Press. NRC. 2003. Progress in Improving Project Management at the Department of Energy, 2002 Assessment. Washington, D.C.: The National Academies Press.