The National Research Council (NRC) completed a planned 3-year review and assessment of the U.S. Department of Energy’ s project management with publication of Progress in Improving Project Management at the Department of Energy: 2003 Assessment (NRC, 2004). In that final assessment report, the Committee for Oversight and Assessment of U.S. Department of Energy Project Management made the following observation:
DOE does not have a uniform set of objective measures for assessing the quality of project management. The lack of objective measures or even reliable historic project data makes it difficult to assess progress in improving project management. It also makes it difficult to build confidence within GAO, Congress, OMB, and the public in the department’s ability to manage the money it spends on its projects. Evidence continues to be anecdotal rather than objective, quantitative, and verifiable. The absence of objective performance measures prevents the identification of best practices and impedes widespread improvement in project management throughout the agency. (NRC, 2004, pp. 31-32)
The Department of Energy responded to the NRC report by forming an internal task group led by the Office of Engineering and Construction Management (OECM) to develop performance measures and benchmarking procedures and asked the NRC to provide additional assistance to guide this effort. This report, therefore, does not contain new findings or recommendations. The advice and guidance presented are intended to help DOE develop and implement effective performance measures and an effective benchmarking program for project management. The information and guidance should be viewed not as the final work but rather as a first step toward development of a viable methodology to suit the needs and goals of DOE.
For a performance measurement and benchmarking system to be effective, it should be crafted to fill multiple organizational needs, carry the imprimatur of the users, and be accepted at all levels of the organization.
The committee suggests 30 possible performance measures in four sets:
Project-level input/process measures. Assess the resources provided to deliver an individual project and the management of the project against standard procedures.
Project-level output/outcome measures. Assess the cost and schedule variables of an individual project and the degree to which the project achieves the stated objectives.
Program- and department-level input/process measures. Assess the total resources provided for all projects within a program or department and the degree to which program- and department-wide goals for projects and their management are met.
Program- and department-level output/outcome measures. Assess overall project performance and the effectiveness of completed projects in supporting program and department missions.
The value of an individual performance measure is limited, but, combined, the measures provide a robust assessment of the quality of project management for individual projects and programs. If applied
consistently over time and used for internal and external benchmarking, the measures will provide the information needed for day-to-day management and long-term process improvement. It remains the responsibility of DOE to select and use the measures that work best for it and its program offices.
For performance measures to have meaning and provide useful information, it is necessary to establish comparisons. The comparisons may evaluate progress in achieving given goals or targets, assess trends in performance over time, or weigh the performance of one organization against another.
Benchmarking is an integral part of process improvement that provides a mechanism for making comparisons of project and program performance internally and externally. To be successful, benchmarking should be implemented as a structured, systematic process based on an understanding of critical success factors. Benchmarking can be applied during various phases of a project for different purposes. When applied early on, such as at project authorization, it can be used to identify characteristics that may be associated with potential future problems and to identify aspects of project management (e.g., risk management) that need special attention to ensure project success. When applied during project execution, it can serve as a management tool to guide project decisions. Postproject benchmarking is usually used to assess performance of a project delivery system and to establish benchmarks for future comparisons.
The benchmarking process described in this report involves nine activities:
Determine what to benchmark,
Define the measures,
Develop data collection methodology,
Identify deficiencies in the use of best practices and project management performance,
Identify reasons for deficiencies (root causes),
Develop an action plan (select best practices to reduce deficiencies),
Integrate best practices into the project delivery process, and
Institutionalize benchmarking as part of a continuous improvement program.
This report primarily addresses the first four of the nine steps. The remaining five steps define the essence and purpose of benchmarking, which is to continuously improve project management. For the effort to yield any benefits, it is essential that DOE establish a means for implementing the five steps after the collection of data and a system for continuous feedback and evaluation.
Measuring performance and benchmarking should be viewed as a routine, integral part of project management processes rather than a separate function. This requires that advocacy be built into the system. The most difficult step is establishing an organizational culture that is ready to assess, compare, and analyze performance and to adopt best practices used by others when they are identified. This requires an organizational commitment to continuous improvement, acceptance of new ideas, and open communication and cooperation at all levels of the organization.
Development of the necessary organizational readiness for benchmarking can be facilitated by taking incremental steps, starting with a limited number of measures and internal benchmarking within a program, then expanding the effort to include more diverse measures and comparisons with other programs. The objective over time should be to develop a full set of measures and to benchmark both internally and externally with organizations in other government agencies and private industry.