Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 64
Progress in Improving Project Management at the Department of Energy: 2001 Assessment 7 Acquisition and Contracting INTRODUCTION The committee reiterates the finding that acquisition planning and acquisition and contracting techniques play an integral role in successful project management (NRC, 1998, 1999). This role is particularly critical at DOE, where approximately 90 percent of the department’s budget is expended by contract (GAO, 2001). The findings and recommendations of the Phase II report stress the importance of developing and employing contracting methods that ensure accountability, adequately address risk, and focus the government and the contractor on achieving the outcomes sought. The committee continues to advocate these recommendations. In accomplishing its mission, DOE relies on different types of contractor-managed activities. They include the large national laboratories, generally university-run or consortium-run, which serve as federally funded R&D centers for critical nuclear weapons and their design, development, and stockpile stewardship efforts. These activities also include large private sector management and operation (M&O) and management and integration (M&I) contractors that maintain operations at sites such as the Nevada Test Site or perform cleanups of hazardous and radioactive waste at places such as Rocky Flats, Colorado. Under an M&O contract, the contractor/subcontractor team performs much of the work under a cost-plus, award-fee contracting arrangement. Under an M&I contract, the contractor conducts continuing competitions among subcontractors and awards fixed-price tasks where possible to get the most cost-effective solution. DOE also uses contractor support for other purposes, such as ammunition assembly at the Pantex plant in Texas.
OCR for page 65
Progress in Improving Project Management at the Department of Energy: 2001 Assessment PERFORMANCE-BASED CONTRACTING Over the past decade, a number of contracting reforms have been put in place across the government to streamline and simplify processes and to sharpen the focus on agency mission and results. Performance-based contracting (PBC), promulgated as a government-wide policy in 1991 by the Office of Federal Procurement Policy (OFPP), is one such reform. It is applicable to some if not all aspects of the various types of projects operated by the department. Its effective use depends on the ability of the government/contractor team to understand and manage the risks inherent in any contracted effort. PBC emphasizes that all aspects of an acquisition should be structured around the purpose of the work to be performed, as opposed to the manner in which it is performed. It offers contractors flexibility to determine how best to meet the government’s requirements, while ensuring that desired performance levels are achieved and that payment is made only for results that meet negotiated performance standards. DOE has included this technique as part of its own contract-reform agenda and has been using a performance-based approach for major projects at Rocky Flats, Oak Ridge, the Nevada Test Site, and elsewhere. Successful PBC is based on defining existing conditions, specific requirements, and the desired results or outcomes, along with objective, meaningful, and measurable performance and quality standards. In addition, incentives are used to focus contractor efforts and to reward success. In a successful performance-based contract, expectations must be made clear, with agency and contractor teams working together in a business partnership to achieve well-defined and measurable results. Agencies across the government are increasingly relying on PBC. Moreover, a March 9, 2001, memorandum to all agencies from the deputy director of the Office of Management and Budget (OMB) requires that 20 percent by dollar value of all agency service contracts for FY2002 be performance-based. This mandated percentage would increase in future years. A 1998 study conducted by OFPP found cost savings on the order of 15 percent and increases in customer satisfaction (up almost 20 percent) when agencies used performance-based contracts rather than traditional requirements-based procurements (EOP, 1998). Much of the success of a PBC approach results from effective use of a cross-functional team for identifying desired outcomes and establishing effective performance metrics. Integrated Project Teams The integrated project team (IPT) concept included in DOE Order O413.3 is an essential element in implementing a performance-based approach (DOE, 2000). The committee strongly supports the use of these IPTs and suggests the following PBC methodology:
OCR for page 66
Progress in Improving Project Management at the Department of Energy: 2001 Assessment Establish a cross-functional government team of program, contracting, and project management personnel to work together to develop, award, implement, and manage a performance-based contract or task. If the contract has already been awarded, then a joint government/contractor team should develop useful performance metrics and incentives. Develop a performance-based statement of work (SOW), in matrix format, to be used as the basis for all subsequent solicitation, proposal, or contract documents. Define, for each major project or task, the desired outcomes, required services, standards of performance, and methods of performance evaluation and measurement. While DOE has followed similar procedures in requiring performance metrics for service contracts, the General Accounting Office (GAO) has been critical of DOE’s employment of PBC (GAO, 2001). It cited, among other things, inadequacies in DOE financial accounting and reporting systems and difficulties in establishing firm baselines from which to measure contractor performance. In its January 2001 report, the GAO continues to cite PBC implementation problems. Balanced Scorecard The DOE Office of Procurement and Assistance Management has conducted its own study of this area and has required headquarters review of the various performance metrics and incentives being used. In addition, a department-wide balanced scorecard self-assessment training program includes PBC. The balanced scorecard is an approach for measuring an organization’s performance and long-term success. Measurements are made in the areas of finance, customer service, internal business processes, and employee learning and growth. This tool is in use in a number of agencies to measure performance and identify strengths and weaknesses. For FY2000, 28 of DOE’s major site and facility management contractors participated in a balanced scorecard process. Examples of performance metrics include a customer satisfaction index, with firms on average achieving a customer satisfaction score of 90 percent. BWX Technologies at the Mound site received the top score of 100 percent. From an internal business perspective, the performance measure used is the percentage of systems in compliance with stakeholder requirements. Here the average for DOE was 92 percent, with Bechtel Nevada achieving a 100 percent rating. The scorecard has been in use for 4 years. All of the major site and facility contracts awarded in the past 4 years were performance based. For FY2001 a new hands-on training program was rolled out, with a new performance-based management-contracting course under development. This training should be included as part of the project manager training program discussed in Chapter 9, “Project Manager Training and Development.”
OCR for page 67
Progress in Improving Project Management at the Department of Energy: 2001 Assessment Benefits of Performance-Based Contracting PBC is not only a contracting technique, it is also critical to the requirements development process, because it entails defining desired outcomes up front as well as assessing risks. These steps cannot be taken without the full involvement of the IPT. The committee believes that effective use of PBC will give DOE more confidence in the likely success of its major projects. To be fully effective, program managers need to be integrated into this process in every respect. The key benefits of adopting this approach are the following: Requirements and processes with no value added are eliminated, enabling the contractor to achieve objectives faster and at lower cost. Contractor innovation is encouraged. Expectations and accountability are clearly defined. A win-win partnership is established between the contractor and the customer, with risks and rewards shared. PERFORMANCE-BASED CONTRACTING IN POLICY AND PROCEDURE DOCUMENTS Given the potential impact of PBC on successful project management and the benefits to all players in this process, the committee believes that PBC objectives and methodology should be defined and discussed in DOE policy and procedure documents. The following subsections suggest the information that might be included in the draft PPM manual, along with examples of PBC applicability to the DOE environment. Define Desired Outcomes Outcome definition should be functional and substantive, using terms that can be easily understood by an external stakeholder and focused on results rather than on work processes. These include the desired outcomes for the overall contract, project, or task and the desired outcomes for major task areas. Each of the major task-level outcomes should contribute to the overall program or project outcome. All outcomes should be based on business results for which the contractor can reasonably be held accountable and should not depend on performance or events outside the contractor’s control. Examples of outcomes for contractor purchasing systems that are currently included in the DOE balanced scorecard assessment are the following: On-time delivery is a measure of effective supplier management. The performance metric is the percentage of on-time delivery to be achieved, with the target set at 85 percent.
OCR for page 68
Progress in Improving Project Management at the Department of Energy: 2001 Assessment Streamlining processes to enable DOE to more efficiently meet its mission needs. The performance metric is the number of critical processes reengineered, redesigned, or revalidated. In one case the target is two annually. These examples demonstrate types of measures that may be used to see whether performance criteria have been achieved. Define Required Services for Specific Milestones or Tasks All required services should contribute to the achievement of the project or task-level outcome. These services might include major deliverables or work products. All requirements should focus on the outcomes to be provided and should not define the process or technical approach used to perform the work. Define Performance Standards for Each Required Service Each required service may have multiple associated performance standards. All performance standards should measure services for which the contractor can be held fully accountable and should be precise, meaningful, and attainable. The following considerations apply: Define performance standards only for those areas where the data collected are sufficiently meaningful to be evaluated and can be used to determine whether or not the desired results have been achieved. Consider the cos/benefit of collecting the performance data. Typical performance standards address quality, timeliness, completeness, accuracy, reliability, and cost. In determining the specific quantitative metrics, consider the cost/benefit of achieving varying levels of performance. Industry standards, departmental policies, and regulations that the government requires the contractor to comply with may be used as performance standards. This point is particularly relevant in the DOE context, since compliance with legal or regulatory directives is often a prerequisite for performing the work. The performance metrics described above are examples of the ways in which success can be measured. Define Acceptable Quality Levels for Each Performance Standard Acceptable quality levels (AQLs) should be expressed as a percentage of conformance with the standard (e.g., 95 percent) or the deviation (e.g., 0 percent
OCR for page 69
Progress in Improving Project Management at the Department of Energy: 2001 Assessment deviation), or as a range (e.g., not to exceed 2 days following a negotiated deadline). Every performance standard must have an AQL. The cost implications of requiring 100 percent compliance with or 0 percent deviation from the standard should be fully considered before setting tolerance levels. However, if safety or compliance requirements are at issue, then 100 percent compliance may be the only acceptable approach. Define Surveillance or Monitoring Methods to Document Performance and Evaluate Whether Performance Standards and AQLs Have Been Met There should be at least one surveillance or monitoring method for each performance standard. In general, surveillance includes both contractor-provided data and government validation. The government’s primary role should be to validate and assess performance. Identify Incentives and Disincentives Relating to the Critical Results Disincentives, including loss of fee, can be used especially where activities relate to the health and safety of workers or the public or where serious environmental impacts may occur. While monetary incentives are generally used, other potential incentives may relate to reduced reporting requirements or to extending automatically the term of a contract based on a contractor’s superior record of performance. The latter approach is called award term contracting. In the DOE context, with a relatively limited base of bidders, an award term approach may further limit competition and therefore may not be as suitable as it is at other agencies. Frequently, in the DOE context, timeliness incentives are critical, whether they involve speeding cleanup operations at a site like Rocky Flats or an additional fee for accelerating a schedule, as is the case for the new contract at Yucca Mountain. An innovative incentive approach currently in development by the National Nuclear Security Agency (NNSA) is the use of multisite integrated incentives for key programs. These multiyear incentives, which will become effective in FY2002, link the sites in achieving cost efficiencies. As discussed above, effective PBC approaches have been developed through the balanced-scorecard self-assessment process. Other examples include the M&I contract at Oak Ridge, Tennessee, which uses effective performance-based tasks and subcontractor competition to effect its cleanup operations. Oak Ridge has also established a clear government-contractor partnership document to ensure accountability and to identify each party’s responsibilities in getting the sought-after results. Similarly, the Rocky Flats, Colorado, cleanup operation has developed performance metrics and incentives that focus both the government and the contractor on the desired end state, that is, successful site cleanup and closure of
OCR for page 70
Progress in Improving Project Management at the Department of Energy: 2001 Assessment operations. Finally, the Nevada Test Site contract has used specific, objective, fee-based performance incentives to focus all aspects of the contractor’s operations on achieving the government’s goals. Develop a Performance Matrix to Assess PBC As noted previously, the committee strongly supports the OECM policy to use IPTs to develop an acquisition plan for the contract and then to play a continuing role throughout the acquisition cycle. The IPT should develop a performance matrix that uses the categories of effort described above as column headings. This matrix will display desired business results, performance metrics, and performance monitoring or surveillance methods, as well as incentives or disincentives associated with the contract all in one easily readable and understandable document. The matrix in effect serves as the work statement for the contractor and should be incorporated into the resulting contract or task order. Any references, assumptions, or dependencies should also be included in the matrix to the extent possible. As a practical matter, understanding the various dimensions of the PBC matrix can take some time. Moreover, it is essential that all team members have a common understanding of what each term means and how it is to be used. For this reason, the committee believes that PBC training should be provided to all IPTs, if possible using a just-in-time training method, so that all key players will be fully informed on the benefits of the approach as well as on its methods and objectives. In sum, the performance matrix defines the performance and results expected by the government but does not dictate how the work is to be performed unless law or regulation mandates certain steps. RISK, PERFORMANCE-BASED CONTRACTING, AND CONTRACT TAILORING A central part of the IPT’s effort in developing the performance matrix is the need to consider risks very early in the acquisition strategy and acquisition planning process. Only after the risks have been fully assessed can the contracting approach and type of contract be determined. PBC supports fixed-price contract vehicles by focusing on results as opposed to level of effort. The contractor is given considerable operational leeway but is held strictly accountable for meeting the outcomes established by the IPT for tasks and for the overall effort. Of course, by moving toward a fixed-price environment, the government is effectively transferring risk to the contractor. As the level of uncertainty surrounding technical risk (e.g., uncharacterized elements in a cleanup operation or scalability and integration issues in an information technology or scientific environment) or business risk (e.g., the likelihood
OCR for page 71
Progress in Improving Project Management at the Department of Energy: 2001 Assessment of adequate funding to achieve project milestones) increases, the manageability of the risk comes into question. Where the uncertainties are too great, the contractor is less likely to be held accountable for the desired result. The same is true if interdependencies cannot be adequately identified. In a fixed-price environment, the government transfers certain cost risks to the contractor, as defined by the contract specification. That is, the fixed price puts a ceiling on the cost to the government for risks deemed to fall under the contractor’s control or risks considered normal in the particular line of business (weather and other things), provided the risks are identified in the contract. However, a fixed-price contract is also a floor on the cost to the government, unless the project is adequately managed to eliminate changed conditions, delays, and other sources of additional cost to the owner. Also, the risk of delays, schedule overruns, or quality of the facility may devolve to the government unless the required performance and desired outcomes are clearly defined and monetary or other incentives provided for superior performance. Risks may also revert to the government if there are changed conditions or design changes for which the government may be held responsible. Again, the PBC matrix provides a very effective tool for assuring contractor accountability for results. In an environment where uncertainty is great, fixed-price contracts would be unreasonable or imprudent. That does not mean, however, that the contracted effort could not be structured in such a way as to allow performance-based approaches and competition for those aspects of the effort where risk is both clear and manageable. DOE is undertaking an acquisition risk management (ARM) study to be completed by December 2001 that focuses on the points included in DOE O413.3 (DOE, 2000) and Part 7 of the Federal Acquisition Regulation (FAR) that project risks need to be managed early in the acquisition planning stage. The study will result in a guide to be used by IPTs in the development of a project’s acquisition strategy. This guide should also help the team in deciding the amount of contingency to be included in execution estimates. As an IPT works through these questions and issues, it will have much greater confidence that it has in fact structured an appropriate contracting approach for meeting DOE’s mission. This front-end planning effort requires both time and the right mix of staff to see that all key questions are identified and addressed, for if these questions are not addressed early in the project, the prospects for problems later on are much greater. While incentive contracting has been identified as a key tool for achieving cost savings and better results, the IPT should also consider other contracting methods to see if they might be more effective in meeting the department’s needs. For example, at Los Alamos National Laboratory (LANL), the M&O contractor used a straightforward design/build project delivery method for the Advanced Computing Facility and more recently for the Non-Proliferation Center Building. Where the work to be done can be defined in traditional design and construction
OCR for page 72
Progress in Improving Project Management at the Department of Energy: 2001 Assessment terms, as they were here, schedule advantages, project quality, and cost savings can be realized. More information on design/build and other contracting modes that might meet DOE contracting needs are found at Part 36 of the FAR, “Construction and Architect-Engineer Contracts.” In summary, defining desired outcomes and performance standards, as well as assessing risk early on in the acquisition process, is key to selection of the right contract vehicle and offers the best chance of meeting agency and project and program officer expectations. DOE should consider these innovative contracting methods when looking for the best method to achieve the outcomes sought. A recurring issue in performance-based contracting is how to set a price on performance. In general, this might be done in one of two ways: The owner is highly knowledgeable, has an excellent database, and therefore knows the appropriate costs for a given level of performance and can negotiate equitable prices with prospective contractors. The owner is not knowledgeable and relies on the marketplace (that is, competitive bidding) to set the price. Clearly, the second method requires a large pool of responsible bidders. The Phase II report noted that the DOE bidder pool was shrinking and recommended that DOE act to arrest and reverse this trend. In 2001, a report by the DOE Contract Reform and Privatization Project Office, Analysis of the DOE Contractor Base: Readiness, Willingness, Profitability, and Trends: A Focus on the Environmental Management Program, showed that for EM, at least, the contractor base has shrunk even further since 1999 (DOE, 2001). To pursue performance-based contracting effectively, DOE should either (1) take steps to increase the contractor base in order to carry out the second method or (2) fall back on the first method and become an owner knowledgeable about costs and performance (or both). Execution of the first method is very difficult if only the contractors know the costs. FINDINGS AND RECOMMENDATIONS Finding. The extent of training and use of PBC in DOE contracting efforts is unclear. There is no DOE-wide database that shows the extent of use of PBC or the number of staff trained in PBC techniques. Recommendation. The committee reaffirms the recommendations made in previous reports (NRC 1999, 2001) on using PBC and encourages OECM to play a lead role in supporting this practice. OECM should work closely with the Office of Procurement and Assistance Management to see that PBC training is provided as part of the career development process for project management personnel and
OCR for page 73
Progress in Improving Project Management at the Department of Energy: 2001 Assessment just-in-time training for the IPT. In the near term, OECM should bring on board a cadre of experts, skilled in performance-based contracting, to provide technical assistance to IPTs responsible for new major system initiatives. Finding. The draft Program and Project Management (PPM) manual and draft Project Management Practices (PMP) developed by the OECM fail to address PBC adequately. Recommendation. The detailed descriptions of PBC alternatives and their application to DOE projects should be included in the revised PMP and PPM. Finding: There have been continuing efforts on the part of DOE to move toward a more effective use of PBC methods and to support these efforts. Recommendation. Contract approaches should be tailored to use fixed-price and performance-based methods where practicable to assist the DOE to get the most cost-effective results and to stimulate competition. In addition, the department should continue to explore other innovative commercial contracting approaches to meet its needs. REFERENCES DOE (Department of Energy). 2000. Program and Project Management for the Acquisition of Capital Assets (Order O413.3). Washington, D.C.: Department of Energy. DOE, Contract Reform and Privatization Project Office. 2001. Analysis of the DOE Contractor Base: Readiness, Willingness, Profitability, and Trends: A Focus on the Environmental Management Program. Washington D.C.: Department of Energy. EOP (Executive Office of the President). 1998. A Report on the Performance-Based Service Contracting Pilot Project. Washington, D.C.: Executive Office of the President. GAO (General Accounting Office). 2001. Major Management Challenges and Program Risks: Department of Energy. GAO/GAO-01–246. Washington, D.C.: Government Printing Office. NRC (National Research Council). 1998. Assessing the Need for Independent Project Reviews in the Department of Energy. Washington, D.C.: National Academy Press. NRC. 1999. Improving Project Management in the Department of Energy. Washington, D.C.: National Academy Press. NRC. 2001. Improved Project Management in the Department of Energy. Letter report, January. Washington, D.C.: National Academy Press.
Representative terms from entire chapter: