Click for next page ( 58


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 57
9 Acquisition and Contracting INTRODUCTION The committee in each of its previous reports noted the critical relationship between the use of good acquisition practices and successful project management (NRC, 2001a, 2001b, 2002~. Given that contractors perform the preponderance of the agency's work, special attention must be given to three critical aspects of the acquisition process: Creating early on an effective and complete strategy for selecting a con- tractor that can later be updated and refined, Developing a clear, performance-based contracting plan that allows an objective means for measuring results, and Establishing an effective set of incentives to align contractor performance with DOE goals and outcomes. The committee continues to emphasize the importance of sound up-front acquisition planning, performance-based contracting, and the effective use of incentives as key elements in a successful project management process. DOE has taken positive steps over the past year in each of these areas. The following sections discuss its actions regarding these acquisition methods and provide find- ings and recommendations for each. 57

OCR for page 57
58 PROGRESS IN IMPROVING PROJECT MANAGEMENT AT THE DOE DEVELOPING AN ACQUISITION STRATEGY Chapter 5 of the August 2002 version of the draft Program and Project Management (PPM) manual, entitled "Definition," discusses the project defini- tion phase (DOE, 2002~. This phase includes activities that occur between the approval of mission need (CD-0) and the approval of system requirements and alternatives (CD-1~. A major focus of this early phase is the development of an acquisition strategy that will be used to guide the project throughout all of its subsequent phases. As such, it provides a foundation for the overall project. A draft of the strategy is needed even before mission need is agreed to, but this draft of the strategy will be refined as the project progresses. The PPM states that the strategy should address the following topics: Requirement (including a summary project description); Project structure (including an organization chart and a listing of acquisi- tion steps); Risk assessment (schedule, cost, technical as well as mitigation strategies); Approach to managing program/project cost and performance; Acquisition trade-offs and streamlining (including a discussion of the pros and cons of alternative acquisition approaches); Project management (general philosophy and approach); Support concepts and IT strategy; Business and contracting strategy (including ensuring maximum competi- tion for an award, contract types, and incentives); and Other important considerations (i.e., other agency involvement). For each of the items noted, the PPM provides additional guidance on ensuring that critical issues that may affect project performance are thought through early on and continue to remain a focus of subsequent efforts. In addition to providing useful guidance to project management staff on issues to consider in developing an acquisition strategy, the PPM includes the Acquisition Strategy Approval Form. This form is to be signed by the project manager and by various senior staff up to and including the agency acquisition executive. DOE Order 413.3 requires approval of mission need and acquisition plans by senior department management (DOE, 2000~. The deputy secretary approves projects of $400 million or higher, the under secretaries approve projects of less than $400 million and may delegate this authority to the assistant secretaries/ program secretarial officers as they deem appropriate. As part of this process, the Office of Engineering and Construction Management (OECM) reviews draft project acquisition strategy documents and provides comments back to project staff. Although OECM does not have sign-off authority for the documents, it provides recommendations to the director of the Office of Management, Budget and Evaluation (OMBE) on whether projects should be allowed to proceed or if

OCR for page 57
ACQUISITION AND CONTRACTING 59 deficiencies in documentation and justification need to be corrected. As such, OECM staff has worked informally with project staff to address issues before the project goes through the formal Energy Systems Acquisition Advisory Board (ESAAB) process. From December 2001 through May 2002, OECM reviewed 21 acquisition strategy documents, and 9 proceeded on for formal approval. The committee examined some of the draft acquisition plans submitted by various DOE compo- nents. These included, among others, plans for the development of a materials facility for special nuclear materials, a technology center upgrade construction project, and a communications infrastructure modernization program. These documents reveal a significant amount of inconsistency in project team responses to the specific items listed in the PPM as part of an acquisition strategy. For example, in some, risks are detailed as well as efforts to mitigate them. In others, risks are mentioned but there is no discussion of how they are to be handled. Also, technical risk is addressed but business risk ignored in some of these acquisition strategies. Other shortcomings that have been observed as a part of the OECM's review include risk plans that were 3 years old when submitted and therefore very likely out of date, with contractors, as opposed to federal managers, defining the acqui- sition strategy and outcomes. In addition, while some integrated project teams (IPTs) were identified, they had not begun to function as real teams. Acquisition strategies and, in particular, risk-management plans cannot be effectively devel- oped without the active participation of all project interests (i.e., the IPT). Finally, the draft documents have been weak on identifying the pros and cons of alterna- tives to be considered as part of the critical decision on mission need, and frequently project costs at the very earliest stage are presented as point estimates rather than as a likely range of costs that would later be refined. Another area that requires special attention in developing an acquisition strategy is the ability to attract a sufficient number of bidders to ensure an accept- able level of competition for awards. Recently there has been a decline in the bidder pool and a concentration of DOE contracts among a very few large con- tractors. This concentration has increased over time and is worse than the concentration observed in the 1999 NRC report (NRC, 1999~. Draft acquisition plans should focus on ways to increase these pools to maximize the benefits of competition for DOE procurements. DOE should address the problem of inade- quate and declining competition in all programs. This is especially critical for an agency that is so highly dependent on contractors to function. DOE should take remedial actions and set goals for increasing the bidder pools for DOE projects. It is apparent that the project teams remain somewhat unclear about what constitutes an acquisition plan that would allow a project to proceed toward successful completion. The core elements of the acquisition plan should be a full description of the planning rationale developed by the IPT, a cost and schedule range that should be integrated with the budget, a clear focus on competition, and

OCR for page 57
60 PROGRESS IN IMPROVING PROJECT MANAGEMENT AT THE DOE a risk assessment and mitigation plan relevant to each project stage. These elements will help connect the project acquisition and financial management aspects of the project right from the start. The OECM review process has clearly been useful in helping teams develop the appropriate level of detail for each of the critical elements of the acquisition strategy. As such, the process is forcing a greater level of front-end planning than was the case in previous projects the committee has reviewed. However, and perhaps because the process is new, the response to critical decision requirements seems to be more a matter of compliance than of using analytic tools to strengthen overall project management. The committee advocates the use of risk analysis and other analytical tools not merely for the sake of analytic rigor but rather because the analysis, if properly performed, forces the IPT to identify and con- front the critical issues facing the project. The committee believes that as more acquisition strategies are developed and as more staff become familiar with the benefits of addressing these issues early in the project, support for these manage- ment tools will increase. PERFORMANCE-BASED CONTRACTING It is vital that DOE project management staff be well versed in techniques that allow them to identify key outcomes to be achieved through contractor efforts, to develop performance incentive targets, and to monitor and oversee contractor performance. One contracting technique in particular performance- based contracting (PBC) has been integral in shifting the overall focus of the acquisition process from process to outcomes. The committee strongly recom- mended in its previous reports that more training be provided to staff on this methodology (NRC, 2001b, 2002~. Under a performance-based contract, acquisition is structured around the purpose of the work to be performed as opposed to how it is to be accomplished. Contractors should therefore be clear about the project objectives but should also have more flexibility in determining how to meet the government' s requirements. Such flexibility would be more likely to interest contractors in bidding on DOE efforts, thereby addressing the issue of declining competition, referred to above. However, it is essential that the government continue to monitor contractor performance. Much of the success of this approach is dependent on effective and ongoing communications between the government and the contractor. The gov- ernment does not abdicate or delegate to a contractor its oversight responsibilities but rather works with the contractor to accomplish identified outcomes. The main point is that the judicious use of incentives can bring both the government's objectives and the contractor's objectives into complete alignment. When the government succeeds, the contractor succeeds. DOE included PBC as part of its contract reform agenda of the early 1990s and has been employing this approach in major projects at Rocky Flats, Oak Ridge, the Nevada Test Site, and elsewhere.

OCR for page 57
ACQUISITION AND CONTRACTING 61 The federal government has encouraged the use of this technique for more than a decade, and the Bush administration has given it even greater emphasis, making it a key element of the President's management agenda. In March 2001, the Office of Management and Budget (OMB) required that 20 percent of all services contracts for FY 2002 over $25,000 be performance-based. The President's management council monitors progress related to this goal. The committee's 2001 assessment described the basic components of a performance-based contracting template and defined how outcomes, performance metrics, quality assurance surveillance plans, and incentives all work together to produce a framework for contractor accountability (NRC, 2001~. As mentioned previously, a 1998 OMB study found both cost savings (of about 15 percent) and significant increases in customer satisfaction when government agencies shifted to performance-based contracts (EOP, 1998~. Even though PBC has been a longstanding and important aspect of many DOE contracts, the initial draft PPM made no mention of it. The committee is pleased to see that the August 2002 draft points out the importance of using performance-based contracting methods as a part of good project management. It also refers to a seven-step process that has been followed by a number of govern- ment agencies in conducting performance-based acquisitions (DOE, 2002~. How- ever, PBC is an integral part of project management, and as noted in the PPM, it is one of a number of techniques that join to produce a successful project. Of interest to the committee is how effective and objective performance metrics can be developed and used in accomplishing departmental objectives. An analysis sponsored by DOE's Office of Procurement and Assistance Management and completed in June 2002 provides some useful insight into this issue. The purpose of the analysis was to assess management techniques used by other agencies in contracts similar to those operated by DOE. One focus was the use of objective measures to assess and reward contractor performance. The categories of contracts from which comparable non-DOE contracts were selected include the following: Operation of complex government-owned industrial-type facilities, Operation of complex government-owned research and development facilities, Construction of unique facilities, and Environmental cleanup and remediation efforts. The reviewers also sought contracts that involved large sums of money, had multiyear commitments, and presented similar risks to those faced by DOE. In all, contracts awarded by the following agencies were explored in depth: The National Cancer Institute of the National Institutes of Health, in Frederick, Maryland;

OCR for page 57
62 PROGRESS IN IMPROVING PROJECT MANAGEMENT AT THE DOE . The Air Force's range technical support contract for Patrick Air Force Base, in Florida; The joint NASA/Air Force base operations and support contract at the Kennedy Space Center, Florida; The NASA Jet Propulsion Laboratory, in California; The Air Force Pacific Missile Range Facility, in Hawaii; The Army Tooele Chemical Agent Disposal Facility, in Utah; and The NASA Plum Brook Research Station, in Ohio. Virtually all of these contracts were in the hundreds of millions of dollars range and involved a host of support activities to be performed by the contractor. In every case, a cost-reimbursement contract was used, as is the case with DOE's management and operation (M&O) contracts. Moreover, six of the seven con- tracts used an award fee process for determining incentives. In this type of contract, the government sets a maximum fee amount for the contracted period and then determines after the fact how much of the fee should be awarded to the contractor. The seventh contract (Plum Brook) used an award term incentive approach. Under this model, superior performance does not result in a higher fee but in an extension of contract length. Each of the above contracts follows a similar process for determining an award fee. Basically, performance standards and their relative importance are conveyed to contractors at the beginning of the evaluation period, which gener- ally is 6 months. Teams of government officials are assigned to monitor and document contractor performance against these objectives, with performance information collected at the end of the period. An evaluation board then reviews the information and provides a recommendation on the fee to the fee-determining official, who is the ultimate decision maker. In each case, contractors are offered an opportunity to provide written comments or oral presentations to the evalua- tion committee. What was of concern in each of these contracts was the amount of subjectivity in the evaluation process. Some of the contracts identified spe- cific performance metrics (as, for example, safety or production performance at the Tooele depot for eliminating nerve agents). However, there was always an ability to modify the fee based on subjective considerations, using such ratings as "good" or "excellent." In one of the contracts, the contractor had to successfully achieve 87 metrics as a threshold requirement before any fee could be awarded. In another, 107 metrics were monitored and the results factored into the award fee process. But these were just factors to be considered in the overall award. Clearly, having 87 or 107 metrics does little to focus the contractor's attention on the critical project issues and turns performance-based contracting into an accounting exercise. Some of the arguments that were presented against relying on purely objec- tive metrics for fee determination are the following:

OCR for page 57
ACQUISITION AND CONTRACTING 63 What can be easily measured is not critical to the contract's execution. If this is the case, objective measures would fail to bring about the right outcomes. There are too many uncertainties to tie the fee to such measures. For example, new areas of emphasis may arise after performance has started, or there may be unanticipated reductions in funding. Objective measures fail to account for unknowns. For example, an incen- tive for chemical disposal based solely on pure production would be dependent on knowing the liquid/gel proportion of nerve agents in shells that must be incinerated. Only then could a consistent rate of production be determined and the contractor held to that schedule. Objective measures do not allow the government to communicate its concerns if it believes the contractor is not being sufficiently responsive. While the committee believes that objective measures may not necessarily be applicable to all types of contracting efforts, it believes that many of these issues or complaints can be overcome with a diligent review of the basic tasks specified in the contract. There are ways to accommodate changes to the incen- tive schedule just as there are ways to accommodate changes to the project scope, budget, and schedule. These views inhibit greater use of performance-based techniques and help to explain the government's reluctance to adopt a more objectively determined fee process. One of the positive aspects of using a performance-based approach was that such measures allow a better understanding of up-front expectations on performance and outcomes for all parties, while reducing the impact of unsubstantiated information on the award fee process. INCENTIVES The use of incentives or disincentives has been an effective contracting tool to focus the contractor and the government on the overall outcome being sought. Both are key factors in the development of virtually any type of performance- based contract. An incentive or disincentive clarifies for all parties the priority that the government is placing on a particular aspect of the contractor's perfor- mance. Ordinarily, the government provides incentives to the contractor by offering a bonus for performance above and beyond satisfactory accomplishment of contracted tasks. This is usually accomplished through a subjective award fee process similar to the one described above or through a more objective performance- based process. Cost performance can also be incentivized. Generally this type of fee involves the contractor and government sharing, by a prearranged formula, in cost underruns (the amount by which actual costs are less than target costs) and overruns (the amount by which actual costs are more than target costs). Clearly, the government must be able to make reliable target cost estimates and not simply adopt those of the contractor. If the government lacks the experience and knowl-

OCR for page 57
64 PROGRESS IN IMPROVING PROJECT MANAGEMENT AT THE DOE edge to estimate cost targets (not just historical costs incurred) through activity- based costing, then it should seek independent cost estimates. An incentive contracting approach can include monetary and other types of rewards. In some cases, the contractor can be awarded outstanding past perfor- mance ratings that can be used by the contractor in bidding on additional work. In others, some agencies have been using an award term scheme that offers the contractor a contract extension, as described above for the National Aeronautics and Space Administration (NASA) Plum Brook facility. In a hazardous work environment, disincentives, such as loss of fee, can be used where outcomes related to the health and safety of workers or the public are important or where there could be serious environmental impacts. Disincentives are also frequently used where vital agency operations are at stake, such as maintaining computer network operations. In every case, if additional funds are to be provided for contractor performance, the government has an obligation to ensure that the outcome (in terms of costs or benefits) is commensurate with the incentive or disincentive employed. Contractor incentives should consider the value added for the government, not merely the costs to the contractor. The DOE has a long history of using incentives to direct contractor efforts in many of its major management and operating contracts. However, Under Secretary Robert Card recently adopted a new approach to the use of incentives for the department's major environmental cleanup efforts. Under this approach, DOE is setting new funding priorities based on which sites can achieve closure most cost effectively by accelerating operations. By focusing on and incentivizing accelerated closure, and by rewarding contractors for task completion rather than ongoing efforts, DOE is sending a strong message to all of its environmental cleanup contractors to align their goals with those of the department. The committee applauds these efforts. As the department increases its emphasis on closure, it will be required to reassess funding priorities and redirect funds to those projects offering the great- est opportunity for success. In this restructuring process, the department needs to ensure that the necessary coordination is carried out with state and other regula- tors. In addition, it should reassess incentives already in place, particularly those focused on interim progress and goals, to determine whether they are still mean- ingful. The River Corridor project at DOE's Hanford site provides a good opportunity to evaluate the effectiveness of DOE's new performance-based, closure-incentivized policy. FINDINGS AND RECOMMENDATIONS Acquisition Strategy Finding: The committee believes that the August 2002 draft PPM provides a good framework for addressing acquisition strategy issues and offers a useful

OCR for page 57
ACQUISITION AND CONTRACTING 65 model for DOE project managers to follow in preparing and planning their efforts. However, it finds that the acquisition strategy documents being reviewed by OECM are of mixed quality and believes that this indicates a need for more training and development of additional reference documents. The iterative process of review and correction will also improve the overall quality of planning docu- ments over time. The most recent draft PPM now provides clear and consistent guidance on what needs to be addressed in each draft acquisition strategy and, as such, should significantly increase the quality of the documents submitted for review. Recommendation: The committee recommends that senior management con- tinue to require project teams to focus on up-front acquisition planning and that it continue to use the approval process for ensuring compliance and consistency. DOE management should return documents that do not meet management expec- tations and should follow up by asking why these inadequacies were not fixed at lower levels. Project teams should be trained in developing effective acquisition strategies. DOE leadership should also continue to focus on competition to obtain a range of innovative approaches from a variety of contractors to meet its management, operating, and development needs. Performance-Based Contracting Finding: For large cost-reimbursement contracts, many factors compromise the ability of the government to use purely objective measures for assessing perfor- mance. Moreover, federal agencies are comfortable using a more traditional cost reimbursement award fee approach, in which the award fee is at the discretion of the federal project manager. Recommendation: A significant amount of up-front planning by the IPT is needed to specify outcomes and identify those aspects of an overall project for which a contractor can effectively be held accountable. The committee reiterates its recommendation that training in performance-based contracting methods be provided to IPT members. In addition, DOE should collect best practices infor- mation on the use of performance-based contracting in DOE contracts and iden- tify those activities most suitable for use of these metrics. Incentives Finding: DOE is reassessing its cleanup efforts, giving them a new focus on cost-effective and rapid closure of sites, and setting up incentives that can best achieve that goal.

OCR for page 57
66 PROGRESS IN IMPROVING PROJECT MANAGEMENT AT THE DOE Recommendation: DOE should reassess its use of incentives in existing con- tracts to ensure that they focus on closure and that interim goals are effective in driving this overall objective. REFERENCES DOE (Department of Energy). 2000. Program and Project Management for the Acquisition of Capital Assets, Order O 413.3. Washington, D.C.: Department of Energy. DOE. 2002. Program and Project Management. Draft. Washington, D.C.: Department of Energy. EOP (Executive Office of the President). 1998. A Report on the Performance-Based Service Con- tracting Pilot Project. Washington, D.C.: Executive Office of the President. NRC (National Research Council). 1999. Improving Project Management in the Department of Energy. Washington, D.C.: National Academy Press. NRC. 2001a. Improved Project Management in the Department of Energy. Letter report, January. Washington, D.C.: National Academy Press. NRC. 2001b. Progress in Improving Project Management at the Department of Energy, 2001 Assess- ment. Washington, D.C.: National Academy Press. NRC. 2002. Progress in Improving Project Management at the Department of Energy, 2002 Interim Assessment. Letter report, May. Washington, D.C.: National Academy Press.