|This page in the original is blank.|
A Evidence of Problems with DOE Project Performance
A variety of organizations have documented the cost and schedule problems associated with U.S. Department of Energy (DOE) projects and programs. A few have provided solid evidence to validate the existence of these problems, and some have offered recommendations for solutions. The committee reviewed documents produced by a variety of organizations:
- reports by the General Accounting Office (GAO)
- reports by the DOE Office of Inspector General
- independent external reviews of DOE projects
- statistical analyses by Independent Project Analysis, Inc., for DOE
- project histories
Reports by the General Accounting Office
GAO, which examines matters relating to the receipt and disbursement of public funds, has performed numerous audits and evaluations related to DOE construction, procurement, and contracting practices. Many of these reports have been focused on contractor performance and the appropriateness of contractor costs. In its Performance and Accountability Series, GAO recently summarized more than 15 reports that found problems in the following areas: the completion of large projects; modifications of DOE's organizational structure to correct problems; DOE contracting practices; and technical and managerial skills (GAO, 1999). The reports note the challenges facing DOE in carrying out its mission, DOE's performance, and efforts to meet these challenges and implement reform.
A few of the reports are directly related to project performance, but most are indirectly related, focusing more on organization and management. The most relevant are described below.
In 1996, GAO reported on DOE's difficulties in delivering major projects in keeping with baseline costs, schedules, and scope (GAO, 1996a). Of the 80 major systems projects initiated between 1980 and 1996, only 15 were completed, many of them behind schedule and over cost; 31 were terminated before completion. GAO concluded that there were four causes of failure:
- a poor system of incentives for contractors
- insufficient DOE personnel with the skills to oversee contractor operations
- DOE's poorly defined or changing missions
- the incremental funding of projects
GAO cited frequently deficiencies in DOE's contracting practices. In 1990, GAO concluded that DOE's contracting practices were at high-risk for fraud, waste, and abuse (GAO, 1997a). Although GAO has commended DOE's efforts to reform its contracting since 1994, it continues to monitor contract management as a high-risk area and has encouraged DOE to do more to increase competition among bidders (GAO, 1996b), link contractor fees to performance (GAO, 1998a), and determine the best contracting type and strategy to use for each project (GAO, 1997b, 1998b, 1998c).
The effect of organizational structure on project performance has been another area of focus for GAO. The absence of clear lines of authority or defined roles and responsibilities throughout the department has made it difficult to demand accountability from contractors or staff (GAO, 1997c).
Office of Inspector General
The mission of the DOE Office of Inspector General (OIG) is to promote effective, efficient, and economical operations of DOE programs through audits, inspections, investigations, and other reviews. The OIG has continually advised DOE headquarters and field office managers of the significant issues affecting project performance. The major areas where improvements in efficiency and effectiveness can be made are described in the following section.
DOE's contract administration has many weaknesses. Recent reports have noted problems in the development and use of performance measures in various management contracts. In one case, the Nevada Operations Office did not establish performance milestones until the contractor had completed the work
(DOE, 1997a). The OIG found that the Rocky Flats Field Office had approved a contractor's cost reduction proposals that did not meet DOE's basic criteria for reducing cost by using innovative practices. The OIG also found that this contract included performance measures that were not supported by objective data, were not structured to encourage and reward superior performance, and were often focused on the process rather than the results of the process (DOE, 1997b). In 1997 OIG reviewed the Cost-Reduction Incentive Program with Westinghouse Savannah River Company and found that most of the savings identified by the program could not be attributed to innovative changes in work methods or processes. The internal assessment team recommended that the Savannah River Operations Office either modify the program so that payments were made only for innovative ideas or cancel the program and initiate performance-based incentives that would reward cost savings above an established threshold. Although several options were available, OIG found that the operations office did not act on the suggestions in the internal assessment (DOE, 1998a).
Architecture and Engineering Costs
The OIG has issued several reports over the past few years criticizing the high cost DOE pays for architecture and engineering (A-E) services. The OIG concluded that the costs of A-E services at six locations averaged more than twice those of industry for comparable projects (DOE, 1990). In a report on 65 conventional construction projects at the Idaho National Engineering Laboratory (INEEL), the OIG found that the costs incurred were $5.8 million higher than comparable industry standards (DOE, 1996a). In a recent review of projects by Sandia (SNL) and Los Alamos National Laboratories (LANL), the OIG found that Sandia's A-E costs were reasonable in comparison to adjusted industry standards, but Los Alamos' costs were 65 percent, or $2.5 million, higher than the adjusted standards for the seven projects reviewed. The success at Sandia was attributed to cost-control measures and competition among firms for A-E procurement. Higher costs at Los Alamos were attributed to inadequate and ineffective cost controls and performance measures (DOE, 1998b).
DOE has not established mechanisms for controlling changes in the costs, schedules, and scope of projects. Some projects were constructed without a full assessment of alternatives, changes to the mission, or realistic budgets (DOE, 1996b). An audit of renovation and new construction projects at Lawrence Livermore National Laboratory found that the laboratory could not demonstrate that it had selected the best alternatives for meeting the DOE's mission needs while minimizing costs (DOE, 1997c).
Independent External Reviews of DOE Projects
DOE is currently contracting for independent reviews to assess the quality of the technical scope, cost estimates, schedules, and supporting data for DOE projects in accordance with congressional language and criteria provided by the National Research Council (NRC, 1998). The findings and recommendations of the initial reviews transmitted to Congress have documented deficiencies similar to those found by this committee and others. The results to date have indicated that DOE's problems with project performance continue. The findings range from project-specific deficiencies to overall issues and concerns with DOE policies and practices related to project delivery and management. The findings of the 10 independent project reviews that have been completed so far are summarized below.
- The Spent Nuclear Fuel Dry Storage Project, at INEEL, is a "privatization" project with a total project cost (TPC) of $105 million. The independent reviewer found many deficiencies in planning, execution, and contracting of this project, including the omission of a project execution plan and associated schedules. INEEL had not sent formal documentation to DOE headquarters to alert the department that the project was likely to exceed its TPC. INEEL had no formal procedures for selecting the privatization contract form, and the reviewers found no record of an analysis underlying that decision. Finally, INEEL did not have sufficient project staff because funding for project staff must come from operating budgets rather than project budgets for privatization projects (Lockwood-Greene Technologies, 1998).
- The Stockpile Management Restructuring Initiative Project at DOE's Pantex facility has many weaknesses in budgeting and planning. The schedules for some of the subprojects were "extremely conservative," and cost estimates for many of the more routine demolition and construction activities were "very high." Too many layers of management were involved in approving "small, straightforward projects." The reviewers found that 8 of the 12 subprojects lacked adequate justification for proceeding (questionable economic benefits, reduced operational flexibility, or failure to take account of changing circumstances) (Foster Wheeler Environmental Corporation, 1998a).
- An independent review of the Stockpile Management Restructuring Initiative at the Kansas City Plant called it "a well run, well managed project." The review, however, did identify that an independent review of the project's cost estimate found that level-of-effort estimating may have been used rather than an activity-based approach. The assessment also identified $3.2 million worth of start-up engineering that was not included in the original cost estimate (Foster Wheeler Environmental Corporation, 1998b).
- The Nuclear Materials Storage Facility Renovation project of LANL had a number of deficiencies in cost estimating and scheduling. At the time of the review, there was no baseline total cost estimate document, and the construction schedule needed to be revised to support the completion of the project cost estimate. The review team also found that the LANL did not have a comprehensive plan integrating all stockpile management projects and programs (CETROM, 1998a).
- CETROM Consulting Engineering, Inc., also reviewed the Rapid Reactivation Project initiated to protect the limited life component manufacturing capabilities of SNL, LANL, and the Kansas City Plant. The review found certain deficiencies although the project was generally successful. The schedule contingency allowances were too small management documents were not complete; LANL did not have a project execution plan (PEP); and the PEP for the Kansas City Plant needed to be updated (CENTROM, 1998b).
- The review of the Nuclear Materials Safeguards and Security Upgrades Project at LANL found that although a detailed schedule risk analysis was performed as part of the critical decision review process, risks such as construction performance and the potential impacts of ongoing changes to the project management process at LANL had not been taken into account. The reviewers also noted that although LANL was to be a subcontractor, there was no documentation to hold LANL accountable for the cost and schedule of its deliverables (CETROM, 1998c).
- The Jupiter Corporation reviewed the Pit Disassembly and Conversion Facility Project of the DOE Office of Fissile Material Disposition and found that the project managers had not adequately considered the applicability of Nuclear Regulatory Commission licensing standards and other regulatory requirements, although Congress requires that DOE submit to those standards. In addition, the project had failed to document the analysis supporting its choice of contract type (Jupiter Corporation, 1998a).
- An independent review of the Dual Axis Radiographic Hydrotest Facility Project found it to be generally successful and well managed. The reviewers suggested that a simpler project management process be adopted and that procurement documentation be better coordinated among the national laboratories involved with the project (Jupiter Corporation, 1998b).
- The independent review of the Chemistry and Metallurgy Research Facility Upgrades project of LANL (which was suspended in 1997 because of rapidly growing costs and scope and was partly resumed in 1998) found serious weaknesses. The reviewers found that the original cost, scope, and schedule baselines had not been changed to meet current DOE missions, and they recommended that new baselines be established. They also observed that DOE management at DOE headquarters, disregarding stated
- policy, had inserted itself into the process for approving changes in the baseline (Jupiter Corporation, 1998c).
- An independent review of the Atlas Project to improve simulation of nuclear weapons performance at LANL found that the project was unlikely to be completed within the baseline cost estimate. The reviewers noted that up until very recently, DOE had focused on the technical challenges of the project and the design changes precipitated by the change in mission for the project. Consequently project management and cost had been given little attention. The emphasis is now on meeting the TPC of $48.9 million although no updated project-specific risk and uncertainty analysis has been conducted since the 100 percent draft CDR was issued (Cadmus Group and Project Performance Corporation, 1999).
Statistical Analyses of DOE Projects
Three statistical analyses of DOE project performance in the areas of environmental remediation (ER) and waste management (WM) were performed by an outside contractor, Independent Project Analysis, Inc. (IPA): Project Performance Study, November 30, 1993 (IPA, 1993); Project Performance Study, Waste Management Addendum, December 1995 (IPA, 1995); and Project Performance Study Update, April 1996 (IPA, 1996). These studies involved statistical comparisons of DOE ER and WM projects with a database of comparable projects performed by private industry and other government agencies (primarily the U.S. Army Corps of Engineers), including multivariate regression models derived from these data. Although the committee was unable to inspect the models developed and used by IPA in these analyses, these Project Performance Studies are believed to be the only cross-sectional and longitudinal analyses of DOE project performance as a whole. The results of these analyses have not been challenged by DOE, and in fact were the basis for a two-day "stand-down" initiated by the secretary of energy and assistant secretary for environmental management on January 26 and 27, 1994 (Improving Project Performance: A Federal Hands-on Initiative) (IPA, 1994). The Project Performance Study Update in April 1996 was commissioned by the assistant secretary in an attempt to demonstrate progress made as a result of the EM stand-down.
Although the projects, sample sizes (76 projects in December 1990, 65 projects in November 1993, 22 projects in December 1995, and 48 projects in April 1996), statistical models, and database of industrial projects used for comparison all varied from study to study, some common themes emerged. The studies focused on four factors:
- cost performance, or the absolute cost of DOE projects compared to cost for projects by industry and other government agencies, normalized for comparability
- cost overruns, or the relative increase of DOE project costs compared to the original budgets
- schedule performance, or the absolute time duration of DOE projects compared to the duration of projects by industry and other government agencies, normalized for comparability
- schedule slippages, or the relative increase of DOE project durations compared to the original schedules
Project Costs and Cost Overruns
Combining the results of the Project Performance Study of November 30, 1993 (IPA, 1993), with the Waste Management Addendum of December 1995 (IPA, 1995), DOE WM projects cost an average of 48 percent more than comparable projects performed by industry and other government agencies; and DOE ER projects cost about 33 percent more. On the same basis, the average cost overruns for ER projects were about 48 percent and for WM project's about 42 percent compared to an average of about 3 percent for industry and other government agencies. Moreover, the variability in cost growth from project to project for DOE was much higher than industry. Thus, not only did DOE projects cost roughly 40 percent more than comparable industrial projects (the "DOE tax"), they also overran their initial cost estimates by about 45 percent, indicating that DOE either has a problem controlling costs or a problem estimating costs. The Project Performance Study Update in April 1996 (IPA, 1996) stated that "We expect that the DOE WM '96 project (sic) will average 33 to 43 percent more than the private sector. Additionally, we expect that cost overruns will be lower . . . . about 22 to 36 percent for the DOE WM '96 project set" (p. 80). For ER projects, the same report stated, "We expect DOE ER's average cost to go from 33 percent more than the private sector to 25 percent more" (p. 119). However, the ER cost overruns were not expected to change; "DOE ER projects are still likely to have 50 percent cost growth. . . ." (p. 123).
Project Durations and Schedule Slippages
Combining the results of the Project Performance Study (IPA, 1993), with the Waste Management Addendum (IPA, 1995), DOE WM projects took an average of three times longer to complete them comparable projects by industry and other government agencies, and the original schedules slipped an average of "about 22 months, or 52 percent" (IPA, 1996, p. 80), compared to an average of 17 percent in industry. Thus, even though DOE WM initial project schedules were very long compared to similar projects done by others, they nevertheless slipped more. The Project Performance Study Update of April 1996, however, projected that "the average slip [was to be reduced] to about 15 months, or 35 to 43 percent. The reduced schedule slip should reduce the relative schedule
duration of DOE WM projects from 300 percent of private sector norm (i.e., three times as long as the private sector) to 250 to 280 percent of the private sector norm. This is still too long . . . " (p. 80). The DOE ER project schedule performance was slightly better, with durations only about 18 percent longer than comparable projects, but the average schedule slippage was about 42 percent. In the 1996 Update, IPA expected "DOE ER schedules to speed up slightly, from 18 percent slower than the private sector to 15 percent slower" (p. 121). However, the average ER schedule slip of 42 percent was expected to "remain about the same" (p. 124).
Thus, the 1996 Update expected some improvements in costs and durations compared to the previous studies, but these were extrapolations because the projects in the study had not yet been completed. No follow-up study has been made since April 1996 to determine whether these expectations were realized, but even with these projected improvements, DOE project costs and schedules would be very much higher than for comparable projects in industry.
In addition to the outcomes or dependent variables (project costs, overruns, durations, and slippages), the IPA studies also identified some causal factors or independent variables that influenced these outcomes. According to the DOE DP Project Cost Growth Study of December 1990 (IPA, 1990):
The cost growth of these projects is distorted by the frequent use of scope as a contingency. Of the 59 projects that could be analyzed for this, 11 (nearly 20 percent) used an average of 12 percent of their actual expenditures for items outside of the original intent of the projects. Seven of the projects (over 10 percent) decreased their actual costs by an average of 10 percent by reducing the scope from that which was authorized. . . . An example of the type of discretionary scope change would be if a laboratory project, which was to have included equipment purchases, met with a favorable bidding climate. The remaining money might be spent on equipment purchases beyond those initially planned (p. 19).
Even though these observations were made in 1990, there is no reason to believe that the situation had changed by 1998; according to a presentation made by Dr. David A. Gottschlich of IPA to this committee on August 3, 1998, "Scope reduction is a primary method for cost control." Moreover, the DOE Inspector General (Audit Report: The U.S. Department of Energy's Value Engineering Program, HQ-B-98-01, July 1998, [DOE, 1998c]) reported that the Chicago operations office reported as savings from value engineering "about $1.2 million . . . from refining cost estimates and project scopes when contract bids exceeded available funding" (p. 2). Scope adjustments are likely to bias project costs upward, on the average, even if the original cost estimates were unbiased because its easier to add reasons to spend any underruns.
Project definition in DOE projects has been a continuing problem. According to the Project Performance Study of November 30, 1993, "The statistical analysis shows that project definition [at the completion of design] accounts for 50 percent (r2 = 0.50) of the cost growth variance [in environmental remediation projects]. This relationship is statistically significant at the 0.0001 level" (IPA, 1993, p. 47). In other words, for these projects, the level of project definition explains more of the variance in cost growth than all other variables put together. IPA used a composite front end loading index (FEL) comprised of a combination of several parameters (Gottschlich, 1998). Based on the results of the Project Performance Study of November 30, 1993 (IPA, 1993) and the Waste Management Addendum of December 1995 (IPA, 1995), "DOE WM [waste management] project definition is poor [with] only 6 percent design complete at authorization vs. 15 percent design complete for industry." Moreover, "[the] FEL index of complex projects is worse than [the] FEL of routine projects," indicating an inversion: more complex projects are authorized with less design definition than routine projects.
The 1996 Update (IPA, 1996) showed some improvement: "At baseline, DOE WM process projects in the sample fell into the Screening Study [the lowest] category, substantially short of the Industry average; since the stand-down, their definition has improved from Screening Study to Poor" (p. 41). However, "the project definition improvement is about one-quarter of what is needed. . . ." (p. 43). For WM projects, "site definition improved, but engineering status . . . improved only slightly" (p. 47), and "there has been negligible improvement [in project execution planning] since the Stand-Down" (p. 49). For ER projects, the 1996 Update (IPA, 1996) found that "the level of front end loading attained by DOE's remediation projects is not improving" (p. 89), and there was no improvement in site definition, engineering definition, or project execution planning (p. 91). As noted above, project definition is highly correlated with cost growth. Of the DOE premium for "WM projects . . . an average of 48 percent more than the same designs executed by the private sector," eleven percentage points of this 48 percent premium could be eliminated if the average DOE project definition index were equal to the average for the private sector. That is, a WM project that would cost $10 million in the private sector would cost DOE $14.8 million, but this could be reduced by $1.1 million if the DOE project definition were equal to the average project definition in the private sector. As the incremental engineering cost of improving this project definition would be far less than $1.1 million, this improvement would have substantial benefits to cost ratio. Nevertheless, as IPA summarized the situation after the 1996 Update (IPA, 1996), "we do not expect to see a significant change in these metrics: no significant improvement in FEL" (Gottschlich, 1998).
The 1996 IPA Update indicates that "the estimated cost to conduct . . . value engineering . . . [is] about 0.5 percent of the accepted savings" (p. 53). In the DOE WM projects, "although value engineering was used on almost half of the projects, we expected more frequent usage. . . ." The DOE Inspector General agreed with the benefits to cost ratio: "The Corps [of Engineers] has reported a $20 return for each $1 spent on the VE [value engineering] effort. . . .[and] the General Accounting Office . . . reported that VE usually produces a net savings of 3 to 5 percent of project costs" (DOE, 1998c, p. 1). Moreover, "Public law 104-106 and Office of Management and Budget (OMB) Circular A-131 require Federal agencies to use VE." Nevertheless, the DOE IG concluded, as of July 1998, that "The Department had not fully developed and implemented an effective VE program" (DOE, 1998c, p. 2), and documented savings from value engineering were less than 3 percent of project costs.
Project Team Turnover
The DOE WM projects in the baseline sample experienced a high rate of project manager turnover (over 80 percent) during execution. . . . This is nearly twice the turnover levels experienced by Industry. . . . Nearly 70 percent of the projects in the DOE WM '96 project set have already [that is, in April 1996] experienced a change in their project managers; since only one of these projects was complete, we expect that the incidence of turnover in the current set of projects will exceed that of the baseline (IPA, 1996, p. 63).
In short, DOE project manager turnover was getting worse in 1996.
DOE Involvement at Project Level
In the private sector, virtually every project is run by a project manager from the owner's organization. In those very rare cases-less than 2 percent-where the project manager is from the contractor's organization, the owner almost always has a strong presence in the project team. Those projects without good owner control perform distinctly worse as a class than those with strong owner control. DOE WM projects are usually led by a project manager from the contractor organization and often have little or no DOE/owner representation on the project team (IPA, 1996, p. 65).
Compared to the 1993 Study and the 1995 Addendum, the 1996 Update showed a significant increase in the involvement of DOE personnel at the project level for WM projects but still much less than the average for the private sector projects (IPA, 1996, p. 66). Since then, DOE has emphasized privatization; the
effect of privatization on the involvement of DOE personnel at the project level is not known.
Histories of Specific Projects
Various analyses of DOE projects were either part of the preconstruction planning process or, in some cases, as postconstruction evaluations. A preconstruction predictive project analysis can be used as a basis for decisions on the potential project outcomes based on the identified risks. Postconstruction assessments tend to evaluate results so that the lessons learned can be used for future projects. The following project performance assessments included statistical analyses that benchmarked performance against other projects with similar characteristics. The committee felt that these projects were representative of DOE's project portfolio.
Hanford Waste Vitrification Project
A Project Risk Analysis of the Hanford Waste Vitrification Plant (IPA, 1990), was prepared in June 1990 by IPA for the DOE Office of Program Management. On the basis of statistical analysis using the IPA project database for comparison, this report recommended that DOE make the following changes:
- Increase the contingency allowance in the capital cost estimate from 19 to 39 percent to allow an even chance of avoiding a cost overrun. The three factors for additional contingency include: (1) project is not yet fully defined; (2) new technology is approximately one-fifth of the project cost; and (3) nuclear materials processing facilities usually require higher contingencies.
- The current schedule, accelerated by two years over the original schedule, is somewhat optimistic.
- The cold start-up schedule of eight months is optimistic.
- Production performance for this project during the first year is expected to be only 25 percent of design attainment.
The statistical risk analysis of this project indicated that "the chances of meeting or underrunning the $965 million estimate are less than 15 percent. There is an even chance of overrunning the estimated cost of $965 million by $155 million. There is a 16 percent chance of overrunning the $965 million cost estimate by $335 million" (p. 13). The "$965 million cost estimate" included a "base estimate" of $806 million and a 20 percent contingency of $159 million. That is, the risk analysis showed a less than 15 percent chance of meeting DOE's estimate, even including DOE's contingency, and about one chance in six of an overrun of more than $335 million.
The Hanford Waste Vitrification Plant was canceled in August 1996 after an expenditure of $418 million. According to the IPA risk analysis, this project was clearly in trouble. Nevertheless, the project proceeded.
Rocky Flats 881 Hillside Project
A Post Analysis of the 881 Hillside Project was performed by IPA in September 1994 for EG&G Rocky Flats (IPA, 1994), the site maintenance and operating (M&O) contractor. The goal of this ER project was to stop the migration of a groundwater plume contaminated with trichloroethylene (TCE) and heavy metals. The proposed solution was the design and construction of a 1,500-foot French drain to intercept the groundwater migrating down the 881 hillside and the development of an associated treatment facility. The project was authorized in 1988, reauthorized in 1990, and completed in April 1992, at a total cost of $11.31 million. "The 1988 estimate included approximately 13 percent contingency; the 1990 estimate included 11 percent contingency" (p. 3). In fact, the actual cost growth was 443 percent compared to the 1988 estimate, and 91 percent compared to the 1990 estimate.
The ex post statistical analysis showed that "The 881 Hillside remedial action was 4.41 times more expensive than the industry average cost for comparable work. . . The project results included a faulty design, excessive cost overruns, and poor cost performance relative to comparable projects performed by other organizations. . . . In addition to the poor project coordination, the project team experienced a lot of turnover. . . . The project overrun was due to a poor understanding of the project at its authorization points and to poor project practices. In other words, the scope of work was very poorly understood and that scope was performed badly. . . . In 1988 the project was poorly defined, but the project was authorized anyway. . . . The project team consisted of numerous organizations that were poorly coordinated. Turnover was a constant problem, both with companies and individuals at companies. . . . The remedial design was flawed and resulted in numerous change orders on fixed price contracts. . . . The project's contractual strategy appears to have been to distribute the work to as many organizations as feasible" (emphasis added) (IPA, 1994, pp. ii-iv).
According to the results of this report, the following recommendations were made:
- Ensure that projects receive a minimum level of FEL (project definition) at key project milestones.
- Ensure rigorous design reviews.
- Do not use diverse contracting strategies for poorly defined projects.
- Set cost contingencies in line with project risks.
Cadmus Group and Project Performance Corporation. 1999. External Independent Review: The Atlas Project. Defense Programs Project No. 96-D-103. Final report prepared for the Office of Field Management, U.S. Department of Energy.
CETROM (CETROM Consulting Engineering Corporation). 1998a. External Independent Review: Nuclear Materials Storage Facility Renovation. Defense Programs Project No. 97-D-122. Final report prepared for the Office of Field Management, U.S. Department of Energy.
CETROM. 1998b. External Independent Review: Rapid Reactivation Project. Defense Programs Project No. 99-D-122. Final report prepared for the Office of Field Management, U.S. Department of Energy.
CETROM. 1998c. External Independent Review: Nuclear Materials Safeguards and Security Upgrades, Defense Programs Project No. 97-D-132. Final report prepared for the Office of Field Management, U.S. Department of Energy.
DOE (U.S. Department of Energy). 1990. Department-wide Audit of Architecture and Engineering Design Costs. DOE/IG-0289. Washington, D.C.: U.S. Department of Energy, Office of Inspector General.
DOE. 1996a. Audit of Architecture and Engineering Costs at the Idaho Engineering National Laboratory. DOE/IG-0387. Washington, D.C.: U. S. Department of Energy, Office of Inspector General.
DOE. 1996b. Special Report on the Audit of the Management of Department of Energy Construction Projects. DOE/IG-0398. Washington, D.C.: U.S. Department of Energy, Office of Inspector General.
DOE. 1997a. Report on Audit of the Contractor Incentive Program at the Nevada Operations Office. DOE/IG-0412. Washington, D.C.: U.S. Department of Energy, Office of Inspector General.
DOE. 1997b. Report on Audit of the Contractor Incentive Programs at the Rocky Flats Environmental Technology Site. DOE/IG-0411. Washington, D.C.: U.S. Department of Energy, Office of Inspector General.
DOE. 1997c. Audit of Renovation and New Construction Projects at Lawrence Livermore National Laboratory. WR-B-97-06. Washington, D.C.: U.S. Department of Energy, Office of Inspector General.
DOE. 1998a Audit Report, The Cost Reduction Incentive Program at the Savannah River Site. ERB-98-08. Washington, D.C.: U.S. Department of Energy, Office of Inspector General.
DOE. 1998b. Audit Report, Architect and Engineering Costs at Los Alamos and Sandia National Laboratories. August 1998. DOE/IG-0424. Washington, D.C.: U.S. Department of Energy, Office of Inspector General.
DOE. 1998c. Audit Report, DOE's Value Engineering Program. HQ-B-98-01. Washington, D.C.: U.S. Department of Energy, Office of Inspector General.
Foster Wheeler Environmental Corporation. 1998a. External Independent Review: Stockpile Management Restructuring Initiative Project, DOE Pantex Facility. Defense Programs Project No. 99-D-128. Final report prepared for the Office of Field Management, U.S. Department of Energy.
Foster Wheeler Environmental Corporation. 1998b. Independent Assessment: Stockpile Management Restructuring Initiative, Kansas City Plant. Report prepared for the Office of Field Management, U.S. Department of Energy.
GAO (General Accounting Office). 1996a. Department of Energy: Opportunities to Improve Management of Major System Acquisitions. Report to the Chairman, Committee on Governmental Affairs, U.S. Senate. GAO/RCED-97-17. Washington, D.C.: Government Printing Office.
GAO. 1996b. Department of Energy: Contract Reform is Progressing, But Full Implementation Will Take Years. GAO/RCED-97-18. Washington, D.C.: Government Printing Office.
GAO. 1997a. Department of Energy Contract Management. GAO High-Risk Series, GAO/HR-97-13. Washington, D.C.: Government Printing Office.
GAO. 1997b. Nuclear Waste: Department of Energy's Project to Clean Up Pit 9 at Idaho Falls Is Experiencing Problems. Report to the Committee on Commerce, U.S. House of Representatives. GAO/RCED-97-180. Washington, D.C.: Government Printing Office.
GAO. 1997c. Department of Energy: Information on the Tritium Leak and Contractor Dismissal at the Brookhaven National Laboratory. GAO/RCED-98-26. Washington, D.C.: Government Printing Office.
GAO. 1998a. Department of Energy Can Improve Linkages among Plans and between Resources and Performance. GAO/RCED-98-94. Washington, D.C.: Government Printing Office.
GAO. 1998b. Department of Energy's Hanford Tank Waste Project: Schedule, Cost, and Management Issues. GAO/RCED-99-13. Washington, D.C.: Government Printing Office.
GAO. 1998c. Department of Energy: Alternative Financing and Contracting Strategies for Cleanup Projects. Report to the Subcommittee on Energy and Water Development, Committee on Appropriations, House of Representatives. GAO/RCED-98-169. Washington, D.C.: Government Printing Office.
GAO. 1999. Major Management Challenges and Program Risks: Department of Energy. GAO/OCG-99-6. Washington, D.C.: U.S. Government Printing Office.
Gottschlich, D. 1998. Project performance studies performed for DOE environmental restoration and waste management projects, presentation by D. Gottschlich, area manager, Independent Project Analysis, Inc., to the Committee to Assess the Policies and Practices of the Department of Energy to Design, Manage, and Procure Environmental Remediation, Waste Management, and Other Construction Projects, August 3, 1998, National Research Council, Washington, D.C.
IPA (Independent Project Analysis, Inc.). 1990. U.S. Department of Energy, Office of Environmental Restoration and Waste Management, A Project Risk Analysis of the Hanford Waste Vitrification Plant. Reston, Va.: Independent Project Analysis, Inc.
IPA. 1993. U.S. Department of Energy, Office of Environmental Restoration and Waste Management, Project Performance Study. Reston, Va.: Independent Project Analysis, Inc.
IPA. 1994. A Post Analysis of the 881 Hillside Project. An Analysis for DOE EG&G Rocky Flats. Reston, Va.: Independent Project Analysis, Inc.
IPA. 1995. U.S. Department of Energy, Office of Environmental Restoration and Waste Management, Project Performance Study, Waste Management Addendum. Reston, Va.: Independent Project Analysis, Inc.
IPA. 1996. U.S. Department of Energy, Office of Environmental Restoration and Waste Management, Project Performance Study Update. Reston, Va.: Independent Project Analysis, Inc.
Jupiter Corporation. 1998a. External Independent Review: Pit Disassembly and Conversion Facility. Project #99-D-141. Final report prepared for the Office of Field Management, U.S. Department of Energy.
Jupiter Corporation. 1998b. External Independent Review: Dual Axis Radiographic Hydrotest Facility. Defense Programs Project No. 97-D-102. Final report prepared for the Office of Field Management, U.S. Department of Energy.
Jupiter Corporation. 1998c. External Independent Review: Chemistry and Metallurgy Research Facility Upgrades. Project No. 95-D-102. Final report prepared for the Office of Field Management, U.S. Department of Energy.
Lockwood-Greene Technologies. 1998. External Independent Review: Spent Nuclear Fuel Dry Storage Project. Defense Environmental Management Privatization Project No. 98-PVT-2. Readiness Review Report prepared for the Office of Field Management, U.S. Department of Energy.
NRC (National Research Council). 1998. Assessing the Need for Independent Project Reviews in the Department of Energy. Board on Infrastructure and the Constructed Environment, National Research Council. Washington, D.C.: National Academy Press.