B Communication of Project Costs and Durations
The committee has observed that the U.S Department of Energy (DOE), Congress, and other stakeholders do not always use the same definitions for project terms, such as cost estimate, contingency, and risk. DOE documents talk a great deal about risk but offer no precise meaning of the term or how to compute it. Clarifications of terminology and frames of reference, including quantitative assessments of risk and uncertainties, could reduce misunderstandings about project estimates, contingencies, and commitments.
A cost estimate is a prediction about a future event, namely the final project cost, and because future events are uncertain, they ought to be described probabilistically. Cost estimates ought to reflect the uncertainties and risks inherent in the project at the time the estimates are made. Confidence factors or ranges should be included with all cost estimates at all stages of a project, to give proponents, participants, and sponsors a realistic idea of the risks and uncertainties related to cost and schedule overruns. Point estimates should be avoided because they give a misleading impression of precision, especially when the reliability of the estimate is low. The General Accounting Office (GAO) criticized "DOE's practice of presenting rough order of magnitude numbers as point estimates" (GAO, 1998, p. 11).
Allowances for cost uncertainties and unknown cost factors can be developed through risk assessments, scenario analyses, contingency assessments, sensitivity analyses, and related methods. Sensitivity analyses and independent external reviews of the assumptions used in the cost and duration estimates should be used to assure that cost and duration estimates are robust against changes in assumptions. "Studies show that the amount of contingency required in the estimate is
directly related to how well the project is defined. Projects with poorly defined scopes require larger contingencies than projects with well-defined scopes. . . . [A] contingency is added to adjust for the estimator's incomplete or uncertain knowledge" (Diekmann, 1996, p. 12).
Two basic types of uncertainties may be identified. One type is internal to the project and the other external. The internal uncertainties or unknowns relate to such estimating factors as labor rates or productivity, unexpected foundation conditions, prices and quantities of commodities, such as concrete, steel, etc. Best estimates of these factors must be included in the initial estimated cost of the project, and allowances for changes to the estimated values should be included in the contingency.
External uncertainties are related to external influences and externally-mandated changes beyond the control of the project. They include the effects of political change, Congressional actions, changes in general DOE policies, local, state, or tribal influences, and all changes in cost or schedule originating outside the project for reasons unrelated to the project's purpose or objectives.
The different types of uncertainties have often been treated differently, and estimates often do not include external unknowns, uncertainties, or risks because cost estimators did not know how to estimate them and because they are externally controlled, hence deemed not to be the responsibility of the project. But not allowing for external risks is the same as estimating them to be zero. This practice may be acceptable in projects for which the external uncertainties are very small, but this is not usually the case for DOE projects. DOE cannot apply conventional thinking to unconventional situations. DOE should estimate both classes of uncertainty and include all uncertainties in the contingencies.
Objective, statistical evidence demonstrates that DOE's cost estimates are biased on the low side, as documented in Appendix A. The Project Performance Study Update by Independent Project Analysis, Inc., (IPA, 1996) also found that "contingencies of 20 to 25 percent [are assigned for waste management projects] regardless of the individual project risks" (pp. 69-70). When contingencies are based on flat percentages rather than on risk or uncertainty assessments, it is not surprising that costs frequently exceed the assigned contingencies.
The same study found that "DOE underestimates the contingency requirement for their ER [environmental restoration] projects. . . . DOE ER estimated contingencies are not in line with industry norms and are set too low" for the projects' risks (IPA, 1996, pp. 115-116). Contingencies averaged about 15 to 20 percent, but average cost growth was 48 percent. Obviously these costs were not covered by contingency allowances.
DOE's schedule duration estimates are also biased on the low side, as documented in Appendix A. There is no indication that DOE includes any contingency allowances for project durations when setting schedules.
Figure B-l, adapted from Figure 4.3 of the RAND study performed for DOE in 1981, shows the general trend in cost estimates over the life cycle of a project (Merrow et al., 1981). The figure shows that the average estimates are biased on the low side; they approach the true or final cost asymptotically from below; and the 67 percent confidence limits (+/- one standard deviation from the estimate) do not even cover the actual cost until late in the project.
General guidance for conducting benefit-cost and cost-effectiveness analyses was spelled out by the Office of Management and Budget (OMB) in 1992. OMB Circular No. A-94, Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs, states:
9. Treatment of Uncertainty. Estimates of benefits and costs are typically uncertain because of imprecision in both underlying data and modeling assumptions. Because such uncertainty is basic to many analyses, its effects should be analyzed and reported. Useful information in such a report would include the key sources of uncertainty, expected value estimates of outcomes, the sensitivity of results to important sources of uncertainty, and where possible, the probability distributions of benefits, costs, and net benefits.
a. Characterizing Uncertainty. Analyses should attempt to characterize the sources of uncertainty. Ideally, probability distributions of potential benefits, costs, and net benefits should be presented. It should be recognized that many phenomena that are treated as deterministic or certain are, in fact, uncertain. In analyzing uncertain data, objective estimates of probabilities should be used whenever possible. Market data, such as private insurance payments or interest rate differentials, may be useful in identifying and estimating relevant risks. Stochastic simulation methods can be useful for analyzing such phenomena and developing insights into the relevant probability distributions. In any case, the basis for the probability distribution assumptions should be reported. Any limitations of the analysis because of uncertainty or biases surrounding data or assumptions should be discussed.
b. Expected Values. The expected values of the distributions of benefits, costs and net benefits can be obtained by weighing each outcome by its probability of occurrence, and then summing across all potential outcomes. If estimated benefits, costs and net benefits are characterized by point estimates rather than as probability distributions, the expected value (an unbiased estimate) is the appropriate estimate for use. Estimates that differ from expected values (such as worst-case estimates) may be provided in addition to expected values, but the rationale for such estimates must be clearly presented. For any such estimate, the analysis should identify the nature and magnitude of any bias. For example, studies of past activities have documented tendencies for cost growth beyond initial expectations; analyses should consider whether past experience suggests that initial estimates of benefits or costs are optimistic.
DOE cost estimates violate OMB Circular No. A-94 in a number of ways:
- The effects of uncertainty are not analyzed.
- No probability distributions are given for costs.
- Past experience showing bias in cost estimates is not taken into account in new cost estimates.
- Sensitivity analyses are not performed.
- Cost estimates are not expected values.
If the probability distribution of costs were symmetrical (e.g., a normal distribution [Figure B-2]), the expected value, or mean, would correspond to the
median, or 50 percent likelihood value. That is, for a symmetric probability distribution, 50 percent of all projects on the average would ultimately cost less than the expected value, and 50 percent would cost more.
If the probability distribution on costs is skewed to the left (Figure B-3), the mean (expected value) of the costs would be to the right of (higher than) the median. In this case, on the average, more than 50 percent of all projects would cost less than the estimated value, and fewer than 50 percent would ultimately cost more than the mean value. It is clear from the statistical analyses cited in Appendix A that DOE estimates do not fit this pattern. Therefore, the evidence is that the costs estimated for DOE projects are not the expected costs.
Communications between project participants and stakeholders could be improved if the definitions of estimated costs, budgets, and contingencies were clarified. This could be achieved if DOE adhered to the requirements of OMB Circular No. A-94 and included uncertainties in its estimates. Clarification could also be achieved by reporting the probability distribution (also required by OMB Circular No. A-94). However, the situation might be better conveyed by reporting the complement of the cumulative probability distribution, which is the probability that the cost will be exceeded for any value of cost, as shown in Figure B-4.
This format directly shows the likelihood of the cost overrunning any given amount. "An accurate project contingency will allow a project team to avoid a cost overrun by establishing a project budget large enough to absorb cost increases driven by project uncertainties." (Diekmann, 1996, p. 34) When presented in this format, the authorized budget (i.e., the sum of the estimate plus the contingency), can be assigned based on an acceptable probability that the actual costs will exceed the budget.
A probability distribution for project durations, or the timing of significant project milestones, is shown in Figure B-5. An alternative would be to give the median duration (the duration that would, on the average, be exceeded 50 percent of the time) and the 90 or 95 percent confidence duration (the duration that would, on the average, be exceeded 5 percent of the time). In fact, the latter method of reporting uncertainties in project durations and milestone dates was used in the DOE Report to Congress: Treatment and Immobilization of Hanford Radioactive Tank Waste (DOE, 1998), in which "British Nuclear Fuel, Ltd. (BNFL) . . .provided two sets of milestone dates that differ depending on BNFL's estimate of their likelihood of achievement (i.e., either 50 percent confidence or 90 percent confidence."
OMB Circular No. A-94 outlines two methods for obtaining probability distributions or confidence factors: (1) by estimating probabilities objectively, and (2) by stochastic simulation methods.
- According to OMB, objective estimates should be used whenever possible. Objective estimates can be obtained through statistical models using multivariate regression on previous project data. This is the type of analysis used by IPA (under contract to DOE) in the Project Performance Study Update (IPA, 1996) and other studies (IPA, 1993, 1995). Regression coefficients are determined by an objective statistical analysis of previous costs; then specific project parameters are used as independent variables in these models to predict expected costs and establish confidence limits.
Similar regression models can be used to predict durations and the confidence limits on durations. (Typically, however, an assumption is made of normal distributions, so that the computed standard deviation of the estimate is used to give symmetric, rather than asymmetric, probability distributions and confidence intervals.)
- Stochastic simulations are commonly carried out through Monte Carlo computer simulations. A number of computer models have been developed for the type of stochastic simulation described in OMB Circular No. A-94. DOE has actually sponsored some for use on environmental restoration projects. Diekmann and Featherman (1998) discuss risk analysis simulation models developed with funding by EM-432 and the Center for Risk Management at Oak Ridge National Laboratory; and Diekmann developed a methodology for predicting cost and schedule growth "based on data from eight ER [environmental restoration] projects" (Diekmann, 1996).
DOE has also used uncertainty analysis and simulation to evaluate the effects of technological risk on life cycle costs (von Winderfeldt and Schweitzer, 1998). Although the analysis was program-related rather than project-related (the focus was on selecting a tritium supply alternative based on operability, productivity, and availability), analogous methods could be used for estimating the probability distributions for project costs and durations. The results were presented in figures similar to Figures B-4 and B-5. Other DOE-sponsored studies have focused on evaluations of technical risk, comparisons of alternative technologies, and the development of "decision analysis methodology [that] can analyze uncertainties about site characterization and remedial alternative effectiveness" (Parnell et al., 1997).
Although some individuals at DOE have supported the development and use of methods for risk analysis in keeping with OMB Circular No. A-94, their use is not DOE standard practice. "Modeling these risks and interpreting the results requires skills that are not common on a given ER [environmental restoration] project team. This means that if cost risk studies are to be commonplace some training will be needed" (Diekmann, 1996). Although some individuals in DOE have used these risk assessment tools, their general use, even when developed with DOE funds, has not been institutionalized in the DOE organization.
DOE (U.S. Department of Energy). 1998. Report to Congress: Treatment and Immobilization of Hanford Radioactive Tank Waste. Washington, D.C.: U.S. Department of Energy, Office of Environmental Management.
Diekmann, J.E. 1996. Cost Risk Analysis for U.S. Department of Energy Environmental Restoration Projects. A Report to the Center for Risk Management, Oak Ridge National Laboratory. Boulder, Colo.: University of Colorado Construction Research Series.
Diekmann, J.E., and W.D. Featherman. 1998. Assessing cost uncertainty: lessons from environmental restoration projects. Journal of Construction Engineering and Management 124(6): 445-451.
GAO (General Accounting Office). 1998. Nuclear Waste: Department of Energy's Hanford Tank Waste Project: Schedule, Cost, and Management Issues. GAO/RCED-9913. Washington, D.C.: Government Printing Office.
IPA (Independent Project Analysis). 1993. U.S. Department of Energy, Office of Environmental Restoration and Waste Management, Project Performance Study. Reston, Va.: Independent Project Analysis, Inc.
IPA. 1995. U.S. Department of Energy, Office of Environmental Restoration and Waste Management, Project Performance Study, Waste Management Addendum. Reston, Va.: Independent Project Analysis, Inc.
IPA. 1996. U.S. Department of Energy, Office of Environmental Restoration and Waste Management, Project Performance Study Update. Reston, Va.: Independent Project Analysis, Inc.
Merrow, E.W., K.E. Phillips, and C.W. Myers. 1981. Understanding Cost Growth and Performance Shortfalls in Pioneer Process Plants. Washington, D.C.: RAND.
OMB (Office of Management and Budget). 1992. Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs. OMB Circular No. A-94, October 29, 1992. Washington, D.C.: Government Printing Office.
Parnell, G.S., J.A. Jackson, J.M. Kloeber, Jr., and R.F. Deckro. 1997. Improving DOE Environmental Management Using CERCLA-Based Decision Analysis for Remedial Alternative Evaluation in the RI/FS Process. Report VCU-MAS-97-2. Butte, Montana: MSE Technology Applications, Inc.
von Winderfeldt, D., and E. Schweitzer. 1998. An assessment of tritium supply alternatives in support of the U.S. nuclear weapons stockpile. Interfaces 28(1): 92-112.