Progress in Improving Project Management at the Department of Energy: 2001 Assessment (2001)

Risk management is probably the most difficult aspect of project management, and for many DOE projects it is also the most critical. The Phase II report noted that DOE does not always use proven techniques for assessing, allocating, and managing risks. Discussions with DOE project managers reveal that DOE has not yet established effective risk-management methodologies or systems and that managers lack the tools and training to adequately manage risk for projects with high levels of uncertainty. The Acquisition Risk Management (ARM) pilot program by the DOE Contract Reform Office is beginning to address many risk management issues and has defined three iterative phases of risk management for DOE projects:

1. Risk identification,

2. Risk analysis and evaluation, and

3. Risk response.

The committee believes that all three phases of risk management are critical to effective project management in DOE.

The ARM process emphasizes the identification of project risks very early in the project, the development of a risk management plan during front-end project planning, and the updating of the plan throughout the project. This process, if adequately defined and implemented, has the potential to improve the mitigation and management of risks on future DOE projects. The committee supports and

encourages this effort. Unfortunately, the ARM process is still in the pilot stage, and the DOE guidance on risk management issued to date is insufficient (see Chapter 8, “Documentation of Project Management Policies and Procedures”).

The proposed ARM process correctly points out that the first objective of risk analysis is to identify, define, and characterize the risks. However, simply examining the activities and work packages of a project and qualitatively assessing them as high, medium, or low risk (a process observed on several DOE projects reviewed by the committee) does not necessarily achieve the desired objectives. The disaggregation of a project into work packages, which may be very suitable for construction management and contracting, may be of little value in identifying important project risks. For example, although the performance, delivery, and cost of a critical, but yet to be developed, technology may be a major risk to project success, the technology development may not appear as a work package in a project work package analysis. Assessment of past project performance shows that risks that generated delays and cost overruns were ignored or left out in DOE work package risk analyses (NRC, 1999).

Risk factors are not only technical or environmental. Human factors—including uncertainties related to human behavior, human failures to perform, changes in critical personnel on the project or in the DOE, changes in mission or loss of mission, and other issues related to the performance of people—should be identified, quantified, and given due consideration in risk assessments.

Effective risk analysis requires an examination of the nature of the project to identify the root causes of risks and to trace these causes though the project to their consequences. As an example, if a project has a work package called Design Advanced Superconducting Magnets Using New High-Temperature Superconducting Material, there may be uncertainties surrounding this activity, insofar as the more advanced the design, the longer the design process may take and the more it will cost. Moreover, there may be significant uncertainties in the ultimate cost of fabrication of the equipment, the delivery date, and the ultimate performance. All of these may suggest that there are substantial risks associated with the budget, schedule, and scope of this activity. Furthermore, the activity may have even greater impact on the uncertainties associated with other work packages. In turn, the uncertainties in the physical size of the equipment, its power requirements, cooling requirements, safety requirements, maintenance requirements, reliability, and other factors may have significant impacts on still other activities during design, construction, and operation, and greatly increase the uncertainty associated with these work packages. Even so, it is not sufficient to evaluate just the primary interactions, because they, in turn, may impact other activities, and so on, making it necessary to examine the primary, secondary, tertiary impacts, or even further, to evaluate the potential for a ripple effect, in

which the effects of one event propagate through many other events. Not tracking sources of risk through the causal relationships in the project can cause important project risks to be overlooked or understated. Neglect or underestimation of the ripple effect is a common deficiency in risk analysis, leading almost always to an underassessment of risk.

Several approaches are available for handling the kind of risk assessment commonly associated with DOE projects:

• Systems analysis and systems thinking. Systems analysis and systems thinking emphasize the relationships among activities in a project and the understanding of basic feedback structures that drive projects, through the development of shared maps of the processes, participants, and their interactions.

• Causal loop diagrams. Causal loop diagrams are a systems analysis and thinking tool that shows how activities are related through feedback loops; they help explain why some variables have little or no effect (negative feedback) and some have highly amplified effects (positive feedback).

• Event trees. Event trees (or fault trees or probability trees) are commonly used in reliability studies, probabilistic risk assessments, and analysis of failure modes and effects. Each event tree shows the results of a top event and other variables, leading to the determination of outcomes and the likelihood of these outcomes.

After qualitative identification of risk factors, it is necessary to quantify them. Although there are many approaches for quantifying uncertainty, the most generally accepted methods are based on probability theory. If uncertainties are expressed as probabilities, then the entire set of methods derived from probability theory can be drawn upon. In practice, however, there are certain difficulties with their application. One problem is that probabilities are generally based on the relative frequencies of events derived from historical data, but in the absence of such data for project costs, durations, and scopes, the probabilities are not objective but subjective. A second problem arises if the person with detailed knowledge of the project is not experienced in analyzing uncertainties as probabilities. A third problem arises from the tendency of those directly involved in estimating a project’s schedule or costs to underestimate the uncertainties associated with the schedule and costs. Conversely, more objective persons familiar with probabilistic methods may know little about the root causes of risks on a particular project. Collaboration and interaction between these two groups, starting from

the conceptual planning stages, are typically required to produce realistic, unbiased quantification of project risks.

Many times, inputs based on subjective judgments and experience are the only information available and give useful results. However, DOE should assure that these inputs are provided by persons experienced with similar projects and that the reasons for choosing specific probability distributions are thoroughly documented. The most effective antidote to bias in risk assessments is to use an open process, with documentation, justification, and review of all assumptions by disinterested parties. Also, it is essential to the success of a risk assessment program that actual costs, schedule, and other relevant project data be collected and compared with the original estimates, in order to build up a database that can be used for estimating future projects. Without a system that provides feedback on actual performance of projects to project planners, future risk assessments will not improve.

The committee agrees with the ARM approach that emphasizes the breakdown of the uncertainties into manageable parts. Figure 4–1, presented to the committee by DOE, shows a Pareto graph of the top 40 sources of uncertainty in the River Protection Project integrated schedule. It is clear from this presentation which activities have the greatest impact on the project completion date and require the greatest attention to risk mitigation and management.

One of the important results of a good risk analysis is that it allows determining where to apply management resources and what to leave alone. Unfortunately, from the presentation in Figure 4–1 one cannot determine if the elements referenced (e.g., AZ 101 HLW Start-up HLW Vitrification Production, the highest-ranking source of uncertainty) are truly root causes or simply work packages or activities. The top events should be root causes so that this analysis can serve as a map for DOE managers to find ways to reduce, mitigate, buffer, or otherwise manage the sources of uncertainty.

The objective of risk assessment is not just to compute risk values but to increase capacity to mitigate and manage the risks. Characterizing some risks as completely out of project management influence—acts of God, for instance— might be helpful in understanding total project uncertainty, but the primary goal of risk assessment should be the identification of active measures for risk management. Risk management should be an active not a passive endeavor.

Calculation of a single project risk estimate may be useful as input to a decision on whether to execute the project (in which case there may be biases toward underestimating it) or as a basis for setting contingency (in which case there may be biases toward overestimating it). The DOE PSOs should be in a position to know the magnitudes of risk associated with each project under their control. There are many available methods for combining risks.

Data-based analysis, or “objective” analysis, is one of two methods explicitly cited in Office of Management and Budget (OMB) Circular No. A-94, Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs (OMB, 1992). This method is objective in that it does not rely on subjective probability distributions elicited from (possibly biased) project advocates. It builds a statistical model based on data for many projects and then compares the proposed new project with this model. Its use is highly desirable as an independent benchmark for evaluating risk (and other factors) for a specific project. Unfortunately, it requires a large database of projects, and DOE, despite its history of many projects, does not have such a database. (See NRC, 1999, Appendix B, for more information and references.)

The Monte Carlo method is a generic term for simulations that use random number generators to draw variates from probability distributions. It is the second of two methods explicitly cited in OMB Circular No. A-94 (OMB, 1992). (See NRC, 1999, Appendix B, for more information and references.) Stochastic simulations can be very useful in the absence of real data. Their advantage is that they are based on subjective assessments of probability distributions and therefore do not depend upon large databases of project information. Their weakness is that because they are based on subjective assessments of probability distributions, their objectivity may be suspect. Stochastic simulation models that are based on event trees or feedback models can give reasonable estimates of total project risk. Monte Carlo simulations that simply add up the uncertainties associated with various activities or work packages may be biased, because the typical approach is to assume that all these activities are statistically independent. If one performs an elementary risk analysis, as described above, in which root causes of uncertainties are identified and the ripple effects tracked through the entire project, it is obvious that activities affected by the same root cause cannot be statistically independent. Therefore the inappropriate assumption of independence in Monte Carlo models can severely underestimate risks.

If the objective is simply to find the probability distribution of the project cost estimate as the sum of a number of work package costs, stochastic simulation is unnecessary overkill. It is well known from elementary statistics that the moments of a sum are the sum of the moments, if all the terms are statistically independent. This summation needs to be modified if the variables are not statistically independent, but the summation method can still be readily applied. Simple

programs based on summation of moments have been used for many years to combine risks for dependent as well as independent variables, using the first (mean), second (variance), and third (skewness) moments, permitting use of highly skewed probability distributions (project cost distributions are generally skewed to the right, that is, with long tails in the direction of higher costs). One advantage of simple additive models is that they are easily understood, and it is usually obvious which activities contribute the most to the total project uncertainty and which activities contribute relatively little. Ease of understanding is more important than a false indication of accuracy, when all the probability distributions are based on qualitative judgments anyway. Unfortunately, the summation method does not work for project durations (critical path lengths) unless one can assume, as the program evaluation and review technique (PERT) method does, that the critical path is not affected by uncertainty in the activities.

System dynamics models are typically based on quantitative causal loop diagrams that show how activities are related through feedback loops. The models may be deterministic or probabilistic. Commercial, off-the-shelf programs are available to perform the calculations. Although systems dynamics models are more often deterministic than stochastic, because they are based on dynamic feedback principles, they can nevertheless be used to evaluate the ripple effect of various changed conditions or root causes. Experience with systems dynamics models on real projects has shown that this method generally gives much more realistic estimates of the consequences of changes or other events than methods that do not adequately account for the ripple effect. Such models are not only useful in the early stages of a project for risk assessment, they can also be very valuable in later stages for managing change.

As previously stated, the primary function of risk analysis is to break down the problem into essential elements that are capable of mitigation and management throughout the life of a project. Therefore, regardless of what method of combining risks is used, it is highly desirable to perform a sensitivity analysis of the results. Not all project managers are well versed in probability theory and stochastic simulation, but experience shows that most can relate well to sensitivity analyses, which indicate the relative influence of certain variables on the outcomes. In the absence of real data, sensitivity analysis can be very useful in checking the reasonability of risk models. In fact, when a project system is tightly coupled, it may be impossible to evaluate the effect of various variables separately and in isolation; in such cases it is more effective to perform a system simulation and then use sensitivity analysis on the systems model to identify the most important sources of uncertainty.

Finding. With rare exceptions, there are no risk models for ongoing DOE projects, and back-fitting risk assessment to ongoing legacy projects does not seem to be part of the acquisition risk management (ARM) study. There is no consistent system for evaluating the relative risks of projects with respect to scope, cost, or duration, so the deputy secretary, the chief financial officer, and the PSO managers have no objective basis for knowing which projects are riskier (and therefore require more management attention) than others.

Recommendation. DOE should develop the ability to perform quantitative risk assessments. These assessments should be carried out by DOE personnel with experience in such analyses working with persons who have an in-depth understanding of a given project. Internal project risk assessments should be separately evaluated by independent assessors or reviewers who are not project proponents for reasonableness of assumptions, estimates, and results. Risk mitigation and management plans should be prepared that can deal with significant risks identified.

Recommendation. DOE project management personnel should be trained in risk assessment methodology. This training should cover not only risk analysis methodology and techniques, but also the managerial responsibilities related to interpretation of risk assessments and mitigation and management of risks.

Recommendation. Risk analyses should explicitly consider the interdependence of the various activities due to common modes (root causes), or document why there is no dependence.

Finding. DOE has not implemented statistical models (the “objective” analysis cited in OMB Circular No. A-94, Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs, because it has no usable database of past and current projects.

Recommendation. DOE should develop an internal database of project data on its own projects and on projects of other owners. A system should be established to capture data on current and future projects. Data on comparable projects performed by other federal agencies and by industry should be obtained and included. The current development of the project analysis and reporting system (PARS) (discussed in Chapter 5) could be a step toward this goal, and the committee plans to follow this work with interest. Although its early stage of development prevents assessing its effectiveness at this time, the level of participation by projects, accuracy of data, completeness of data, and avoidance of duplication should be addressed by OECM. The architecture of this data system should be specifically designed to provide support for the analysis of risks for ongoing and future projects.

Some risks, once identified, can be readily eliminated or reduced. Most risks are much more difficult to handle, and risk mitigation and management require long-term efforts by project managers.

If a project has a low level of uncertainty, then the optimal policy is to proceed as fast as possible. Decisions should be made as early as possible, because in a project with low uncertainty, there is by definition little chance of making bad decisions. Fixed-price contracts, perhaps with schedule performance incentives, are appropriate. In projects with cost-benefit analyses (and all projects should have cost-benefits analyses under GPRA), the present value of the project will be increased by completing the project earlier and thereby obtaining the benefits of the project sooner. The introduction of new uncertainties over time will also be minimized.

In general, everything else being equal, projects that take longer cost more. Many DOE projects take longer than they should, in part owing to dilatory decision making inside DOE and in part owing to the budgeting-authorization-appropriation cycle. Many projects seem to stall while awaiting authorization and funding and then try to make up for this lost time by rushing forward. However, if a project has a high degree of uncertainty, a full-speed-ahead approach may not be optimal or desirable. For projects with high levels of uncertainty, performance-based incentive contracts are generally more appropriate than fixed-price contracts.

In the front-end planning model in Figure 3–1, all the arrows show movement from left to right; there is no provision for rework or iteration. This may be a realistic description of conventional infrastructure projects, but rework and iteration are common in DOE projects because of factors such as design and scope changes resulting from inherent uncertainties in science, technology, and environmental characterization. Regulatory issues also provide a source of uncertainty that can cause conceptual project planning and design to be reworked many times. In high-uncertainty projects, rework is the norm, not the exception.

Failure to recognize and anticipate changes and iteration in preparing schedules and budgets can lead to unfortunate results. The use of techniques and skills that are appropriate to low-uncertainty projects can yield poor results when applied to high-uncertainty projects with great potential for changes and high sensitivity to correct decisions. For high-risk projects, a flexible decision-making approach is much more successful. Management of uncertainty cannot just be delegated to contractors; instead, attempts to assign all uncertainties to contractors have generally resulted in increased costs, unsatisfactory performance, and litigation. Effective risk management requires the active attention of federal project managers and senior program managers.

Finding. By and large, DOE’s practices in risk assessment and risk management have not significantly improved since the Phase II report. The committee reviewed some project risk assessment studies but did not see an example of a risk assessment or risk mitigation plan that it finds acceptable. The discussion in the draft PPM is merely an outline, and the material in the draft PMP is not useful as a guide for practicing risk management. Conversely, the current ongoing acquisition risk management (ARM) pilot study at three DOE sites and by the Contract Reform and Privatization Office and the EM Division Steering Group/Working Group, due for completion by December 2001, is a positive move and shows promise. The committee intends to follow this study with interest as it evolves.

Recommendation. The current acquisition risk management (ARM) pilot study should be continued and expanded beyond budget risks to cover the issues addressed in the Phase II report and in this report, such as schedule, scope, quality, and performance risks.

Finding. DOE’s deficiencies in risk analysis lead to inadequate risk mitigation planning and execution. Plans often address symptoms but not causes. Execution is typically reactive or nonexistent. To be useful during project implementation, this planning should, at a minimum, do the following:

• Characterize the root causes of major risks that were identified and quantified in earlier portions of the risk management process.

• Identify alternative mitigation strategies, methods, and tools for each major risk.

• Evaluate risk interaction effects.

• Identify and assign priorities to mitigation alternatives.

• Select and commit required resources to specific risk mitigation alternatives.

• Communicate planning results to all project participants for implementation.

Recommendation. DOE should develop and implement risk mitigation planning processes and standards. Project risk assessment and management should be carried out throughout the project life cycle and should be part of the documentation for each critical decision point. Risk mitigation plans should be reviewed, critiqued, returned for additional work if needed, and approved by an independent organization such as the ESAABs at each critical decision point and prior to project approval for design or construction funding.

Recommendation. Until DOE project managers can be adequately trained in risk management, OECM should establish a cadre of experienced risk assessment personnel, who can be detailed or seconded to projects in the very early stages, to

provide risk assessment expertise from the beginning of projects and incorporate risk management into the initial project management plan. (Also see the recommendation for human resources in Chapter 3, “Front-End Planning.”)

Flexibility in project plans to address foreseeable risks and flexibility in organization, management, and control to address unforeseeable events are required to successfully manage highly uncertain projects. The value of management flexibility increases in direct proportion to the uncertainty in the project. To paraphrase a quotation attributed to General Eisenhower, who said after D-day: “It is absolutely necessary to prepare battle plans, but it is equally necessary to know when to deviate from them when actually in battle.” The same thought may be applied to project management. A flexible decision-making structure requires that project managers be active and show initiative. Under these circumstances, project managers should not be constrained by organizational culture, bureaucratic restrictions, fear, self-interest, or those who are likely to apply rigid management principles rather than initiative and flexibility.

Many DOE projects experience high levels of uncertainty in many critical project components. Most of these uncertainties cannot be significantly reduced through project planning alone. They require risk management approaches different from those used for traditional projects. Some of them cannot be adequately characterized and optimal actions chosen during front-end project planning. This is common when uncertainties will be reduced only over time or through the execution of some project tasks. For example, uncertainty about the presence or strength of specific chemicals in a groundwater supply or solid waste may be reduced only after project initiation and partial completion. Under these circumstances, committing to specific risk management actions during planning makes project success a gamble that the uncertainty will be resolved as assumed in planning. A classic example of commitment to a specific course of action and the maintenance of that course without developing alternative plans is the In-Tank Precipitation project at Savannah River (GAO, 1999).

Strategic flexibility can provide tools for effectively planning for and mitigating such risks. Incorporating flexibility into risk management plans can reduce project costs and durations. Flexibility can be incorporated into project planning in several ways. One approach is to subordinate the project components that are impacted by an uncertain component to the uncertain component’s design. For example, if the size of a to-be-developed piece of equipment is not known, other components such as the facilities to house and service the equipment could be oversized to accommodate the 95th percentile equipment size. However this approach can generate conflicting constraints from different uncertain components, can commit the project during planning to a single action, and can be very costly in time, money, or both. Better risk management decisions can often be

made after a portion of the uncertainty has been resolved through some additional testing or preliminary design of the equipment. Purposefully and strategically postponing some risk management decisions and incorporating flexibility into risk management can improve project performance. However, postponing important risk management decisions without plans for when and how those decisions will be made invites failure by allowing inappropriate reaction to short-term conditions or ad hoc decision making. Additionally, postponing decisions might be less cost effective than committing to specific actions when needed.

Finding. DOE needs to take a flexible approach in managing risk because of the high levels of uncertainty. To be effective in risk management, flexibility should be structured. A process is needed for designing, assessing, evaluating, and implementing risk-management alternatives that include decisions made during front-end project planning and decisions made after project initiation.

Recommendation. DOE should develop cutting-edge abilities to manage high-risk projects. It should adopt a process of identifying, designing, evaluating, and selecting risk management alternatives. The process should explicitly include and address alternatives that take advantage of opportunities for the partial resolution of important uncertainties after project initiation. Reviews at critical decision points should always entertain Plan B, that is, the alternatives to be pursued if the primary approach is adversely affected by subsequent information or events.

The committee observed that DOE in the past tried to shift risks to contractors (K.A.Chaney and J.J.Mocknick, 1998; IPA, 1993). DOE briefings, too, have implied that risks should be allocated to the parties best able to manage them, which is difficult to accomplish when DOE has no quantitative assessment of the risks. Under some circumstances, risk allocation can degenerate into attempts by project participants to shift risks to others instead of searching for equitable allocation. There are two critical starting points for risk allocation: (1) the government initially owns all the risk and the other project participants (particularly prospective contractors) own none until a contract is signed and (2) contractors generally agree to take risks only in exchange for money. There is a price that the DOE can (but not necessarily should) pay a project participant to accept the gains or losses generated by specific uncertainties, but to determine this price, it is necessary to quantify the risks. Hence, quantitative risk assessment is essential to effective contracting.

Finding. An objective assessment is essential to performance-based contracting, to assure that DOE does not shift to other project participants risks that it should retain or vice versa, or shift risks at more cost than they are worth.

Recommendation. DOE should explicitly identify all project risks to be allocated to the contractors and all those that it will retain, and these risks should be made known to prospective bidders. To use a market-based approach to allocating risks and to avoid unpleasant surprises and subsequent litigation, it is necessary that all parties to an agreement have full knowledge of the magnitude of risks and who is to bear them.

The management of risk during many DOE projects appears to be passive and ad hoc without the benefits of tracking the root causes of risk identified during characterization or making proactive decisions and taking actions to mitigate risks. This practice has contributed to serious project performance problems, such as at the National Ignition Facility, where some major risks were not recognized or were ignored after project initiation until the budget and schedule problems they created forced rebaselining. A passive and reactive approach is often used in which risks are generally ignored until undesired events occur, at which time solutions are sought that often assume the availability of additional resources. Such an approach precludes preventing some undesirable events and increases the costs of addressing others. Inadequate front-end risk management planning and a tradition of budget increases may be the primary contributors to these behaviors and may deter proactive risk management during projects.

Risks need to be rigorously and aggressively managed during projects. Planning for risk mitigation is an important aid for this but is not sufficient. Rigorous risk management includes monitoring every risk factor and assigning management and mitigation responsibility to project parties. One tool for this purpose is a project risk registry. This management tool is initially constructed during project planning by identifying all types of uncertainties that could impact project performance (e.g., scope, schedule, technology, permits, site conditions, and environmental) and estimating the likelihood of occurrence and the nature and magnitude of the impacts. These estimates are used for prioritizing uncertainties for managerial focus, contingency sizing, and decision making. If funds appropriated are less than requested, the project risk registry acts as a basis for rescoping or redesigning the project, so that it remains consistent with the funds allocated. After the project has started, the project risk registry provides a tool for allocating managerial responsibility for specific uncertainties and reporting and monitoring their status. The most effective use of this tool includes regular and frequent reporting on each risk until the project passes the point where the risk is no longer an issue. Risks, as the term is used here, are distinguished from work packages, which can contain risks but are not typically defined to reflect them.

The consequences of inadequate risk management were amply demonstrated by highly visible projects such as Pit 9, in-tank precipitation, and TWRS. In these

examples, the risk of the initial approach not working was not adequately reassessed; they exemplify the failure to manage important risks during the project.

Finding. DOE project risks are not aggressively managed after project initiation. Risk management during projects is an inadequately developed project management capability at DOE.

Recommendation. DOE should initiate a program to improve the knowledge, skills, and abilities of project managers and develop tools and information needed to manage risk throughout the life of a project. Project participants who manage risks actively and achieve successful project performance should be appropriately rewarded.

The committee observed that many ongoing DOE projects are characterized by a high level of uncertainty and a minimal understanding and management of their risks. Most of these projects were initiated before the Phase II report and before DOE initiated project management reforms. Further, the committee observed a deficiency in DOE risk management methods. DOE has a need to manage risks to project schedules, cost, and scope. Doing so would prevent unpleasant surprises, enable remedial action, and avoid breaching baselines.

The committee believes that DOE should conduct a risk analysis of all ongoing large projects to establish their risks and vulnerabilities with respect to schedule, cost, and performance. The analysis could be used to establish a department-wide assessment of the risks remaining in each project and for the department as a whole, as well as to identify projects that are the most vulnerable and need the most attention. Because consistency is necessary for department-wide, cross-project comparisons, it is recommended that this risk study be led by OECM.

Finding. The committee observed an ongoing deficiency in risk management that undermines DOE’s ability to avoid surprises and take timely remedial action to avoid baseline breaches and to predict the actual cost to complete ongoing projects.

Recommendation. DOE should conduct an immediate and thorough risk assessment of all ongoing DOE projects with significant remaining time and costs. Such an assessment would establish, on a consistent basis, the risks and vulnerabilities of projects with respect to schedule, cost, and performance. It should assess the actual status of current projects and compare them with the project’s original baselines, the current project schedules and budgets, and performance for comparable completed projects. The assessment should evaluate the risks of future scope shortfalls and budget and schedule overruns.

It is not unusual for DOE projects to have unusually high levels of uncertainty in many critical project components. The successful management of these risks is often critical to project success. However, traditional risk management tools, methods, and practices may be inadequate. The committee believes that the methods in the draft PPM and PMP documents are inadequate if applied in a piecemeal fashion to the task of assuring successful project management practices under the conditions pervading DOE projects. Given the circumstances, new risk management tools and methods should be developed, tested, and implemented within DOE. Several existing tools and methods, such as those cited earlier and below, and the successful (or unsuccessful) management of risks in engineering projects that match DOE projects in size and duration (Miller and Lessard, 2000) can guide this effort and form the basis for developing risk management excellence at DOE. DOE has funded the development of a number of risk models (Diekmann, 1996; Parnell et al., 1997; Diekmann and Featherman, 1998), but there is little evidence that they have been used on actual projects.

The DOE could set up a project simulation program that would let project managers simulate the activities in a project before doing it. This was done successfully in private industry as well as in the military. Simulation could be manual or computerized or both. A project simulation facility might be expensive, but it would certainly be less expensive than making big mistakes on real projects and would pay for itself in the long run. Computer simulation models have been used to study the feedback loops and the effects of change in projects, and they have been used successfully to describe and to predict project completion rates and costs.

High-risk projects are not well described by conventional critical-path network models (which prohibit recycling), and efforts to apply conventional methods inappropriately to these projects can lead to incorrect conclusions and counterproductive solutions. One approach to developing useful computer project simulations is system dynamics. This computer simulation modeling methodology can specifically depict the characteristics of dependencies among project processes, resources, and management and their impact on project performance. By focusing on specific issues, modeling can clarify and test the assumptions used by project participants and be used to design and test existing and proposed project process improvements and managerial policies.

Finding. Innovative, cutting-edge, and exceptional risk management abilities are needed by DOE to identify and address the risks in many of its projects. DOE needs to develop expertise and excellence in managing very risky development projects. The DOE complex has the intellectual, computational, and other resources necessary to produce significant improvements in this area.

Recommendation. DOE should develop more expertise and improved tools for risk management. Nontraditional and innovative approaches, tools, and methods should be investigated for their adaptability to DOE project conditions and use in DOE risk management. They would include those cited earlier in this report and in the Phase II report (NRC, 1999, Appendix B), such as systems analysis, event trees, causal loop diagrams, system dynamics, and stochastic simulation, which have been tested and shown to be valuable on similar projects or in addressing similar challenges.

The discussion above has addressed risks mainly at the individual project level. However, of at least equal concern is the management of risks at the PSO level and at the departmental CFO or AE level. It is often said that project budgets and contingencies should be based on risk assessments, that is, on probabilities. Although probabilistic statements are impossible to verify on the basis of a single observation, DOE performs a large number of projects, so that statistical statements could in principle be verified over the population of all projects. The following is an elementary example.

The committee has been informed that the appropriate level of authorization for a project, assuming that the uncertainty in the ultimate project cost can be described by a probability distribution, is some value that has been called the risk-adjusted cost estimate (RACE). This might be, as one example, the dollar value at, say, the 85th percentile or confidence level. That is, using this number, there would be an 85 percent probability that the project will actually cost less than the RACE and a 15 percent probability that it will cost more. So, if there are 100 such projects, all funded at their respective RACEs, one would expect that 85 of these projects would be completed within their budgets and 15 would return to Congress for additional funds. In other words, if budgets are set at the RACE of the 85th percentile, statistically, 85 percent of the projects should return some unused funds to the treasury. This does not appear to be the case. Of the projects presented to the committee, only one, a DP project at Los Alamos (Infrastructure Renewal—Water Well Replacement, funded at approximately $17 million), claimed to have given back the contingency. No systematic data were available to the committee, as DOE does not seem to track contingency funds or management reserves. The conclusion, therefore, is that whatever the budget allocated to a project, the project will rarely or never underspend this budget, although it might overspend it.

For there to be any accountability, not to mention management of contingency funds, it would be necessary to state who authorizes the transfer of contingency funds to the baseline. DOE policy and procedures should define whether this is the federal project manager, the change control board, the ESAAB, the CFO, the contracting officer, or some other entity. DOE documentation should

define policies when more than one PSO, or multiple offices or laboratories, are involved in a single project—that is, whether each laboratory controls (i.e., is able to reallocate) its own management reserve or whether the management reserve on a multilaboratory project should be under the control of the overall project manager, controlled by the DOE site office manager, or controlled at the PSO level.

Contingency in the schedule is as important as contingency in the budget and should be also be covered by DOE policy. None of the above policy issues are covered in O413.3 or in the draft PPM manual or the draft PMP.

Finding. DOE does not seem to have a consistent or explicit policy on the use of management reserves, what size they should be, and who should control them.

Recommendation. The deputy secretary as secretarial acquisition executive, and the chief financial officer, assisted by the PSOs and OECM, should define and state DOE policy on management reserves. This policy should be clarified in a future release of O413.3.

Chancy, K.A., and J.J.Mocknick. 1998. Privatization: A Business Strategy Under Siege. WM 98 Proceedings, Tucson, Ariz.

Diekmann, J.E. 1996. Cost Risk Analysis for U.S. Department of Energy Environmental Restoration Projects. A Report to the Center for Risk Management, Oak Ridge National Laboratory. Boulder, Colo.: University of Colorado Construction Research Series.

Diekmann, J.E., and W.D.Featherman. 1998. “Assessing Cost Risk Uncertainty: Lessons from Environmental Restoration Projects.” Journal of Construction Engineering and Management 124 (6):445–451.

GAO (General Accounting Office). 1999. Process to Remove Radioactive Waste from Savannah River Tanks Fails to Work (GAO/RCED-99–69). Washington, D.C.: General Accounting Office.

IPA (Independent Project Analysis). 1993. Project Performance Study for U.S. Department of Energy, Office of Environmental Restoration and Waste Management. Reston, Va.: Independent Project Analysis.

Miller, Roger, and Donald Lessard. 2000. The Strategic Management of Large Engineering Projects. Cambridge, Mass.: MIT Press.

NRC (National Research Council). 1999. Improving Project Management in the Department of Energy. Washington, D.C.: National Academy Press.

OMB (Office of Management and Budget). 1992. Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs (Circular No. A-94). Washington, D.C.: Executive Office of the President.

Parnell, G.S., J.A.Jackson, J.M.Kloeber, Jr., and R.F.Deckro. 1997. Improving DOE Environmental Management Using CERCLA-Based Decision Analysis for Remedial Alternative Evaluation in the RI/FS Process. Report VCU-MAS-97–2. Butte, Mont.: MSE Technology Applications.