Click for next page ( 97


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 96
APPEND TX E Key Questions for Readiness to Proceed INTRODUCTION Critical decisions depend on asking the right questions during readiness reviews. The committee notes in its 2001 assessment report and in this report that it is important for managers to ask the right questions and not allow projects to proceed until these questions have been adequately addressed. Assumptions that things will go well are inadequate. There is no uniformly correct answer, but the manager charged with making a critical decision should be satisfied that the project team has addressed each issue and has developed an answer with enough certainty to allow the project to proceed with reasonable confidence. The lists of questions that follow are intended as a guide to preparing for and making critical . . declslons. PROJECT ORGANIZATION Project Execution Plans The Project Execution Plan (PEP), including the Project Risk Assessment and Risk Management Plan, should identify the planned actions and solutions to all risk factors to be considered by the project manager. Questions that specifi- cally address the adequacy of risk management procedures are discussed in the risk management section. 96

OCR for page 96
APPENDIX E 97 Is there a PEP? If not, when will it be completed? An acceptable PEP should be the sine qua non for approval to move forward with the project. Without an acceptable PEP, the project is undefined. Is there a Project Procurement Plan? Does it cover expediting? It may be necessary to expedite procure- ments on the project critical path. How many viable suppliers and contractors are available for major systems and components? If there is an inadequate number of suppliers or contractors, the project is at risk and steps should be taken to broaden the logistic base and strengthen the supply chain. This is one reason that delegating the development of PEPs to contractors may lead to conflicts of interest. Are there alternative or backup plans in case the supplier of a major component fails? The difference between a contingency plan and a contingency allowance is that a contingency plan provides the basis for assurance that the failure of a major supplier does not severely impact the project (i.e., it addresses developing alternative suppliers), whereas a contingency allowance merely provides funds to cover the additional costs if a supplier does fail. DOE PEPs typically emphasize contin- gency allowances to pay for risks if they occur rather than contingency plans to prevent the risks from occurring. Integrated Project Team Late assignment of critical personnel to the integrated project team is one of the most common causes of project delays in the front end delays that can never be recovered. Projects often try to recover delays at the end, when reducing delivery time is very expensive, or even impossible. . Is the experience of the project team consistent with the size and com- plexity of the project? Experience has shown repeatedly that experience on small, run-of-the-mill, and virtually risk-free projects is not suitable preparation for managing large, complex, and risky projects. Have appropriate members of the integrated project team received project management training or certification? Project management is a profes- sion, and the managers of large DOE projects should be professionals. Has the team worked before on similar projects? People who have worked together previously will progress up the learning curve much faster. If they have not, additional time for team building should be allocated at the beginning (or, rather, before the beginning). What is the turnover rate of the project team? High turnover means changes in direction and indecision. Major projects require consistency.

OCR for page 96
98 APPENDIX E . . High turnover of managers is a symptom of poor management and a sure recipe for project failure. Is there an employee motivation plan to maintain the morale of the project team over the long term of the project? At the beginning of a project, everyone is optimistic and enthusiastic. Nevertheless, the project needs to plan for motivating the project members when, inevitably, things do not go well. Has the Integrated Project Team (IPT) been identified? If not, when will it be named? Naming of the project team should be a requirement for moving the project forward. Are the members of the IPT assigned full time to this project? There is a tendency for personnel assigned to new projects to continue working on their old projects. This is detrimental to the success of new projects, where getting the right start in the preplanning process is critical. If not, how many are full time on the project? When will all members of the IPT be full time? How many times has the IPT team met face to face? Successful team building requires face-to-face contact, not just e-mail. If the IPT has not met several times, the project is not ready to proceed. Project Organization The Work Breakdown Structure (WBS) defines the assignments of project team members and the activities that they will be held accountable for. The WBS defines the project, and it should specifically include all activities related to resolution of uncertainty and risk mitigation, even if some of these activities are contingent. Note that the depth to which the WBS is defined increases over time. Without a WBS, the project remains undefined. . Is there a WBS, including all R&D, simulation, and prototyping activities, if any? Does the WBS match the organization chart? For successful project management, the organizational breakdown structure and the work breakdown structure should be in alignment. If they are not, insufficient thought has been given to the project to permit it to proceed. Does the WBS match the reporting structure? See comment just above. Is there a WBS dictionary? There should be, so that all project partici- pants understand the WBS. Is the WBS established down to (depending on the time of the decision): Level 0 (agency)? Level 1 (total project)? Level 2 (major systems milestones)? Level 3 (systems and components)? Level 4 (reporting)?

OCR for page 96
APPENDIX E . 99 Is the WBS broken down by system or by area? The WBS evolves with time. The initial project specification and engineering design may be broken down by system. Then, construction activities are broken down by area. Finally, start-up and testing activities are again broken down by system and function. Does each Level 3 item have an assigned manager? Responsibility and accountability should be assigned based on the WBS. Cost Estimates Cost estimates are at the core of project planning. When reviewing estimates it is important to recognize the facts and assumptions that they are based on and the degree of certainty of these facts and assumptions. . Is there a top-down cost estimate? The top-down estimate is based on previous history, experience, records, and perhaps statistical analyses of similar projects, corrected and adjusted to the conditions of the present project. It is, in effect, benchmarking of this project against others. What is the confidence level in this estimate? Point estimates are never adequate, and they are particularly inadequate when dealing with top- down estimates. Is there a bottom-up cost estimate? The bottom-up estimate is prepared from the engineering design. If not, when will it be completed? The engineering cost estimate will be refined over time as the design is developed, quantities developed, and bids received. What is the level of confidence in this estimate? The confidence band should depend on the level of refinement. Do error bars or confidence limits indicate the uncertainty in all cost estimates? The establishment of confidence bands is essential to a deci- sion to proceed. There is no reason why a DOE senior manager should have more confidence in the cost estimate than the estimators have, and the decision maker should demand to see evidence of this confidence. Is the cost estimate integrated with the schedule? If not, why not? Inte- grated cost estimates and schedules have been industry best practice for 20 years and are essential for the application of an earned value manage- ment system (EVMS). A project that states that it is in compliance with O 413.3 with regard to EVMS but does not have a cost-integrated schedule is not ready to proceed. Has there been a cost contingency analysis? This is related to the Project Risk Assessment and Risk Management Plan, which should be part of the PEP. Without it, there is no rational basis for setting the contingency. If not, when will it be performed?

OCR for page 96
100 APPENDIX E . . . . What are the cost contingency and/or the management reserve? The purpose of having a contingency should be explicitly defined. Then the amount of the contingency should be adequate to achieve this purpose. Contingency funds should cover contingencies and should not be just another way to increase the project budget. Who controls the contingency and/or the management reserve? More important than the amount of the contingency is who controls it. The manager who controls contingency should be held accountable for it. Contingency should be controlled at a level high enough in the project organization that its purpose can be achieved. Has there been a schedule contingency analysis? See comments just above regarding budget contingency. If not, when will it be available? What are the schedule contingency and/or management reserve? Who controls the schedule contingency and/or the management reserve? Who controls the technical performance contingency? Although in industry best practice technical performance and scope are considered to be fixed objectives, it appears that some DOE projects explicitly treat cost as fixed and use project scope and technical performance as contingency. The committee does not endorse this approach, because a project cannot pos- sibly be essential to the mission of the agency if its scope and performance are variables. The decision maker should know when this is happening; it may be grounds for project termination. Constructability and Value Engineering Constructability is an important aspect of cost and schedule control. Lack of a Constructability evaluation should be considered evidence of low confidence in the cost and duration estimates. Value engineering examines design alternatives to identify those that will achieve the project objectives most efficiently. . Has a Constructability analysis of the proposed (conceptual, preliminary) design been performed? If not, when will it be performed? How many Constructability analyses are planned and at what points? Constructability analyses may be desirable as the design is developed and refined. Constructability analysis should not be skimped on to save money; in virtually every case, a well-done, independent, and objective Constructability saves far more money than it costs. Has a value engineering study been performed? Value engineering is explicitly required by O 413.3. If not, when will it be done? How many value engineering studies are planned, and at what points?

OCR for page 96
APPENDIX E . 101 Are the results of the value engineering studies and constructibility analyses fed back into the engineering design process? Are they then fed back into the cost estimation and project scheduling process? Change Control Change control is an area in which DOE projects have been notably poor. DOE project managers have often left change control to the contractors. Who- ever controls changes, controls the project. Has a configuration management system been set up? If not, when will it be operational? Has the Change Control Board (CCB) been established? If not, when will it be operational? A CCB is required by Order O 413.3. No project should proceed without one. Does the CCB control design scope and all changes to the baseline technical specs and reserve margins on performance? It should. If not, who does? Does the CCB control the cost contingency and allocate contingency to changes? It should. If not, who does? Does the CCB control the schedule contingency? It should. If not, who does? Reporting and Controls Cost tracking for EVMS is not the same as project accounting, and accounting systems are generally incapable of providing cost data on a timely, accrual basis. . . . Has a commitment tracking system been established to track and control commitments to peer review boards, outside advisors, regulators, etc.? If it has not, it is certain that some commitments will be missed, to the detriment of the budget and schedule, if not to the project's overall success. Has an EVMS been established? If not, when will it be in place? It is a fallacy that earned value applies only to construction activities. It should apply to engineering, design, procurement, planning, and other project activities as well. If a project is not tracking earned value, how does the project manager know where the project is with respect to scope, time, and cost? Is there a project management control system in place to track all costs and earned value? The successful use of EVMS depends on timely reporting accurate insofar as possible, but in any case timely. Note that the function of an EVMS implementation should not be just to document the past and what went wrong with the project, but to predict the future in time to do something about it.

OCR for page 96
102 APPENDIX E If not, when will it be in place? Are costs reported on an accrual basis as required by EVMS? Accrual-based cost reporting is essential. Are all contractors and subcontractors capable of supplying timely, accu- rate cost and progress data to roll up in EVMS? This point requires specific management attention, as some contractors may be inclined to promise what they cannot deliver. After the contract is signed it is too late to ask this question. Are costs collected at Level 5 of the WBS? If not, at what level are they collected? Any other level may require subjective judgment, and this should be questioned. Is the cost estimate fully integrated with the schedule? (The effective use of EVMS requires this integration.) Is progress reported against activities in the network schedule consistent with the costs in the WBS? EVMS requires reporting of the budgeted cost of work performed (the earned value) compared with the plan (the bud- geted cost of work scheduled.) Are all technical performance parameters tracked and reported? In DOE projects. it is just as important to track performance and scope parameters relative to mission requirements as to track costs. DOE management should be continually aware of the ability of a project to fulfill its mission. In the past, DOE had a record not only of projects overrunning their budgets, but also of projects that had no use when they were completed. Is the contingency used or allocated tracked through the project? There should be a mechanism (configuration management) for tracking and reporting the release of contingency funds. Is the remaining unallocated contingency reported? The project manager should be able to know how much contingency is left and who controls it. Are the floats of all activities off the critical path controlled? Much attention is focused on the critical path. It is necessary to assure that this focus does not lead to some other path becoming critical. . . ~ . Readiness It is astonishing how many projects that push for early funding decisions turn out to be unprepared to start when the funding is approved. Consequently, they start behind schedule and then try to catch up. Management should determine whether all the prerequisites (the IPT, other project members, etc.) are in place for immediate startup when the project is authorized, and whether the project is capable of meeting the approved spending profile, before funding a project or even recommending a project for funding. . Is this project ready to begin when funded?

OCR for page 96
APPENDIX E 103 Start-up and Operations Operational factors are prime causes of uncertainty and performance failures. . . Have all operational factors and issues been identified in the Project Execution Plan and the Project Risk Management Plan? For some critical systems? For most systems? For all systems? Is there a start-up and turnover organization? This should be established early in the project. It is not acceptable to have spent millions, even more than a billion dollars, before determining who will be responsible for operations and maintenance of the completed facility. Are representatives of the operational users and of the start-up and turn- over organizations active participants in the IPT? Have they participated in risk identification and assessment studies? Project risk covers not only the risks of not meeting budget and schedule but also the risks of not meeting operational requirements. Have start-up issues been identified and a start-up plan initiated? The start-up plan controls the work breakdown structure in the start-up and turnover phase. In this phase, it is necessary to be able to measure progress not by quantities of materials installed but by completion of items on the punch lists required for system completion and turnover. RISK MANAGEMENT Critical Questions for Managing Risk Every project has a unique set of factors that may or may not increase cost, delay schedules, and reduce the end performance of the project. The factors need to be identified and understood and steps taken to minimize any impact they have on the project. Have major risk factors and events been identified? Have all risk factors affecting technical performance, cost, and schedule been identified? How many risk factors are there in each category? Has a probabilistic risk assessment or a failure modes and effects analysis been performed? If not, one should be performed. It should also be kept in mind that the purpose of this exercise is not to compute the probability that the project will fail, but to use probabilities to prioritize the risks to be eliminated, mitigated, or managed so that the project does not fail.

OCR for page 96
104 APPENDIX E . . Have all moderate risk factors been identified? (See above for major risks.) Have all minor risk factors been identified? The purpose of identifying minor risks is to be sure they are minor and do not have to be mitigated. Have common causes or root causes (common mode failures) been iden- tified? This is a major deficiency in DOE project risk assessments, in which typically all risks are assumed to be independent of each other. Insofar as this is untrue, the risk analysis is not conservative enough. Has a risk management plan been prepared for all the significant risk factors? It is the function of DOE management to assess whether project risks are acceptable. If not, when will it be prepared? Does the risk management plan have an item that lists specific actions to eliminate, reduce, or mitigate each risk factor? Without an action plan, the risk assessment is an academic exercise. Has the project manager signed off as accepting responsibility for the risk management plan? Is it recognized that the risk assessment plan is a continuous process, not a product? Some provision should be made for continual review and updating of the risk assessments. Technology Risks Some issues related to technological decisions may need research and devel- opment to resolve uncertainties about performance, efficiency, reliability, and other risks. DOE has often used major capital acquisition projects as R&D projects, to determine whether the technology will work after being scaled up by orders of magnitude. . . . If technology is a potential risk factor, have alternatives solutions or suppliers been identified for all technologies, processes, and systems with high or moderate risk factors? As in many other areas, diversification is one effective way to mitigate risks. Is there a backup plan, alternative plan, or a "Plan B" for every major technical, scientific, or engineering issue? In the past, DOE has locked in on one technology before it was proven to work. Even the Manhattan District had two technical alternatives (uranium and plutonium) to improve the probability of success. Effective project management requires plan- ning for the case that all the optimistic scientific projections do not turn out to be correct. How far are these alternatives carried in parallel through the design process? A budget should be provided for carrying parallel designs or technology development, because having technological options increases project value.

OCR for page 96
APPENDIX E . . . 105 At what decision point will alternatives be selected or eliminated? These decision points should be identified beforehand, in the PEP. They will probably be developed by working backward from the required project delivery date to the latest time by which a decision must be made to avoid delaying completion. Has all the information necessary to making a technological decision been identified? Have all the technical challenges and/or issues needing research and devel- opment been identified, in order to eliminate, to mitigate, or to reduce risks to acceptable levels? Have these risks been documented system by system? Is the necessary R&D program under way? It is prudent to perform the necessary R&D prior to commitment to a capital acquisition. If not, when will it start? When will it finish? Will all the R&D issues be resolved prior to the critical decision points? Give a schedule for resolution. Prototypes, Pilots, and Bench-Scale Tests Prototyping is one effective way to resolve risks prior to commitment to a full project. Cutting the budgets for prototyping is not a cost-effective method. DOE management should assure at critical decision points that sufficient prototyping or R&D has been completed to support the decision to proceed. Have prototypes been identified and built where needed to resolve ques- . lions (risks) concerning project solutions? If not, when? Are these bench scale or laboratory scale? Are these prototypes pilot plants? What is the scale-up factor from the prototype to the real system? Is this scale-up reasonable? How does risk increase with scale-up factor? Will the prototype resolve all the outstanding technical issues prior to the critical decision points? Provide the schedule for resolution. Regulatory Risks Regulatory requirements should, for the most part, be predictable. Managers need to identify which regulatory agencies and which regulations will be applied to the project. nave environmental, safety, and health issues been identified? Have public relations issues been identified and a public affairs office established?

OCR for page 96
106 APPENDIX E . . Have all regulators been consulted about possible regulatory issues and delays with the project? Regulatory issues are important in EM projects and may become more important for all DOE projects. Are important regulators part of the working IPT? Regulators should be involved with the project from the start. That is, regulators should be considered part of the project, not external to it. Are regulatory decision points identified on the project schedule? It is not adequate to simply assume in making a schedule that regulatory decisions will be made automatically. Are regulatory decision points on the project critical path? If so, why? Are there alternatives? Schedule Risks Are there backup plans in case of regulatory difficulty? Is there a Level 0 network schedule (in addition to bar charts or Gantt charts)? Is there a Level 1 network schedule? If not, when will it be available? Is there a Level 2 network schedule? If not, when will it be available? Is there a Level 3 network schedule? If not, when will it be available? QUESTIONS FOR FIRST-OF-A-KIND PROJECTS First-of-a-kind projects are usually inherently riskier, but sometimes risky projects need to be approved as long as they are managed accordingly. Before making critical decisions senior managers should ask critical questions to ensure that there has been sufficient planning to consider all risks and sufficient analysis to manage them. Some questions that management might wish to pose, espe- cially for first-of-a-kind projects, are these: Have all the necessary technical specifications been identified? Are all the technical specifications complete? If not, for how many sys- tems are the technical specifications complete? What is the percentage of completed technical specifications (of all systems)? Have all systems been identified? What is the likelihood that new systems will be added? What is the likelihood that the technical specification will require rework, iteration, and recycling through the approval channels? Is there demonstrated adequate reserve margin in project technical perfor- mance specifications? Who controls the technical performance reserve margin? What is the recovery plan in case there are performance shortfalls?

OCR for page 96
APPENDIX E 107 Simulation Models Is there a simulation model for project performance? If not, should there be one? What questions should the simulation model address? How long would it take to develop and when would results be available? Will the simulation assist in understanding the outstanding technical, cost, or duration issues prior to the critical decision points? Give a schedule for resolution. (To be useful, results should be obtained prior to the scheduled date of a decision that depends on those results.) Is this model based on firm and well-understood scientific principles, or on speculation and subjective judgments? Has this model been used to address what-if questions? Has there been a sensitivity analysis of the impacts of project technical performance shortfalls? How long should alternatives be carried on in the project? How often and by whom will reviews and analysis of the critical parameters values be evaluated? What should be done in case of performance shortfalls? What is the highest degree of uncertainty that allows proceeding to the next phase? . . CONCLUSIONS Projects have been conducted in which serious problems could have been avoided if DOE management had asked some critical questions often simple ones. DOE managers should consider asking some of the questions outlined above at critical decision points, to avoid unanticipated delays, costs, or other obstacles to completing the project successfully, and to ensure that projects are truly ready to proceed.