C
Summary Notes of a Workshop on DOE EIRs

The DOE External Independent Review Workshop was conducted June 8, 2006, at the Keck Center of the National Academies, Room 203, 500 Fifth Street, NW, in Washington, D.C. The agenda and list of workshop participants appear at the end of this summary.

The Committee on Assessing the Results of External Independent Reviews for U.S. Department of Energy Projects conducted a fact-finding workshop to solicit comments and opinions from DOE staff, including project directors, program and project management support managers from the Office of Engineering Construction and Management (OECM) and the program secretarial offices (PSOs) Operations and Management (O&M) contractors, and external independent review (EIR) contractors. The workshop consisted of open discussions to address a series of questions regarding the planning and execution of EIRs, EIR reports and corrective action plans, the value of EIRs, and suggestions for improving EIRs in the future.

The following summarizes the discussion based on notes taken by Michael Cohn, NRC program officer. The summary reflects the comments of the participants but is not a verbatim transcript. The comments were interpreted and condensed so as to allow focusing on the issues being considered by the committee. The sources of the comments are identified by job title or affiliation to provide context.

PLANNING AND IMPLEMENTING REVIEWS

Tailoring

Committee: Please comment on the process used to plan EIRs, the people involved in planning, and if the reviews are effectively tailored to the conditions presented by individual projects.


Federal project director (FPD): EIRs definitely need to be tailored. Each review should take into account the maturity and type of project being reviewed. New capital construction projects are different from a continuing operation or new baseline in a continuing acquisition. The 16 elements in the EIR guideline can be used, but the emphasis needs to be adapted to the project. It is important that the reviews be planned with the federal project director (FPD) as an active participant. The planning process also needs to recognize that the projects continue to mature as the EIR planning proceeds. Documents are given to the review team 5 weeks before the review, but they are not static. A lot of communication between the project team and the review team is needed to keep the review team current. The project team’s involvement is also needed to guide the preparation of on-site briefings.


Program/project management support office (PMSO) manager: It is important for the review team to recognize the implications of the various types of contracts used by DOE. A cost plus incentive fee contract is very different from a fixed-price contract. In the latter type of contract, the contractor bears the risk and the role of the EIR changes. The EIR for a specific project should reflect how risk is identified and quantified and what roles DOE staff and contractors play in managing risk.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects C Summary Notes of a Workshop on DOE EIRs The DOE External Independent Review Workshop was conducted June 8, 2006, at the Keck Center of the National Academies, Room 203, 500 Fifth Street, NW, in Washington, D.C. The agenda and list of workshop participants appear at the end of this summary. The Committee on Assessing the Results of External Independent Reviews for U.S. Department of Energy Projects conducted a fact-finding workshop to solicit comments and opinions from DOE staff, including project directors, program and project management support managers from the Office of Engineering Construction and Management (OECM) and the program secretarial offices (PSOs) Operations and Management (O&M) contractors, and external independent review (EIR) contractors. The workshop consisted of open discussions to address a series of questions regarding the planning and execution of EIRs, EIR reports and corrective action plans, the value of EIRs, and suggestions for improving EIRs in the future. The following summarizes the discussion based on notes taken by Michael Cohn, NRC program officer. The summary reflects the comments of the participants but is not a verbatim transcript. The comments were interpreted and condensed so as to allow focusing on the issues being considered by the committee. The sources of the comments are identified by job title or affiliation to provide context. PLANNING AND IMPLEMENTING REVIEWS Tailoring Committee: Please comment on the process used to plan EIRs, the people involved in planning, and if the reviews are effectively tailored to the conditions presented by individual projects. Federal project director (FPD): EIRs definitely need to be tailored. Each review should take into account the maturity and type of project being reviewed. New capital construction projects are different from a continuing operation or new baseline in a continuing acquisition. The 16 elements in the EIR guideline can be used, but the emphasis needs to be adapted to the project. It is important that the reviews be planned with the federal project director (FPD) as an active participant. The planning process also needs to recognize that the projects continue to mature as the EIR planning proceeds. Documents are given to the review team 5 weeks before the review, but they are not static. A lot of communication between the project team and the review team is needed to keep the review team current. The project team’s involvement is also needed to guide the preparation of on-site briefings. Program/project management support office (PMSO) manager: It is important for the review team to recognize the implications of the various types of contracts used by DOE. A cost plus incentive fee contract is very different from a fixed-price contract. In the latter type of contract, the contractor bears the risk and the role of the EIR changes. The EIR for a specific project should reflect how risk is identified and quantified and what roles DOE staff and contractors play in managing risk.

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects Planning Process FPD: There does not seem to be a clear process for FPDs to become involved in planning and tailoring the EIRs, nor is there a way to ensure that the FPD’s concerns are addressed by the review. OECM: The planning process has evolved in the past few years with increasing participation of the FPDs. Nonetheless, OECM has the final say on the scope of the EIR, which is intended to support validation of the baseline, and serves as an assessment of the readiness of a project to proceed. The process varies depending on the OECM, program, project, and contractor personnel involved. The intent is to have a collaborative process. Program/PMSO manager: The Office of Science (SC) recently issued a directive that all EIR planning should be approved by the PMSO. SC will require agreement on the scope, review team, and costs before providing funds for the review. This had been discussed in the past, but the procedure had not been clearly defined. In the past, SC relied on the site office to ensure the quality of the EIR but was not sufficiently involved in the planning decisions. The process needs to be more collaborative than it has been. We need to assess the objective of the EIRs and focus more on the baseline cost and schedule and less on compliance with the details of acquisition policy. Small Projects Program/PMSO manager: The SC laboratory infrastructure program undertakes many projects that are significant (generally from $5 million to $20 million), but relatively small and less complex when compared to the new capital acquisition projects in SC. In some cases, tailoring has been applied to these small projects, eliminating the on-site portion of the review and relying solely on the review of documents and conference calls. These so called “desk-top reviews” do not work. The EIRs are of value to these projects not only as a validation of the baseline but also as a means to transfer lessons learned and develop the knowledge and abilities of the junior project staff who are routinely assigned to the smaller projects. Even with the proposed shift in the lower limit for mandated EIRs to projects costing more than $100 million, we would like to continue conducting EIRs for selected laboratory infrastructure projects. OECM: Proposed changes to O 413.3 for projects less than $100 million will allow either OECM or the program office to manage the review. OECM will provide the contract for employing the EIR contractor, but PSOs programs will be responsible for selecting the contractor and planning the review. EIR contractor: EIRs provide exposure to lessons learned on past projects and training opportunities for less experienced project directors assigned to the smaller projects. This opportunity could be lost if reviews are not conducted for projects with less than $100 million total estimated cost (TEC). OECM: Reviews will continue to be required, but they will be the responsibility of the program or site office. They will have the latitude to use either internal or external reviews. M&O contractor: There is value in having an independent review, even for smaller, less complex projects. In the past, however, some of the reviews were too detailed for simple infrastructure projects, with the same EIR process having been used for them as for large new construction projects. Program/PMSO manager: There has often been a cookie cutter approach to planning EIRs. They are all treated the same and follow the same lines of inquiry and the same level of detail regardless of the size and complexity of the project. OECM insists that scope is not negotiable, because the independence of

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects the review would be compromised if the program or project teams were allowed to have significant influence on the scope of the review. The result of this insistence has been ineffective tailoring. Review Criteria Committee: If program offices were responsible for determining if an EIR is needed, what criteria would they use? Program/PMSO manager: The complexity of the project would be the primary criterion. The experience of the project team would also be important. Committee: Should EIRs for smaller projects include all the current lines of inquiry? Should the emphasis be on baseline validation or project manager development? Program/PMSO manager: Baseline validation is the primary purpose. Most infrastructure projects are developed as design-to-budget projects, and DOE needs to know that what is proposed can be accomplished for the allotted budget. Program/PMSO manager: If O 413.3 is revised as proposed, the project management support office in SC will conduct independent reviews of all projects with TECs between $5 to $100 million, including laboratory infrastructure projects. The reviews will be tailored to critical issues presented by each project. They will look at the technical design issues, cost and schedule baselines, and, most important, the readiness of the management team. Past EIRs focused more on compliance with procedural requirements and less on the effectiveness of the management team. EIRs Versus IPRs Committee: Will SC continue to conduct EIRs for projects under $100 million TEC or rely solely on IPRs? Program/PMSO manager: SC believes that reviews conducted by SC personnel who are not affiliated with the project are independent. SC will primarily use IPRs and only use EIRs as needed. Program/PMSO manager: One of the problems with past EIRs is that the review team did not know enough about the project or the technical issues. A review team is definitely independent, but that is not enough to add value to the project. Program/PMSO manager: The previous statement describes what happened during the course of several SC projects, particularly those involving highly technical physics experimental systems. Committee: How do the Office of Environmental Management (EM) and the National Nuclear Security Administration (NNSA) anticipate conducting reviews for projects with a TEC of less than $100 million? Program/PMSO manager: The change in the dollar value threshold is not likely to have much of an impact on EM since most EM projects are greater than $100 million TEC. EM conducts IPRs that are similar to SC’s reviews prior to the EIRs and independent of the EIR approximately every 9 months. The IPRs rely on EM staff from headquarters and field offices and on consultants.

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects Program/PMSO manager: NNSA conducts IPRs prior to CD-1 and CD-3 (except for major systems, when OECM conducts EIRs). If the threshold is changed to $100 million TEC, NNSA will be conducting more IPRs for baseline validation prior to CD-2. IPRs primarily use NNSA personnel from other sites. Contractors are used if the needed expertise is not available in NNSA. FPD: I have found NNSA IPRs to be very helpful. Program/PMSO manager: NNSA will conduct an IPR when issues arise and the project team determines that it needs outside review and input. Review Teams Committee: For projects with a TEC greater than $100 million that will continue to have EIRs planned by OECM and executed by contractors, is there a need to pay more attention to the technical subject matter experts on the review teams? Program/PMSO manager: Having a review team with the necessary technical experts is a critical issue for SC. Because of the specialized nature of SC projects, this expertise is difficult to find outside SC. The Liniac Coherent Light Source (LCLS) project is an example of the kind of project where a lack of expertise on the review team can cause delays and reduce the effectiveness of the EIR. Committee: Could SC, or other program offices, work with OECM to identify the experts that need to be on a team without infringing the independence of the review? EIR contractor: The EIR contractors bring experts to the review teams to address technical issues and would include specific expertise if so encouraged by DOE. In some cases, particularly joint IPR/EIR reviews, the contractors have been told not to provide the expertise because the areas requiring that expertise would be addressed by DOE. EIR contractor: Most of the contractor’s EIR personnel are independent experts who are called in as necessary to take on the issues presented by a specific EIR assignment. Contractors generally respond to OECM suggestions. Coordination of EIRs and IPRs Committee: What are the factors that determine the extent of coordination, sharing of expertise, and joint activities for IPRs and EIRs and if the reviews are conducted sequentially or concurrently? Program/PMSO manager: Joint sessions with open meetings and breakout sessions are an effective model. EIR contractor: Problems can arise because the IPR and EIR teams are competing for the same project resources to conduct their reviews, but this can be overcome with sufficient planning and coordination. Committee: Do joint IPR/EIR reviews blur the lines of accountability for the outcome of the review? EIR contractor: The juxtaposition of two teams with separate leadership increases the challenge of coordinating the review agenda and arriving at recommendations.

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects Program/PMSO manager: Joint IPR/EIR reviews generally share data and conduct simultaneous briefings but each team develops it own report and recommendations. EIR contractor: Reviews have included a combination of joint and separate sessions that are focused on the needs of each project. This variety of arrangements provides an opportunity to share project resources and individual observations. EIR contractor: The EIR and IPR teams often focus on different aspects of a project; e.g., an EIR might focus on cost and schedule and an IPR on scope. Because project outcomes are determined by the interaction of these issues, each team needs to be aware of the changes being recommended by the other. Conducting the reviews simultaneously makes coordination more difficult. Committee: Are reviews planned and coordinated for the life cycle of a project? Program/PMSO manager: In addition to the reviews set by DOE policy, individual program offices plan project review schedules independently to support their needs. Committee: Does DOE plan on using the same review participants through the entire life of the project or does it plan to adjust the expertise in response to changing project issues? Program/PMSO manager: IPR teams selected by the program offices do try to balance continuity and the changing needs of the project. Committee: If the IPR addresses scope and the EIR cost and schedule, is it helpful to the EIR team if there is an IPR prior to the EIR? EIR contractor: Yes, it is necessary to set the scope before the cost and schedule can be validated. Generally, if an IPR has been conducted, the results are available to the EIR team. The EIR team does not undertake a technical design/scope review but verifies that one has been completed and outstanding issues have been resolved. Committee: It appears that EIRs and IPRs are conducted either concurrently or consecutively. What are the advantages or disadvantages of these two approaches? Program/PMSO manager: For a large project, concurrent EIR/IPR reviews are less disruptive and require less of the project team’s time. FPD: Consecutive reviews benefit the project team because the IPR can serve as a high-level review, identifying issues that can be corrected before the EIR. It can also be used to help focus the EIR so that it is more effective. Program/PMSO manager: Consecutive reviews offer more learning opportunities for less-experienced project managers. The cost of an EIR is not insignificant. Conducting an IPR before the EIR facilitates assessment of the readiness of a project for baseline validation. Program/PMSO manager: IPRs also cover readiness issues not covered in EIRs, such as safety and management concerns. Neither approach is clearly better. The best approach is determined by the circumstance of the project and the surrounding issues. EIR contractor: Every EIR is tailored. They cover the same 16 lines of inquiry but to varying levels of detail. They may appear similar on the surface, but they are all tailored to the issues presented by the project.

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects Committee: An effective IPR should identify outstanding issues; the EIR would then only verify that they had been resolved. Is an EIR effective if it does not find any issues that were not identified and resolved? OECM: We have never come across such a case, but the answer is yes. The goal is to validate the baseline, not find mistakes. Committee: Is the timing of IPRs and EIRs properly distributed in the life cycle of a project? Does the timing support continuous process improvement? Program/PMSO manager: We would hope there is continuous process improvement and we would hope that at some point we get good enough at project management that we will need fewer reviews. Lessons Learned Committee: Does the current program of reviews have a mechanism to capture lessons learned from one project and transfer them throughout DOE? OECM: There is currently no DOE-wide lessons-learned system for project management. Some of the programs have undertaken initiatives to identify and distribute lessons learned. Program/PMSO manager: NNSA is developing a lessons learned database by mining past IPRs. NNSA also conducts workshops where project managers share the successes and problems experienced on various projects. Peer Review or Audit Program/PMSO manager: The EIR process was established to ensure senior managers that policy and procedures are being followed and that project management outcomes meet a vaguely defined level of quality. The EIRs are like audits while the IPRs are more like a peer review. The audience for the first EIRs was Congress. Reports were sent directly to the appropriation committees. The audience has shifted more to DOE senior managers, but Congress is still watching. EIRs are becoming more like peer reviews, but they are still considered by field staff to be audits. Committee: How can that staff perception be changed? Program/PMSO manager: Changing that perception will be difficult because people from the outside the PSO are viewed as adversaries because the customers for an EIR are the OECM, the senior management, and Congress, not the PSO. The reviews are not being conducted for the PSO’s benefit. The external perspective is important for the larger projects but it creates an adversarial atmosphere between the PSO, the OECM, and the EIR contractors. FPD: FPDs do not feel that they can influence the scope or focus of the EIR. EIR contractor: From our point of view, OECM is the customer. Although it is clearly beneficial to involve the program and project in the planning process, that is not the way planning is being done. From the contractor personnel’s perspective, OECM determines the scope of the review. If others are involved in the planning process, there needs to be one DOE person who can resolve conflicts that may arise. We try to overcome the perception that the EIR team members are auditors, but the limited involvement of program and project personnel in the planning process makes that more difficult.

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects Planning Process OECM: There should be broad participation from the project and program in the planning process. In some cases the PMSOs choose not to participate. OECM needs to ensure that the scope of the review will allow verifying the baseline. The quality of communications in the planning process depends on the people involved in a project. Program/PMSO manager: Neither the planning process itself or the roles of the program and project in planning EIRs are clearly defined. In one case the program and project took the initiative to define the scope of the EIR, which then became the basis for discussions with OECM and the contractor. The results were very productive. EIR contractor: For a recent project, the contractor developed a review plan, which was then distributed to OECM, the program, and project for comments. It was revised and all parties signed off on it before the site visit. This seemed to provide sufficient communication for a relatively straightforward project. EIR contractor: The 16 elements in the lines of inquiry are fairly inclusive. Most of the tailoring involves the attention to various details within these elements. Committee: It appears the current process does not work for mega projects. Are alternatives to the EIR, such as U.S. Army Corps of Engineers (USACE) reviews and project-contractor-led “best and brightest” reviews, more appropriate for mega projects? Program/PMSO manager: EM is seeing good results from the “best and brightest” approach at CD-1 to ensure that projects are well defined from the beginning. This should make EIRs more effective at CD-2. For EM, many of the issues relate to environmental safety and health (ES&H). OECM: We probably should put more emphasis on CD-1. If the project plan and structure are fully aligned at that point, the project will be more likely to have a valid baseline at CD-2. OECM: We need to separate the need for external, fully independent peer reviews from the mission assigned to OECM—that is, to validate the baselines. Independent Cost Estimates Committee: There does not appear to be much reference to independent cost estimates (ICE) as a part of the current review process. Are they still used? Program/PMSO manager: There is a role for them, but it is not well defined in O 413.3. After not conducting ICEs for a while, we are just starting to do them again. The USACE is doing one for the Hanford Waste Treatment and Immobilization Plant (WTP) project, but that is an exception. FPD: The value of the ICE depends on the specific circumstances of a project. Program/PMSO manager: EM has cost-estimating capability in house. Committee: What are the current criteria for ICEs? Program/PMSO manager: An ICE is a bottom-up cost estimate. This is different from a cost review or a government estimate. Generally DOE does not have a lot of cost data or cost estimating capability. We need independent input. ICEs are generally not part of EIRs.

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects EIR contractors: EIRs usually include a focused independent cost review (ICR) that reviews cost estimating assumptions for selected work packages. A bottom-up estimate is usually a major undertaking that is too intensive to be part of an EIR. Assessing the Management Team Committee: Can an EIR evaluate the adequacy of a project management team? OECM: Yes. They do that by evaluating the performance of the project team. EIR contractor: The EIR usually does not address the qualification of individuals, but will usually comment if a potential problem is observed. Program/PMSO manager: The EIR teams often comment informally about the performance and potential performance of the project team. Program/PMSO manager: The government spends a lot of time and effort in source selection and does not need an EIR to second guess its source selection by evaluating the contractor’s project management team. Program/PMSO manager: The management capabilities of the project team are critical. Judgments about their ability to execute the project need to be an internal matter. The EIR can identify potential problems, but responsibility for determining if a change is needed stays with the department. EIR contractor: The O 413.3 and the EIR directives have not been adapted to design-build projects. The project management requirements and the EIR process need refinement. Reviews need to be incorporated into the project management plan. Value of On-site Reviews Committee: Do EIRs need to include on-site reviews or can they be accomplished off-site by conference calls and a review of documents? EIR contractor: The value of face-to-face meetings and seeing the job site is difficult to know, but that direct contact definitely helps an EIR team evaluate the project. Some parts of a review can be done off-site but they need to be focused. FPD: For large projects, the EIR team definitely needs to be on-site. EIR contractor: It is not ideal, but when necessary, individual members can participate from a distance. The goal should be to have the entire team on site. DOCUMENTATION OF REVIEWS AND CORRECTIVE ACTION PLANS Review Findings and Recommendations Committee: What is the best way to communicate the outcome of a review to the project team and the OECM?

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects EIR contractor: Generally there is an open session briefing by the EIR team at the end of the on-site review. The session brings together the project team and other contractors and DOE personnel involved in the project. People who are not at the site are linked by phone. View graphs are prepared and distributed the night before. The session includes time for discussion. The EIR team then prepares a written report, which is transmitted to OECM about a week after the on-site review. OECM reviews the report to determine if it is ready to send to the site office. The project team reviews the report for factual accuracy. The final report includes an outline of recommendations for the corrective action plan (CAP). It is an iterative process. OECM: We have also conducted some EIRs where the final report was submitted before the team left the site. This approach is still being evaluated. EIR contractor: For a large project whose review may take several weeks, there are periodic briefings of findings rather than a single briefing at the end of the review. This allows the project team to resolve most issues before the review team leaves the site. Committee: The term “corrective action plan” may have negative connotations. Would it help if such plans were called “action plans” or “follow-up plans”? The new terminology might also change the false perception that an EIR is an audit. OECM: That issue has never come up. EIR contractor: The report sometimes includes appendixes that identify best practices that the project team could use to improve the management of the project. Program/PMSO manager: In reality, the review team does not always follow this process. When the program office or project team does not agree with the EIR team the process gets bogged down. Generally, there is insufficient attention paid to the project team’s perspective. There should be an option that allows the project team to disagree with the review findings. EIR contractor: The reports have three levels of findings. Only the most significant findings require corrective actions. The project team can disagree and ignore the less important findings and still have the project baseline validated. In the end, it is up to OECM to determine what actions need to be taken to validate the baseline. Committee: Is there a problem with review teams adding findings to the final report that were not discussed at the site briefing? Program/PMSO manager: If the review team cannot identify all the major findings before they leave the site, they are not doing their job. The project team needs an opportunity to discuss the outcome of the review and, if necessary, voice objections. EIR contractor: Issues that arise after the site briefing are discussed with OECM. The FPD may also be brought into the discussion. There is no set procedure for communicating additional findings. Corrective Action Plans Committee: What happens to CAPs? Are they tracked beyond validation of the baseline? Program/PMSO manager: CAPs are implemented immediately. Some of the responses are ongoing and will continue until the end of the project.

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects OECM: In some cases a plan of action is sufficient; in others, the plan needs to be completed or implemented. OECM needs to determine, based in part on its judgment, when to validate the project. EIR contractor: Many of the recommendations address the cost and schedule issues that should be addressed before validation. By segregating the recommendations into three levels of importance, the review reports also sets priorities for action. The significant findings are generally those that need to be addressed before the baseline is validated. The EIR contractor is providing recommendations, but it is up to OECM to determine what is required from the project before it is allowed to proceed and up to the program to determine what requires long-term attention. VALUE OF EXTERNAL INDEPENDENT REVIEWS Committee: Is there a sense that EIRs result in a better outcome for a project? Program/PMSO manager: A project is generally improved by conducting IPRs and EIRs, but it is difficult to determine if the benefit comes from the project team’s preparation for the review or the review itself. Anticipation of a review increases the rigor of the project team’s planning and documentation. Program/PMSO manager: In some instances the EIRs did not identify any issues that the program and project team were not already aware of. The concept of the EIR is valuable, but the implementation does not always provide the anticipated value. Problems have occurred when communications falter and EIRs address issues that are not germane to the project. For SC, an IPR is more productive. The relative importance of the standard 16 lines of inquiry can vary greatly from project to project. The review team does not always identify which issues are critical and which will not affect the outcome of the project. The EIR is not a substitute for effective project management. SC tries to have a transparent process so that OECM should know beforehand if an EIR project is ready to proceed. EIRs do not add value that is consistent with their cost. Program/PMSO manager: Projects receive benefits from EIRs, but there is a question about their value and if the benefit justifies the cost. The program offices pay for the EIRs. Since 1999 DOE has expended more than $50 million on EIRs. NNSA is spending $5.3 million this year, which is more than four times PMSO’s budget to provide IPRs, training, and other activities that are needed to ensure effective project management. That is why we question the value of EIRs. They need to be done, but they should be limited to the large projects where they can make a difference. The cost of an EIR is probably offset by reduced project costs, but that money is not available for the other activities that need to be done to improve project management. From a program perspective, an IPR provides much greater value. Committee: There was earlier reference to a lessons learned program to increase the benefits gained from reviews. Does that apply to EIRs? Program/PMSO manager: That was an NNSA program that applies to IPRs only. It could be used for EIRs. Committee: Why are EIRs not funded out of project funds? Program/PMSO manager: DOE policies determine the funding sources. EIRs will be funded through the working capital fund (WCF) starting in FY07. Programs will contribute operating dollars to the WCF to pay for the reviews. Committee: Are the projects required to conduct IPRs?

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects Program/PMSO manager: IPRs are only required prior to CD-3. Most IPRs are done at the discretion of the PSO. SC performs IPRs more frequently than the other programs. NNSA conducted 35 IPRs last year that were requested by the FPD or the program office. EM conducts IPRs prior to CD-1 and then through CD-3. Committee: Could IPRs replace EIRs? Program/PMSO manager: Not entirely, especially for the larger projects and for the program offices that do not have PMSOs. Committee: What is the average cost of an EIR? Program/PMSO manager: The average cost is approximately $150,000. One recent EIR cost $1.2 million. EIR contractor: EIRs for smaller projects with a three-member review team cost approximately $50,000. Committee: The rate at which projects were rebaselined drove the need for EIRs. Has project performance been improved by conducting EIRs? Could internal reviews achieve the same results? OECM: The complexity of the issues and number of external factors affecting project performance make it difficult to measure the impact of reviews. EIR contractor: There are examples of projects that have been canceled based on EIR findings. Here, EIRs clearly provided a benefit. O&M contractor: For one fairly large but not complex project, the EIR identified approximately 20 issues; however, none of them were of any consequence. There are also examples of projects that underwent several EIRs before the project progressed to the point that it could proceed past CD-2. Program/PMSO manager: Benefits are definitely gained from conducting EIRs, but they should be limited to larger projects that have the potential to embarrass DOE. It is easy for an EIR or IPR to provide significant results when it is applied to projects that are known to be in trouble. The challenge is identifying the precursors of those problems and applying corrective action before the problems arise. It is also more difficult to assess the impact of preventive actions that result in cost avoidance as opposed to cost savings. Committee: Does a project that experiences problems after an EIR and acceptance of the CAP mean the EIR and the validation process have failed? Program/PMSO manager: Every successful project and every problematic project has had an EIR or IPR. The question is not the type of review but whether or not it addressed the critical issues. Sometimes the critical issues were not identified, sometimes the risks could not be avoided, and other times external requirements, such as the design basis threat, are not known at the time of the review. The interval between reviews can contribute to the impact of all these scenarios. There are instances when problems arose because the designs were not ready for construction or the project team did not understand the bidding market, but the IPRs and EIRs did not assess those aspects of project management. Committee: Should the review team have recognized these issues as a threat to the project baseline?

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects OECM: We see evidence that in some instances the EIRs need more rigor, but at the same time there are pressures to reduce the cost of reviews. EIR contractor: The EIRs are an evolving process. They used to be conducted in greater depth and included larger samples. Cost pressures have led to reduced scopes and smaller samples. The challenge is to identify the critical projects. Committee: Is the fact that a project has not undertaken an IPR or EIR for an extended period of time a leading indicator of management problems? Program/PMSO manager: It could be a sign that the process is not open or transparent. Committee: Does the value of an EIR depend on employing the right EIR contractor for the project, and how can that contractor be identified? Is there a process to assess the capabilities of the various EIR contractors? Program/PMSO manager: Yes, the experience and expertise of the review team are critical to the success of the review. The programs give OECM feedback on the quality of review teams, but the process is informal. OECM: A more formal assessment process is under development. Committee: How are the contractors selected? OECM: The program offices are asked if they have a preference, but OECM has the primary responsibility for selecting the contractor. Factors such as previous experience at a site, expertise, and current workload are used to select from the group of contractors with multiyear indefinite quantity contracts. We just added two additional contractors to the group for a total of five, which seems to be sufficient. Committee: Do the contractors receive enough feedback to help them improve future reviews? EIR contractor: Contractor performance is reviewed annually by OECM, but there is no feedback on particular projects. OPPORTUNITIES FOR IMPROVEMENT Committee: Could the composition of the EIR team be improved by including people designated by DOE as having the required expertise? Would DOE’s involvement in selecting the team interfere with the contractor’s control of the EIR? EIR contractor: Using DOE-designated participants on the review team is not a problem as long as they have no conflict of interest, i.e., no prior or future involvement in the project. Each member of the team contributes to the final review report, but the contractor controls the content of the final report. The externality of the review would be compromised and its independence could be questioned. However, DOE’s participation in the selection would be appropriate if it was necessary to obtain the necessary expertise. The final judge of the independence of the review is OECM. OECM: We try to keep it as independent as possible, but the objective is to obtain a review team that is both informed and independent.

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects EIR contractor: Members of the review team are relied on to maintain their professional integrity. Committee: Could project outcomes be improved or costs reduced with better coordination of IPRs and EIRs? FPD: If sufficient time elapses between the IPR and the EIR they can form a complementary iterative process. The key is timing. There are examples where this has worked. OECM: The EIR team is informed of the outcome of prior reviews, which helps the team to better focus its review. Committee: Is there a more efficient method for conducting EIRs and completing the review report? EIR contractor: Giving the contractor more lead time improves selection of the review team and planning of the review. The cost of the review may not be reduced but it will be more effective. OECM: For some NNSA reviews, the review team was kept on site until the report was completed. This reduced the time required to complete the report, but it increased the cost and presumably did not affect the quality of the review. EIR contractor: It generally takes about a week to develop a draft EIR report. This seems to be a reasonable time and applies whether the report is written on-site or back in the office. Because the reports and the review team briefings are preceded by dialogue between the review team and the project team, the final briefing and report should not hold any surprises for the project team. The review benefits somewhat from having time to reflect and putting some distance between the review team and the project site. EIR contractor: OECM has developed a guide for the reports that facilitates the writing process and makes the reports more consistent. Committee: During the project reviews the committee heard that early planning and close coordination with the project team, on-site participation of OECM, and discussion of preliminary findings led to an effective review. Is there anything to add to this list? OECM: Pre-EIR site visits also seem to increase the effectiveness of the reviews. Committee: Would projects benefit more from reviews (both IPR and EIR) if they were conducted periodically rather than prior to CDs? Program/PMSO manager: SC tries to conduct reviews on a 6-month schedule but also tries to schedule them ahead of CDs. A 6-month interval is typical after construction starts. Reviews are also conducted when performance issues arise. Reviews for NNSA and EM projects are not as frequent or regular. Committee: Are there any concerns about the proposed changes to O 413.3 that will eliminate the requirement for EIRs for projects with TECs of less than $100 million? EIR contractor: Contractors are not unhappy about this change because there will continue to be a sufficient workload. They have seen improvement in DOE project management activities that may be the result of the EIRs. If the IPRs are implemented for projects with TEC less than $100 million and project managers continue to be held accountable, then this improvement should continue. Some form of review will continue to be needed.

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects New Issues EIR contractor: The reviews are conducted when the design is approximately 30 percent complete. The review focuses primarily on cost and schedule to deliver the scope as defined at the time of the review. The review does not verify that the scope is accurate, and subsequent baseline changes are often the result of changes in scope that are implemented during completion of the design. Should the EIRs be delayed until the design is more complete and the scope becomes fixed? Committee: A scope can be clearly defined at that point in a project when design is 30 percent complete. Lack of a clear scope is a failure of project management. An assessment of the completeness of the scope definition should be part of the EIR. Program/PMSO manager: DOE has adapted a project definition rating index (PDRI) tool developed by the CII. The tool helps project managers determine if the scope has been defined sufficiently to allow the project to proceed. It is used by EM and NNSA for IPRs prior to CD-1. OECM: The EIR should occur when the project is ready to be baselined. If the scope is not defined, the project is not ready. If the design needs to be more than 30 percent complete before the scope can be fixed, then the project probably should wait. The need to have a well-defined scope is often ignored because various external budget or political pressures make it urgent to start the project. Following a call for final comments, the workshop was adjourned at 4:00 p.m. WORKSHOP AGENDA June 8, 2006 8:30 a.m. Welcome and Introductions–Joseph Colaco   Objectives of the Study and the Workshop–Michael Cohn 9:00 a.m. Workshop Discussions–Gary Sanders, committee member, workshop facilitator   Planning and Implementing EIRs   Are EIRs planned effectively to address the project issues that have the greatest impact on the performance baseline? Are the reviews tailored consistently, or is there evidence of inconsistent response to project conditions? Who should be responsible for tailoring the review? Should EIRs be tailored by adjusting duration, team expertise, issues/project management documents reviewed, or other aspects? Should EIRs be tailored to program requirements (i.e., a different starting point for each program) and then tailored to the specific project requirements? Are there a limited or infinite number of approaches? Are EIRs carried out with some consistency? Do EIR teams have all of the necessary expertise and experience? Which types of expertise and experience are missing? Which are not necessary?

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects   Is there a separate role for an independent cost estimator? Should an ICE be part of an EIR or separate? Is the time needed to complete a review consistent with the complexity and importance of the issues motivating it? Can EIRs be effective without on-site activities? Are CAPs developed, communicated, and implemented effectively? Are they (or should they be) part of the project management system and tracked beyond validation of the baseline? Does the involvement of multiple parties (e.g., OECM, program HQ, DOE project personnel, contractor’s project personnel, EIR contractor) lead to communication problems? Are the current 16 elements in the EIR Standard Operating Procedures document appropriate? Are there elements that should be included in all reviews? Conversely, should some elements be considered in exceptional circumstances only? Is there (or should there be) a follow-up to improve future EIRs, and are such follow-up findings used? Do the advantages (e.g., knowledge of DOE procedures and expectations for EIRs) of using a limited number of EIR contractors outweigh the disadvantages (e.g., risk of losing independence)? 12:00 p.m. Lunch 1:00 p.m. Documentation of EIRs and Corrective Actions   Is the review plan formulated jointly by the program office, the OECM, the review contractor, and the project team documented sufficiently? Do all parties have a clear understanding of the EIR objectives? Are results of EIRs communicated effectively at the time of the site visit and, in written documentation? Do review reports produce findings characteristic of an audit, checklist review, and/or a substantive analysis? Is there a way to collect and transfer EIR lessons learned? Are there continuous project and program records of all reviews (IPRs, EIRs, etc.), CAPs, and corrective actions? Are these data tracked and compared to project performance? 2:00 p.m. Determining the Value of EIRs   Do reviews consistently produce findings and recommendations that impact (improve or reduce risk) the project baseline cost, schedule, and scope, and risk management and project management strategies? How often are EIR findings and recommendations of little or no consequence? How often are there problems in projects that should have been identified and corrected during an EIR? Do projects that experience problems after CD-2 indicate a weakness in the EIR? Do projects that follow the performance baseline indicate an effective EIR? Does an EIR finding that a project has no problems or issues have value? Do project directors/managers value the output of EIRs? Do project directors/managers benefit from preparing for EIRs? Is preparation for a pre-CD-2 EIR the same as preparation for CD-2? Does preparation for the EIR make it easier to prepare for the CD review?

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects   Which elements of the EIR provide the greatest benefit to the project team and management? Is the value gained from an EIR consistent with its direct and indirect costs? 3:00 p.m. Break 3:15 p.m. Opportunities for Improvement   Are there more effective and efficient alternative methods for validating performance baselines? Can reviews be made more effective by including personnel from other government agencies on the review teams? Should EIR review teams include DOE personnel from other programs? Is there sufficient support from senior DOE managers to facilitate cross-program reviews? Can EIR contractors be encouraged to include others and still be held responsible for the outcomes? Is it more effective to review some issues/documents (e.g., WBS, risk management plan, cost, and schedule estimates) with an IPR or EIR? Can an equivalent outcome be accomplished with a smaller, more limited, or more numerous reviews? Would increased coordination between IPRs and EIRs increase the effectiveness of both? Should EIRs be used for more than CD-1 alternative finance, CD-2 performance baseline validation, rebaseline validation, and CD-3 readiness for major systems projects? There is a proposal to raise the lower limit of the cost of a project that will require an EIR to $100 million. Is this appropriate starting point? Should it be higher, or should some form of EIR be used for projects with smaller TECs? Should there be a point where EIRs managed by OECM are no longer necessary, or necessary only for major systems acquisition or large complex projects? 4:15 p.m. Observations and Conclusions 4:30 p.m. Adjourn PARTICIPANTS In addition to the NRC study committee, the following Department of Energy (DOE) staff, M&O contractors, and EIR contractors participated in the workshop: David Arakawa, DOE, Office of Science* Terence Bates, Systematic Management Services, Inc., M&O contractor Kevin Bazzell, DOE, Office of Environmental Management, Richlands Operations Office Kin Chao, DOE, Office of Science Raymond P. Chuebon, Burns and Roe, EIR contractor A. Scott Dam, Logistics Management Institute, EIR contractor Mike Donnelly, DOE, Office of Engineering and Construction Management Joe Eng, DOE, Office of Science* Gregg Flett, Oak Ridge National Laboratory, M&O contractor* Geoff Greene, Oak Ridge National Laboratory, M&O contractor* Mike Hickman, DOE, National Nuclear Security Administration Linda Horton, DOE, Oak Ridge National Laboratory*

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects Jeff Hoy, DOE, Office of Science* Tom Hull, DOE, Office of Environmental Management Clyde Jupiter, Jupiter Corporation, EIR contractor Brian Kong, DOE, Office of Engineering and Construction Management Thad Konopnicki, DOE, National Nuclear Security Administration Jim Krupnick, Oak Ridge National Laboratory, M&O contractor* Dan Lehman, DOE, Office of Science Katy Makeig, DOE, National Nuclear Security Administration Caryle Miller, DOE, Office of Science John “Bill” Newton, Jupiter Corporation, EIR contractor Jeanette Norte, DOE, National Nuclear Security Administration, Sandia* Andrew Reape, Project Time and Cost, EIR contractor Jay Rhoderick, DOE, Office of Environmental Management Steve Rossi, DOE, Office of Engineering and Construction Management Lisa Saldano, DOE, Office of Engineering and Construction Management Jehanne Simon-Gillo, DOE, Office of Science Jack Stellern, Oak Ridge National Laboratory, M&O contractor* Chris Watson, Project Time and Cost, EIR contractor Gerald W. Westerbeck, Logistics Management Institute, EIR contractor * Participated by phone.

OCR for page 45
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects This page intentionally left blank.