National Academies Press: OpenBook
« Previous: B EIR Documents Reviewed by the Committee
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

C
Summary Notes of a Workshop on DOE EIRs

The DOE External Independent Review Workshop was conducted June 8, 2006, at the Keck Center of the National Academies, Room 203, 500 Fifth Street, NW, in Washington, D.C. The agenda and list of workshop participants appear at the end of this summary.

The Committee on Assessing the Results of External Independent Reviews for U.S. Department of Energy Projects conducted a fact-finding workshop to solicit comments and opinions from DOE staff, including project directors, program and project management support managers from the Office of Engineering Construction and Management (OECM) and the program secretarial offices (PSOs) Operations and Management (O&M) contractors, and external independent review (EIR) contractors. The workshop consisted of open discussions to address a series of questions regarding the planning and execution of EIRs, EIR reports and corrective action plans, the value of EIRs, and suggestions for improving EIRs in the future.

The following summarizes the discussion based on notes taken by Michael Cohn, NRC program officer. The summary reflects the comments of the participants but is not a verbatim transcript. The comments were interpreted and condensed so as to allow focusing on the issues being considered by the committee. The sources of the comments are identified by job title or affiliation to provide context.

PLANNING AND IMPLEMENTING REVIEWS

Tailoring

Committee: Please comment on the process used to plan EIRs, the people involved in planning, and if the reviews are effectively tailored to the conditions presented by individual projects.


Federal project director (FPD): EIRs definitely need to be tailored. Each review should take into account the maturity and type of project being reviewed. New capital construction projects are different from a continuing operation or new baseline in a continuing acquisition. The 16 elements in the EIR guideline can be used, but the emphasis needs to be adapted to the project. It is important that the reviews be planned with the federal project director (FPD) as an active participant. The planning process also needs to recognize that the projects continue to mature as the EIR planning proceeds. Documents are given to the review team 5 weeks before the review, but they are not static. A lot of communication between the project team and the review team is needed to keep the review team current. The project team’s involvement is also needed to guide the preparation of on-site briefings.


Program/project management support office (PMSO) manager: It is important for the review team to recognize the implications of the various types of contracts used by DOE. A cost plus incentive fee contract is very different from a fixed-price contract. In the latter type of contract, the contractor bears the risk and the role of the EIR changes. The EIR for a specific project should reflect how risk is identified and quantified and what roles DOE staff and contractors play in managing risk.

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Planning Process

FPD: There does not seem to be a clear process for FPDs to become involved in planning and tailoring the EIRs, nor is there a way to ensure that the FPD’s concerns are addressed by the review.


OECM: The planning process has evolved in the past few years with increasing participation of the FPDs. Nonetheless, OECM has the final say on the scope of the EIR, which is intended to support validation of the baseline, and serves as an assessment of the readiness of a project to proceed. The process varies depending on the OECM, program, project, and contractor personnel involved. The intent is to have a collaborative process.


Program/PMSO manager: The Office of Science (SC) recently issued a directive that all EIR planning should be approved by the PMSO. SC will require agreement on the scope, review team, and costs before providing funds for the review. This had been discussed in the past, but the procedure had not been clearly defined. In the past, SC relied on the site office to ensure the quality of the EIR but was not sufficiently involved in the planning decisions. The process needs to be more collaborative than it has been. We need to assess the objective of the EIRs and focus more on the baseline cost and schedule and less on compliance with the details of acquisition policy.

Small Projects

Program/PMSO manager: The SC laboratory infrastructure program undertakes many projects that are significant (generally from $5 million to $20 million), but relatively small and less complex when compared to the new capital acquisition projects in SC. In some cases, tailoring has been applied to these small projects, eliminating the on-site portion of the review and relying solely on the review of documents and conference calls. These so called “desk-top reviews” do not work.


The EIRs are of value to these projects not only as a validation of the baseline but also as a means to transfer lessons learned and develop the knowledge and abilities of the junior project staff who are routinely assigned to the smaller projects. Even with the proposed shift in the lower limit for mandated EIRs to projects costing more than $100 million, we would like to continue conducting EIRs for selected laboratory infrastructure projects.


OECM: Proposed changes to O 413.3 for projects less than $100 million will allow either OECM or the program office to manage the review. OECM will provide the contract for employing the EIR contractor, but PSOs programs will be responsible for selecting the contractor and planning the review.


EIR contractor: EIRs provide exposure to lessons learned on past projects and training opportunities for less experienced project directors assigned to the smaller projects. This opportunity could be lost if reviews are not conducted for projects with less than $100 million total estimated cost (TEC).


OECM: Reviews will continue to be required, but they will be the responsibility of the program or site office. They will have the latitude to use either internal or external reviews.


M&O contractor: There is value in having an independent review, even for smaller, less complex projects. In the past, however, some of the reviews were too detailed for simple infrastructure projects, with the same EIR process having been used for them as for large new construction projects.


Program/PMSO manager: There has often been a cookie cutter approach to planning EIRs. They are all treated the same and follow the same lines of inquiry and the same level of detail regardless of the size and complexity of the project. OECM insists that scope is not negotiable, because the independence of

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

the review would be compromised if the program or project teams were allowed to have significant influence on the scope of the review. The result of this insistence has been ineffective tailoring.

Review Criteria

Committee: If program offices were responsible for determining if an EIR is needed, what criteria would they use?


Program/PMSO manager: The complexity of the project would be the primary criterion. The experience of the project team would also be important.


Committee: Should EIRs for smaller projects include all the current lines of inquiry? Should the emphasis be on baseline validation or project manager development?


Program/PMSO manager: Baseline validation is the primary purpose. Most infrastructure projects are developed as design-to-budget projects, and DOE needs to know that what is proposed can be accomplished for the allotted budget.


Program/PMSO manager: If O 413.3 is revised as proposed, the project management support office in SC will conduct independent reviews of all projects with TECs between $5 to $100 million, including laboratory infrastructure projects. The reviews will be tailored to critical issues presented by each project. They will look at the technical design issues, cost and schedule baselines, and, most important, the readiness of the management team. Past EIRs focused more on compliance with procedural requirements and less on the effectiveness of the management team.

EIRs Versus IPRs

Committee: Will SC continue to conduct EIRs for projects under $100 million TEC or rely solely on IPRs?


Program/PMSO manager: SC believes that reviews conducted by SC personnel who are not affiliated with the project are independent. SC will primarily use IPRs and only use EIRs as needed.


Program/PMSO manager: One of the problems with past EIRs is that the review team did not know enough about the project or the technical issues. A review team is definitely independent, but that is not enough to add value to the project.


Program/PMSO manager: The previous statement describes what happened during the course of several SC projects, particularly those involving highly technical physics experimental systems.


Committee: How do the Office of Environmental Management (EM) and the National Nuclear Security Administration (NNSA) anticipate conducting reviews for projects with a TEC of less than $100 million?


Program/PMSO manager: The change in the dollar value threshold is not likely to have much of an impact on EM since most EM projects are greater than $100 million TEC. EM conducts IPRs that are similar to SC’s reviews prior to the EIRs and independent of the EIR approximately every 9 months. The IPRs rely on EM staff from headquarters and field offices and on consultants.

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

Program/PMSO manager: NNSA conducts IPRs prior to CD-1 and CD-3 (except for major systems, when OECM conducts EIRs). If the threshold is changed to $100 million TEC, NNSA will be conducting more IPRs for baseline validation prior to CD-2. IPRs primarily use NNSA personnel from other sites. Contractors are used if the needed expertise is not available in NNSA.


FPD: I have found NNSA IPRs to be very helpful.


Program/PMSO manager: NNSA will conduct an IPR when issues arise and the project team determines that it needs outside review and input.

Review Teams

Committee: For projects with a TEC greater than $100 million that will continue to have EIRs planned by OECM and executed by contractors, is there a need to pay more attention to the technical subject matter experts on the review teams?


Program/PMSO manager: Having a review team with the necessary technical experts is a critical issue for SC. Because of the specialized nature of SC projects, this expertise is difficult to find outside SC. The Liniac Coherent Light Source (LCLS) project is an example of the kind of project where a lack of expertise on the review team can cause delays and reduce the effectiveness of the EIR.


Committee: Could SC, or other program offices, work with OECM to identify the experts that need to be on a team without infringing the independence of the review?


EIR contractor: The EIR contractors bring experts to the review teams to address technical issues and would include specific expertise if so encouraged by DOE. In some cases, particularly joint IPR/EIR reviews, the contractors have been told not to provide the expertise because the areas requiring that expertise would be addressed by DOE.


EIR contractor: Most of the contractor’s EIR personnel are independent experts who are called in as necessary to take on the issues presented by a specific EIR assignment. Contractors generally respond to OECM suggestions.

Coordination of EIRs and IPRs

Committee: What are the factors that determine the extent of coordination, sharing of expertise, and joint activities for IPRs and EIRs and if the reviews are conducted sequentially or concurrently?


Program/PMSO manager: Joint sessions with open meetings and breakout sessions are an effective model.


EIR contractor: Problems can arise because the IPR and EIR teams are competing for the same project resources to conduct their reviews, but this can be overcome with sufficient planning and coordination.


Committee: Do joint IPR/EIR reviews blur the lines of accountability for the outcome of the review?


EIR contractor: The juxtaposition of two teams with separate leadership increases the challenge of coordinating the review agenda and arriving at recommendations.

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

Program/PMSO manager: Joint IPR/EIR reviews generally share data and conduct simultaneous briefings but each team develops it own report and recommendations.


EIR contractor: Reviews have included a combination of joint and separate sessions that are focused on the needs of each project. This variety of arrangements provides an opportunity to share project resources and individual observations.


EIR contractor: The EIR and IPR teams often focus on different aspects of a project; e.g., an EIR might focus on cost and schedule and an IPR on scope. Because project outcomes are determined by the interaction of these issues, each team needs to be aware of the changes being recommended by the other. Conducting the reviews simultaneously makes coordination more difficult.


Committee: Are reviews planned and coordinated for the life cycle of a project?


Program/PMSO manager: In addition to the reviews set by DOE policy, individual program offices plan project review schedules independently to support their needs.


Committee: Does DOE plan on using the same review participants through the entire life of the project or does it plan to adjust the expertise in response to changing project issues?


Program/PMSO manager: IPR teams selected by the program offices do try to balance continuity and the changing needs of the project.


Committee: If the IPR addresses scope and the EIR cost and schedule, is it helpful to the EIR team if there is an IPR prior to the EIR?


EIR contractor: Yes, it is necessary to set the scope before the cost and schedule can be validated. Generally, if an IPR has been conducted, the results are available to the EIR team. The EIR team does not undertake a technical design/scope review but verifies that one has been completed and outstanding issues have been resolved.


Committee: It appears that EIRs and IPRs are conducted either concurrently or consecutively. What are the advantages or disadvantages of these two approaches?


Program/PMSO manager: For a large project, concurrent EIR/IPR reviews are less disruptive and require less of the project team’s time.


FPD: Consecutive reviews benefit the project team because the IPR can serve as a high-level review, identifying issues that can be corrected before the EIR. It can also be used to help focus the EIR so that it is more effective.


Program/PMSO manager: Consecutive reviews offer more learning opportunities for less-experienced project managers. The cost of an EIR is not insignificant. Conducting an IPR before the EIR facilitates assessment of the readiness of a project for baseline validation.


Program/PMSO manager: IPRs also cover readiness issues not covered in EIRs, such as safety and management concerns. Neither approach is clearly better. The best approach is determined by the circumstance of the project and the surrounding issues.


EIR contractor: Every EIR is tailored. They cover the same 16 lines of inquiry but to varying levels of detail. They may appear similar on the surface, but they are all tailored to the issues presented by the project.

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

Committee: An effective IPR should identify outstanding issues; the EIR would then only verify that they had been resolved. Is an EIR effective if it does not find any issues that were not identified and resolved?


OECM: We have never come across such a case, but the answer is yes. The goal is to validate the baseline, not find mistakes.


Committee: Is the timing of IPRs and EIRs properly distributed in the life cycle of a project? Does the timing support continuous process improvement?


Program/PMSO manager: We would hope there is continuous process improvement and we would hope that at some point we get good enough at project management that we will need fewer reviews.

Lessons Learned

Committee: Does the current program of reviews have a mechanism to capture lessons learned from one project and transfer them throughout DOE?


OECM: There is currently no DOE-wide lessons-learned system for project management. Some of the programs have undertaken initiatives to identify and distribute lessons learned.


Program/PMSO manager: NNSA is developing a lessons learned database by mining past IPRs. NNSA also conducts workshops where project managers share the successes and problems experienced on various projects.

Peer Review or Audit

Program/PMSO manager: The EIR process was established to ensure senior managers that policy and procedures are being followed and that project management outcomes meet a vaguely defined level of quality. The EIRs are like audits while the IPRs are more like a peer review. The audience for the first EIRs was Congress. Reports were sent directly to the appropriation committees. The audience has shifted more to DOE senior managers, but Congress is still watching. EIRs are becoming more like peer reviews, but they are still considered by field staff to be audits.


Committee: How can that staff perception be changed?


Program/PMSO manager: Changing that perception will be difficult because people from the outside the PSO are viewed as adversaries because the customers for an EIR are the OECM, the senior management, and Congress, not the PSO. The reviews are not being conducted for the PSO’s benefit. The external perspective is important for the larger projects but it creates an adversarial atmosphere between the PSO, the OECM, and the EIR contractors.


FPD: FPDs do not feel that they can influence the scope or focus of the EIR.


EIR contractor: From our point of view, OECM is the customer. Although it is clearly beneficial to involve the program and project in the planning process, that is not the way planning is being done. From the contractor personnel’s perspective, OECM determines the scope of the review. If others are involved in the planning process, there needs to be one DOE person who can resolve conflicts that may arise. We try to overcome the perception that the EIR team members are auditors, but the limited involvement of program and project personnel in the planning process makes that more difficult.

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Planning Process

OECM: There should be broad participation from the project and program in the planning process. In some cases the PMSOs choose not to participate. OECM needs to ensure that the scope of the review will allow verifying the baseline. The quality of communications in the planning process depends on the people involved in a project.


Program/PMSO manager: Neither the planning process itself or the roles of the program and project in planning EIRs are clearly defined. In one case the program and project took the initiative to define the scope of the EIR, which then became the basis for discussions with OECM and the contractor. The results were very productive.


EIR contractor: For a recent project, the contractor developed a review plan, which was then distributed to OECM, the program, and project for comments. It was revised and all parties signed off on it before the site visit. This seemed to provide sufficient communication for a relatively straightforward project.


EIR contractor: The 16 elements in the lines of inquiry are fairly inclusive. Most of the tailoring involves the attention to various details within these elements.


Committee: It appears the current process does not work for mega projects. Are alternatives to the EIR, such as U.S. Army Corps of Engineers (USACE) reviews and project-contractor-led “best and brightest” reviews, more appropriate for mega projects?


Program/PMSO manager: EM is seeing good results from the “best and brightest” approach at CD-1 to ensure that projects are well defined from the beginning. This should make EIRs more effective at CD-2. For EM, many of the issues relate to environmental safety and health (ES&H).


OECM: We probably should put more emphasis on CD-1. If the project plan and structure are fully aligned at that point, the project will be more likely to have a valid baseline at CD-2.


OECM: We need to separate the need for external, fully independent peer reviews from the mission assigned to OECM—that is, to validate the baselines.

Independent Cost Estimates

Committee: There does not appear to be much reference to independent cost estimates (ICE) as a part of the current review process. Are they still used?


Program/PMSO manager: There is a role for them, but it is not well defined in O 413.3. After not conducting ICEs for a while, we are just starting to do them again. The USACE is doing one for the Hanford Waste Treatment and Immobilization Plant (WTP) project, but that is an exception.


FPD: The value of the ICE depends on the specific circumstances of a project.


Program/PMSO manager: EM has cost-estimating capability in house.


Committee: What are the current criteria for ICEs?


Program/PMSO manager: An ICE is a bottom-up cost estimate. This is different from a cost review or a government estimate. Generally DOE does not have a lot of cost data or cost estimating capability. We need independent input. ICEs are generally not part of EIRs.

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

EIR contractors: EIRs usually include a focused independent cost review (ICR) that reviews cost estimating assumptions for selected work packages. A bottom-up estimate is usually a major undertaking that is too intensive to be part of an EIR.

Assessing the Management Team

Committee: Can an EIR evaluate the adequacy of a project management team?


OECM: Yes. They do that by evaluating the performance of the project team.


EIR contractor: The EIR usually does not address the qualification of individuals, but will usually comment if a potential problem is observed.


Program/PMSO manager: The EIR teams often comment informally about the performance and potential performance of the project team.


Program/PMSO manager: The government spends a lot of time and effort in source selection and does not need an EIR to second guess its source selection by evaluating the contractor’s project management team.


Program/PMSO manager: The management capabilities of the project team are critical. Judgments about their ability to execute the project need to be an internal matter. The EIR can identify potential problems, but responsibility for determining if a change is needed stays with the department.


EIR contractor: The O 413.3 and the EIR directives have not been adapted to design-build projects. The project management requirements and the EIR process need refinement. Reviews need to be incorporated into the project management plan.

Value of On-site Reviews

Committee: Do EIRs need to include on-site reviews or can they be accomplished off-site by conference calls and a review of documents?


EIR contractor: The value of face-to-face meetings and seeing the job site is difficult to know, but that direct contact definitely helps an EIR team evaluate the project. Some parts of a review can be done off-site but they need to be focused.


FPD: For large projects, the EIR team definitely needs to be on-site.


EIR contractor: It is not ideal, but when necessary, individual members can participate from a distance. The goal should be to have the entire team on site.

DOCUMENTATION OF REVIEWS AND CORRECTIVE ACTION PLANS

Review Findings and Recommendations

Committee: What is the best way to communicate the outcome of a review to the project team and the OECM?

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

EIR contractor: Generally there is an open session briefing by the EIR team at the end of the on-site review. The session brings together the project team and other contractors and DOE personnel involved in the project. People who are not at the site are linked by phone. View graphs are prepared and distributed the night before. The session includes time for discussion. The EIR team then prepares a written report, which is transmitted to OECM about a week after the on-site review. OECM reviews the report to determine if it is ready to send to the site office. The project team reviews the report for factual accuracy. The final report includes an outline of recommendations for the corrective action plan (CAP). It is an iterative process.


OECM: We have also conducted some EIRs where the final report was submitted before the team left the site. This approach is still being evaluated.


EIR contractor: For a large project whose review may take several weeks, there are periodic briefings of findings rather than a single briefing at the end of the review. This allows the project team to resolve most issues before the review team leaves the site.


Committee: The term “corrective action plan” may have negative connotations. Would it help if such plans were called “action plans” or “follow-up plans”? The new terminology might also change the false perception that an EIR is an audit.


OECM: That issue has never come up.


EIR contractor: The report sometimes includes appendixes that identify best practices that the project team could use to improve the management of the project.


Program/PMSO manager: In reality, the review team does not always follow this process. When the program office or project team does not agree with the EIR team the process gets bogged down. Generally, there is insufficient attention paid to the project team’s perspective. There should be an option that allows the project team to disagree with the review findings.


EIR contractor: The reports have three levels of findings. Only the most significant findings require corrective actions. The project team can disagree and ignore the less important findings and still have the project baseline validated. In the end, it is up to OECM to determine what actions need to be taken to validate the baseline.


Committee: Is there a problem with review teams adding findings to the final report that were not discussed at the site briefing?


Program/PMSO manager: If the review team cannot identify all the major findings before they leave the site, they are not doing their job. The project team needs an opportunity to discuss the outcome of the review and, if necessary, voice objections.


EIR contractor: Issues that arise after the site briefing are discussed with OECM. The FPD may also be brought into the discussion. There is no set procedure for communicating additional findings.

Corrective Action Plans

Committee: What happens to CAPs? Are they tracked beyond validation of the baseline?


Program/PMSO manager: CAPs are implemented immediately. Some of the responses are ongoing and will continue until the end of the project.

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

OECM: In some cases a plan of action is sufficient; in others, the plan needs to be completed or implemented. OECM needs to determine, based in part on its judgment, when to validate the project.


EIR contractor: Many of the recommendations address the cost and schedule issues that should be addressed before validation. By segregating the recommendations into three levels of importance, the review reports also sets priorities for action. The significant findings are generally those that need to be addressed before the baseline is validated. The EIR contractor is providing recommendations, but it is up to OECM to determine what is required from the project before it is allowed to proceed and up to the program to determine what requires long-term attention.

VALUE OF EXTERNAL INDEPENDENT REVIEWS

Committee: Is there a sense that EIRs result in a better outcome for a project?


Program/PMSO manager: A project is generally improved by conducting IPRs and EIRs, but it is difficult to determine if the benefit comes from the project team’s preparation for the review or the review itself. Anticipation of a review increases the rigor of the project team’s planning and documentation.


Program/PMSO manager: In some instances the EIRs did not identify any issues that the program and project team were not already aware of. The concept of the EIR is valuable, but the implementation does not always provide the anticipated value. Problems have occurred when communications falter and EIRs address issues that are not germane to the project. For SC, an IPR is more productive. The relative importance of the standard 16 lines of inquiry can vary greatly from project to project. The review team does not always identify which issues are critical and which will not affect the outcome of the project. The EIR is not a substitute for effective project management. SC tries to have a transparent process so that OECM should know beforehand if an EIR project is ready to proceed. EIRs do not add value that is consistent with their cost.


Program/PMSO manager: Projects receive benefits from EIRs, but there is a question about their value and if the benefit justifies the cost. The program offices pay for the EIRs. Since 1999 DOE has expended more than $50 million on EIRs. NNSA is spending $5.3 million this year, which is more than four times PMSO’s budget to provide IPRs, training, and other activities that are needed to ensure effective project management. That is why we question the value of EIRs. They need to be done, but they should be limited to the large projects where they can make a difference. The cost of an EIR is probably offset by reduced project costs, but that money is not available for the other activities that need to be done to improve project management. From a program perspective, an IPR provides much greater value.


Committee: There was earlier reference to a lessons learned program to increase the benefits gained from reviews. Does that apply to EIRs?


Program/PMSO manager: That was an NNSA program that applies to IPRs only. It could be used for EIRs.


Committee: Why are EIRs not funded out of project funds?


Program/PMSO manager: DOE policies determine the funding sources. EIRs will be funded through the working capital fund (WCF) starting in FY07. Programs will contribute operating dollars to the WCF to pay for the reviews.


Committee: Are the projects required to conduct IPRs?

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

Program/PMSO manager: IPRs are only required prior to CD-3. Most IPRs are done at the discretion of the PSO. SC performs IPRs more frequently than the other programs. NNSA conducted 35 IPRs last year that were requested by the FPD or the program office. EM conducts IPRs prior to CD-1 and then through CD-3.


Committee: Could IPRs replace EIRs?


Program/PMSO manager: Not entirely, especially for the larger projects and for the program offices that do not have PMSOs.


Committee: What is the average cost of an EIR?


Program/PMSO manager: The average cost is approximately $150,000. One recent EIR cost $1.2 million.


EIR contractor: EIRs for smaller projects with a three-member review team cost approximately $50,000.


Committee: The rate at which projects were rebaselined drove the need for EIRs. Has project performance been improved by conducting EIRs? Could internal reviews achieve the same results?


OECM: The complexity of the issues and number of external factors affecting project performance make it difficult to measure the impact of reviews.


EIR contractor: There are examples of projects that have been canceled based on EIR findings. Here, EIRs clearly provided a benefit.


O&M contractor: For one fairly large but not complex project, the EIR identified approximately 20 issues; however, none of them were of any consequence. There are also examples of projects that underwent several EIRs before the project progressed to the point that it could proceed past CD-2.


Program/PMSO manager: Benefits are definitely gained from conducting EIRs, but they should be limited to larger projects that have the potential to embarrass DOE. It is easy for an EIR or IPR to provide significant results when it is applied to projects that are known to be in trouble. The challenge is identifying the precursors of those problems and applying corrective action before the problems arise. It is also more difficult to assess the impact of preventive actions that result in cost avoidance as opposed to cost savings.


Committee: Does a project that experiences problems after an EIR and acceptance of the CAP mean the EIR and the validation process have failed?


Program/PMSO manager: Every successful project and every problematic project has had an EIR or IPR. The question is not the type of review but whether or not it addressed the critical issues. Sometimes the critical issues were not identified, sometimes the risks could not be avoided, and other times external requirements, such as the design basis threat, are not known at the time of the review. The interval between reviews can contribute to the impact of all these scenarios. There are instances when problems arose because the designs were not ready for construction or the project team did not understand the bidding market, but the IPRs and EIRs did not assess those aspects of project management.


Committee: Should the review team have recognized these issues as a threat to the project baseline?

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

OECM: We see evidence that in some instances the EIRs need more rigor, but at the same time there are pressures to reduce the cost of reviews.


EIR contractor: The EIRs are an evolving process. They used to be conducted in greater depth and included larger samples. Cost pressures have led to reduced scopes and smaller samples. The challenge is to identify the critical projects.


Committee: Is the fact that a project has not undertaken an IPR or EIR for an extended period of time a leading indicator of management problems?


Program/PMSO manager: It could be a sign that the process is not open or transparent.


Committee: Does the value of an EIR depend on employing the right EIR contractor for the project, and how can that contractor be identified? Is there a process to assess the capabilities of the various EIR contractors?


Program/PMSO manager: Yes, the experience and expertise of the review team are critical to the success of the review. The programs give OECM feedback on the quality of review teams, but the process is informal.


OECM: A more formal assessment process is under development.


Committee: How are the contractors selected?


OECM: The program offices are asked if they have a preference, but OECM has the primary responsibility for selecting the contractor. Factors such as previous experience at a site, expertise, and current workload are used to select from the group of contractors with multiyear indefinite quantity contracts. We just added two additional contractors to the group for a total of five, which seems to be sufficient.


Committee: Do the contractors receive enough feedback to help them improve future reviews?


EIR contractor: Contractor performance is reviewed annually by OECM, but there is no feedback on particular projects.

OPPORTUNITIES FOR IMPROVEMENT

Committee: Could the composition of the EIR team be improved by including people designated by DOE as having the required expertise? Would DOE’s involvement in selecting the team interfere with the contractor’s control of the EIR?


EIR contractor: Using DOE-designated participants on the review team is not a problem as long as they have no conflict of interest, i.e., no prior or future involvement in the project. Each member of the team contributes to the final review report, but the contractor controls the content of the final report. The externality of the review would be compromised and its independence could be questioned. However, DOE’s participation in the selection would be appropriate if it was necessary to obtain the necessary expertise. The final judge of the independence of the review is OECM.


OECM: We try to keep it as independent as possible, but the objective is to obtain a review team that is both informed and independent.

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

EIR contractor: Members of the review team are relied on to maintain their professional integrity.


Committee: Could project outcomes be improved or costs reduced with better coordination of IPRs and EIRs?


FPD: If sufficient time elapses between the IPR and the EIR they can form a complementary iterative process. The key is timing. There are examples where this has worked.


OECM: The EIR team is informed of the outcome of prior reviews, which helps the team to better focus its review.


Committee: Is there a more efficient method for conducting EIRs and completing the review report?


EIR contractor: Giving the contractor more lead time improves selection of the review team and planning of the review. The cost of the review may not be reduced but it will be more effective.


OECM: For some NNSA reviews, the review team was kept on site until the report was completed. This reduced the time required to complete the report, but it increased the cost and presumably did not affect the quality of the review.


EIR contractor: It generally takes about a week to develop a draft EIR report. This seems to be a reasonable time and applies whether the report is written on-site or back in the office. Because the reports and the review team briefings are preceded by dialogue between the review team and the project team, the final briefing and report should not hold any surprises for the project team. The review benefits somewhat from having time to reflect and putting some distance between the review team and the project site.


EIR contractor: OECM has developed a guide for the reports that facilitates the writing process and makes the reports more consistent.


Committee: During the project reviews the committee heard that early planning and close coordination with the project team, on-site participation of OECM, and discussion of preliminary findings led to an effective review. Is there anything to add to this list?


OECM: Pre-EIR site visits also seem to increase the effectiveness of the reviews.


Committee: Would projects benefit more from reviews (both IPR and EIR) if they were conducted periodically rather than prior to CDs?


Program/PMSO manager: SC tries to conduct reviews on a 6-month schedule but also tries to schedule them ahead of CDs. A 6-month interval is typical after construction starts. Reviews are also conducted when performance issues arise. Reviews for NNSA and EM projects are not as frequent or regular.


Committee: Are there any concerns about the proposed changes to O 413.3 that will eliminate the requirement for EIRs for projects with TECs of less than $100 million?


EIR contractor: Contractors are not unhappy about this change because there will continue to be a sufficient workload. They have seen improvement in DOE project management activities that may be the result of the EIRs. If the IPRs are implemented for projects with TEC less than $100 million and project managers continue to be held accountable, then this improvement should continue. Some form of review will continue to be needed.

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
New Issues

EIR contractor: The reviews are conducted when the design is approximately 30 percent complete. The review focuses primarily on cost and schedule to deliver the scope as defined at the time of the review. The review does not verify that the scope is accurate, and subsequent baseline changes are often the result of changes in scope that are implemented during completion of the design. Should the EIRs be delayed until the design is more complete and the scope becomes fixed?


Committee: A scope can be clearly defined at that point in a project when design is 30 percent complete. Lack of a clear scope is a failure of project management. An assessment of the completeness of the scope definition should be part of the EIR.


Program/PMSO manager: DOE has adapted a project definition rating index (PDRI) tool developed by the CII. The tool helps project managers determine if the scope has been defined sufficiently to allow the project to proceed. It is used by EM and NNSA for IPRs prior to CD-1.


OECM: The EIR should occur when the project is ready to be baselined. If the scope is not defined, the project is not ready. If the design needs to be more than 30 percent complete before the scope can be fixed, then the project probably should wait. The need to have a well-defined scope is often ignored because various external budget or political pressures make it urgent to start the project.


Following a call for final comments, the workshop was adjourned at 4:00 p.m.

WORKSHOP AGENDA

June 8, 2006

8:30 a.m.

Welcome and Introductions–Joseph Colaco

 

Objectives of the Study and the Workshop–Michael Cohn

9:00 a.m.

Workshop Discussions–Gary Sanders, committee member, workshop facilitator

 

Planning and Implementing EIRs

 

  • Are EIRs planned effectively to address the project issues that have the greatest impact on the performance baseline?

  • Are the reviews tailored consistently, or is there evidence of inconsistent response to project conditions?

  • Who should be responsible for tailoring the review?

  • Should EIRs be tailored by adjusting duration, team expertise, issues/project management documents reviewed, or other aspects?

  • Should EIRs be tailored to program requirements (i.e., a different starting point for each program) and then tailored to the specific project requirements?

  • Are there a limited or infinite number of approaches?

  • Are EIRs carried out with some consistency?

  • Do EIR teams have all of the necessary expertise and experience? Which types of expertise and experience are missing? Which are not necessary?

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

 

  • Is there a separate role for an independent cost estimator? Should an ICE be part of an EIR or separate?

  • Is the time needed to complete a review consistent with the complexity and importance of the issues motivating it?

  • Can EIRs be effective without on-site activities?

  • Are CAPs developed, communicated, and implemented effectively? Are they (or should they be) part of the project management system and tracked beyond validation of the baseline?

  • Does the involvement of multiple parties (e.g., OECM, program HQ, DOE project personnel, contractor’s project personnel, EIR contractor) lead to communication problems?

  • Are the current 16 elements in the EIR Standard Operating Procedures document appropriate? Are there elements that should be included in all reviews? Conversely, should some elements be considered in exceptional circumstances only?

  • Is there (or should there be) a follow-up to improve future EIRs, and are such follow-up findings used?

  • Do the advantages (e.g., knowledge of DOE procedures and expectations for EIRs) of using a limited number of EIR contractors outweigh the disadvantages (e.g., risk of losing independence)?

12:00 p.m.

Lunch

1:00 p.m.

Documentation of EIRs and Corrective Actions

 

  • Is the review plan formulated jointly by the program office, the OECM, the review contractor, and the project team documented sufficiently? Do all parties have a clear understanding of the EIR objectives?

  • Are results of EIRs communicated effectively at the time of the site visit and, in written documentation?

  • Do review reports produce findings characteristic of an audit, checklist review, and/or a substantive analysis?

  • Is there a way to collect and transfer EIR lessons learned?

  • Are there continuous project and program records of all reviews (IPRs, EIRs, etc.), CAPs, and corrective actions? Are these data tracked and compared to project performance?

2:00 p.m.

Determining the Value of EIRs

 

  • Do reviews consistently produce findings and recommendations that impact (improve or reduce risk) the project baseline cost, schedule, and scope, and risk management and project management strategies?

  • How often are EIR findings and recommendations of little or no consequence?

  • How often are there problems in projects that should have been identified and corrected during an EIR?

  • Do projects that experience problems after CD-2 indicate a weakness in the EIR?

  • Do projects that follow the performance baseline indicate an effective EIR?

  • Does an EIR finding that a project has no problems or issues have value?

  • Do project directors/managers value the output of EIRs?

  • Do project directors/managers benefit from preparing for EIRs?

  • Is preparation for a pre-CD-2 EIR the same as preparation for CD-2? Does preparation for the EIR make it easier to prepare for the CD review?

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

 

  • Which elements of the EIR provide the greatest benefit to the project team and management?

  • Is the value gained from an EIR consistent with its direct and indirect costs?

3:00 p.m.

Break

3:15 p.m.

Opportunities for Improvement

 

  • Are there more effective and efficient alternative methods for validating performance baselines?

  • Can reviews be made more effective by including personnel from other government agencies on the review teams?

  • Should EIR review teams include DOE personnel from other programs? Is there sufficient support from senior DOE managers to facilitate cross-program reviews?

  • Can EIR contractors be encouraged to include others and still be held responsible for the outcomes?

  • Is it more effective to review some issues/documents (e.g., WBS, risk management plan, cost, and schedule estimates) with an IPR or EIR?

  • Can an equivalent outcome be accomplished with a smaller, more limited, or more numerous reviews?

  • Would increased coordination between IPRs and EIRs increase the effectiveness of both?

  • Should EIRs be used for more than CD-1 alternative finance, CD-2 performance baseline validation, rebaseline validation, and CD-3 readiness for major systems projects?

  • There is a proposal to raise the lower limit of the cost of a project that will require an EIR to $100 million. Is this appropriate starting point? Should it be higher, or should some form of EIR be used for projects with smaller TECs?

  • Should there be a point where EIRs managed by OECM are no longer necessary, or necessary only for major systems acquisition or large complex projects?

4:15 p.m.

Observations and Conclusions

4:30 p.m.

Adjourn

PARTICIPANTS

In addition to the NRC study committee, the following Department of Energy (DOE) staff, M&O contractors, and EIR contractors participated in the workshop:


David Arakawa, DOE, Office of Science*

Terence Bates, Systematic Management Services, Inc., M&O contractor

Kevin Bazzell, DOE, Office of Environmental Management, Richlands Operations Office

Kin Chao, DOE, Office of Science

Raymond P. Chuebon, Burns and Roe, EIR contractor

A. Scott Dam, Logistics Management Institute, EIR contractor

Mike Donnelly, DOE, Office of Engineering and Construction Management

Joe Eng, DOE, Office of Science*

Gregg Flett, Oak Ridge National Laboratory, M&O contractor*

Geoff Greene, Oak Ridge National Laboratory, M&O contractor*

Mike Hickman, DOE, National Nuclear Security Administration

Linda Horton, DOE, Oak Ridge National Laboratory*

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

Jeff Hoy, DOE, Office of Science*

Tom Hull, DOE, Office of Environmental Management

Clyde Jupiter, Jupiter Corporation, EIR contractor

Brian Kong, DOE, Office of Engineering and Construction Management

Thad Konopnicki, DOE, National Nuclear Security Administration

Jim Krupnick, Oak Ridge National Laboratory, M&O contractor*

Dan Lehman, DOE, Office of Science

Katy Makeig, DOE, National Nuclear Security Administration

Caryle Miller, DOE, Office of Science

John “Bill” Newton, Jupiter Corporation, EIR contractor

Jeanette Norte, DOE, National Nuclear Security Administration, Sandia*

Andrew Reape, Project Time and Cost, EIR contractor

Jay Rhoderick, DOE, Office of Environmental Management

Steve Rossi, DOE, Office of Engineering and Construction Management

Lisa Saldano, DOE, Office of Engineering and Construction Management

Jehanne Simon-Gillo, DOE, Office of Science

Jack Stellern, Oak Ridge National Laboratory, M&O contractor*

Chris Watson, Project Time and Cost, EIR contractor

Gerald W. Westerbeck, Logistics Management Institute, EIR contractor

*

Participated by phone.

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

This page intentionally left blank.

Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 45
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 46
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 47
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 48
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 49
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 50
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 51
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 52
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 53
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 54
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 55
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 56
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 57
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 58
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 59
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 60
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 61
Suggested Citation:"C Summary Notes of a Workshop on DOE EIRs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 62
Next: D Acronyms and Abbreviations »
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Peer review is an essential component of engineering practice and other scientific and technical undertakings. Peer reviews are conducted to ensure that activities are technically adequate, competently performed, and properly documented; to validate assumptions, calculations, and extrapolations; and to assess alternative interpretations, methodologies, acceptance criteria, and other aspects of the work products and the documentation that support them. Effective peer reviews are conducted in an environment of mutual respect, recognizing the contributions of all participants. Their primary objective is to help the project team achieve its goals. Reviews also contribute to quality assurance, risk management, and overall improvement of the management process.

The U.S. Department of Energy (DOE) conducts different types of peer reviews at the different stages of a project, including reviews to assess risks and other factors related to design, safety, cost estimates, value engineering, and project management. Independent project reviews (IPRs) are conducted by federal staff not directly affiliated with the project or program and management and operations (M&O) contractors. External independent reviews (EIRs) are overseen by the Office of Engineering and Construction Management and conducted by contractors external to the department. EIRs are the primary focus of this report. However, the committee found that, in many cases, IPRs are explicitly used as preparation for or as preliminary reviews prior to EIRs. Thus, because IPRs are integral to the review process in DOE, they are also discussed because they might have an effect on EIRs.

In October 2000, DOE issued Order 413.3, Program and Project Management for the Acquisition of Capital Assets (DOE, 2000). The order established a series of five critical decisions (CDs), or major milestones, that require senior management review and approval to ensure that a project satisfies applicable mission, design, security, and safety requirements: approve mission need, approve alternative selection and cost range, approve performance baseline, approve start of construction, and approve start of operations or project closeout.

Assessment of the Results of External Independent Reviews for U. S. Department of Energy Projects summarizes the results.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!