Conclusions, Findings, and Recommendations
The committee’s charge calls for advice on how EIRs can be tailored to a mix of project types to ensure that essential information is provided at the optimum point in the CD process and that resources are used effectively. DOE policies designate EIRs as the primary process for validating project performance baselines before submitting a request for project funding to Congress.
If EIRs are intended only to validate project performance baselines, then the current review planning procedures, the practice of employing outside contractors responding directly to the OECM, and the use of a standardized set of key review elements might be appropriate. However, DOE project directors and program managers reported additional benefits from both EIRs and IPRs including professional development, management process improvement, transfer of lessons learned from other sites and projects, and improved communication between DOE staff and contractor personnel. Further, the benefits of peer reviews often accrue in the form of avoidance of future problems and increased confidence in project performance, benefits which are difficult to quantify. The value and cost-effectiveness of EIRs would be enhanced if they were planned more carefully with the broader involvement of all stakeholders, tailored in a more flexible manner using a collaborative process, and integrated into the complete portfolio of reviews that are used to monitor and support DOE projects. In addition, EIR reports can be a source of valuable lessons learned which could be used to help improve project management within the DOE. The committee’s findings and recommendations below address this view of the enhanced role of EIRs.
Finding 1: DOE employs a variety of peer reviews to assess risk and other factors that bear on design, safety, cost estimates, value engineering, and project management. Because the responsibility for organizing the reviews is dispersed throughout the department, the planning of project reviews is not well coordinated.
EIRs and IPRs are the predominant types of peer reviews initiated by the OECM and the PSOs. Current DOE practices disperse the responsibility for organizing the various types of reviews: contractors manage independent design and technical reviews, the PSOs manage IPRs, and the OECM manages EIRs. The result is an uncoordinated use of peer reviews throughout the department.
Currently, EIR review teams focus on validation of the cost and schedule baseline. The scope of work or the specific technical approaches and risks are assessed by IPRs. Because the cost and schedule baseline can be valid only if the scope is valid and stable, it is essential that the planning of IPRs and EIRs be coordinated to ensure that the risk associated with future scope changes are addressed in the baseline budget.
Finding 2: The value of a peer review (IPR or EIR) depends, in large part, on the experience and expertise of the review team, its capacity to understand the requirements of the project, its independence, and its ability to make objective assessments and recommendations. Assembling teams that have both expertise and independence can be difficult, especially for the complex, one-of-a-kind projects sometimes undertaken by DOE.
DOE projects often involve specialized scientific equipment or innovative technologies with significant risks and implementation challenges. The nonroutine, sometimes unique, nature of DOE projects requires a case-by-case treatment. For this reason, EIRs often require peer reviewers who have unique expertise. In addition, critical issues change over the course of a project’s development, with planning issues dominating in early stages and implementation issues becoming more important at later stages. As a result, the composition of an appropriate review team must be carefully considered to ensure that members have the appropriate expertise.
Currently, IPRs primarily rely on reviewers from within the DOE community, who may lack full objectivity, while EIRs rely on reviewers from outside the DOE community, who may sometimes not have the requisite expertise. In the long term, reviews can be conducted by teams using both internal and external reviewers as needed to address the critical issues presented by the project. For projects with specialized requirements, a team composed of external and internal reviewers could provide a balance between independence and the technical focus needed to effectively assess the inherent risks in proposed project scopes of work, budgets, and schedules, and to serve a broader audience. Validation of a project scope may require more internal reviewers, while validation of cost and schedule may require more external reviewers.
Finding 3: Total project cost is the leading criterion in guiding the planning and scheduling of reviews. Additional criteria may be equally appropriate.
Decisions for planning appropriate peer reviews need to be based in part on TPC, but should also consider other critical project success factors such as inherent risks, complexity, technology readiness, the capabilities of the project management team, and the amount of time elapsing between CD milestones.
Finding 4: Peer reviews (IPRs or EIRs) conducted prior to approval of mission need (CD-0) and again prior to approval of the alternative selection and cost range (CD-1) could play a valuable role in risk management and quality assurance during front-end project planning. Some projects could probably benefit from annual reviews for quality assurance purposes when the time between critical decision milestones exceeds 1 year.
The Construction Industry Institute’s (CII’s) benchmarking program has shown that the quality and completeness of a project’s front-end planning are linked to the probability of success of a project and that an owner’s best opportunities to influence the outcome of a project occur at the front-end (CII, 1996). The greater the complexity and risks of a project, the greater the need to ensure the validity of early critical decisions.
Risks in large and complex projects change as the project develops. The potential for change justifies the need for some form of periodic peer review. Such reviews could provide a fresh look at the project, account for changing circumstances, identify emerging problems, and revisit project assumptions and the information upon which decisions were made to ensure they remain valid.
Finding 5: Problems in communication stemming from the prereview planning process have occurred when an EIR team arrived at a significant finding after the on-site exit briefing but did not relay the finding to DOE staff until delivery of the final report.
The committee observed that the development of a report guide by the OECM improved the overall organization and consistency of the reports. Some concern was expressed by DOE personnel that the reports took too long to produce after completion of the on-site review sessions, but this time, generally about one week to complete the first draft, did not seem to the committee to be excessive. Problems in communication stemming from the pre-review planning process seemed to be present in instances where the EIR team developed additional significant findings after the on-site exit interview but did not communicate those findings until delivery of the final report, which may occur several months after the exit briefing. Such a delay in communication on major findings hinders the ability of the IPT to begin corrective actions expeditiously.
Finding 6: The purpose of tailoring is to refine the focus of the EIR to suit the particular characteristics of a project. DOE policies emphasize the need for tailoring, but the committee observed little evidence of effective tailoring.
DOE O 413.3 recognizes the essential need to tailor EIRs to specific project requirements. The Order gives responsibility for tailoring to the project team but places the OECM in control of the EIR.
Determining how to tailor an EIR requires experience as well as judgment about the value of peer reviews per se. For example, tailoring that involved eliminating the on-site sessions for smaller projects was reported to severely diminish the effectiveness of the EIR. Because much of the value of a peer review lies in the interaction of the participants, it is not surprising that eliminating the on-site review was counterproductive. In contrast, tailoring that was done by a collaboration of the EIR contractor, the IPT, the PSO, and the OECM produced effective reviews.
Finding 7: A clearly defined, systematic process for capturing data from peer reviews along with a system to catalogue, store, and disseminate this information is needed and, if implemented, would greatly enhance the benefits derived from peer reviews.
Peer reviews (EIRS and IPRs) can be a primary vehicle for gauging an organization’s project management and maturing it. When EIR contractors are exposed to problems and solutions from multiple projects at multiple sites they can disseminate lessons learned through on-site discussions and EIR reports. DOE staff and M&O contractors who participate in IPRs are exposed to methods and procedures from other sites and bring their experience to benefit the personnel associated with the project being reviewed. For small projects undertaken by less experienced personnel, EIRs were reported to provide training and contribute to their professional development. However, lessons learned from EIRs and IPRs have not been documented systematically.
Finding 8: EIRs and other peer reviews are an intrinsic part of project management and are an integral part of project costs. The current policy of funding EIRs from program direction funds places them in competition with other program activities.
The current practice of funding EIR contract costs (travel and consultant personnel) from the program direction budgets puts pressure on DOE program managers to choose between funding EIRs and funding personnel training and other activities to improve project management. IPRs are favored by the PSOs because the majority of costs, except for travel, are absorbed by existing personnel budgets. Because IPRs require direct funding only for the travel expenses of the review team, there is a bias in favor of IPRs and a bias against the value of EIRs.
The mechanism for funding EIRs and other reviews affects more than the availability of funds to conduct reviews. It is counterproductive to place these reviews in a zero-sum game with program direction funds for overall department management. This competition leads to a less than enthusiastic approach to EIRs on the part of the PSOs and the project teams. The selection of the type of review
should be based on which one provides the most benefit to the project and the department rather than the funding source.
Recommendation 1: EIRs should be integrated into a comprehensive peer review process. To accomplish this,
A comprehensive plan to coordinate the planning for all project peer reviews should be developed by the integrated project team (IPT), with input from the OECM and the PSOs. Plans for peer reviews should be included in the project execution plan and the quality assurance program.
The IPT, the PSOs, and the OECM should determine the types of and schedules for these reviews using the guidelines shown in Table 4.1. It is important to note that the committee’s recommendations for reviews to be conducted by default do not necessarily parallel Order 413.3A.
In addition to the TPC, the criteria for determining the appropriate type, form, and timing of a review should include project complexity; severity and types of inherent risks; project management maturity; and the qualifications and experience of the project teams. These criteria should be developed jointly by the OECM and the PSOs.
EIRs should be planned and executed jointly by the OECM, the PSOs, and the IPTs. The office responsible for validating the performance baseline (CD-2) should take the lead for reviews that occur prior to CD-2.
For projects with TPCs less than $100 million, an EIR should be required to validate the performance baseline (CD-2) unless project complexity, inherent risks, project management capabilities, or other factors warrant conducting an IPR. The determination of the most appropriate type of review should be made by the PSO.
For some projects, a review (IPR or EIR) should be conducted prior to CD-0 and then again prior to CD-1 if warranted by the project complexity, inherent risks, project management capabilities, or other factors. After CD-2, when the elapsed time between critical decision milestones exceeds 1 year, an annual IPR should be conducted for quality assurance purposes.
Recommendation 2: EIRs should assess whether the particular project management team—i.e., project manager, project team, federal project director, and IPT—has the ability to execute the project successfully. This assessment should evaluate the qualifications and experience of the project management team, the project’s organizational structure, and the communication and decision-making processes in place.
Recommendation 3: EIR review teams should be selected to provide the experience and expertise necessary to evaluate a project’s baseline cost, schedule, scope, inherent risks, and the capabilities of the project management team. The acquisition executive could add DOE contractor personnel who are not project advocates to EIR teams when outside technical expertise is not available. In the long term, the distinction between the EIR and IPR review teams can be lessened and review team members from within DOE and external to it can be selected as needed to address the critical issues presented by each project.
Recommendation 4: The OECM, the PSOs, and the IPT should collaborate on the tailoring of reviews. They should determine which key review elements should be emphasized and which ones should be streamlined based on their assessment of project risks, complexity, and cost.
TABLE 4.1 Recommended Guidelines for Planning Peer Reviews
Recommendation 5: Communication among all EIR participants should be improved from initial planning of the reviews through taking corrective actions:
If significant findings are arrived at subsequent to the formal on-site exit briefing, the EIR team should notify the IPT immediately and not wait until delivery of the final report.
Postreview assessments of the quality of the review process for EIRs should be conducted and led by the OECM with input from the IPT and the PSO. The purpose is to improve future reviews and provide input to the annual contractor performance appraisal.
Recommendation 6: The DOE should document lessons learned from peer reviews and disseminate them throughout the department.
The OECM should establish a process to document and disseminate the lessons learned from all peer reviews across all program offices. The process should be developed through a coordinated effort with the full participation of the PSOs.
EIRs should clearly identify best practices observed in order to facilitate the lessons learned process.
Recommendation 7: DOE should view peer reviews as an intrinsic component of project management.
The costs for EIRs and IPRs should be covered by project funding—that is, be part of the preliminary engineering design funds prior to CD-2 and a part of project funds when these funds have become available to the PSO.
Prereview planning decisions about the most appropriate type of review (EIR or IPR) should be based on full-cost accounting of either type of review.