National Academies Press: OpenBook
« Previous: 2 Characteristics of Effective External Independent Reviews
Suggested Citation:"3 Criteria for Structuring Review Programs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

3
Criteria for Structuring Review Programs

DOE Order O 413.3 established criteria for the timing of peer reviews based on total project cost (TPC) and identified the DOE office responsible for conducting the reviews (Table 1.1). A revised Order, O 413.3A was issued on July 28, 2006. The timing and types of peer reviews and the responsible office remained the same as outlined in Table 1.1. However, the TPC threshold levels that trigger the requirement for an EIR or IPR (not shown on Table 1.1) were changed significantly. The Office of Engineering and Construction Management (OECM) is now required to conduct an external independent review (EIR) prior to CD-2 for a project with a TPC of $100 million or more and prior to CD-3 for a major systems project of $750 million or more (DOE, 2006). The revised Order, 413.3A, requires the program secretarial offices (PSOs) to conduct independent project reviews (IPRs) prior to CD-2 and CD-3 for projects with a TPC less than $100 million. Order 413.3A also addresses the need for IPRs prior to other critical decision (CD) points and for peer reviews of technical design, quality, risk management, and safety.

DOE projects vary significantly in size, complexity, scope, and inherent risk. An analysis of 141 projects in the Project Assessment and Reporting System (PARS) database indicated that the average total project cost was more than $1 billion, whereas the median total project cost was $175 million. The PARS data demonstrate skewing due the small percentage of projects having very high costs. Table 3.1 shows the distribution of the 141 projects by TPC. Although 70 percent of all projects cost less than $750 million each, their combined costs total only 11 percent of all project estimated costs. Considered another way, 30 percent of the total number of DOE projects accounted for 89 percent of total estimated costs. Many of the projects costing more than $750 million are in the Office of Environmental Management.

The committee was asked to advise DOE on how it could focus its resources on those projects with the greatest risk. In the committee’s opinion, the distribution of projects at various TPCs and the relative level of risk of the larger projects support Order 413.3A’s revised thresholds at which an IPR or an EIR is triggered. However, the committee also believes that more emphasis should be placed on factors other than cost that influence project risk, such as the extent to which a technology was used previously in its planned application, the number of stakeholders and their influence on project requirements, and the certainty and detail in the definition of the project scope.

TIMING OF REVIEWS

DOE policy recognizes the importance of peer reviews for performance baseline validation and prior to the start of construction. The Construction Industry Institute’s (CII’s) benchmarking program has shown that the quality and completeness of a project’s front-end planning affect the probability that a project will succeed and that an owner’s best opportunities to influence the outcome of a project occur at the front end (CII, 1996). The more complex and risky a project, the greater the need to ensure the validity of early CDs.

Suggested Citation:"3 Criteria for Structuring Review Programs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

TABLE 3.1 Distribution of Sample Projects in the Project Assessment and Reporting System

Total Project Cost (million $)

Percentage of Projects (Cumulative) (million $)

Percentage of Costs (Cumulative) (million $)

<20

11

<1

<100

38

2

<400

62

7

<750

70

11

Previous NRC reports focusing on DOE’s project management procedures emphasized again and again the importance of effective risk management to the success of projects (NRC, 2001, 2003, 2004). Risk management is important element of EIRs because it is critical to the validation of baseline budgets and schedules. The nature and severity of risks vary greatly from project to project, requiring their review to be closely tailored to specific project needs.

The NRC report The Owner’s Role in Project Risk Management addresses the role of risk management in DOE projects and supports a proactive approach in which owners take the following actions:

  • Start risk management early in the life cycle of all projects, prior to CD-0, approval of mission need.

  • Include key stakeholders in the process, with the DOE project director as the lead and the integrated project team (IPT) also intimately involved.

  • Evaluate project risks and risk responses periodically during project life (CD-0 through approval of the start of operations [CD-4]).

  • Develop risk mitigation plans and update them as the project progresses.

  • Follow through with mitigation actions until risks are acceptable.

  • Tie a project’s level of risk to cost and schedule contingencies.

  • Communicate progress and changes to project risks and mitigation plans to all key stakeholders.

The report also noted that risk needs to be managed by qualified personnel working within a project management process that includes review and approval by senior management. CDs, such as those defined in DOE O 413.3, must be made by senior management to ensure the quality of the risk management process and that the risks inherent in a project are necessary and acceptable. Reliance on team experience alone, without subjecting decisions to reviews, can lead to gaps in analysis and lack of consistency (NRC, 2005).

Peer reviews conducted prior to CD-0 and CD-1 could play an important role in risk management and quality assurance. Moreover, since risk can change as a project—especially a large and complex project—develops, some form of periodic peer review is justified. The committee believes, furthermore, that additional reviews should be considered to assure quality if the time elapsing between CDs is greater than 1 year.

RESPONSIBLE OFFICE

DOE Orders 413.3 and 413.3A assign responsibility for project oversight, baseline validation, and control of reviews based on the TPC. The overall result of DOE policies is that the OECM controls EIRs and the PSOs control the IPRs. In the committee’s opinion, this distinction is unnecessary and it may hamper the selection of the most appropriate type of review. Depending on the TPC, complexity, and

Suggested Citation:"3 Criteria for Structuring Review Programs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

risks of a project, EIRs could be successfully planned and executed by the PSOs and the project teams. The OECM could then assume an oversight role to ensure that the reviews are adequately planned and executed and that the review teams have the necessary expertise and are sufficiently independent, that is, sufficiently detached from the project to be able to provide objective advice. The office responsible for validating the performance baseline (CD-2) should take the lead for EIRs that occur prior to CD-2. Such an approach would place responsibility for conducting the EIR with the group that is most directly accountable for the project outcome.

TAILORING

DOE Order 413.3 recognizes that the complexity and variations inherent in DOE capital asset projects require a project-by-project adjustment or tailoring of the peer review process (DOE, 2000). The order states that while all key review elements are to be addressed, the approach to meeting the requirements should be consistent with a project’s risk, complexity, visibility, cost, safety, security, and schedule. The federal project director is required to develop a tailoring strategy to be approved by the Acquisition Executive.

The parameters for tailoring, such as risk, complexity, visibility, safety, security, cost, and schedule, are influenced by project size, but they are also independent. Although O 413.3 review criteria are based primarily on TPC, peer review team requirements, lines of inquiry, and duration of reviews, depend primarily on the risks inherent in the project and the project management team’s ability to control or mitigate those risks.

Considering the benefits to be gained from all types of peer review, DOE procedures should require more extensive and more independent reviews as the default for most projects. Tailoring should be applied to streamline key review elements if project risks are well understood and are minimal. The committee believes that EIRs and IPRs should be considered on a continuum and that reviews should be tailored to provide a balance of independence and expertise specific to the risks and characteristics of the project. As noted in O 413.3, tailoring should be undertaken by the leaders of the project management team, overseen by the Acquisition Executive or the OECM. DOE should seek to instill a culture of unbiased peer review throughout the organization. The OECM could assume an oversight role to ensure that the reviews are unbiased and to address issues critical to the success of the project.

CRITERIA FOR STRUCTURING A PEER REVIEW PROGRAM

Based on its observations, the committee believes the frequency of peer reviews throughout the life of a project, from CD-0 to CD-4, should depend on several factors, including the TPC, project complexity, inherent risks, capabilities of the project management team, and the amount of time elapsing between CDs.

Evaluating technological risk is a problem faced by many business and government agencies. The process for tailoring a peer review to technological risks requires understanding the magnitude of the potential risk. Faced with quantifying the technological risk in a space vehicle design, NASA developed a technological readiness scale (Mankins, 1995) that evaluates risk in each subsystem: The NASA scale ranges from 1 to 9, with 1 indicating that only the basic principles of the technology have been observed and reported and 9 indicating that a system’s capabilities have been proven during space flight. This scale allows NASA to better understand the risks in a project and make the appropriate decisions.

Technological risk is just one factor affecting the requirements for peer reviews of DOE projects. Potential environmental safety and health risks often drive the need for independent reviews. The organizational complexity of DOE projects, which involve multiple labs and funding organizations, often with different management cultures, and the varying levels of experience of the IPTs and federal project directors also present risks that must be reviewed to ensure that a project can be successfully executed.

Suggested Citation:"3 Criteria for Structuring Review Programs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

Setting EIR requirements based on threshold criteria for risk factors and project team capabilities is beyond the scope of this study. However, such criteria could be developed to assess risk using tools similar to the NASA technology readiness scale or to assess the project management team’s capabilities based on DOE’s project manager career development program (PMCDP) (DOE, 2004). And criteria to assess risks associated with project planning could be based on tools like the Construction Industry Institute’s Project Definition Rating Index (CII, 2006). Such criteria and tools should be reliable but they do not need to be precise. They could be used to categorize projects as high risk or low risk and, when combined with the TPC, would provide some direction for tailoring and developing a peer review plan.

Table 3.2 summarizes the committee’s guidelines for determining the appropriate type of review, the timing of reviews, and the responsible office. Projects have been categorized as small, intermediate, large, and major systems, based on TPC thresholds in O 413.3A (DOE, 2006).

Under each category and each critical decision on Table 3.2, the committee has first identified the type of review to be conducted by default. Thus, for large projects prior to CD-0, an IPR is the default, whereas an EIR is the default for small projects prior to CD-2. It is important to note that the committee’s recommendation for the review to be conducted by default does not necessarily parallel Order 413.3A.

The committee has also identified the types of reviews that should be used if the project is determined to be either high or low risk. Thus, an EIR would be required for a large project at CD-2 unless the project was determined to be low risk, in which case an IPR would be appropriate.

The level of risk should be determined based on a number of factors, including TPC, complexity, technological readiness, the capabilities of the project management team, or other factors. For example, if a project is costly, complex, inherently risky, or if the project management team’s capabilities are poor, the project could be considered high risk. In contrast, if a project is routine in nature, with few risks, and/or the project management team is very capable, the project might be considered low risk. Additional explanatory text follows.

Reviews Prior to Approval of Mission Need (CD-0)

For small, routine projects, no peer reviews should be required unless the project is determined to be high risk, in which case an IPR is recommended. For intermediate projects, an IPR should be required. For large projects and major systems, an IPR should be required unless the project is determined to be high risk, in which case some form of EIR may be warranted. The determination that a project is high risk should be made by the PSO, and all reviews conducted by the PSO.

Reviews Prior to Approval of Alternative Selection and Cost Range (CD-1)

An IPR should be required for all projects. For large projects and major systems determined to be high risk, an EIR may be warranted. This determination should be made by the PSO and all reviews conducted by the PSO.

Reviews Prior to Approval of Baseline Performance (CD-2)

For small and intermediate projects, an EIR should be required unless the project is evaluated by the PSO and determined to be low risk, in which case an IPR is warranted. This determination should be made by the PSO with the OECM’s concurrence; the IPR or EIR should be conducted by the PSO.

Suggested Citation:"3 Criteria for Structuring Review Programs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

TABLE 3.2 Recommended Guidelines for Peer Review Planning

For large projects and major systems, both an IPR and an EIR should be required. These reviews should be coordinated so that either the IPR occurs first to validate the technical scope of work prior to the EIR or, if warranted by the characteristics of the project, the reviews occur simultaneously. The IPR should be conducted by the PSO, and the EIR should be conducted by the OECM, unless the OECM determines the project is low risk, in which case both reviews should be conducted by the PSO.

For major systems, if more than 1 year elapses between CD-2 and CD-3, an annual IPR should be required and be conducted by the PSO.

Suggested Citation:"3 Criteria for Structuring Review Programs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×

Reviews Prior to Start of Construction (CD-3)

For small projects, an IPR should be required and conducted by the PSO. For intermediate and large projects, an IPR should be required unless the project is determined to be high risk, in which case an EIR may be warranted. This determination should be made by the PSO with the OECM’s concurrence. The IPR should be conducted by the PSO, and the EIR should be conducted by the OECM.

For major systems, both an IPR and an EIR should be required. The two reviews should be coordinated such that either the IPR occurs first to validate that the project is ready for construction prior to the EIR or they occur simultaneously. The IPR should be conducted by the PSO and the EIR by the OECM.

Reviews Prior to Approval of Start of Operations or Project Closeout (CD-4)

No reviews should be required for small and intermediate projects unless the PSO determines the project is high risk, in which case an IPR may be warranted. For large projects and major systems, an IPR should be required. If the time elapsing between CD-3 and CD-4 is more than 1 year, an annual IPR should also be required and should be conducted by the PSO.

The committee believes that all peer reviews conducted during the course of a project are important. However, the committee supports an emphasis on reviews prior to CD-2 because it is during front-end planning when the ability to influence the outcome of a project is greatest.

REFERENCES

CII (Construction Industry Institute). 2006. PDRI: Project Definition Rating Index, Building Projects. Second Edition. Austin, Tex.: The Construction Industry Institute.

DOE (Department of Energy). 2000. Program and Project Management for the Acquisition of Capital Assets (Order O 413.3). Washington, D.C.: Department of Energy.

DOE. 2004. Acquisition Career Development Program (DOE Order O 361.1a), Chapter IV. Project Management Career Development Program Module. Washington, D.C.: Department of Energy. Available at http://www.er.doe.gov/opa/PDF/o3611a1.pdf#search=%22doe%20pmcdp%22. Accessed September 19, 2006.

DOE. 2006. Program and Project Management for the Acquisition of Capital Assets (Order O 413.3A). Washington, D.C.: Department of Energy.

Mankins, John C. 1995. Technology Readiness Levels. Washington, D.C.: National Aeronautics and Space Administration. Available at http://www.hq.nasa.gov/office/codeq/trl/trl.pdf#search=%22nasa%20technological%20readiness%20scale%22. Accessed September 19, 2006.

NRC (National Research Council). 2001. Progress in Improving Project Management at the Department of Energy, 2001 Assessment. Washington, D.C.: National Academy Press.

NRC. 2003. Progress in Improving Project Management at the Department of Energy, 2002 Assessment. Washington, D.C.: The National Academies Press.

NRC. 2004. Progress in Improving Project Management at the Department of Energy, 2003 Assessment. Washington, D.C.: The National Academies Press.

NRC. 2005. The Owner’s Role in Project Risk Management. Washington, D.C.: The National Academies Press.

Suggested Citation:"3 Criteria for Structuring Review Programs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 21
Suggested Citation:"3 Criteria for Structuring Review Programs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 22
Suggested Citation:"3 Criteria for Structuring Review Programs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 23
Suggested Citation:"3 Criteria for Structuring Review Programs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 24
Suggested Citation:"3 Criteria for Structuring Review Programs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 25
Suggested Citation:"3 Criteria for Structuring Review Programs." National Research Council. 2007. Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects. Washington, DC: The National Academies Press. doi: 10.17226/11887.
×
Page 26
Next: 4 Conclusions, Findings, and Recommendations »
Assessment of the Results of External Independent Reviews for U.S. Department of Energy Projects Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Peer review is an essential component of engineering practice and other scientific and technical undertakings. Peer reviews are conducted to ensure that activities are technically adequate, competently performed, and properly documented; to validate assumptions, calculations, and extrapolations; and to assess alternative interpretations, methodologies, acceptance criteria, and other aspects of the work products and the documentation that support them. Effective peer reviews are conducted in an environment of mutual respect, recognizing the contributions of all participants. Their primary objective is to help the project team achieve its goals. Reviews also contribute to quality assurance, risk management, and overall improvement of the management process.

The U.S. Department of Energy (DOE) conducts different types of peer reviews at the different stages of a project, including reviews to assess risks and other factors related to design, safety, cost estimates, value engineering, and project management. Independent project reviews (IPRs) are conducted by federal staff not directly affiliated with the project or program and management and operations (M&O) contractors. External independent reviews (EIRs) are overseen by the Office of Engineering and Construction Management and conducted by contractors external to the department. EIRs are the primary focus of this report. However, the committee found that, in many cases, IPRs are explicitly used as preparation for or as preliminary reviews prior to EIRs. Thus, because IPRs are integral to the review process in DOE, they are also discussed because they might have an effect on EIRs.

In October 2000, DOE issued Order 413.3, Program and Project Management for the Acquisition of Capital Assets (DOE, 2000). The order established a series of five critical decisions (CDs), or major milestones, that require senior management review and approval to ensure that a project satisfies applicable mission, design, security, and safety requirements: approve mission need, approve alternative selection and cost range, approve performance baseline, approve start of construction, and approve start of operations or project closeout.

Assessment of the Results of External Independent Reviews for U. S. Department of Energy Projects summarizes the results.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!