3
Criteria for Structuring Review Programs
DOE Order O 413.3 established criteria for the timing of peer reviews based on total project cost (TPC) and identified the DOE office responsible for conducting the reviews (Table 1.1). A revised Order, O 413.3A was issued on July 28, 2006. The timing and types of peer reviews and the responsible office remained the same as outlined in Table 1.1. However, the TPC threshold levels that trigger the requirement for an EIR or IPR (not shown on Table 1.1) were changed significantly. The Office of Engineering and Construction Management (OECM) is now required to conduct an external independent review (EIR) prior to CD-2 for a project with a TPC of $100 million or more and prior to CD-3 for a major systems project of $750 million or more (DOE, 2006). The revised Order, 413.3A, requires the program secretarial offices (PSOs) to conduct independent project reviews (IPRs) prior to CD-2 and CD-3 for projects with a TPC less than $100 million. Order 413.3A also addresses the need for IPRs prior to other critical decision (CD) points and for peer reviews of technical design, quality, risk management, and safety.
DOE projects vary significantly in size, complexity, scope, and inherent risk. An analysis of 141 projects in the Project Assessment and Reporting System (PARS) database indicated that the average total project cost was more than $1 billion, whereas the median total project cost was $175 million. The PARS data demonstrate skewing due the small percentage of projects having very high costs. Table 3.1 shows the distribution of the 141 projects by TPC. Although 70 percent of all projects cost less than $750 million each, their combined costs total only 11 percent of all project estimated costs. Considered another way, 30 percent of the total number of DOE projects accounted for 89 percent of total estimated costs. Many of the projects costing more than $750 million are in the Office of Environmental Management.
The committee was asked to advise DOE on how it could focus its resources on those projects with the greatest risk. In the committee’s opinion, the distribution of projects at various TPCs and the relative level of risk of the larger projects support Order 413.3A’s revised thresholds at which an IPR or an EIR is triggered. However, the committee also believes that more emphasis should be placed on factors other than cost that influence project risk, such as the extent to which a technology was used previously in its planned application, the number of stakeholders and their influence on project requirements, and the certainty and detail in the definition of the project scope.
TIMING OF REVIEWS
DOE policy recognizes the importance of peer reviews for performance baseline validation and prior to the start of construction. The Construction Industry Institute’s (CII’s) benchmarking program has shown that the quality and completeness of a project’s front-end planning affect the probability that a project will succeed and that an owner’s best opportunities to influence the outcome of a project occur at the front end (CII, 1996). The more complex and risky a project, the greater the need to ensure the validity of early CDs.
TABLE 3.1 Distribution of Sample Projects in the Project Assessment and Reporting System
Total Project Cost (million $) |
Percentage of Projects (Cumulative) (million $) |
Percentage of Costs (Cumulative) (million $) |
<20 |
11 |
<1 |
<100 |
38 |
2 |
<400 |
62 |
7 |
<750 |
70 |
11 |
Previous NRC reports focusing on DOE’s project management procedures emphasized again and again the importance of effective risk management to the success of projects (NRC, 2001, 2003, 2004). Risk management is important element of EIRs because it is critical to the validation of baseline budgets and schedules. The nature and severity of risks vary greatly from project to project, requiring their review to be closely tailored to specific project needs.
The NRC report The Owner’s Role in Project Risk Management addresses the role of risk management in DOE projects and supports a proactive approach in which owners take the following actions:
-
Start risk management early in the life cycle of all projects, prior to CD-0, approval of mission need.
-
Include key stakeholders in the process, with the DOE project director as the lead and the integrated project team (IPT) also intimately involved.
-
Evaluate project risks and risk responses periodically during project life (CD-0 through approval of the start of operations [CD-4]).
-
Develop risk mitigation plans and update them as the project progresses.
-
Follow through with mitigation actions until risks are acceptable.
-
Tie a project’s level of risk to cost and schedule contingencies.
-
Communicate progress and changes to project risks and mitigation plans to all key stakeholders.
The report also noted that risk needs to be managed by qualified personnel working within a project management process that includes review and approval by senior management. CDs, such as those defined in DOE O 413.3, must be made by senior management to ensure the quality of the risk management process and that the risks inherent in a project are necessary and acceptable. Reliance on team experience alone, without subjecting decisions to reviews, can lead to gaps in analysis and lack of consistency (NRC, 2005).
Peer reviews conducted prior to CD-0 and CD-1 could play an important role in risk management and quality assurance. Moreover, since risk can change as a project—especially a large and complex project—develops, some form of periodic peer review is justified. The committee believes, furthermore, that additional reviews should be considered to assure quality if the time elapsing between CDs is greater than 1 year.
RESPONSIBLE OFFICE
DOE Orders 413.3 and 413.3A assign responsibility for project oversight, baseline validation, and control of reviews based on the TPC. The overall result of DOE policies is that the OECM controls EIRs and the PSOs control the IPRs. In the committee’s opinion, this distinction is unnecessary and it may hamper the selection of the most appropriate type of review. Depending on the TPC, complexity, and
risks of a project, EIRs could be successfully planned and executed by the PSOs and the project teams. The OECM could then assume an oversight role to ensure that the reviews are adequately planned and executed and that the review teams have the necessary expertise and are sufficiently independent, that is, sufficiently detached from the project to be able to provide objective advice. The office responsible for validating the performance baseline (CD-2) should take the lead for EIRs that occur prior to CD-2. Such an approach would place responsibility for conducting the EIR with the group that is most directly accountable for the project outcome.
TAILORING
DOE Order 413.3 recognizes that the complexity and variations inherent in DOE capital asset projects require a project-by-project adjustment or tailoring of the peer review process (DOE, 2000). The order states that while all key review elements are to be addressed, the approach to meeting the requirements should be consistent with a project’s risk, complexity, visibility, cost, safety, security, and schedule. The federal project director is required to develop a tailoring strategy to be approved by the Acquisition Executive.
The parameters for tailoring, such as risk, complexity, visibility, safety, security, cost, and schedule, are influenced by project size, but they are also independent. Although O 413.3 review criteria are based primarily on TPC, peer review team requirements, lines of inquiry, and duration of reviews, depend primarily on the risks inherent in the project and the project management team’s ability to control or mitigate those risks.
Considering the benefits to be gained from all types of peer review, DOE procedures should require more extensive and more independent reviews as the default for most projects. Tailoring should be applied to streamline key review elements if project risks are well understood and are minimal. The committee believes that EIRs and IPRs should be considered on a continuum and that reviews should be tailored to provide a balance of independence and expertise specific to the risks and characteristics of the project. As noted in O 413.3, tailoring should be undertaken by the leaders of the project management team, overseen by the Acquisition Executive or the OECM. DOE should seek to instill a culture of unbiased peer review throughout the organization. The OECM could assume an oversight role to ensure that the reviews are unbiased and to address issues critical to the success of the project.
CRITERIA FOR STRUCTURING A PEER REVIEW PROGRAM
Based on its observations, the committee believes the frequency of peer reviews throughout the life of a project, from CD-0 to CD-4, should depend on several factors, including the TPC, project complexity, inherent risks, capabilities of the project management team, and the amount of time elapsing between CDs.
Evaluating technological risk is a problem faced by many business and government agencies. The process for tailoring a peer review to technological risks requires understanding the magnitude of the potential risk. Faced with quantifying the technological risk in a space vehicle design, NASA developed a technological readiness scale (Mankins, 1995) that evaluates risk in each subsystem: The NASA scale ranges from 1 to 9, with 1 indicating that only the basic principles of the technology have been observed and reported and 9 indicating that a system’s capabilities have been proven during space flight. This scale allows NASA to better understand the risks in a project and make the appropriate decisions.
Technological risk is just one factor affecting the requirements for peer reviews of DOE projects. Potential environmental safety and health risks often drive the need for independent reviews. The organizational complexity of DOE projects, which involve multiple labs and funding organizations, often with different management cultures, and the varying levels of experience of the IPTs and federal project directors also present risks that must be reviewed to ensure that a project can be successfully executed.
Setting EIR requirements based on threshold criteria for risk factors and project team capabilities is beyond the scope of this study. However, such criteria could be developed to assess risk using tools similar to the NASA technology readiness scale or to assess the project management team’s capabilities based on DOE’s project manager career development program (PMCDP) (DOE, 2004). And criteria to assess risks associated with project planning could be based on tools like the Construction Industry Institute’s Project Definition Rating Index (CII, 2006). Such criteria and tools should be reliable but they do not need to be precise. They could be used to categorize projects as high risk or low risk and, when combined with the TPC, would provide some direction for tailoring and developing a peer review plan.
Table 3.2 summarizes the committee’s guidelines for determining the appropriate type of review, the timing of reviews, and the responsible office. Projects have been categorized as small, intermediate, large, and major systems, based on TPC thresholds in O 413.3A (DOE, 2006).
Under each category and each critical decision on Table 3.2, the committee has first identified the type of review to be conducted by default. Thus, for large projects prior to CD-0, an IPR is the default, whereas an EIR is the default for small projects prior to CD-2. It is important to note that the committee’s recommendation for the review to be conducted by default does not necessarily parallel Order 413.3A.
The committee has also identified the types of reviews that should be used if the project is determined to be either high or low risk. Thus, an EIR would be required for a large project at CD-2 unless the project was determined to be low risk, in which case an IPR would be appropriate.
The level of risk should be determined based on a number of factors, including TPC, complexity, technological readiness, the capabilities of the project management team, or other factors. For example, if a project is costly, complex, inherently risky, or if the project management team’s capabilities are poor, the project could be considered high risk. In contrast, if a project is routine in nature, with few risks, and/or the project management team is very capable, the project might be considered low risk. Additional explanatory text follows.
Reviews Prior to Approval of Mission Need (CD-0)
For small, routine projects, no peer reviews should be required unless the project is determined to be high risk, in which case an IPR is recommended. For intermediate projects, an IPR should be required. For large projects and major systems, an IPR should be required unless the project is determined to be high risk, in which case some form of EIR may be warranted. The determination that a project is high risk should be made by the PSO, and all reviews conducted by the PSO.
Reviews Prior to Approval of Alternative Selection and Cost Range (CD-1)
An IPR should be required for all projects. For large projects and major systems determined to be high risk, an EIR may be warranted. This determination should be made by the PSO and all reviews conducted by the PSO.
Reviews Prior to Approval of Baseline Performance (CD-2)
For small and intermediate projects, an EIR should be required unless the project is evaluated by the PSO and determined to be low risk, in which case an IPR is warranted. This determination should be made by the PSO with the OECM’s concurrence; the IPR or EIR should be conducted by the PSO.
TABLE 3.2 Recommended Guidelines for Peer Review Planning
For large projects and major systems, both an IPR and an EIR should be required. These reviews should be coordinated so that either the IPR occurs first to validate the technical scope of work prior to the EIR or, if warranted by the characteristics of the project, the reviews occur simultaneously. The IPR should be conducted by the PSO, and the EIR should be conducted by the OECM, unless the OECM determines the project is low risk, in which case both reviews should be conducted by the PSO.
For major systems, if more than 1 year elapses between CD-2 and CD-3, an annual IPR should be required and be conducted by the PSO.
Reviews Prior to Start of Construction (CD-3)
For small projects, an IPR should be required and conducted by the PSO. For intermediate and large projects, an IPR should be required unless the project is determined to be high risk, in which case an EIR may be warranted. This determination should be made by the PSO with the OECM’s concurrence. The IPR should be conducted by the PSO, and the EIR should be conducted by the OECM.
For major systems, both an IPR and an EIR should be required. The two reviews should be coordinated such that either the IPR occurs first to validate that the project is ready for construction prior to the EIR or they occur simultaneously. The IPR should be conducted by the PSO and the EIR by the OECM.
Reviews Prior to Approval of Start of Operations or Project Closeout (CD-4)
No reviews should be required for small and intermediate projects unless the PSO determines the project is high risk, in which case an IPR may be warranted. For large projects and major systems, an IPR should be required. If the time elapsing between CD-3 and CD-4 is more than 1 year, an annual IPR should also be required and should be conducted by the PSO.
The committee believes that all peer reviews conducted during the course of a project are important. However, the committee supports an emphasis on reviews prior to CD-2 because it is during front-end planning when the ability to influence the outcome of a project is greatest.
REFERENCES
CII (Construction Industry Institute). 2006. PDRI: Project Definition Rating Index, Building Projects. Second Edition. Austin, Tex.: The Construction Industry Institute.
DOE (Department of Energy). 2000. Program and Project Management for the Acquisition of Capital Assets (Order O 413.3). Washington, D.C.: Department of Energy.
DOE. 2004. Acquisition Career Development Program (DOE Order O 361.1a), Chapter IV. Project Management Career Development Program Module. Washington, D.C.: Department of Energy. Available at http://www.er.doe.gov/opa/PDF/o3611a1.pdf#search=%22doe%20pmcdp%22. Accessed September 19, 2006.
DOE. 2006. Program and Project Management for the Acquisition of Capital Assets (Order O 413.3A). Washington, D.C.: Department of Energy.
Mankins, John C. 1995. Technology Readiness Levels. Washington, D.C.: National Aeronautics and Space Administration. Available at http://www.hq.nasa.gov/office/codeq/trl/trl.pdf#search=%22nasa%20technological%20readiness%20scale%22. Accessed September 19, 2006.
NRC (National Research Council). 2001. Progress in Improving Project Management at the Department of Energy, 2001 Assessment. Washington, D.C.: National Academy Press.
NRC. 2003. Progress in Improving Project Management at the Department of Energy, 2002 Assessment. Washington, D.C.: The National Academies Press.
NRC. 2004. Progress in Improving Project Management at the Department of Energy, 2003 Assessment. Washington, D.C.: The National Academies Press.
NRC. 2005. The Owner’s Role in Project Risk Management. Washington, D.C.: The National Academies Press.