. "3 The Independent Review Process ." Assessing the Need for Independent Project Reviews in the Department of Energy. Washington, DC: The National Academies Press, 1998.
The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
ASSESSING THE NEED FOR INDEPENDENT PROJECT REVIEWS IN THE DEPARTMENT OF ENERGY
3
The Independent Review Process
Construction projects have traditionally been subject to various kinds of independent review. The specific type of review depends on interrelated factors, such as the culture of the institution, the structure of the organization and its supporting components, the competence of the staff, the technical complexity and public and political sensitivity of the project, and the level of risk and amount of investment. Many engineers seek independent opinions of their projects, particularly if public safety or health is involved. Many organizations, including DOE, perform reviews routinely as part of their quality management programs.
Independent reviews can take various forms. Private sector architect-engineer firms review their projects internally, both by individuals other than the designer and as part of general overviews by higher echelon individuals. In most cases, the client (owner) also conducts a review of the product, the detail of which depends on the talent in the client organization and the complexity or sensitivity of the project. Also, it is not uncommon for a client to hire another architect-engineer firm to perform a peer review of work by another firm.
Federal organizations that have construction missions strive to have independent reviews of their own projects, as well as of the projects of private sector architect-engineers. Usually, a contract with a private sector firm requires that a quality management plan be submitted. In some cases, federal organizations have hired private firms to review the output of another architect-engineer, as well as experts to review specific features of a project.
Even though some of the reviews described above are internal, their value should not be peremptorily dismissed. If an organization has the proper internal culture and commitment to quality management, internal reviews can be as valuable as external reviews. The International Organization for Standardization (ISO) has introduced a rigorous methodology (ISO9000) for assessing the quality of professional services that can improve the quality of internal review procedures. However, external reviews are considered desirable to promote competition and advance the state of the art.
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole. Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 17
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true
to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please
THE INDEPENDENT REVIEW PROCESS 17
3
The Independent Review Process
Construction projects have traditionally been subject to various kinds of independent review. The specific
type of review depends on interrelated factors, such as the culture of the institution, the structure of the
organization and its supporting components, the competence of the staff, the technical complexity and public and
political sensitivity of the project, and the level of risk and amount of investment. Many engineers seek
independent opinions of their projects, particularly if public safety or health is involved. Many organizations,
including DOE, perform reviews routinely as part of their quality management programs.
Independent reviews can take various forms. Private sector architect-engineer firms review their projects
internally, both by individuals other than the designer and as part of general overviews by higher echelon
individuals. In most cases, the client (owner) also conducts a review of the product, the detail of which depends
on the talent in the client organization and the complexity or sensitivity of the project. Also, it is not uncommon
for a client to hire another architect-engineer firm to perform a peer review of work by another firm.
Federal organizations that have construction missions strive to have independent reviews of their own
projects, as well as of the projects of private sector architect-engineers. Usually, a contract with a private sector
firm requires that a quality management plan be submitted. In some cases, federal organizations have hired
private firms to review the output of another architect-engineer, as well as experts to review specific features of a
project.
Even though some of the reviews described above are internal, their value should not be peremptorily
dismissed. If an organization has the proper internal culture and commitment to quality management, internal
reviews can be as valuable as external reviews. The International Organization for Standardization (ISO) has
introduced a rigorous methodology (ISO9000) for assessing the quality of professional services that can improve
the quality of internal review procedures. However, external reviews are considered desirable to promote
competition and advance the state of the art.
use the print version of this publication as the authoritative version for attribution.
OCR for page 18
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true
to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please
THE INDEPENDENT REVIEW PROCESS 18
INDEPENDENT REVIEW PROCESSES OF OTHER AGENCIES AND ORGANIZATIONS
During the course of this study, several independent review processes, both inside and outside of
government, were examined. Review processes are routinely used by the U.S. Department of Defense (DoD), the
National Aeronautics and Space Administration (NASA), the World Bank, and by DOE. Although these
processes have been adapted to meet the specific needs of the agencies that use them, they are all based on
independent assessments of major projects and programs. The need for independent reviews is based on the
supposition that those who use or benefit from a project may not be the most objective reviewers of that project.
Independent reviewers are more likely to present all of the attributes of a project, both positive and negative.
Brief descriptions of some review practices are presented below, but the degree to which the review practices are
implemented in agencies other than DOE is not discussed.
U.S. Department of Defense
DoD has stated its rationale for using independent reviews (DoD, 1996):
Assessments, independent of the developer and the user, are extremely important to ensure an impartial evaluation
of program status. Consistent with statutory requirements and good management practice, DoD shall use
independent assessments of program status. Senior acquisition officials shall consider these assessments when
making decisions. Staff offices that provide independent assessments shall support the orderly progression of
programs through the acquisition process.
DoD has established a Cost Analysis Improvement Group (CAIG) to serve as the independent advisory
body to the principal decision authority for major systems acquisitions. CAIG is charged with reviewing
programs, estimating assumptions, and evaluating their validity (DoD, 1992):
In preparing its estimate, the CAIG shall employ the best current professional practice for that task. The CAIG may
incorporate in its estimate, with or without adjustment, specific portions of the program office cost estimate…if it
has independently established that the portions included are valid.
The CAIG is chaired by the Deputy Assistant Secretary of Defense (Resource Analysis) in the Office of the
Assistant Secretary of Defense (Program Analysis and Evaluation) and is comprised of members appointed by
each permanent member of the Defense Acquisition Board, one member appointed by the Assistant Secretary of
Defense (Force Management and Personnel), and ad hoc representatives appointed by the CAIG chair for special
purposes. Although CAIG is primarily a program management or systems acquisition tool, the
use the print version of this publication as the authoritative version for attribution.
OCR for page 19
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true
to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please
THE INDEPENDENT REVIEW PROCESS 19
principles of the underlying graded approach can be readily applied to reviews of individual projects.
National Aeronautics and Space Administration
NASA uses a series of independent reviews in the management of its programs and projects. These include
non-advocate reviews (NARs), independent annual reviews (IARs), and independent assessments (IAs) (NASA,
1997). NARs provide independent verification of candidate programs or project plans, life-cycle cost status, and
readiness to proceed to the next phase. These reviews provide NASA management with independent assessments
of the adequacy of the program or project's formulation effort. IARs are yearly reviews of the status of the
performance, cost, and schedule commitments in the performance agreement between the NASA administrator
and the cognizant associate administrator. IARs are designed to measure performance against the program or
project baseline and established thresholds. IAs are a more traditional peer reviews for validating advanced
systems or scientific concepts. All three kinds of reviews are conducted by teams of highly knowledgeable
management, technical, and budget specialists from organizations outside the advocacy chain of the program and
the project being reviewed; members of the reviewing team do not participate in the implementation of the
proposed program or project under review.
The World Bank
The World Bank has launched a program to assess the quality of supervision of its extensive portfolio of
international infrastructure investments. The reviews are intended to complement other programs designed by the
Quality Assurance Group to improve operations, notably the quality of projects at initiation and the performance
of the investment portfolio. Although the World Bank reviews focus on a specific aspect of project management,
namely oversight by the bank staff, the review process is instructive because it recognizes the need for ongoing
assessment as part of a comprehensive program of quality assurance and continuous improvement. The
assessments are performed by independent panels of senior professionals with recognized knowledge of, and
experience in, the geographic region and the type of project. The reviewers have had no previous contact with
the project under review. The results of the assessments are used to promote systemic improvements in the
efficiency and effectiveness of bank supervision (The World Bank, 1997).
use the print version of this publication as the authoritative version for attribution.
OCR for page 20
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true
to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please
THE INDEPENDENT REVIEW PROCESS 20
U.S. Department of Energy
In the last few years, DOE upper management has emphasized the need for more and better project reviews
and has increased its commitment to quality improvement. Although most of DOE's reviews cannot be classified
as totally independent or non-advocacy reviews, their value should not be discounted. For example, an
independent cost estimate (ICE) for the Tritium Extraction Facility at the Savannah River site was performed by
a private contractor at the behest of DOE's independent oversight organization. The ICE team estimate was $140
million less than the estimate prepared by the Savannah River site M&O contractor. A thorough reevaluation of
the technical approach by the site project team identified additional cost reductions, and the project cost estimate
was ultimately reduced by $257 million.
DOE has developed an extensive body of documentation on its program and project management. The
primary requirements document for project planning and management is DOE Order 430.1, Life Cycle Asset
Management (LCAM) (DOE, 1995a). To facilitate implementation of the LCAM Order, DOE has developed
“Good Practice Guides” for many areas of project and fixed asset management, including descriptions of key
processes, performance measures, best-in-class examples, and lessons learned.
Good Practice Guide, GPG-FM-015, Project Reviews (DOE, 1996b), for example, describes the purpose,
scope, and circumstances of different types of reviews that can be performed at various stages of a project for
effective project management. The potential benefits of this sound project guidance are diminished, however,
because use of the guides is not mandated by DOE Headquarters, and consequently, the field offices do not
always follow them. Regarding the independence of review bodies, the guide states:
The project manager should also consider using external, unemotionally attached resources for performance of
reviews, because project control systems often affect most project participants and can provoke bias in internal
project review personnel.
In DOE's Office of Energy Research, a practice of using independent reviews has been in place for more
than 15 years. Because a significant constituency outside of government is interested in energy research (e.g.,
high energy and plasma physics, applied genetics, and advanced materials), DOE has traditionally engaged the
scientific community in evaluating its projects. First a group of knowledgeable individuals familiar with the
subject is assembled. The members of the group (peers) are drawn from DOE program elements not involved in
the proposed project and the national laboratories, augmented by relevant representatives of industry and
academia. Reviews are carried out at the proposal stage for projects that exceed approximately $20 million prior to
use the print version of this publication as the authoritative version for attribution.
OCR for page 21
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true
to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please
THE INDEPENDENT REVIEW PROCESS 21
submittal to Congress. The reviews focus on the baseline (technical scope, costs, and schedules) but do not
generally address higher level issues, such as the relationship to mission objectives, i.e., whether the facility is
needed or not. Once a project has been approved and funded, the peer review team makes semiannual reviews
during the execution phase.
In the area of environmental management, where poor cost and schedule performance are acknowledged by
both the Subcommittee on Energy and Water and DOE, no formal system of external or independent reviews has
been established. However, the Office of Environmental Management did commission two major external
studies, which produced substantive findings and recommendations for improving the process and for achieving
cost savings in the preconceptual and conceptual phases of a project. Environmental management projects
involve either site restoration or waste management and processing, both of which require highly complex
technical solutions with inherently high degrees of uncertainty. The NRC (National Research Council) has also
undertaken a series of reviews and assessments of the science and technology base of the environmental
management program (NRC, 1996, 1997a, 1997b).
For civil infrastructure projects, the LCAM Order largely governs the review process. Many of these
projects are comparable to conventional construction projects in that they are less complex and uncertain than
environmental management projects, and historical experience can be used to develop cost estimates and
schedules. However, the high cost of construction on secure DOE facilities, or, construction “inside the fence,” is
also an issue. These higher costs are attributable to the operating cost of the DOE staff and management and
operating contractors, stringent security requirements, investigative and preparatory costs, complicated and
sometimes duplicative regulations, and the inherent costs of operating in a somewhat noncompetitive
environment. These factors are discussed in reports prepared for DOE by Independent Project Analysis, Inc.
(IPA, 1993, 1996).
In addition to the reviews encouraged by the LCAM Order, field offices and prime site contractors engage
other governmental organizations or private contractors/individuals to perform independent reviews for selected
projects. DOE Headquarters, through the Office of Field Management, also has a quarterly review process to
track project performance and address current and anticipated problems.
EXTERNAL REVIEWS OF U.S. DEPARTMENT OF ENERGY PROGRAMS AND PROJECTS
In the past few years, three external reviews (with varying levels of detail) have been conducted of DOE
programs and projects. The General Accounting
use the print version of this publication as the authoritative version for attribution.
OCR for page 22
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true
to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please
THE INDEPENDENT REVIEW PROCESS 22
Office (GAO), Independent Project Analysis, Inc. (IPA), and the U.S. Army Corps of Engineers (USACE)
undertook these reviews.
General Accounting Office
The GAO study assessed DOE's ability to complete its largest and most significant projects and addressed
three specific topics: DOE's performance in completing its major system acquisitions; key factors that hinder
timely, cost-effective completion of the acquisitions; and measures taken by DOE to improve performance
(GAO, 1996). The GAO report found that, from 1980 to 1996, 31 of 80 major systems acquisitions were
terminated prior to completion for a number of reasons. In some cases, conditions had changed and projects were
no longer needed. In other cases, cancellation could be traced to changes in administration policy. In still other
cases, however, management problems or ineffective oversight by DOE had led to cost overruns and schedule
slippages that caused the projects to be terminated.
GAO identified four key factors that had hindered the successful completion of major systems acquisitions:
• a continuously changing mission for DOE, which often makes it difficult to maintain departmental and
congressional support for long-term high cost projects
• incremental funding of projects from year to year, which often causes delays or even terminations
• a flawed system of incentives that does not always reward organizations for doing the right thing or
punish them for poor contract performance;
• difficulty in hiring, training, and retaining people with requisite skills to provide effective oversight
The GAO report came to the following conclusions (GAO, 1996):
DOE's management initiatives [contract reform, strategic systems and life-cycle asset management, strategic
planning, information resources management, and financial planning] offer the opportunity for the agency to begin
addressing some of the key factors affecting the Department's management of its major system acquisitions.
Independent Project Analysis, Inc.
In 1993, the Office of Environmental Management commissioned IPA to assess the progress of its
environmental restoration program. IPA is a private firm that specializes in measuring project performance
according to key parameters
use the print version of this publication as the authoritative version for attribution.
OCR for page 23
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true
to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please
THE INDEPENDENT REVIEW PROCESS 23
developed (in a proprietary database) from assessments of many similar projects both in the private sector and by
other government agencies. Even though IPA analyzed individual environmental restoration and waste
management projects, the study is a macro-level review that measures trends and highlights problem areas (IPA,
1993). A follow-up study completed in April 1996 compares the progress of the Office of Environmental
Management against the benchmarks established in the earlier study.
The follow-up study found that performance had improved since 1993 and that better project definition at
the front-end of the process is the single most important step for further improving project performance. Project
definition requires integrating financial resources, facilities, personnel, and organizations in a manner that
exploits technical opportunities or satisfies regulatory requirements. The primary objective of project definition
is to minimize changes and define what will be done, who will do it, when it will be done, and what resources
will be required. In other words, the project baseline depends directly on project definition. Earlier and better
project definition should lead to fewer design changes and shorten schedules. The construction industry has
already recognized that most of the benefits of management attention and independent review occur in the early
stages of a project and can favorably affect investments.
The IPA study also suggested that the turnover of project personnel (government and contractor) should be
kept to a minimum, that DOE should be more actively involved in project management, and that contingencies
comparable to industry norms should be included in DOE project estimates. IPA concluded that DOE generally
lags behind the private sector in project definition, even though private contractors largely define the projects for
DOE. The report recognizes that not all DOE projects suffer from a lack of definition and that many have been
successful. The IPA follow-up study was directed primarily toward assessing the effects of project management
on the TEC portion of the budget estimate (IPA, 1996).
U.S. Army Corps of Engineers
In late 1996, USACE was commissioned to provide an independent program assessment of DOE's Ten Year
Plan for integrating environmental remediation projects (USACE, 1997). USACE reviewed the scope of the
projects, cost estimates, and schedules in the program baseline and recommended ways to improve the baseline
components.
Phase 1, which was completed in February 1996, was a reconnaissance-level assessment of 13 sites. This
assessment disclosed that more than 40 percent of cost estimates had been developed using an assumed level-of-
effort rather than activity-based cost elements. USACE acknowledged that a lack of relevant cost
use the print version of this publication as the authoritative version for attribution.
OCR for page 24
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true
to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please
THE INDEPENDENT REVIEW PROCESS 24
data for environmental restoration projects favored the level-of-effort approach, but activity-based costing results
in a more detailed representation of labor, equipment, and materials. The study also found that the scope, costs,
and schedules of about three-quarters of the projects evaluated had not been reviewed independently to ensure
that the necessary measures for fulfilling a given mission were being performed and that they were being
performed as cost effectively as possible. Another finding was that only about one-third of the schedules
evaluated were in a form that could be integrated into the program baseline. These findings reinforced the
findings of the IPA study that project definition at the front-end must be improved.
USACE's Phase 1 study identified 58 targets, with a potential cost savings of $4 to $5 billion, for further
investigation in a Phase 2 study. Unfortunately, the Phase 1 study included only a few FY98 projects. The Phase
2 study is under way and will include a more detailed analysis of each project using activity-based cost
estimates, critical analyses of scope, costs and schedules, and suggestions for re-engineering and improved
integration. The estimates being developed in the USACE study are fairly detailed and reflect the TPC.
SUMMARY
All of the studies cited are considered to be of high caliber and should help DOE improve its budgetary data
from the perspective of higher-level program management (GAO), better project management procedures (IPA),
and better recognition of the mechanics and elements involved in producing good cost estimates (USACE). All
three also point out the wisdom of having good project definition and good estimates before moving into the
budgetary phase of a project.
DOE environmental remediation has long been regarded by the people involved as a program-level rather
than project-level activity. Programs continue at a level of effort until completed, while projects are discrete
operations with defined objectives and specific costs and schedules. Therefore, one of the significant issues for
DOE is to integrate program-based attitudes with project-based thinking, in other words, to convert waste
management programs into definable projects. Providing specific project definitions in the early stages of
environmental restoration projects (as well as some waste management projects) may prove to be very difficult,
however, because of the overwhelming scope and complexity of the problems. Establishing baselines for projects
that must meet massive demands, for which defining the extent and characteristics of the problem is a major
undertaking, and for which technological solutions are still being explored during the preconceptual and
conceptual phases, poses major challenges. In fact, adjusting or fine tuning the baseline periodically and
carefully documenting the changes, might be a preferable approach (each successive reanalysis should refine the
scope, schedules, and cost estimates).
use the print version of this publication as the authoritative version for attribution.
OCR for page 25
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true
to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please
THE INDEPENDENT REVIEW PROCESS 25
The technical scope, cost estimates, and schedules presented in future budgetary data should improve as
DOE and its contractors gain experience and because of DOE management's commitment to continuous quality
improvement. After examining the history of DOE's performance in energy, science, waste management, and
environmental restoration programs, this study concludes that independent reviews are generally warranted and
would be beneficial. Subsequent reviews should complement the existing ones as much as possible. However,
independent reviews are not justified for all projects, and all projects that are reviewed should not be subject to
the same type and intensity of review. In practice, application of the proposed criteria will vary with the type and
size of the specific project, and, ultimately, DOE must exercise judgment in selecting projects for review.
Recommendations for independent reviews of specific projects appear in Chapter 4.
use the print version of this publication as the authoritative version for attribution.
OCR for page 26
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true
to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please
use the print version of this publication as the authoritative version for attribution.
THE INDEPENDENT REVIEW PROCESS
26
