3

The Independent Review Process

Construction projects have traditionally been subject to various kinds of independent review. The specific type of review depends on interrelated factors, such as the culture of the institution, the structure of the organization and its supporting components, the competence of the staff, the technical complexity and public and political sensitivity of the project, and the level of risk and amount of investment. Many engineers seek independent opinions of their projects, particularly if public safety or health is involved. Many organizations, including DOE, perform reviews routinely as part of their quality management programs.

Independent reviews can take various forms. Private sector architect-engineer firms review their projects internally, both by individuals other than the designer and as part of general overviews by higher echelon individuals. In most cases, the client (owner) also conducts a review of the product, the detail of which depends on the talent in the client organization and the complexity or sensitivity of the project. Also, it is not uncommon for a client to hire another architect-engineer firm to perform a peer review of work by another firm.

Federal organizations that have construction missions strive to have independent reviews of their own projects, as well as of the projects of private sector architect-engineers. Usually, a contract with a private sector firm requires that a quality management plan be submitted. In some cases, federal organizations have hired private firms to review the output of another architect-engineer, as well as experts to review specific features of a project.

Even though some of the reviews described above are internal, their value should not be peremptorily dismissed. If an organization has the proper internal culture and commitment to quality management, internal reviews can be as valuable as external reviews. The International Organization for Standardization (ISO) has introduced a rigorous methodology (ISO9000) for assessing the quality of professional services that can improve the quality of internal review procedures. However, external reviews are considered desirable to promote competition and advance the state of the art.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 17
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please THE INDEPENDENT REVIEW PROCESS 17 3 The Independent Review Process Construction projects have traditionally been subject to various kinds of independent review. The specific type of review depends on interrelated factors, such as the culture of the institution, the structure of the organization and its supporting components, the competence of the staff, the technical complexity and public and political sensitivity of the project, and the level of risk and amount of investment. Many engineers seek independent opinions of their projects, particularly if public safety or health is involved. Many organizations, including DOE, perform reviews routinely as part of their quality management programs. Independent reviews can take various forms. Private sector architect-engineer firms review their projects internally, both by individuals other than the designer and as part of general overviews by higher echelon individuals. In most cases, the client (owner) also conducts a review of the product, the detail of which depends on the talent in the client organization and the complexity or sensitivity of the project. Also, it is not uncommon for a client to hire another architect-engineer firm to perform a peer review of work by another firm. Federal organizations that have construction missions strive to have independent reviews of their own projects, as well as of the projects of private sector architect-engineers. Usually, a contract with a private sector firm requires that a quality management plan be submitted. In some cases, federal organizations have hired private firms to review the output of another architect-engineer, as well as experts to review specific features of a project. Even though some of the reviews described above are internal, their value should not be peremptorily dismissed. If an organization has the proper internal culture and commitment to quality management, internal reviews can be as valuable as external reviews. The International Organization for Standardization (ISO) has introduced a rigorous methodology (ISO9000) for assessing the quality of professional services that can improve the quality of internal review procedures. However, external reviews are considered desirable to promote competition and advance the state of the art. use the print version of this publication as the authoritative version for attribution.

OCR for page 17
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please THE INDEPENDENT REVIEW PROCESS 18 INDEPENDENT REVIEW PROCESSES OF OTHER AGENCIES AND ORGANIZATIONS During the course of this study, several independent review processes, both inside and outside of government, were examined. Review processes are routinely used by the U.S. Department of Defense (DoD), the National Aeronautics and Space Administration (NASA), the World Bank, and by DOE. Although these processes have been adapted to meet the specific needs of the agencies that use them, they are all based on independent assessments of major projects and programs. The need for independent reviews is based on the supposition that those who use or benefit from a project may not be the most objective reviewers of that project. Independent reviewers are more likely to present all of the attributes of a project, both positive and negative. Brief descriptions of some review practices are presented below, but the degree to which the review practices are implemented in agencies other than DOE is not discussed. U.S. Department of Defense DoD has stated its rationale for using independent reviews (DoD, 1996): Assessments, independent of the developer and the user, are extremely important to ensure an impartial evaluation of program status. Consistent with statutory requirements and good management practice, DoD shall use independent assessments of program status. Senior acquisition officials shall consider these assessments when making decisions. Staff offices that provide independent assessments shall support the orderly progression of programs through the acquisition process. DoD has established a Cost Analysis Improvement Group (CAIG) to serve as the independent advisory body to the principal decision authority for major systems acquisitions. CAIG is charged with reviewing programs, estimating assumptions, and evaluating their validity (DoD, 1992): In preparing its estimate, the CAIG shall employ the best current professional practice for that task. The CAIG may incorporate in its estimate, with or without adjustment, specific portions of the program office cost estimate…if it has independently established that the portions included are valid. The CAIG is chaired by the Deputy Assistant Secretary of Defense (Resource Analysis) in the Office of the Assistant Secretary of Defense (Program Analysis and Evaluation) and is comprised of members appointed by each permanent member of the Defense Acquisition Board, one member appointed by the Assistant Secretary of Defense (Force Management and Personnel), and ad hoc representatives appointed by the CAIG chair for special purposes. Although CAIG is primarily a program management or systems acquisition tool, the use the print version of this publication as the authoritative version for attribution.

OCR for page 17
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please THE INDEPENDENT REVIEW PROCESS 19 principles of the underlying graded approach can be readily applied to reviews of individual projects. National Aeronautics and Space Administration NASA uses a series of independent reviews in the management of its programs and projects. These include non-advocate reviews (NARs), independent annual reviews (IARs), and independent assessments (IAs) (NASA, 1997). NARs provide independent verification of candidate programs or project plans, life-cycle cost status, and readiness to proceed to the next phase. These reviews provide NASA management with independent assessments of the adequacy of the program or project's formulation effort. IARs are yearly reviews of the status of the performance, cost, and schedule commitments in the performance agreement between the NASA administrator and the cognizant associate administrator. IARs are designed to measure performance against the program or project baseline and established thresholds. IAs are a more traditional peer reviews for validating advanced systems or scientific concepts. All three kinds of reviews are conducted by teams of highly knowledgeable management, technical, and budget specialists from organizations outside the advocacy chain of the program and the project being reviewed; members of the reviewing team do not participate in the implementation of the proposed program or project under review. The World Bank The World Bank has launched a program to assess the quality of supervision of its extensive portfolio of international infrastructure investments. The reviews are intended to complement other programs designed by the Quality Assurance Group to improve operations, notably the quality of projects at initiation and the performance of the investment portfolio. Although the World Bank reviews focus on a specific aspect of project management, namely oversight by the bank staff, the review process is instructive because it recognizes the need for ongoing assessment as part of a comprehensive program of quality assurance and continuous improvement. The assessments are performed by independent panels of senior professionals with recognized knowledge of, and experience in, the geographic region and the type of project. The reviewers have had no previous contact with the project under review. The results of the assessments are used to promote systemic improvements in the efficiency and effectiveness of bank supervision (The World Bank, 1997). use the print version of this publication as the authoritative version for attribution.

OCR for page 17
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please THE INDEPENDENT REVIEW PROCESS 20 U.S. Department of Energy In the last few years, DOE upper management has emphasized the need for more and better project reviews and has increased its commitment to quality improvement. Although most of DOE's reviews cannot be classified as totally independent or non-advocacy reviews, their value should not be discounted. For example, an independent cost estimate (ICE) for the Tritium Extraction Facility at the Savannah River site was performed by a private contractor at the behest of DOE's independent oversight organization. The ICE team estimate was $140 million less than the estimate prepared by the Savannah River site M&O contractor. A thorough reevaluation of the technical approach by the site project team identified additional cost reductions, and the project cost estimate was ultimately reduced by $257 million. DOE has developed an extensive body of documentation on its program and project management. The primary requirements document for project planning and management is DOE Order 430.1, Life Cycle Asset Management (LCAM) (DOE, 1995a). To facilitate implementation of the LCAM Order, DOE has developed “Good Practice Guides” for many areas of project and fixed asset management, including descriptions of key processes, performance measures, best-in-class examples, and lessons learned. Good Practice Guide, GPG-FM-015, Project Reviews (DOE, 1996b), for example, describes the purpose, scope, and circumstances of different types of reviews that can be performed at various stages of a project for effective project management. The potential benefits of this sound project guidance are diminished, however, because use of the guides is not mandated by DOE Headquarters, and consequently, the field offices do not always follow them. Regarding the independence of review bodies, the guide states: The project manager should also consider using external, unemotionally attached resources for performance of reviews, because project control systems often affect most project participants and can provoke bias in internal project review personnel. In DOE's Office of Energy Research, a practice of using independent reviews has been in place for more than 15 years. Because a significant constituency outside of government is interested in energy research (e.g., high energy and plasma physics, applied genetics, and advanced materials), DOE has traditionally engaged the scientific community in evaluating its projects. First a group of knowledgeable individuals familiar with the subject is assembled. The members of the group (peers) are drawn from DOE program elements not involved in the proposed project and the national laboratories, augmented by relevant representatives of industry and academia. Reviews are carried out at the proposal stage for projects that exceed approximately $20 million prior to use the print version of this publication as the authoritative version for attribution.

OCR for page 17
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please THE INDEPENDENT REVIEW PROCESS 21 submittal to Congress. The reviews focus on the baseline (technical scope, costs, and schedules) but do not generally address higher level issues, such as the relationship to mission objectives, i.e., whether the facility is needed or not. Once a project has been approved and funded, the peer review team makes semiannual reviews during the execution phase. In the area of environmental management, where poor cost and schedule performance are acknowledged by both the Subcommittee on Energy and Water and DOE, no formal system of external or independent reviews has been established. However, the Office of Environmental Management did commission two major external studies, which produced substantive findings and recommendations for improving the process and for achieving cost savings in the preconceptual and conceptual phases of a project. Environmental management projects involve either site restoration or waste management and processing, both of which require highly complex technical solutions with inherently high degrees of uncertainty. The NRC (National Research Council) has also undertaken a series of reviews and assessments of the science and technology base of the environmental management program (NRC, 1996, 1997a, 1997b). For civil infrastructure projects, the LCAM Order largely governs the review process. Many of these projects are comparable to conventional construction projects in that they are less complex and uncertain than environmental management projects, and historical experience can be used to develop cost estimates and schedules. However, the high cost of construction on secure DOE facilities, or, construction “inside the fence,” is also an issue. These higher costs are attributable to the operating cost of the DOE staff and management and operating contractors, stringent security requirements, investigative and preparatory costs, complicated and sometimes duplicative regulations, and the inherent costs of operating in a somewhat noncompetitive environment. These factors are discussed in reports prepared for DOE by Independent Project Analysis, Inc. (IPA, 1993, 1996). In addition to the reviews encouraged by the LCAM Order, field offices and prime site contractors engage other governmental organizations or private contractors/individuals to perform independent reviews for selected projects. DOE Headquarters, through the Office of Field Management, also has a quarterly review process to track project performance and address current and anticipated problems. EXTERNAL REVIEWS OF U.S. DEPARTMENT OF ENERGY PROGRAMS AND PROJECTS In the past few years, three external reviews (with varying levels of detail) have been conducted of DOE programs and projects. The General Accounting use the print version of this publication as the authoritative version for attribution.

OCR for page 17
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please THE INDEPENDENT REVIEW PROCESS 22 Office (GAO), Independent Project Analysis, Inc. (IPA), and the U.S. Army Corps of Engineers (USACE) undertook these reviews. General Accounting Office The GAO study assessed DOE's ability to complete its largest and most significant projects and addressed three specific topics: DOE's performance in completing its major system acquisitions; key factors that hinder timely, cost-effective completion of the acquisitions; and measures taken by DOE to improve performance (GAO, 1996). The GAO report found that, from 1980 to 1996, 31 of 80 major systems acquisitions were terminated prior to completion for a number of reasons. In some cases, conditions had changed and projects were no longer needed. In other cases, cancellation could be traced to changes in administration policy. In still other cases, however, management problems or ineffective oversight by DOE had led to cost overruns and schedule slippages that caused the projects to be terminated. GAO identified four key factors that had hindered the successful completion of major systems acquisitions: • a continuously changing mission for DOE, which often makes it difficult to maintain departmental and congressional support for long-term high cost projects • incremental funding of projects from year to year, which often causes delays or even terminations • a flawed system of incentives that does not always reward organizations for doing the right thing or punish them for poor contract performance; • difficulty in hiring, training, and retaining people with requisite skills to provide effective oversight The GAO report came to the following conclusions (GAO, 1996): DOE's management initiatives [contract reform, strategic systems and life-cycle asset management, strategic planning, information resources management, and financial planning] offer the opportunity for the agency to begin addressing some of the key factors affecting the Department's management of its major system acquisitions. Independent Project Analysis, Inc. In 1993, the Office of Environmental Management commissioned IPA to assess the progress of its environmental restoration program. IPA is a private firm that specializes in measuring project performance according to key parameters use the print version of this publication as the authoritative version for attribution.

OCR for page 17
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please THE INDEPENDENT REVIEW PROCESS 23 developed (in a proprietary database) from assessments of many similar projects both in the private sector and by other government agencies. Even though IPA analyzed individual environmental restoration and waste management projects, the study is a macro-level review that measures trends and highlights problem areas (IPA, 1993). A follow-up study completed in April 1996 compares the progress of the Office of Environmental Management against the benchmarks established in the earlier study. The follow-up study found that performance had improved since 1993 and that better project definition at the front-end of the process is the single most important step for further improving project performance. Project definition requires integrating financial resources, facilities, personnel, and organizations in a manner that exploits technical opportunities or satisfies regulatory requirements. The primary objective of project definition is to minimize changes and define what will be done, who will do it, when it will be done, and what resources will be required. In other words, the project baseline depends directly on project definition. Earlier and better project definition should lead to fewer design changes and shorten schedules. The construction industry has already recognized that most of the benefits of management attention and independent review occur in the early stages of a project and can favorably affect investments. The IPA study also suggested that the turnover of project personnel (government and contractor) should be kept to a minimum, that DOE should be more actively involved in project management, and that contingencies comparable to industry norms should be included in DOE project estimates. IPA concluded that DOE generally lags behind the private sector in project definition, even though private contractors largely define the projects for DOE. The report recognizes that not all DOE projects suffer from a lack of definition and that many have been successful. The IPA follow-up study was directed primarily toward assessing the effects of project management on the TEC portion of the budget estimate (IPA, 1996). U.S. Army Corps of Engineers In late 1996, USACE was commissioned to provide an independent program assessment of DOE's Ten Year Plan for integrating environmental remediation projects (USACE, 1997). USACE reviewed the scope of the projects, cost estimates, and schedules in the program baseline and recommended ways to improve the baseline components. Phase 1, which was completed in February 1996, was a reconnaissance-level assessment of 13 sites. This assessment disclosed that more than 40 percent of cost estimates had been developed using an assumed level-of- effort rather than activity-based cost elements. USACE acknowledged that a lack of relevant cost use the print version of this publication as the authoritative version for attribution.

OCR for page 17
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please THE INDEPENDENT REVIEW PROCESS 24 data for environmental restoration projects favored the level-of-effort approach, but activity-based costing results in a more detailed representation of labor, equipment, and materials. The study also found that the scope, costs, and schedules of about three-quarters of the projects evaluated had not been reviewed independently to ensure that the necessary measures for fulfilling a given mission were being performed and that they were being performed as cost effectively as possible. Another finding was that only about one-third of the schedules evaluated were in a form that could be integrated into the program baseline. These findings reinforced the findings of the IPA study that project definition at the front-end must be improved. USACE's Phase 1 study identified 58 targets, with a potential cost savings of $4 to $5 billion, for further investigation in a Phase 2 study. Unfortunately, the Phase 1 study included only a few FY98 projects. The Phase 2 study is under way and will include a more detailed analysis of each project using activity-based cost estimates, critical analyses of scope, costs and schedules, and suggestions for re-engineering and improved integration. The estimates being developed in the USACE study are fairly detailed and reflect the TPC. SUMMARY All of the studies cited are considered to be of high caliber and should help DOE improve its budgetary data from the perspective of higher-level program management (GAO), better project management procedures (IPA), and better recognition of the mechanics and elements involved in producing good cost estimates (USACE). All three also point out the wisdom of having good project definition and good estimates before moving into the budgetary phase of a project. DOE environmental remediation has long been regarded by the people involved as a program-level rather than project-level activity. Programs continue at a level of effort until completed, while projects are discrete operations with defined objectives and specific costs and schedules. Therefore, one of the significant issues for DOE is to integrate program-based attitudes with project-based thinking, in other words, to convert waste management programs into definable projects. Providing specific project definitions in the early stages of environmental restoration projects (as well as some waste management projects) may prove to be very difficult, however, because of the overwhelming scope and complexity of the problems. Establishing baselines for projects that must meet massive demands, for which defining the extent and characteristics of the problem is a major undertaking, and for which technological solutions are still being explored during the preconceptual and conceptual phases, poses major challenges. In fact, adjusting or fine tuning the baseline periodically and carefully documenting the changes, might be a preferable approach (each successive reanalysis should refine the scope, schedules, and cost estimates). use the print version of this publication as the authoritative version for attribution.

OCR for page 17
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please THE INDEPENDENT REVIEW PROCESS 25 The technical scope, cost estimates, and schedules presented in future budgetary data should improve as DOE and its contractors gain experience and because of DOE management's commitment to continuous quality improvement. After examining the history of DOE's performance in energy, science, waste management, and environmental restoration programs, this study concludes that independent reviews are generally warranted and would be beneficial. Subsequent reviews should complement the existing ones as much as possible. However, independent reviews are not justified for all projects, and all projects that are reviewed should not be subject to the same type and intensity of review. In practice, application of the proposed criteria will vary with the type and size of the specific project, and, ultimately, DOE must exercise judgment in selecting projects for review. Recommendations for independent reviews of specific projects appear in Chapter 4. use the print version of this publication as the authoritative version for attribution.

OCR for page 17
About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution. THE INDEPENDENT REVIEW PROCESS 26