National Academies Press: OpenBook

Progress in Improving Project Management at the Department of Energy: 2003 Assessment (2004)

Chapter: Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation

« Previous: Appendix E: Delegation of Authority as Acquisition Executive
Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

APPENDIX F
Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation Prior to Critical Decision 2

This summary was prepared by the DOE Office of Engineering and Construction Management and covers 19 projects with total estimated costs of under $100 million, equally apportioned between the Office of Science (SC), the Office of Environmental Management (EM), and the National Nuclear Security Administration (NNSA).

NATIONAL NUCLEAR SECURITY ADMINISTRATION PROJECTS

Project:

Terascale Simulation Facility

Site:

Sandia National Laboratories

ID:

00-D-103

TPC:

$95.3 million

EIR Date:

August 2001

Key Findings:

  • Resource-loaded schedule was not developed.

  • Fiber-optic system was not included in the cost estimate.

Project:

Test Capabilities Revitalization Phase I

Site:

Sandia National Laboratories

ID:

02-D-103-01

TPC:

$47.3 million

EIR Date:

December 2002

Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

Key Findings:

  • Project has three tiers of equipment. Tiers 2 and 3 appeared to be desirable items that will be purchased if money is available at the end of the project. There was no documented basis of what was in the baseline.

  • No documented connection exists between the results of the risk analysis workshops and the contingency figures the project team actually applied to the project estimate.

  • The backup support for the demolition and disposal estimate does not match the baseline estimate and is not detailed enough to support the $1 million estimated for this activity.

Project:

Gas Transfer Capacity Expansion project

Site:

Kansas City

ID:

03-D-121

TPC:

$31.3 million

EIR Date:

November 2002

Key Findings:

  • Key assumptions for cost and schedule estimates were not documented.

  • Assumptions for overhead and profit markups were unreasonable.

Project:

Joint Computational Engineering Laboratory

Site:

Sandia National Laboratories

ID:

00-D-107

TPC:

$30.8 million

EIR Date:

September 2001

Key Findings:

  • Funding level had not been established consistent with project costs.

  • Project did not have a current and sound estimate of TEC and TPC.

Project:

Sensitive Compartmented Information Facility

Site:

Lawrence Livermore National Laboratory

ID:

00-D-800

TPC:

$25.1 million

EIR Date:

July 2001

Key Findings:

  • Project schedule was not complete.

  • A critical path analysis was not performed. Key equipment requiring long-lead procurement was not identified.

  • Review all drawings and calculations and ensure that the design criteria requirements are met.

  • Hazards analysis, including fire hazards, not complete.

Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

Project:

Bus Upgrades

Site:

Nevada Test Site

ID:

02-D-107

TPC:

$16.7 million

EIR Date:

September 2002

Key Findings:

  • Project schedule does not have sufficient detail for performance baseline.

  • Cost baseline is not in agreement with funding profile.

  • Drawings were not consistent with project baseline. For example, retention pond shown in the drawings was not in the baseline.

Project:

Office Building Replacement

Site:

Los Alamos National Laboratory

ID:

01-D-704

TPC:

$10.5 million

EIR Date:

May 2001

Key Findings:

  • Schedule does not include all significant activities and is not integrated.

  • Requirements are not clearly defined (i.e., document has conflicting statements). For example, section 15101 states that a complete chilled water system shall be provided. Performance specification section 15670 states that the chilled water system is an option based on the results of a life-cycle cost analysis.

  • Project has not been sufficiently planned to implement the federal government’s goals for efficient energy management.

OFFICE OF ENVIRONMENTAL MANAGEMENT PROJECTS

Project:

Glovebox Excavation Method Project—Pit 9

Site:

Idaho National Engineering and Environmental Laboratory

ID:

ID-ER-106

TPC:

$79.6 million

EIR Date:

July 2002

Key Findings:

  • Funding requirements by FY could not be verified due to a lack of crosswalk from estimate to schedule.

  • There was no current approved baseline schedule that reflected the current plan.

  • Some technical issues of risk had not been identified in the risk assessment, there was no crosswalk from the project risk assessment to the project action item list, and no quantitative schedule risk analysis had been done.

Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
  • System design criteria documents contained some outdated and/or incomplete information. In some specific cases, they did not reflect current project requirements for use in detailed design.

  • No systematic consideration or application of value engineering to address life-cycle costs.

Project:

Ashtabula Closure Project

Site:

Ohio

ID:

OH-AB-0030

TPC:

$43.4 million

EIR Date:

June 2003

Key Findings:

  • The development of the site’s long-term stewardship (LTS) plan is not captured in the work breakdown structure.

  • Several contract issues are not adequately addressed.

  • The risk management process is based on a qualitative analysis and provides no definitive information on the potential magnitude of the cost and schedule impacts associated with the risks. Government furnished services and items (GFS&I) deliverables and some contractor risk items were not included in the risk analysis and plan. No contingency or reserve for changes to contractor’s scope or other risks.

  • The sanitary batch reactor is scheduled for demolition in September 2003 and there is no alternative plan in place to handle sanitary wastes in support of remediation activities scheduled through May 2005.

  • There is no sitewide groundwater remediation strategy and no basis for the contractor’s commitment for a pump-and-treat remediation system.

  • There is no plan for a reduction in staffing tied to any environmental safety and health (ES&H) milestones to reflect the project winding down, nor does Human Resources appear to be intimately involved.

  • The cleanup level is well below the typical cleanup levels found at other sites. Significant cost savings/avoidance could be achieved if higher levels are accepted for this site.

  • The contractor does not have a resource-loaded integrated schedule. It uses a combination of Microframe to develop earned value, Primavera P3 for schedules, and Cost Point to collect costs. The systems do not use common milestones.

  • The total project cost is inconsistent with the funds analysis.

  • Statements have been made regarding the acceleration of the remediation schedule with added funding, but there is no supporting documentation.

  • Schedule and cost estimate were developed without an allowance for overtime.

Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

Project:

Plutonium Packaging and Stabilization (Pu P&S) Project

Site:

Savannah River Site

ID:

01-D-414/02-D-420

TPC:

$25.5 million

EIR Date:

April 2002

Key Findings:

  • The manner in which the escalation was calculated did not provide an accurate estimate of project costs.

  • Project contingency appeared to be high for this project at its stage of design development.

  • The performance criteria and requirements regarding functional capabilities were not complete.

  • The draft project execution plan (PEP) was incomplete and not current.

  • The project change control thresholds did not correspond to O 413.3.

  • Some major equipment items included in the baseline were not included in the Acquisition Strategy.

  • There was no crosswalk to assure that project risk implications of open technical issues had been addressed.

Project:

INEEL CERCLA Disposal Facility (ICDF)

Site:

Idaho National Engineering and Environmental Laboratory

ID:

ICDF

TPC:

$46.9 million

EIR Date:

March 2002

Key Findings:

  • The current budget/funding requirements by FY could not be verified.

  • The most current budget guidance for EM projects was not used in setting escalation rates for construction and life-cycle costs.

  • Schedule activities were not man-hour loaded to assure adequate duration of activities and that activities are appropriately sequenced.

  • The PEP was not final and did not conform to O 413.3.

  • The baseline change control process did not conform to O 413.3.

  • Not all risks to the project had been identified. Cost and schedule implications of identified risks had not been quantitatively evaluated. Progress of risk reduction and mitigation activities was not tracked.

  • No time frame for accomplishing risk reduction/mitigation actions was provided.

Project:

Western Sector Dynamic Underground Stripping Project

Site:

Savannah River Site

ID:

CA-1707

TPC:

$18.6 million

EIR Date:

October 2002

Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

Key Findings:

  • Several of the larger work activities had a 32 percent markup. Some project-specific factors support a higher-than-normal overhead and profit markup; this one is too high for large construction activities.

  • The well drilling and installation estimate is based on a few assumptions and a generalized work scope.

  • Escalation was not applied to a major portion of the construction estimate for FY 2003.

  • The 10 percent construction contingency appears to underestimate the risk associated with the installation of the dynamic underground stripping remediation system.

  • The PEP did not clearly identify the schedule baseline or adequately describe the technical baseline.

  • The project summary schedule does not include adequate detail.

  • The schedule of contractor activities was not adequately detailed for negotiation.

  • No signed mission-need approval letter from DOE headquarters had ever been issued.

Project:

Miamisburg Environmental Management Closure Project, Potential Release Site (PRS) 66 Proposed Remedial Action

Site:

Ohio

ID:

OH-MB

TPC:

$11.7 million

EIR Date:

March 2003

Key Findings:

  • No significant issues.

  • The scopes of the PRS 66 remedial action plans are complete and well defined. They are based upon a comprehensive characterization leading to a good definition of the location of waste and the amount of soil needed to be removed.

  • The uncertainties with the remediation effort are defined and are generally provided for in planning for the project.

Project:

INTEC Cathodic Protection System Expansion Project

Site:

Idaho National Engineering and Environmental Laboratory

ID:

02-D-402

TPC:

$6.7 million

EIR Date:

September 2001

Key Findings:

  • The Construction Project Data Sheet does not reflect the current cost estimate.

  • The intended work scope was not clearly outlined in the Title I design documents.

Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
  • The work scope definition and design criteria are not clearly documented.

  • There is no documentation of a formal design review having been conducted when the Title I design was completed.

  • The project management estimate is higher than recommended in DOE cost guidelines.

  • The durations used to calculate the escalation are not consistent with the project schedule, and the assumed average escalation of 2.9 percent is higher than DOE published rates.

  • The cost estimate and the scope of work are conservative in several areas and have a built-in contingency in several areas.

  • The support from radiation contamination technicians and the disposal of mixed and radioactive wastes are included in other project costs (OPC) but should be in the total estimated cost (TEC).

OFFICE OF SCIENCE PROJECTS

Project:

Mechanical and Control Systems Upgrade—Phase 1

Site:

Argonne National Laboratory-East

ID:

MEL-001-17

TPC:

$9.1 million

EIR Date:

September 2002

Key Findings:

  • The project does not have a fully integrated resource and cost loaded critical path method schedule.

  • The cost for risk mitigation contingency shown in the project cost estimate was not derived from the application of the risk model.

  • No commissioning plan exists to ensure that the replacement systems, equipment, and components are ultimately capable of being operated and maintained according to the laboratory’s operational needs.

  • No quality assurance plan that both assigns roles and responsibilities and defines procedures specific to this project has been developed.

  • No construction inspection and acceptance testing plan specific to the systems, equipment, and components to be replaced in this project has been developed.

Project:

Center for Nanophase Materials Sciences

Site:

Oak Ridge National Laboratory

ID:

03-R-312

TPC:

$65 million

EIR Date:

August 2002

Key Findings:

  • Escalation included in the cost estimate did not use the best process and was based on FY 2003 rather than FY 2004 guidance.

Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
  • Life-cycle costs have not been developed for the total project.

  • The current schedule has not been cost and resource loaded.

  • The current summary-level project schedule is incomplete because it does not include some major project activities—for example, Title III, project management, design support, OPC activities, and commissioning.

  • The ES&H plan at the University of Tennessee (UT)-Battelle (O&M contractor) level has not been formally adopted.

  • Analysis supporting design decisions was not well documented.

  • There is no system in place to ensure that the building systems are designed, installed, functionally tested, and capable of being operated and maintained according to UT-Battelle’s operational needs and that the building systems will meet UT-Battelle’s needs.

  • The PEP has not been finalized.

  • An overarching quality assurance plan at the UT-Battelle level, which covers both equipment for the center and construction of the center, has not been established.

  • The level of detail and specificity in the risk assessment should be increased.

  • A system to review the constructibility, buildability, and bidability of the design has not been put in place.

Project:

RUN IIB D-Zero and CDF Detectors

Site:

Fermi National Accelerator Laboratory

ID:

SC-1/2

TPC:

$59 million

EIR Date:

December 2002

Key Findings:

  • The PEP for the project is incomplete. Value engineering, quality assurance, and risk management are neither addressed nor referenced in the PEP.

  • Project-specific configuration management and control process have not been developed, as required by O 413.3. Further, no laboratory configuration management and control policy or procedure is cited in the project management plan (PMP).

  • There is no description or reference in the PEP, or in the PMP, to the flowdown of requirements and processes for quality assurance and quality control to specifics of design, fabrication, procurement, or establishment and maintenance of document approval and authenticity.

Project:

Research Support Center

Site:

Oak Ridge National Laboratory

ID:

MEL-001-25

TPC:

$16.3 million

EIR Date:

August 2002

Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×

Key Findings:

  • The Title I (CD-2) cost baseline has not been formally established, and various versions of the baseline exist throughout the project documentation, negating the usefulness of a controlled baseline estimate.

  • Costs have been shifted to other projects for some items and some functionality has been eliminated from the center. The driving force behind this effort is to keep the total estimated cost at $16.1 million.

  • The project integration estimate lacks backup support, activities are not defined, and a cost basis is not provided.

  • The estimate contains no costs for postconstruction activities such as commissioning, development of maintenance and safety plans, and management of these activities.

  • The baseline schedule does not have sufficient detail to support CD-2.

  • The level of detail for an $11 million construction subcontract is inadequate to justify the activity’s duration. No detail was available to justify further proposed reduction in the duration for this activity from 22 months to 16 months.

  • A cost-loaded schedule is not available, as required by O 413.3.

  • The baseline control and project control milestones included in the PEP use a timeline graduated in FY quarters. This timeline scale is too general for the change control thresholds proposed in PEP.

  • UT-Battelle indicated it intends to purchase standard equipment for the center as a separate item. No line item on the schedule shows these procurements or their coordination with the facility construction.

  • The schedule does not contain an identified critical path.

  • The scope baseline in PEP Section 5.1 states that the center will be a 50,000-square-foot building. The reviewers were advised that this is no longer the case, and a 53,500-square-foot building is currently planned.

  • Project documentation provided to the review team does not consistently reflect the current scope.

  • The project does not have a clearly defined integrated project team (IPT).

  • The PEP approved in May 2002 is now outdated and in need of extensive revision.

  • The risk assessment plan identifies only three risks, leaving many unaddressed—for example, coordination perils resulting from the concurrent general plant projects, state and private sector projects occurring in the same vicinity, and the likelihood of funding shortfalls. The risk mitigation strategies lack specificity and depth.

  • There is no documented quality assurance program, and the review team was informed that a documented program for the project is not intended. The organization charts and descriptions of project responsibilities do not show an individual assigned to quality assurance.

Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
  • The East Research Campus area will become the scene of increasing construction activity as simultaneous projects conducted by different parties for different owners compete for utilities, space, and resources. The PEP does not include any description of a coordination effort or organization necessary to manage this complex project area.

  • The undated acquisition execution plan (AEP) lacks specificity, includes contradictory statements, and is inconsistent with other project documents.

  • The cost and schedule baselines cannot be determined from the documentation. The documents contain no consistent “official” set of cost and schedule figures.

  • The PEP does not adequately describe a baseline change control board function.

  • The configuration management process described in the PEP does not sufficiently describe the process used for configuration management, the coordination requirements with others, or the level of effort planned for this function.

  • The project cost estimates include wildly fluctuating contingency amounts as the project team endeavors to maintain the TEC.

Project:

Laboratory Systems Upgrades

Site:

Pacific Northwest National Laboratory

ID:

MEL-001-18

TEC:

$9.4 million

EIR Date:

May 2002

Key Findings:

  • The install and test control system costs are higher than normal, and Johnson Controls, Incorporated, the contractor designing the system and making preliminary cost estimates, is being considered for a sole-source procurement.

  • The productivity rates assumed for a number of construction activities appear overly conservative.

  • The contingency level in the project team’s estimate is higher than the level recommended in DOE guidance and does not appear justified.

  • The project team has not sufficiently defined milestones included in the schedule baseline to manage the project in accordance with the project baseline change control thresholds and DOE guidance.

  • Several key areas of the preliminary design were missing outline specifications, equipment specifications, general notes, and keyed notes.

  • The documentation did not meet the minimum requirements for receiving CD-2 approval.

  • There is no risk management plan. The Contingency and Risks section of the PEP does not sufficiently identify and quantify risks or define mitigation strategies.

Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 99
Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 100
Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 101
Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 102
Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 103
Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 104
Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 105
Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 106
Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 107
Suggested Citation:"Appendix F: Summary of External Independent Reviews for National Nuclear Security Administration, Office of Environmental Management, and Office of Science Projects, FY 2001 Through FY 2003, Covering Only Baseline Validation." National Research Council. 2004. Progress in Improving Project Management at the Department of Energy: 2003 Assessment. Washington, DC: The National Academies Press. doi: 10.17226/10931.
×
Page 108
Next: Appendix G: Correspondence Between Dr. Orbach and Dr. Alberts Regarding the Committee's 2002 Assessment Report »
Progress in Improving Project Management at the Department of Energy: 2003 Assessment Get This Book
×
Buy Paperback | $45.00 Buy Ebook | $35.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In 1997, Congress, in the conference report, H.R. 105-271, to the FY1998 Energy and Water Development Appropriation Bill, directed the NRC to carry out a series of assessments of project management at the Department of Energy (DOE). This report, the 2003 Assessment, is the final one in that series. It presents an examination of DOE's progress in improving program management over the past three years including the Department's response to the recommendations of the previous assessments in this series. In addition to assessing DOE’s progress, the report also describes opportunities for further improvement and gives a prognosis for future developments.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!