National Academies Press: OpenBook
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×

Evaluation of the Achievement Levels
for Mathematics and Reading on the
National Assessment of Educational Progress

Committee on the Evaluation of NAEP Achievement Levels for
Mathematics and Reading

Christopher Edley, Jr., and Judith A. Koenig, Editors

Board on Testing and Assessment
and
Committee on National Statistics

Division of Behavioral and Social Sciences and Education

images

THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×

THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001

This activity was supported by Contract No. ED-IES-14-C-0124 from the U.S. Department of Education. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.

International Standard Book Number-13: 978-0-309-43817-9
International Standard Book Number-10: 0-309-43817-9
Digital Object Identifier: 10.17226/23409

Additional copies of this publication are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.

Copyright 2017 by the National Academy of Sciences. All rights reserved.

Printed in the United States of America

Suggested citation: National Academies of Sciences, Engineering, and Medicine. (2017). Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×

images

The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.

The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. Mote, Jr., is president.

The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.

The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.

Learn more about the National Academies of Sciences, Engineering, and Medicine at www.national-academies.org.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×

images

Reports document the evidence-based consensus of an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and committee deliberations. Reports are peer reviewed and are approved by the National Academies of Sciences, Engineering, and Medicine.

Proceedings chronicle the presentations and discussions at a workshop, symposium, or other convening event. The statements and opinions contained in proceedings are those of the participants and have not been endorsed by other participants, the planning committee, or the National Academies of Sciences, Engineering, and Medicine.

For information about other products and activities of the National Academies, please visit nationalacademies.org/whatwedo.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×

COMMITTEE ON THE EVALUATION OF NAEP ACHIEVEMENT LEVELS

CHRISTOPHER EDLEY, JR. (Chair), School of Law, University of California, Berkeley School of Law

PETER AFFLERBACH, Department of Teaching and Learning, Policy and Leadership, University of Maryland

SYBILLA BECKMANN, Department of Mathematics, University of Georgia

H. RUSSELL BERNARD, Institute for Social Science Research, Arizona State University, and Department of Anthropology, University of Florida

KARLA EGAN, National Center for the Improvement of Educational Assessment, Dover, NH

DAVID J. FRANCIS, Department of Psychology, University of Houston

MARGARET E. GOERTZ, Graduate School of Education, University of Pennsylvania (emerita)

LAURA HAMILTON, Education Program, RAND Corporation, Pittsburgh, PA

BRIAN JUNKER, Department of Statistics Carnegie Mellon University

SUZANNE LANE, School of Education, University of Pittsburgh

SHARON J. LEWIS, Council of the Great City Schools, Washington, DC (retired)

BERNARD L. MADISON, Department of Mathematics, University of Arkansas

SCOTT NORTON, Standards, Assessment, and Accountability, Council of Chief State School Officers, Washington, DC

SHARON VAUGHN, College of Education, University of Texas at Austin

LAURESS WISE, Human Resources Research Organization, Monterey, CA

JUDITH KOENIG, Study Director

JORDYN WHITE, Program Officer

NATALIE NIELSEN, Acting Board Director (until June 2015)

PATTY MORISON, Acting Board Director (from June 2015)

KELLY ARRINGTON, Senior Program Assistant

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×

BOARD ON TESTING AND ASSESSMENT

DAVID J. FRANCIS (Chair), Texas Institute for Measurement, Evaluation, and Statistics, University of Houston

MARK DYNARSKI, Pemberton Research, LLC, East Windsor, NJ

JOAN HERMAN, National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles

SHARON LEWIS, Council of Great City Schools, Washington, DC

BRIAN STECHER, Education Program, RAND Corporation, Santa Monica, CA

JOHN ROBERT WARREN, Department of Sociology, University of Minnesota

NATALIE NIELSEN, Acting Director (until June 2015)

PATTY MORISON, Acting Director (from June 2015)

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×

COMMITTEE ON NATIONAL STATISTICS

LAWRENCE D. BROWN (Chair), Department of Statistics, Wharton School, University of Pennsylvania

JOHN M. ABOWD, School of Industrial and Labor Relations, Cornell University

FRANCINE BLAU, Department of Economics, Cornell University

MARY ELLEN BOCK, Department of Statistics, Purdue University

MICHAEL E. CHERNEW, Department of Health Care Policy, Harvard Medical School

DON A. DILLMAN, Department of Sociology, Washington State University

CONSTANTINE GATSONIS, Center for Statistical Sciences, Brown University

JAMES S. HOUSE, Survey Research Center, Institute for Social Research, University of Michigan

MICHAEL HOUT, Survey Research Center, University of California, Berkeley

THOMAS L. MESENBOURG, U.S. Census Bureau (retired)

SUSAN A. MURPHY, Department of Statistics, University of Michigan

SARAH M. NUSSER, Department of Statistics, Center for Survey Statistics and Methodology, Iowa State University

COLM A. O’MUIRCHEARTAIGH, Harris Graduate School of Public Policy Studies, University of Chicago

RUTH D. PETERSON, Criminal Justice Research Center, Ohio State University

ROBERTO RIGOBON, Sloan School of Management, Massachusetts Institute of Technology

EDWARD H. SHORTLIFFE, Biomedical Informatics, Columbia University and Arizona State University

CONSTANCE F. CITRO, Director

BRIAN HARRIS-KOJETIN, Deputy Director

Page viii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×

This page intentionally left blank.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×

Preface

Since 1969 the National Assessment of Educational Progress (NAEP) has been providing policy makers, educators, and the public with reports on the academic performance and progress of the nation’s students. Over the years, NAEP has been updated in ways that kept it current with the ever-changing educational and policy context, but maintained the trend lines it was designed to track. By the late 1980s, there were concerns about the extent to which U.S. students were learning what they need to be globally competitive in the 21st century. There were calls to set high, world-class achievement goals for U.S. students. In response, NAEP adopted standards-based reporting for the subjects and grades it assessed.

Today, some 24 years later, the focus on achievement standards is even more intense. At its best, standards-based reporting can provide a quick way to summarize students’ achievement and track their progress. It can clearly demark disparities between what we expect our students to know and be able to do as articulated in the standards—and what they actually know and can do. It can stimulate policy conversations about educational achievement across the country, identifying areas and groups with high performance as well as those with troubling disparities. It can inform policy interventions and reform measures to improve student learning.

There are potential downsides, however. Standards-based reporting can lead to erroneous interpretations. It can overstate or understate progress, particularly when the goal is to monitor the performance of subgroups. In its attempt to be easily understood by all audiences, it can lead to over-simplifications, such as when users do not do the necessary

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×

work to ensure their understandings are correct. At a time when policy makers are vitally interested in ensuring that all of the country’s students achieve high standards, it is critical that test results are reported in a way that leads to accurate and valid interpretations, and that the standards used in reporting deserve the confidence of policy makers, practitioners, and the public.

The U.S. Department of Education turned to the National Academies of Sciences, Engineering, and Medicine to answer these questions, and the National Academies established the Committee on the Evaluation of NAEP Achievement Levels for Mathematics and Reading. The purpose of the project was to evaluate to what extent NAEP’s standards (or achievement levels) are reliable and valid. Are they reasonable? Are they informative to the public? Do they lead to appropriate interpretations? The committee of 15 brought a broad range of education and assessment experience. Our consensus findings, conclusions, and recommendations are documented in this report.

The committee benefited from the work of many others, and we wish to thank the many individuals who assisted us. We first thank the sponsor who supported this work: the U.S. Department of Education and the staff with the department’s Institute for Education Sciences who oversaw our work, Jonathon Jacobson and Audrey Pendleton.

During the course of its work, the committee met three times in person and four times by video conference. The committee’s first meeting included a public session designed to learn more about the sponsor’s goals for the project and the history of achievement-level setting for NAEP. We heard from officials from the National Assessment Governing Board (NAGB) and the National Center for Education Statistics (NCES). We especially thank Cornelia Orr, former executive director, and Sharyn Rosenberg, assistant director for psychometrics with NAGB, and Peggy Carr, acting commissioner of NCES, for the vast amount of historical information they provided.

The committee’s second meeting included a public forum designed to provide an opportunity for committee members to hear first-hand accounts of the variety of ways NAEP users interpret and use the achievement levels. These discussions were enormously enlightening to the committee, and we thank all who participated, including Patte Barth, Sonja Brookins Santelises, Sarah Butrymowicz, Michael Casserly, Enis Dogan, Wendy Geiger, Catherine Gewertz, Renee Jackson, Scott Jenkins, Mike Kane, Jacqueline King, Lyndsey Layton, Nathan Olson, Emily Richmond, Bob Rothman, Lorrie Shepard, and Dara Zeehandlaar. The agenda in Appendix A provides their titles and affiliations.

The committee gratefully acknowledges the dedicated effort provided by the staff of the Board on Testing and Assessment (BOTA) and the

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×

Committee on National Statistics (CNSTAT) who worked directly on this project. We first thank Natalie Nielsen, former acting director of BOTA, for her extensive efforts to make this project a reality and her dedication to ensuring its success. We thank Patty Morison, current acting director of BOTA, for her support and guidance at key stages in this project. We are grateful to Connie Citro, director of CNSTAT, and Robert Hauser, executive director of the Division of Behavioral and Social Sciences and Education (DBASSE), for their sage advice throughout the course of this project. We thank Kelly Arrington, senior program assistant, for her exceptional organizational skills and her close attention to detail. Kelly handled all of the administrative details associated with the in-person and virtual meetings and the public forum, and she provided critical support in preparing the manuscript. We are grateful to Jordyn White, program officer with CNSTAT, for her adept research skills and her expertise in assimilating data and designing understandable presentations. We especially thank Judy Koenig for her intellectual and organizational skills as the study director and for her work in assembling the critical information needed to prepare this report. Judy worked tirelessly to keep us on task and to ensure that we met the multiple challenges presented by our charge.

We also thank members of the Office of Reports and Communication of DBASSE for their dedicated work on this report. We are indebted to Eugenia Grohman for her sage editorial advice on numerous versions of this manuscript. We thank Kirsten Sampson-Snyder for her work in coordinating a very intense review process and Yvonne Wise for shepherding the manuscript through production.

This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the Report Review Committee of the National Academies. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.

We thank the following individuals for their review of this report: Lisa M. Abrams, Department of Foundations of Education, School of Education, Virginia Commonwealth University; Brian E. Clauser, Educational Assessment, National Board of Medical Examiners; Sarah W. Freedman, Graduate School of Education, University of California, Berkeley; Roger Howe, Department of Mathematics, Yale University; Nonie K. Lesaux, Graduate School of Education, Harvard University; Joseph A. Martineau, Center for Assessment, Dover, NH; William Penuel, School of Education, University of Colorado Boulder; Barbara Plake, Boros Center for Testing

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×

(director emeritus), University of Nebraska–Lincoln; Mark D. Reckase, Department of Counseling, Educational Psychology and Special Education, Michigan State University; Lorrie A. Shepard, School of Education, University of Colorado Boulder; and Dale Whittington, Research and Accountability, Shaker Heights City School District, Shaker Heights, OH.

Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations nor did they see the final draft of the report before its release. The review of this report was overseen by Edward H. Haertel, School of Education, Stanford University; and Ronald S. Brookmeyer, Department of Biostatistics, University of California, Los Angeles. Appointed by the National Academies, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.

As chair of the committee, I thank the extraordinary efforts of my fellow panel members, who had a broad range of expertise related to assessment, education policy, equity in education, mathematics and reading, program evaluation, social science, and statistics. All of this expertise was critical to the multifaceted issues that had to be addressed in this evaluation. The panel members freely contributed their time to accomplishing the myriad of tasks associated with assembling information and preparing this report. They actively assisted in all stages of this project, including planning the meetings and the public forum, as well as writing and rewriting multiple versions of this report. They gave generously of their time to ensure that the final product accurately represents our consensus findings, conclusions, and recommendations. Their contributions during the period in which the report was in final preparation and after the external review, when sections of the report had to be turned around on a very truncated schedule, are especially appreciated. These efforts manifested the panel members’ deep dedication to improving student learning across the country.

Christopher Edley, Jr., Chair
Committee on the Evaluation of NAEP
Achievement Levels for Mathematics and Reading

Page xiii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×

Dedication

During the course of this project, the measurement community lost a towering figure. Robert Linn, distinguished professor emeritus of education with the University of Colorado Boulder, worked at the intersection of education measurement and education policy. He was known for his enormous contributions to test equating, fairness, validity, performance assessment, and educational accountability, and he was widely respected for his ability to translate theory into practical advice. His contributions to the National Assessment of Educational Progress (NAEP) spanned several decades, and he continued to serve on the NAEP Validity Studies Panel until his final months. His contributions to the National Academies of Sciences, Engineering, and Medicine were numerous, serving as a board member and chair of the Board on Testing and Assessment, and as a readily sought-after committee member and reviewer for many projects. A prodigious scholar and remarkable intellectual leader, Bob was also unceasingly patient and supportive as teacher, colleague, and friend. We are indebted to his contributions and will miss him. We dedicate this report to his memory.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×

This page intentionally left blank.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×
Page R1
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×
Page R2
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×
Page R3
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×
Page R4
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×
Page R5
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×
Page R6
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×
Page R7
Page viii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×
Page R8
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×
Page R9
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×
Page R10
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×
Page R11
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×
Page R12
Page xiii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×
Page R13
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×
Page R14
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×
Page R15
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
×
Page R16
Next: Summary »
Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress Get This Book
×
Buy Paperback | $69.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Since 1969, the National Assessment of Educational Progress (NAEP) has been providing policymakers, educators, and the public with reports on academic performance and progress of the nation's students. The assessment is given periodically in a variety of subjects: mathematics, reading, writing, science, the arts, civics, economics, geography, U.S. history, and technology and engineering literacy. NAEP is given to representative samples of students across the U.S. to assess the educational progress of the nation as a whole.

Since 1992, NAEP results have been reported in relation to three achievement levels: basic, proficient, and advanced. However, the use of achievement levels has provoked controversy and disagreement, and evaluators have identified numerous concerns. This publication evaluates the NAEP student achievement levels in reading and mathematics in grades 4, 8, and 12 to determine whether the achievement levels are reasonable, reliable, valid, and informative to the public, and recommends ways that the setting and use of achievement levels can be improved.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!