NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This report has been reviewed by a group other than the authors according to procedures approved by a Report Review Committee consisting of members of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine.
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Robert M. White is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Robert M. White are chairman and vice chairman, respectively, of the National Research Council.
The work of the Board on Testing and Assessment is supported by the U.S. Department of Defense, Education, and Labor, through a grant administered by the Employment and Training Administration of the U.S. Department of Labor. The appendix reprinted in this report is not the work of the board.
Additional copies of this report are available from:
Board on Testing and Assessment
National Research Council
2101 Constitution Avenue N.W.
Washington, D.C. 20418
Printed in the United States of America
Copyright 1995 by the National Academy of Sciences. All rights reserved.
BOARD ON TESTING AND ASSESSMENT
Richard C. Atkinson (Chair),
University of California, San Diego
Constance B. Newman (Vice Chair),
Smithsonian Institution, Washington, D.C.
Richard J. Shavelson (Vice Chair),
School of Education, Stanford University
Laurie J. Bassi,
Graduate Public Policy Program, Georgetown University
David C. Berliner,
College of Education, Arizona State University, Tempe
Richard F. Elmore,
Graduate School of Education, Harvard University
Patricia M. Flynn,
Graduate School of Business, Bentley College
Edmund W. Gordon,
Department of Psychology, City University of New York
Sylvia T. Johnson,
School of Education, Howard University
Brigitte Jordan,
Xerox Palo Alto Research Center and Institute for Research on Learning, Palo Alto, Calif.
Carl F. Kaestle,
Department of Education, University of Chicago
Michael W. Kirst,
School of Education, Stanford University
Luis M. Laosa,
Educational Testing Service, Princeton, N.J.
Renee S. Lerche,
Ford Motor Company, Dearborn, Mich.
Alan M. Lesgold,
Learning Research and Development Center, University of Pittsburgh
Robert L. Linn,
School of Education, University of Colorado, Boulder
Miles A. Myers,
National Council of Teachers of English, Urbana, Ill.
James L. Outtz,
Outtz and Associates, Washington, D.C.
Neal W. Schmitt,
Department of Psychology, Michigan State University, East Lansing
Alan H. Schoenfeld,
School of Education, University of California, Berkeley
William L. Taylor,
Attorney At Law, Washington, D.C.
Ewart A.C. Thomas,
Department of Psychology, Stanford University
Michael J. Feuer, Director
Holly Wells, Administrative Assistant
Adrienne Crawford, Administrative Assistant
FOREWORD
In the spring of 1994, when the Board on Testing and Assessment was planning a special program to mark its first anniversary, it sought the wisdom and counsel of a master. The choice was simple. Lee J. Cronbach, whose extraordinary career in psychological measurement has yielded some of the most important and original contributions to education, statistics, and the assessment of human performance, could provide a unique perspective on the pitfalls of testing. Lee graciously accepted an invitation to attend the board's first annual retreat, held in Monterey, California, and to offer reflections from his “60 years in the testing wars.”
Part reminiscence, part cautionary tale, his account ranges over many of the problems associated with the use of educational tests in this country during the past three decades. Of particular interest to those concerned with the vision and ideals of school reform embodied in Goals 2000 and other federal legislation, Lee recounts some of his experiences in the past year with California's effort to institute standardized assessments in its elementary and junior high schools. In 1993, the California Learning Assessment System—known as CLAS—administered an examination of language arts and mathematics to nearly all California students in Grades 4, 8, and 10, using performance measures of achievement rather than relying solely on more traditional multiple-choice questions. Reports on the performance of schools and school districts raised public alarm over the condition of education in the state; the reporting of results for schools, sometimes based on small samples of pupils, caused concern over the management of the assessment system. The state superintendent of public instruction needed the best possible scientific advice, and so asked Lee to chair a small committee (with Norman Bradburn and Daniel Horvitz) to review the CLAS testing system and to comment on the methodology slated for implementation in 1994.
The committee's report to the state of California, which applauds the objectives and accomplishments of CLAS while also identifying certain fundamental technical problems, was released shortly after Lee's visit with the board. The CLAS report is reproduced here as an appendix to Lee' s paper (with the kind permission of Dale Carlson, the director of CLAS), because the board felt that the pioneering experiences of California offer compelling lessons for the design and use of testing as a tool of educational assessment and school reform. The importance of these lessons is
implicit in the committee's encouraging conclusion that “all the shortcomings of CLAS-1993 can be remedied ....”
In preparing his paper and the CLAS report for this publication, Lee has added some additional ideas he developed after the committee 's report was submitted, which appear following the body of the report. Indeed, his final words suggest that he has many more words ahead —Lee may again find it hard to slip into retirement completely—happy news to students and practitioners of education policy and to all who have come to rely on Lee's impeccable scholarship and good sense.
Richard C. Atkinson, Chair
Board on Testing and Assessment
