Page i

Technology and Assessment: Thinking Ahead

Proceedings from a Workshop

Board on Testing and Assessment

Center for Education

Division of Behavioral and Social Sciences and Education

National Research Council

Washington, DC 2002



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
Page i Technology and Assessment: Thinking Ahead Proceedings from a Workshop Board on Testing and Assessment Center for Education Division of Behavioral and Social Sciences and Education National Research Council Washington, DC 2002

OCR for page R1
Page ii NATIONAL ACADEMY PRESS 2101 Constitution Avenue, N.W. Washington, DC 20418 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This study was supported by Contract/Grant No.# 2001-6884 between the National Academy of Sciences and the Hewlett Foundation. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project. Library of Congress Cataloging-in-Publication Data or International Standard Book Number 0-309-08320-6 Suggested citation: National Research Council. (2002). Technology and assessment: Thinking ahead: Proceedings of a workshop. Board on Testing and Assessment, Center for Education, Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press. Additional copies of this report are available from National Academy Press , 2101 Constitution Avenue, N.W. , Lockbox 285, Washington, D.C. 20055 ; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu Printed in the United States of America Copyright 2002 by the National Academy of Sciences . All rights reserved.

OCR for page R1
Page iii THE NATIONAL ACADEMIES National Academy of Sciences National Academy of Engineering Institute of Medicine National Research Council The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Wm. A. Wulf are chairman and vice chairman, respectively, of the National Research Council.

OCR for page R1
Page iv

OCR for page R1
Page v Preface The papers in this collection were commissioned by the Board on Testing and Assessment (BOTA) of the National Research Council (NRC) for a workshop held on November 14, 2001, with support from the William and Flora Hewlett Foundation. Goals for the workshop were twofold. One was to share the major messages of the recently released NRC committee report, Knowing What Students Know: The Science and Design of Educational Assessment (2001), which synthesizes advances in the cognitive sciences and methods of measurement, and considers their implications for improving educational assessment. The second goal was to delve more deeply into one of the major themes of that report—the role that technology could play in bringing those advances together, which is the focus of these papers. For the workshop, selected researchers working in the intersection of technology and assessment were asked to write about some of the challenges and opportunities for more fully capitalizing on the power of information technologies to improve assessment, to illustrate those issues with examples from their own research, and to identify priorities for research and development in this area. BACKGROUND In recent years, BOTA has explored pressing and complex issues in educational assessment, including the role and the appropriate uses of assessment in standards-based reform; how well current assessments are fulfilling the various demands placed on them; and concerns about fairness and equity in testing. In 1998, BOTA decided the time was right to address a long-standing issue noted by numerous researchers interested in problems of educational assessment: the need to bring together scientific understanding of how people learn with methods for assessing what they have learned. An NRC committee was formed to review advances in the cognitive and measurement sciences, as well as initial, promising work done in the intersection between the two disciplines, and to consider the implications for reshaping educational assessment. The National Science Foundation (NSF) recognized the importance and timeliness of such a study and agreed to sponsor the effort. The resulting committee report, Knowing What Students Know: The Science and Design of Educational Assessment, was released in 2001. The underlying premise of the report is that new forms of classroom and large-scale assessments are needed that help all students learn and succeed in school, by making as clear as possible to them, their teachers, and other education stakeholders the nature of their accomplishments and the progress of their learning. Advances in the cognitive sciences have broadened the conception of those aspects of learning that are most important to assess, and advances in measurement have expanded the capability to interpret more

OCR for page R1
Page vi complex forms of evidence derived from student performance. A merger of these two sets of advances could lead to a significant leap forward in the science and practice of assessment, and technology is playing an important role in making such a merger feasible. INFORMATION TECHNOLOGIES: OPPORTUNITIES FOR ADVANCING EDUCATIONAL ASSESSMENT Some of the main conclusions from Knowing What Students Know about the nature of assessment and the role of technology are summarized below. They provide a set of framing ideas for the papers that follow. Assessments are used in both classroom and large-scale contexts for three broad purposes: to assist learning, to measure individual achievement, and to evaluate programs. One type of assessment does not fit all purposes, and assessments used in various contexts often look quite different. But every assessment, regardless of its purpose, rests on three pillars: (1) a model of how students represent knowledge and develop competence in the subject domain; (2) tasks or situations that allow one to observe students' performance; and (3) an interpretation method for drawing inferences from the performance evidence thus obtained. These three elements—cognition, observation, and interpretation—must be explicitly connected and designed as a coordinated whole. If the three elements are not in synchrony, the meaningfulness of inferences drawn from the assessment will be compromised. The three elements and the connections among them are referred to here as the assessment triangle. Improved methods of assessment require a design process that connects the three elements of the assessment triangle to ensure that the theory of cognition, the observations, and the interpretation process work together to support the intended inferences. Fortunately, there are multiple examples of technology tools and applications that enhance specific linkages among cognition, observation, and interpretation, as well as more general linkages among curriculum, instruction, and assessment. A few of these enhancements are mentioned here as a frame of reference for the examples described in the workshop papers. Among the most intriguing applications of technology are those that extend the nature of the problems that can be presented and the knowledge and cognitive processes that can be assessed. When task environments are enriched through the use of multimedia, interactivity, and control over the stimulus display, it is possible to assess a much wider array of cognitive competencies than has heretofore been feasible. New capabilities enabled by technology include directly assessing problem-solving skills, making visible sequences of actions taken by learners to solve problems, and modeling and simulating complex reasoning tasks. Technology also makes possible the collection of data on concept organization and other aspects of students' knowledge structures, as well as representations of their participation in discussions and group projects. A significant contribution of technology has been to the design of systems for implementing sophisticated classroom-based formative assessment practices. Technology-based systems have been developed to support individualized instruction by extracting key features of learners' responses, analyzing patterns of correct and incorrect reasoning, and providing rapid

OCR for page R1
Page vii and informative feedback to both student and teacher. Often such approaches are embedded in complex teaching-learning environments supported by technology, as noted below. A major change in education has resulted from the influence of technology on what is taught and how it is taught. Schools are placing more emphasis on teaching critical content in greater depth. Examples include the teaching of advanced thinking and reasoning skills within a discipline through the use of technology-mediated projects that involve long-term inquiry. Such projects often integrate content and learning across disciplines, as well as integrating assessment with curriculum and instruction in powerful ways. A possibility for the future arises from the projected growth across curricular areas of technology-based assessment embedded in instructional settings. Increased availability of such systems could make it possible to pursue balanced designs representing a more coordinated and coherent assessment system. Information from such assessments could possibly be used for multiple purposes, including the audit function associated with many existing external assessments. Finally, while technology holds great promise for enhancing educational assessment at multiple levels of practice, its use for this purpose also raises issues of utility, practicality, cost, equity, and privacy. These issues will need to be addressed as technology applications in education and assessment continue to expand, evolve, and converge. PAPERS IN THIS VOLUME The papers that follow address some of the opportunities and challenges just noted and provide specific examples of the linkages mentioned above. The first two papers focus on the role of technology in improving assessments used for summative or external evaluation purposes. In the first paper, Gitomer and Bennett illustrate how computer technologies are being used by researchers at the Educational Testing Service to address a long-standing criticism of standardized tests: that they tend to consist of certain types of traditional test items and have lost sight of the underlying constructs, or cognitive competencies, that are the targets of assessment. The authors provide several examples of the use of computer technologies to “unmask” the constructs underlying traditional assessments, such as the PSAT, and to make the constructs more visible and explicit in the design of new assessments. In the second paper, Means and Haertel describe an effort to develop computer-based, quality assessments of scientific inquiry. Despite the visibility of inquiry skills as some of the most highly emphasized skills in science curriculum standards, they are the least likely to be adequately assessed in large-scale accountability systems. In contrast, technology has been widely used over the last decade to develop simulations and complex learning environments that support inquiry processes and the assessment of inquiry skills. Many such environments include embedded assessments that are closely tied to the learning activities. Means and Haertel describe efforts to capitalize on such work and take the next steps toward developing science assessments with broader applicability that can be disentangled from specific instructional contexts and used across various curricula for program evaluation purposes.

OCR for page R1
Page viii The focus of the next three papers shifts to uses of technology to improve classroom assessment that is aimed at monitoring students' understanding and guiding the next steps for instruction. Fletcher's paper reviews evidence of substantial learning gains when instruction is tailored to the needs and capabilities of individual learners, as in one-on-one tutoring situations. While tutoring is the most effective form of instruction, it is also the most expensive. Computer technologies, such as intelligent tutoring systems, are making it feasible to provide some of the advantages of human tutoring on a more widespread basis. Fletcher reviews a variety of examples of computerized instructional programs from the military and other arenas that have assessment components to tailor the pace, content, difficulty, and sequencing of instructional material to the needs of individual learners. In the fourth paper, Williams describes in more depth an example of such a system in the area of early, basic reading skills. Finding time to provide individual feedback during children's reading practice is difficult for teachers who often have 20 or more students in their classes. But recent developments in speech recognition technology are making it possible to increase opportunities for individual reading practice with feedback, as well as to collect assessment information for instructional decision making. In the fifth paper, Corbett provides an example of an intelligent tutoring system in the more complex content domain of algebra. Based on a cognitive model of how students learn algebra, the intelligent tutoring system provides students with rich problem-solving environments and adaptive student support. In the sixth paper, Russell raises ideas about how technologies might not only make assessment more efficient but also more fundamentally “disrupt” current assessment practices. For instance, testing experts have argued for a long time that a single test should be applied to meet a single purpose, but technology may make it feasible for a single source of data to meet multiple purposes. Russell describes several examples of computer-based, complex learning systems that have the potential to capture rich information about student learning and performance during the actual learning process. He argues that information collected in this way could be used for formative assessment purposes to guide next steps for instruction; it could be accumulated and mined for summative assessment purposes, such as program evaluation, and eventually eliminate or reduce the need for separate on-demand, external exams. One of the challenges emphasized in several of the papers is that the development of assessments based on advances in cognitive theory, measurement, and technology is a difficult and time-consuming task; it will be a huge undertaking to develop the kinds of assessments discussed in these papers for multiple areas of the curriculum. In the seventh paper, Baker describes the substantial research and development effort that is needed to make test design more systematic and efficient through the use of technology. This final paper lays out a set of goals and components for automated authoring systems for tests and describes some early work in this area. ACKNOWLEDGMENTS The Board on Testing and Assessment is grateful to Marshall (Mike) Smith of the William and Flora Hewlett Foundation for sponsoring this workshop. Special thanks also go to the steering group that helped plan and facilitate this activity: Jim Pellegrino, University of Illinois, Chicago; Eva Baker, Center for the Study of Evaluation, University of California, Los

OCR for page R1
Page ix Angeles; George Madaus, Boston College; Lauren Resnick, University of Pittsburgh; and Lorrie Shepard, University of Colorado, Boulder. Thanks also to all of the workshop presenters, who contributed to the success of the workshop: Rich Lehrer, University of Wisconsin, Madison; Mark Wilson, University of California, Berkeley; Bob Glaser, University of Pittsburgh; Larry Suter, National Science Foundation; Jose Mestre, University of Massachusetts; Albert Corbett, Carnegie Mellon University; Drew Gitomer and Paul Holland, Educational Testing Service; Mike Russell, Boston College; Susan Williams, University of Texas, Austin; Barbara Means and Geneva Haertel, SRI International; and Dexter Fletcher, Institute for Defense Analyses. The papers included in this collection were written by a subset of these participants, and we greatly appreciate the authors' additional time and effort. Several National Research Council (NRC) staff helped plan the workshop and produce this collection of papers, including Naomi Chudowsky, who directed the project, and Andrew Tompkins, who provided excellent administrative help. Stuart Elliott served as a consultant; Pasquale DeVito, Director of BOTA, and Michael Feuer, Director of the Center for Education, provided guidance to the project. These papers have been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC's Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making this volume as sound as possible and to ensure that the publication meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of these papers: Alan Lesgold, University of Pittsburgh; and Jim Pellegrino, University of Illinois, Chicago. Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations nor did they see the final draft of the papers before their release. The review of this publication was overseen by Milt Hakel, Bowling Green State University. Appointed by the National Research Council, he was responsible for making certain that an independent examination of the papers was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of the papers rests entirely with the individual authors.

OCR for page R1
Page x

OCR for page R1
Page xi Table of Contents 1     Unmasking Constructs Through New Technology, Measurement Theory, and Cognitive Science Drew H. Gitomer and Randy Elliot Bennett 1 2     Technology Supports for Assessing Science Inquiry Barbara Means and Geneva Haertel 12 3     Is It Worth It? Some Comments on Research and Technology in Assessment and Instruction J.D. Fletcher 26 4     Speech Recognition Technology and the Assessment of Beginning Readers Susan M. Williams 40 5     Cognitive Tutor Algebra I: Adaptive Student Modeling in Widespread Classroom Use Albert Corbett 50 6     How Computer-Based Technology Can Disrupt the Technology of Testing and Assessment Michael Russell 63 7     Design of Automated Authoring S ystems for Tests Eva L. Baker 79 Appendix A:     Workshop Agenda 90 Appendix B:     Members of the Board on Testing and Assessment 92

OCR for page R1
Page xii