Page i

Science, Evidence, and Inference in Education

Report of a Workshop

Lisa Towne, Richard J. Shavelson, and Michael J. Feuer, Editors

Committee on Scientific Principles in Education Research

Center for Education

Division of Behavioral and Social Sciences and Education

National Research Council

National Academy Press
Washington, D.C.
May 2001



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
Page i Science, Evidence, and Inference in Education Report of a Workshop Lisa Towne, Richard J. Shavelson, and Michael J. Feuer, Editors Committee on Scientific Principles in Education Research Center for Education Division of Behavioral and Social Sciences and Education National Research Council National Academy Press Washington, D.C.May 2001

OCR for page R1
Page ii NATIONAL ACADEMY PRESS 2101 Constitution Avenue, N.W. Washington, D.C. 20418 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This project has been funded at least in part with Federal funds from the U.S. Department of Education under contract number ED-00-CO-0088. The content of this publication does not necessarily reflect the views or policies of the U.S. Department of Education nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government. International Standard Book Number 0-309-07570-X Additional copies of this report are available from National Academy Press , 2101 Constitution Avenue, N.W. , Lockbox 285, Washington, D.C. 20418 Call (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area) This report is also available online at http://www.nap.edu Copyright 2001 by the National Academy of Sciences . All rights reserved. Suggested citation: National Research Council. (2001). Science, evidence, and inference in education: Report of a workshop. Committee on Scientific Principles in Education Research. L. Towne, R. J. Shavelson, and M. J. Feuer (Eds.). Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press. Printed in the United States of America

OCR for page R1
Page iii THE NATIONAL ACADEMIES National Academy of Sciences National Academy of Engineering Institute of Medicine National Research Council The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. William A. Wulf is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. William A. Wulf are chairman and vice chairman, respectively, of the National Research Council.

OCR for page R1
Page iv COMMITTEE ON SCIENTIFIC PRINCIPLES IN EDUCATION RESEARCH RICHARD J. SHAVELSON (Chair), Center for Advanced Study in the Behavioral Sciences and Stanford University School of Education DONALD I. BARFIELD, WestEd, San Francisco, California ROBERT F. BORUCH, Graduate School of Education and Statistics, University of Pennsylvania, Philadelphia JERE CONFREY, Department of Curriculum and Instruction, University of Texas, Austin RUDOLPH F. CREW, Institute for K-12 Leadership, University of Washington, Seattle ROBERT L. DeHAAN, Department of Cell Biology, Emory University, Atlanta, Georgia MARGARET EISENHART, School of Education, University of Colorado, Boulder JACK McFARLIN FLETCHER, Department of Pediatrics, University of Texas, Houston EUGENE E. GARCIA, Graduate School of Education, University of California, Berkeley NORMAN HACKERMAN, The Welch Foundation, Houston, Texas ERIC HANUSHEK, Hoover Institution, Stanford University ROBERT M. HAUSER, Center for Demography of Health and Aging, University of Wisconsin, Madison PAUL W. HOLLAND, Educational Testing Service, Princeton, New Jersey ELLEN CONDLIFFE LAGEMANN, Spencer Foundation and New York University DENIS C. PHILLIPS, Stanford University School of Education CAROL H. WEISS, Graduate School of Education, Harvard University LISA TOWNE, Director TINA WINTERS, Research Assistant LINDA DePUGH, Senior Project Assistant

OCR for page R1
Page v Acknowledgments The Committee on Scientific Principles in Education Research and the National Research Council's (NRC) Center for Education (CFE) are grateful to the many individuals whose efforts made possible this report, and the workshop it summarizes. The committee's work is supported by a contract with the U.S. Department of Education's National Educational Research Policy and Priorities Board. The guidance offered by Kenji Hakuta, Alba Ortiz, Rafael Valdivieso, Thelma Leenhouts, and Mary Grace Lucier is very much appreciated. We also wish to thank the workshop speakers, whose remarks stimulated a rich and wide-ranging discussion: Michael Agar, Norman Bradburn, Glen Cain, Susan Chipman, Christopher Cross, Larry V. Hedges, Jeremy Kilpatrick, David Klahr, Sharon Lewis, G. Reid Lyon, C. Kent McGuire, Robert Mislevy, William Morrill, William Quinn, Diane Ravitch, Sally Rockey, Steven Ross, Nancy Butler Songer, Judith Sunley, Richard Suzman, Peter Tillers, and Maris Vinovskis. During each session of the workshop, a committee member chaired and skillfully guided the discussions: Donald Barfield, Robert Boruch, Jere Confrey, Robert DeHaan, Paul Holland, and Ellen Condliffe Lagemann. Other committee members, as well as several participants, contributed questions and insights that significantly enhanced the dialogue. The committee conceived the workshop. Lisa Towne, staff director of the committee, Richard J. Shavelson, committee chair, and Michael J. Feuer, director of the Center for Education, composed this summary. This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC's Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: J. Myron Atkin, Stanford University Ellen Guiney, Boston Plan - Boston Annenberg Challenge Paul Hood, WestEd David Myers, Mathematica Policy Research, Inc.

OCR for page R1
Page vi Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by Robert Linn of the University of Colorado. Appointed by the National Research Council, he was responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution. Finally, we wish to thank Kirsten Sampson Snyder for deftly guiding us through this report review process and Yvonne Wise for her helpful editorial suggestions.

OCR for page R1
Page vii Preface As the demand grows for reliable and credible information to guide policy and practice in the current education reform environment, research on education has again come into the political spotlight. Questions about the nature of evidence have recently spilled over from a relatively contained debate among the education research community into the broader policy and practice arenas. In the federal government, several education initiatives have attempted to codify sentiments for more rigorous research into operational laws and programs. 1 These bills use language such as “scientifically valid,” “research-based,” and “standards of quality” for research and its application to practice, with various definitions. Although these legislative trends are relatively new, systematic inquiry into teaching, learning, and schooling in the United States dates back at least to the nineteenth century. This inquiry has generated bodies of scientific knowledge that have profound implications for education. For example, we have seen dramatic advances in understanding how people learn, how young children acquire early reading skills, and how to design and evaluate educational and psychological measurements. However, the highly contextualized nature of education and the wide range of disciplinary perspectives that bear on it have rendered the identification of reducible, generalizable principles—of the sort that typify other scientific disciplines— difficult and slow to achieve. Indeed, a scientific basis for education research has long itself been at issue. It is in this turbulent context that the U.S. Department of Education's National Educational Research Policy and Priorities Board (NERPPB) has asked the NRC to establish a study committee to consider the scientific underpinnings of research in education. The committee's work is one part of a larger federal effort to improve the utility and quality of education research and is designed to contribute to, and learn from, related initiatives (e.g., RAND study panels in reading and mathematics, 2 NRC Strategic Education Research Program, 3 and the Education Quality Institute 4). The committee consists of members with expertise in statistics, psychology, sociology, anthropology, philosophy of science, history of education, economics, chemistry, biology, and education practice. 1 For example, the Bush administration's No Child Left Behind blueprint for the federal role in K-12 education, the Reading Excellence Act, the Comprehensive School Reform Demonstration Act, and a House GOP bill to reauthorize the Office of Educational Research and Improvement. 2 See www.rand.org/multi/achievementforall 3 See http://books.nap.edu/books/0309064899/html 4 The Education Quality Institute is an emerging independent, nonprofit organization that has been created to provide consumers reliable information on what works to raise student acheivement.

OCR for page R1
Page viii Three framing questions summarize the committee's charge: 1. What are the principles of scientific quality in education research? To address this question, the committee is considering how the purposes, norms of inquiry, methods, and traditions of scientific inquiry translate in the study of education. The committee is considering what scientific quality means, both in individual research projects, as well as in programs of research, to better understand how knowledge can be organized, synthesized, and generalized. 2. How can research-based knowledge in education cumulate? Closely related to the first question, the committee is focusing on the mechanisms that support the cumulation of knowledge from scientifically based education research—the organization and synthesis of knowledge generated from multiple investigations. The committee is considering the role of the professional research community, the practitioner community, and the federal government. 3. How can a federal research agency promote and protect scientific quality in the education research it supports? Based on findings about scientific quality in education research and what influences the accumulation of knowledge, the committee will consider design principles necessary for a federal agency to foster the scientific integrity of the work it funds. Among the issues explored by the committee are (a) how internal infrastructure mechanisms, such as peer-review systems, affect research quality, (b) how external forces, such as political influence, affect research quality, and (c) how the federal role can build the capacity of the field to do high-quality work. The committee sponsored a public workshop on March 7-8, 2001 to engage a broad audience of over 125 participants in dialogue about some of the issues and challenges it faces. To help map the terrain related to its charge, the committee organized the event into three main sessions: 1. Supporting Scientific Quality at the Federal Level, 2. The Interface of Research and Practice in Education, and 3. Evidence and Inference. The first session—Supporting Scientific Quality at the Federal Level—was designed to help the committee understand ways in which a range of research agencies conceive and support scientific rigor and integrity (see framing question #3). Specifically, panelists were asked to consider the following questions: A. How do research organizations define scientific quality for both individual projects and programs or portfolios of work? How did these definitions develop and from where were they derived? B. What are the enabling conditions that allow research organizations to promote and sustain scientific quality in the projects and programs they support over time? C. What impediments have hindered quality and how have they been addressed?

OCR for page R1
Page ix The second session—The Interface of Research and Practice in Education—was designed to help the committee understand the relationship between its focus on the scientific underpinnings of education research and the use of research in practice. This topic, and thus the workshop session, relates to two of the three framing questions for the committee's work. The committee wanted to consider the possible trade-offs involved in simultaneously maximizing the utility of the research and scientific rigor (see framing question #1). The committee was also interested in whether and how bodies of knowledge that have accumulated from scientific research inform practice (see framing question #2). To organize this session, the committee developed two roundtable discussions focused on: (1) the relationship between research and education reform and (2) the relationship between scientific quality and the utility of research. To address the relationship between research and reform, roundtable discussants were first asked to consider: A. What is the nature, quantity, and quality of scientific evidence needed to act, scale up? B. In what ways can research provide guidance about various decisions in education policy and practice? What are its limitations? C. In what ways can research provide insights about alternative ways to solve problems? What are its limitations? To address the relationship between scientific quality and research use, the committee asked the same discussants to consider a second set of questions: A. Does “quality” mean the same thing to researchers as it does to the users of research? Should it? B. What are the characteristics of research that influence education decision making, and what are the conditions under which it can drive change? C. Are there trade-offs required in achieving researchers' standards of quality and consumers' standards for utility? Finally, to help the committee articulate scientific principles in education research (see framing question #1), a third session was designed to elucidate similarities and differences in the core concepts of evidence and inference across a range of disciplines and fields. This document summarizes the workshop, and is organized around main session themes. Appendixes include the full agenda of the event ( Appendix A), biosketches of workshop speakers ( Appendix B), and a list of workshop participants ( Appendix C). The verbatim transcript, in text and audio formats, can be found on CFE's webpage at: http://www4.nas.edu/cfe/cfe.nsf/web/other_projects?OpenDocument (under “Scientific Principles in Educational Research: Explorations and Perspectives for OERI”). The committee's intent in issuing this summary is to communicate to an even wider audience the key themes that emerged from the workshop discussions. We hope that this report contributes to ongoing deliberations and discussions about the role of research in generating new knowledge and in informing education decision making, as well as informs policy makers considering reauthorization of the Office of Educational Research and Improvement (OERI).

OCR for page R1
Page x AN IMPORTANT CAVEAT We want to clarify that this report is not intended to reflect the consensus of the committee on findings, conclusions, and recommendations from its study. That consensus is evolving and will be articulated in the committee's final report, which is due in fall 2001.

OCR for page R1
Page xi Contents THEME 1.     Supporting Scientific Quality at the Federal Level: Consistency and Variation 1 THEME 2.     The Interface of Research and Practice in Education: Linking Quality and Utility 7 THEME 3.     Evidence and Inference: Consistency and Variation Revisited 11 SYNTHESIS AND NEXT STEPS FOR THE COMMITTEE 14 APPENDIX A:     WORKSHOP AGENDA 16 APPENDIX B:     WORKSHOP SPEAKER BIOSKETCHES 20 APPENDIX C:     WORKSHOP PARTICIPANTS 29

OCR for page R1
Page xii