National Academies Press: OpenBook
Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×

Page i

Science, Evidence, and Inference in Education

Report of a Workshop

Lisa Towne, Richard J. Shavelson, and Michael J. Feuer, Editors

Committee on Scientific Principles in Education Research

Center for Education

Division of Behavioral and Social Sciences and Education

National Research Council

National Academy Press
Washington, D.C.
May 2001

Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×

Page ii

    NATIONAL ACADEMY PRESS 2101 Constitution Avenue, N.W. Washington, D.C. 20418

    NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.

    This project has been funded at least in part with Federal funds from the U.S. Department of Education under contract number ED-00-CO-0088. The content of this publication does not necessarily reflect the views or policies of the U.S. Department of Education nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government.

    International Standard Book Number 0-309-07570-X

    Additional copies of this report are available from National Academy Press , 2101 Constitution Avenue, N.W. , Lockbox 285, Washington, D.C. 20418

    Call (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area)

    This report is also available online at http://www.nap.edu

    Copyright 2001 by the National Academy of Sciences . All rights reserved.

    Suggested citation: National Research Council. (2001). Science, evidence, and inference in education: Report of a workshop. Committee on Scientific Principles in Education Research. L. Towne, R. J. Shavelson, and M. J. Feuer (Eds.). Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.

    Printed in the United States of America

Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×

Page iii

    THE NATIONAL ACADEMIES

    National Academy of Sciences

    National Academy of Engineering

    Institute of Medicine

    National Research Council

    The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.

    The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. William A. Wulf is president of the National Academy of Engineering.

    The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine.

    The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. William A. Wulf are chairman and vice chairman, respectively, of the National Research Council.

Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×

Page iv

    COMMITTEE ON SCIENTIFIC PRINCIPLES IN EDUCATION RESEARCH

    RICHARD J. SHAVELSON (Chair), Center for Advanced Study in the Behavioral Sciences and Stanford University School of Education

    DONALD I. BARFIELD, WestEd, San Francisco, California

    ROBERT F. BORUCH, Graduate School of Education and Statistics, University of Pennsylvania, Philadelphia

    JERE CONFREY, Department of Curriculum and Instruction, University of Texas, Austin

    RUDOLPH F. CREW, Institute for K-12 Leadership, University of Washington, Seattle

    ROBERT L. DeHAAN, Department of Cell Biology, Emory University, Atlanta, Georgia

    MARGARET EISENHART, School of Education, University of Colorado, Boulder

    JACK McFARLIN FLETCHER, Department of Pediatrics, University of Texas, Houston

    EUGENE E. GARCIA, Graduate School of Education, University of California, Berkeley

    NORMAN HACKERMAN, The Welch Foundation, Houston, Texas

    ERIC HANUSHEK, Hoover Institution, Stanford University

    ROBERT M. HAUSER, Center for Demography of Health and Aging, University of Wisconsin, Madison

    PAUL W. HOLLAND, Educational Testing Service, Princeton, New Jersey

    ELLEN CONDLIFFE LAGEMANN, Spencer Foundation and New York University

    DENIS C. PHILLIPS, Stanford University School of Education

    CAROL H. WEISS, Graduate School of Education, Harvard University

    LISA TOWNE, Director

    TINA WINTERS, Research Assistant

    LINDA DePUGH, Senior Project Assistant

Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×

Page v

Acknowledgments

The Committee on Scientific Principles in Education Research and the National Research Council's (NRC) Center for Education (CFE) are grateful to the many individuals whose efforts made possible this report, and the workshop it summarizes. The committee's work is supported by a contract with the U.S. Department of Education's National Educational Research Policy and Priorities Board. The guidance offered by Kenji Hakuta, Alba Ortiz, Rafael Valdivieso, Thelma Leenhouts, and Mary Grace Lucier is very much appreciated.

We also wish to thank the workshop speakers, whose remarks stimulated a rich and wide-ranging discussion: Michael Agar, Norman Bradburn, Glen Cain, Susan Chipman, Christopher Cross, Larry V. Hedges, Jeremy Kilpatrick, David Klahr, Sharon Lewis, G. Reid Lyon, C. Kent McGuire, Robert Mislevy, William Morrill, William Quinn, Diane Ravitch, Sally Rockey, Steven Ross, Nancy Butler Songer, Judith Sunley, Richard Suzman, Peter Tillers, and Maris Vinovskis. During each session of the workshop, a committee member chaired and skillfully guided the discussions: Donald Barfield, Robert Boruch, Jere Confrey, Robert DeHaan, Paul Holland, and Ellen Condliffe Lagemann. Other committee members, as well as several participants, contributed questions and insights that significantly enhanced the dialogue.

The committee conceived the workshop. Lisa Towne, staff director of the committee, Richard J. Shavelson, committee chair, and Michael J. Feuer, director of the Center for Education, composed this summary.

This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC's Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report:

J. Myron Atkin, Stanford University

Ellen Guiney, Boston Plan - Boston Annenberg Challenge

Paul Hood, WestEd

David Myers, Mathematica Policy Research, Inc.

Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×

Page vi

Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by Robert Linn of the University of Colorado. Appointed by the National Research Council, he was responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.

Finally, we wish to thank Kirsten Sampson Snyder for deftly guiding us through this report review process and Yvonne Wise for her helpful editorial suggestions.

Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×

Page vii

Preface

As the demand grows for reliable and credible information to guide policy and practice in the current education reform environment, research on education has again come into the political spotlight. Questions about the nature of evidence have recently spilled over from a relatively contained debate among the education research community into the broader policy and practice arenas. In the federal government, several education initiatives have attempted to codify sentiments for more rigorous research into operational laws and programs. 1 These bills use language such as “scientifically valid,” “research-based,” and “standards of quality” for research and its application to practice, with various definitions.

Although these legislative trends are relatively new, systematic inquiry into teaching, learning, and schooling in the United States dates back at least to the nineteenth century. This inquiry has generated bodies of scientific knowledge that have profound implications for education. For example, we have seen dramatic advances in understanding how people learn, how young children acquire early reading skills, and how to design and evaluate educational and psychological measurements. However, the highly contextualized nature of education and the wide range of disciplinary perspectives that bear on it have rendered the identification of reducible, generalizable principles—of the sort that typify other scientific disciplines— difficult and slow to achieve. Indeed, a scientific basis for education research has long itself been at issue.

It is in this turbulent context that the U.S. Department of Education's National Educational Research Policy and Priorities Board (NERPPB) has asked the NRC to establish a study committee to consider the scientific underpinnings of research in education. The committee's work is one part of a larger federal effort to improve the utility and quality of education research and is designed to contribute to, and learn from, related initiatives (e.g., RAND study panels in reading and mathematics, 2 NRC Strategic Education Research Program, 3 and the Education Quality Institute 4). The committee consists of members with expertise in statistics, psychology, sociology, anthropology, philosophy of science, history of education, economics, chemistry, biology, and education practice.

1 For example, the Bush administration's No Child Left Behind blueprint for the federal role in K-12 education, the Reading Excellence Act, the Comprehensive School Reform Demonstration Act, and a House GOP bill to reauthorize the Office of Educational Research and Improvement.

2 See www.rand.org/multi/achievementforall

3 See http://books.nap.edu/books/0309064899/html

4 The Education Quality Institute is an emerging independent, nonprofit organization that has been created to provide consumers reliable information on what works to raise student acheivement.

Page viii Cite
Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×

Page viii

Three framing questions summarize the committee's charge:

    1. What are the principles of scientific quality in education research?

    To address this question, the committee is considering how the purposes, norms of inquiry, methods, and traditions of scientific inquiry translate in the study of education. The committee is considering what scientific quality means, both in individual research projects, as well as in programs of research, to better understand how knowledge can be organized, synthesized, and generalized.

    2. How can research-based knowledge in education cumulate?

    Closely related to the first question, the committee is focusing on the mechanisms that support the cumulation of knowledge from scientifically based education research—the organization and synthesis of knowledge generated from multiple investigations. The committee is considering the role of the professional research community, the practitioner community, and the federal government.

    3. How can a federal research agency promote and protect scientific quality in the education research it supports?

    Based on findings about scientific quality in education research and what influences the accumulation of knowledge, the committee will consider design principles necessary for a federal agency to foster the scientific integrity of the work it funds. Among the issues explored by the committee are (a) how internal infrastructure mechanisms, such as peer-review systems, affect research quality, (b) how external forces, such as political influence, affect research quality, and (c) how the federal role can build the capacity of the field to do high-quality work.

The committee sponsored a public workshop on March 7-8, 2001 to engage a broad audience of over 125 participants in dialogue about some of the issues and challenges it faces. To help map the terrain related to its charge, the committee organized the event into three main sessions:

    1. Supporting Scientific Quality at the Federal Level,

    2. The Interface of Research and Practice in Education, and

    3. Evidence and Inference.

The first session—Supporting Scientific Quality at the Federal Level—was designed to help the committee understand ways in which a range of research agencies conceive and support scientific rigor and integrity (see framing question #3). Specifically, panelists were asked to consider the following questions:

    A. How do research organizations define scientific quality for both individual projects and programs or portfolios of work? How did these definitions develop and from where were they derived?

    B. What are the enabling conditions that allow research organizations to promote and sustain scientific quality in the projects and programs they support over time?

    C. What impediments have hindered quality and how have they been addressed?

Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×

Page ix

The second session—The Interface of Research and Practice in Education—was designed to help the committee understand the relationship between its focus on the scientific underpinnings of education research and the use of research in practice. This topic, and thus the workshop session, relates to two of the three framing questions for the committee's work. The committee wanted to consider the possible trade-offs involved in simultaneously maximizing the utility of the research and scientific rigor (see framing question #1). The committee was also interested in whether and how bodies of knowledge that have accumulated from scientific research inform practice (see framing question #2).

To organize this session, the committee developed two roundtable discussions focused on: (1) the relationship between research and education reform and (2) the relationship between scientific quality and the utility of research. To address the relationship between research and reform, roundtable discussants were first asked to consider:

    A. What is the nature, quantity, and quality of scientific evidence needed to act, scale up?

    B. In what ways can research provide guidance about various decisions in education policy and practice? What are its limitations?

    C. In what ways can research provide insights about alternative ways to solve problems? What are its limitations?

To address the relationship between scientific quality and research use, the committee asked the same discussants to consider a second set of questions:

    A. Does “quality” mean the same thing to researchers as it does to the users of research? Should it?

    B. What are the characteristics of research that influence education decision making, and what are the conditions under which it can drive change?

    C. Are there trade-offs required in achieving researchers' standards of quality and consumers' standards for utility?

Finally, to help the committee articulate scientific principles in education research (see framing question #1), a third session was designed to elucidate similarities and differences in the core concepts of evidence and inference across a range of disciplines and fields.

This document summarizes the workshop, and is organized around main session themes. Appendixes include the full agenda of the event ( Appendix A), biosketches of workshop speakers ( Appendix B), and a list of workshop participants ( Appendix C). The verbatim transcript, in text and audio formats, can be found on CFE's webpage at: http://www4.nas.edu/cfe/cfe.nsf/web/other_projects?OpenDocument (under “Scientific Principles in Educational Research: Explorations and Perspectives for OERI”).

The committee's intent in issuing this summary is to communicate to an even wider audience the key themes that emerged from the workshop discussions. We hope that this report contributes to ongoing deliberations and discussions about the role of research in generating new knowledge and in informing education decision making, as well as informs policy makers considering reauthorization of the Office of Educational Research and Improvement (OERI).

Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×

Page x

AN IMPORTANT CAVEAT

We want to clarify that this report is not intended to reflect the consensus of the committee on findings, conclusions, and recommendations from its study. That consensus is evolving and will be articulated in the committee's final report, which is due in fall 2001.

Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×

Page xi

Contents

THEME 1.     Supporting Scientific Quality at the Federal Level: Consistency and Variation 1
THEME 2.     The Interface of Research and Practice in Education: Linking Quality and Utility 7
THEME 3.     Evidence and Inference: Consistency and Variation Revisited 11
SYNTHESIS AND NEXT STEPS FOR THE COMMITTEE 14
APPENDIX A:     WORKSHOP AGENDA 16
APPENDIX B:     WORKSHOP SPEAKER BIOSKETCHES 20
APPENDIX C:     WORKSHOP PARTICIPANTS 29
Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×

Page xii

Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×
Page R1
Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×
Page R2
Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×
Page R3
Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×
Page R4
Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×
Page R5
Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×
Page R6
Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×
Page R7
Page viii Cite
Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×
Page R8
Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×
Page R9
Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×
Page R10
Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×
Page R11
Suggested Citation:"Front Matter." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×
Page R12
Next: Theme 1: Supporting Scientific Quality at the Federal Level: Consistency and Variation »
Science, Evidence, and Inference in Education: Report of a Workshop Get This Book
×
Buy Paperback | $21.00 Buy Ebook | $16.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Research on education has come into the political spotlight as the demand grows for reliable and credible information for the guidance of policy and practice in the education reform environment. Many debates among the education research community feature questions concerning the nature of evidence and these questions have also appeared in broader policy and practice arenas. Inquiry has generally, over the past years, created bodies of scientific knowledge that have profound implications for education. Dramatic advances in understanding how people learn, how young children acquire early reading skills, and how to design and evaluate educational and psychological measurements is a good example of this. However, the highly contextualized nature of education and the wide range of disciplinary perspectives that rely on it have made the identification of reducible, generalizable principles difficult and slow to achieve.

Due to this, the U.S. Department of Education's National Educational Research Policy and Priorities Board (NERPPB) has asked the NRC to establish a study committee to consider the scientific underpinnings of research in education. The committee consists of members with expertise in statistics, psychology, sociology, anthropology, philosophy of science, history of education, economics, chemistry, biology, and education practice. The committee worked with the three questions in mind: What are the principles of scientific quality in education research?, How can research-based knowledge in education cumulate?, and How can a federal research agency promote and protect scientific quality in the education research it supports?.

A workshop was held on March 7-8, 2001 that was organized into three main sessions: Supporting Scientific Quality at the Federal level, The Interface of Research and Practice in Education, and Evidence and Inference. Science, Evidence, and Inference in Education: Report of a Workshop summarizes this workshop through these three ideas. The report also includes what the committee plans to do next, the workshop agenda, and information on the workshop's participants and speakers.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!