On Evaluating Curricular Effectiveness

JUDGING THE QUALITY OF K-12 MATHEMATICS EVALUATIONS

Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials

Mathematical Sciences Education Board

Center for Education

Division of Behavioral and Social Sciences and Education

NATIONAL RESEARCH COUNCIL OF THE NATIONAL ACADEMIES

THE NATIONAL ACADEMIES PRESS
Washington, D.C. www.nap.edu



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations On Evaluating Curricular Effectiveness JUDGING THE QUALITY OF K-12 MATHEMATICS EVALUATIONS Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials Mathematical Sciences Education Board Center for Education Division of Behavioral and Social Sciences and Education NATIONAL RESEARCH COUNCIL OF THE NATIONAL ACADEMIES THE NATIONAL ACADEMIES PRESS Washington, D.C. www.nap.edu

OCR for page R1
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This study/publication was supported by Contract/Grant No. ESI-0102582 between the National Academy of Sciences and the National Science Foundation. Additional funding was provided by an award from the Presidents’ Committee of the National Academies. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project. Library of Congress Cataloging-in-Publication Data On evaluating curricular effectiveness : judging the quality of K-12 mathematics evaluations / Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials, Mathematical Sciences Education Board, Center for Education, Division of Behavioral and Social Sciences and Education. p. cm. Includes bibliographical references. ISBN 0-309-09242-6 (pbk.)—ISBN 0-309-53287-6 (pdf) 1. Mathematics—Study and teaching—Evaluation. I. National Research Council (U.S.). Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials. QA11.2.O5 2004 510'.71—dc22 2004015000 Additional copies of this report are available from the National Academies Press, 500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu. Printed in the United States of America. Copyright 2004 by the National Academy of Sciences. All rights reserved. National Research Council. (2004). On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials. Mathematical Sciences Education Board, Center for Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

OCR for page R1
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations THE NATIONAL ACADEMIES Advisers to the Nation on Science, Engineering, and Medicine The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council. www.national-academies.org

OCR for page R1
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations This page intentionally left blank.

OCR for page R1
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations COMMITTEE FOR A REVIEW OF THE EVALUATION DATA ON THE EFFECTIVENESS OF NSF-SUPPORTED AND COMMERCIALLY GENERATED MATHEMATICS CURRICULUM MATERIALS JERE CONFREY (Chair), Department of Education, Washington University in St. Louis, MO CARLOS CASTILLO-CHAVEZ, Department of Mathematics and Statistics, Arizona State University DOUGLAS A. GROUWS, Department of Learning, Teaching, and Curriculum, University of Missouri CAROLYN MAHONEY, Division of Academic Affairs, Elizabeth City State University, NC DONALD SAARI, Institute for Mathematical Behavioral Sciences, University of California, Irvine WILLIAM SCHMIDT, College of Education, Michigan State University PATRICK W. THOMPSON, Department of Teaching and Learning, Vanderbilt University, TN WILLIAM VELEZ, Department of Mathematics, University of Arizona CAROLE B. LACAMPAGNE, Study Director VICKI STOHL, Research Associate DIONNA J. WILLIAMS, Senior Program Assistant

OCR for page R1
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations MATHEMATICAL SCIENCES EDUCATION BOARD 2004 JOAN LEITZEL (Chair), President Emerita, University of New Hampshire JERE CONFREY (Vice Chair), Department of Education, Washington University in St. Louis, MO THOMAS BANCHOFF, Department of Mathematics, Brown University, RI JAN DE LANGE, Freudenthal Institute, Utrecht University, The Netherlands LOUIS GOMEZ, School of Education and Social Policy, Northwestern University, IL DOUGLAS A. GROUWS, Department of Learning, Teaching, and Curriculum, University of Missouri ARTHUR JAFFE, Department of Mathematics, Harvard University, MA ERIC JOLLY, Science Museum of Minnesota JIM LEWIS, Department of Mathematics and Statistics, University of Nebraska-Lincoln GEORGE MCSHAN, National School Boards Association, TX KAREN MICHALOWICZ, Mathematics Department, The Langley School, VA JUDITH MUMME, WestEd, CA CASILDA PARDO, Teaching and Learning Systems, Valle Vista Elementary School, NM SUE PARSONS, Instructor, Cerritos College, CA MARGE PETIT, The National Center for the Improvement of Educational Assessment, VT DONALD SAARI, Institute for Mathematical Behavioral Sciences, Universityof California, Irvine RICHARD SCHEAFFER, Professor Emeritus, University of Florida FRANCIS SULLIVAN, Center for Computing Sciences, MD HUNG HSI WU, Department of Mathematics, University of California, Berkeley

OCR for page R1
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations Foreword The Mathematical Sciences Education Board (MSEB) of the National Research Council was established in 1985 to provide national leadership and guidance for policies, programs, and practices supporting the improvement of mathematics education at all levels. Curriculum materials for grades K-12 play a central role in what mathematics topics are taught in our schools, how the topics are sequenced and presented to students, what levels of understanding are expected, what skills students will develop and when. Schools, practitioners, policy makers, and the public depend on evaluations of materials undertaken during their development and implementation in making decisions about the appropriate uses of the materials. The MSEB recognized that the nature and quality of the evidence used to judge claims of success and failure are critical elements in enabling the community to make sound judgments. This report presents a synthesis of the evidence used in the evaluations of several sets of recently developed curriculum materials, provides a framework for the design of evaluation studies of curriculum materials, and gives conclusions and recommendations to guide future efforts in evaluating curriculum materials. The report was prepared by a committee of experts who devoted their time, skills, and scholarship to this project over the past two years. On behalf of the MSEB, I want to thank each of them for their commitment to the important and difficult set of issues this study comprised. I especially want to commend Jere Confrey and extend deepest appreciation to her for her extraordinary leadership and commitment as chair of this project. In addition to her leadership of the committee, Jere played an extensive role

OCR for page R1
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations in drafting and redrafting the report through the final stages of committee consultation and the intensive review process. Her dedication to maintaining the highest standards of scholarship and the full engagement of the committee, despite her many other professional obligations, was exemplary. The report bears the imprint of her commitment to intellectual and empirical rigor; the field of mathematics education is the fortunate beneficiary. Joan R. Leitzel Chair, MSEB

OCR for page R1
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations Acknowledgments This report reflects the efforts of many people, and the committee is most grateful for their contributions. First, we wish to acknowledge and thank our sponsor, the National Science Foundation’s Directorate for Education and Human Resources, for their support of this study. The National Academies’ Presidents’ Fund also provided some additional support for the report. The committee was aided greatly by the individuals who participated in our evidence-gathering workshops. They helped us understand the nuances of curriculum development and implication and ways to evaluate the efficacy of such curricula once out in the field. The first evidence gathering workshop was held in September 2002 and focused on how to define or evaluate “effectiveness” of school mathematics curricula. The committee gained valuable insights from the experts who presented at that workshop: Johanna Birckmayer, Pacific Institute of Research and Evaluation; David Francis, University of Houston; Pendred Noyce, The Noyce Foundation; Andrew Porter, Wisconsin Center for Education Research; Frank Wang, Saxon Publishers; Jan Mokros, TERC; Thomas Romberg, University of Wisconsin–Madison; Sheila Sconiers, ARC Implementation Center; Andrew Isaacs, University of Chicago; Judith Zawojewski, Illinois Institute of Technology; Solomon Garfunkel, COMAP, Inc; Norman Webb, Wisconsin Center for Education Research; Richard Askey, University of Wisconsin–Madison; Roger Howe, Yale University; William McCallum, University of Arizona; M. Kathleen Heid, Pennsylvania State University; Richard Lehrer, Vanderbilt University; Richard Lesh, Purdue University; Mark Jenness,

OCR for page R1
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations Western Michigan University; June Mark, Education Development Center, Inc.; Maria Santos, Achieve, Inc; Richard Stanley, University of California, Berkeley; Terri Dahl, Charles M. Russell High School, MT; Eric Gutstein, University of Illinois, Chicago; Mark St. John, Inverness Research Associates; Timothy Wierenga, Naperville Community Unit School District #203, IL; James Stigler, University of California, Los Angeles; and William Tate, Washington University, St. Louis. The second evidence-gathering workshop was held in October 2002 and focused on how commercial publishers evaluate their text series. The committee gained valuable insight from Arthur Block, McGraw-Hill; Karen Usiskin, Scott Foresman; Steve Rasmussen, Key Curriculum Press; Denise McDowell, Houghton Mifflin; Paula Gustafson, Texas Education Agency; and Lisa Brady-Gill, Texas Instruments. Much was learned from those who presented at that these workshops which proved of great value to the committee as it wrote this report. The committee would like to acknowledge the kindness of the principal investigators of the NSF-supported curriculum projects, of the NSF Implementation Centers, and of textbook publishers for supplying us with texts and evaluation studies. The committee is also most grateful to the graduate students who labored through reading and entering data on the almost 700 evaluation studies that we accumulated: Kathleen M. Clark, University of Maryland; Timothy P. Fukawa-Connelly, University of Maryland; Michelle Caren Massey, American University; and Francesca Palik, George Washington University. Among these graduate students, special thanks go to Erica Slate Young, University of Texas, Austin, who worked closely with the chair of the committee. Sibel Kazak and Dustin Mitchell, Washington University in St. Louis also provided assistance to the chair. In addition special thanks go to Alan Maloney, for the long hours he dedicated (including his birthday) to helping the chair prepare the final revised draft of the report. Special thanks go to the NRC staff who worked on this report. Research associate Vicki Stohl worked tirelessly to develop a data base, assemble materials for the committee, and produce part of the preliminary draft of the document. Senior program assistant Dionna Williams worked efficiently under great time pressure to engineer the workshops and other committee meetings and to put the many drafts of this report into good form. Carole Lacampagne provided leadership for the project, Jay Labov gave sage advice along the way, and Patricia Morison was instrumental in bringing this report to successful closure. Martin Orland, director of the Center for Education, also provided leadership and support for the project. This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC’s RRC. The purpose of this independent review is to provide candid and critical comments that will assist the institu-

OCR for page R1
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations tion in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Douglas Carnine, National Center to Improve the Tools of Educators, University of Oregon; Mary M. Lindquist, Department of Mathematics Emeritus, Columbus State University, Columbus, GA; R. James Milgram, Department of Mathematics, Stanford University; Leona Schauble, Teaching and Learning Department, Vanderbilt University; Alan H. Schoenfeld, School of Education, University of California, Berkeley; Norman Webb, Wisconsin Center for Education Research, University of Wisconsin–Madison; and Iris R. Weiss, President, Horizon Research, Inc., Chapel Hill, NC. Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations nor did they see the final draft of the report before its release. The review of this report was overseen by Adam Gamoran, University of Wisconsin–Madison, and William G. Howard, Jr., Independent Consultant, Scottsdale, AZ, appointed by the National Research Council, who were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility of the final content of this report, however, rests entirely with the authoring committee and the NRC.

OCR for page R1
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations This page intentionally left blank.

OCR for page R1
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations Contents     EXECUTIVE SUMMARY   1 1   THE CHALLENGE AND THE CHARGE   11      Need for This Study,   12      Timeliness of the Report,   14      Committee Charge and Approach,   14      Report Layout,   16 2   COMMITTEE PROCEDURES AND CHARACTERISTICS OF STUDIES REVIEWED   18      Curricula Under Review,   19      Data Gathering,   22      The Study Matrix,   24      Study Characteristics,   25      Study Characteristics for Categories 1 Through 4,   30 3   FRAMEWORK FOR EVALUATING CURRICULAR EFFECTIVENESS   36      Primary Components,   40      Secondary Components,   52      Evaluation Design, Measurement, and Evidence,   54

OCR for page R1
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations 4   CONTENT ANALYSIS   65      Literature Review,   68      Dimensions of Content Analyses,   71 5   COMPARATIVE STUDIES   96      Description of Comparative Studies Database on Critical Decision Points,   104      Conclusions from the Comparative Studies,   164 6   CASE STUDIES AND SYNTHESIS STUDIES   167      Case Studies,   167      Synthesis Studies,   180 7   CONCLUSIONS AND RECOMMENDATIONS   185      The Quality of the Evaluations,   188      Framework and Key Definitions,   189      Scientifically Establishing Curricular Effectiveness,   191      Recommended Practices for the Conduct of Evaluation Studies,   193      Recommendations to Federal and State Agencies and Publishers,   201     REFERENCES   206     APPENDIXES     A   Biographical Sketches   219 B   Bibliography of Studies Included in Committee Analysis   223 C   Outcome Measures   259     INDEX   261

OCR for page R1
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations On Evaluating Curricular Effectiveness

OCR for page R1
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations This page intentionally left blank.