Assessing Accomplished Teaching: Advanced-Level Certification Programs (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Assessing Accomplished Teaching Advanced-Level Certification Programs Committee on Evaluation of Teacher Certification by the National Board for Professional Teaching Standards Milton D. Hakel, Judith Anderson Koenig, and Stuart W. Elliott�� Editors , Board on Testing and Assessment Center for Education Division of Behavioral and Social Sciences and Education 

THE NATIONAL ACADEMIES PRESS   500 Fifth Street, N.W.   Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Govern- ing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineer- ing, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropri- ate balance. This study was supported by Contract No. ED-04-CO-0139 between the National Academy of Sciences and the U.S. Department of Education. Any opinions, find- ings, conclusions, or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the organizations or agencies that provided support for the project. Library of Congress Cataloging-in-Publication Data Assessing accomplished teaching : advanced-level certification programs : Committee on Evaluation of Teacher Certification by the National Board for Professional Teaching Standards / Milton D. Hakel, Judith Anderson Koenig, and Stuart W. Elliott, editors.    p. cm.   ISBN 978-0-309-12118-7 (pbk.) — ISBN 978-0-309-12119-4 (pdf)  1. Teachers—Certification—United States.  I. Hakel, Milton D. II. Koenig, Judith A. III. Elliott, Stuart W. IV. National Board for Professional Teaching Standards (U.S.)   LB1771.A77 2008   371.120973—dc22 2008026490 Additional copies of this report are available from the National Academies Press, 500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap. edu. Copyright 2008 by the National Academy of Sciences. All rights reserved. Printed in the United States of America Suggested citation: National Research Council. (2008). Assessing Accomplished Teaching: Advanced-Level Certification Programs. Committee on Evaluation of Teacher Certification by the National Board for Professional Teaching Standards. Milton D. Hakel, Judith Anderson Koenig, and Stuart W. Elliott, editors. Board on Testing and Assessment, Center for Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press. 

The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Acad- emy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding en- gineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineer- ing programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is presi- dent of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Insti- tute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sci- ences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Coun- cil is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council. www.national-academies.org 



COMMITTEE ON EVALUATION OF TEACHER CERTIFICATION BY THE NATIONAL BOARD FOR PROFESSIONAL TEACHING STANDARDS MILTON D. HAKEL (Chair), Department of Psychology, Bowling Green State University JULIAN BETTS, Department of Economics, University of California, San Diego MARK DYNARSKI, Mathematica Policy Research, Inc., Princeton, New Jersey ADAM GAMORAN, Department of Sociology, University of Wisconsin, Madison JANE HANNAWAY, The Urban Institute, Washington, DC RICHARD INGERSOLL, Graduate School of Education, University of Pennsylvania MICHAEL T. KANE, National Conference of Bar Examiners, Madison, Wisconsin DEIRDRE J. KNAPP, Human Resources Research Organization, Alexandria, Virginia SUSANNA LOEB, School of Education, Stanford University JAMES (TORCH) H. LYTLE, Graduate School of Education, University of Pennsylvania C. FORD MORISHITA, Clackamas High School, Clackamas, Oregon LYNN W. PAINE, College of Education, Michigan State University NEIL J. SMELSER (Emeritus), University of California, Berkeley BRIAN STECHER, RAND Corporation, Santa Monica, California ANA MARIA VILLEGAS, College of Education and Human Services, Montclair State University DOROTHY Y. WHITE, Department of Mathematics and Science Education, University of Georgia KAREN K. WIXSON, School of Education, University of Michigan JUDITH A. KOENIG, Study Director STUART W. ELLIOTT, Senior Program Officer ALIX BEATTY, Senior Program Officer VIOLA HOREK, Manager, Program Operations TERESIA WILMORE, Senior Program Assistant  

BOARD ON TESTING AND ASSESSMENT LYLE F. BACHMAN, Department of Applied Linguistics, University of California, Los Angeles STEPHEN B. DUNBAR, College of Education, University of Iowa DAVID J. FRANCIS, Department of Psychology, University of Houston ANDREW J. HARTMAN, Independent Consultant, Denver, Colorado MICHAEL NETTLES, Educational Testing Service, Princeton, New Jersey JAMES W. PELLEGRINO, Department of Psychology, University of Illinois, Chicago DIANA C. PULLIN, School of Education, Boston College STUART W. ELLIOTT, Director vi 

Acknowledgments The Committee on Evaluation of Teacher Certification by the Na- tional Board for Professional Teaching Standards (NBPTS) was formed in response to legislation passed by the U.S. Congress asking the National Academies to develop a framework for evaluating programs that offer advanced-level certification to teachers and to apply that framework in an evaluation of the impacts of the NBPTS. The committee began its work in September 2005 and, over the course of the next 30 months, held six meet- ings, commissioned numerous papers and analyses, and collected a variety of information as part of its evaluation. The committee’s work benefited tremendously from the contributions of many people, and the committee is grateful for their assistance and support. The work was overseen by the U.S. Department of Education with management handled by Susan Sanchez. We thank Susan for her expert guidance and quick response to our many questions. Over the course of the project, we made many requests of NBPTS staff, including numerous requests for information, documentation, research re- ports, and data files; visits to their office and meetings with their staff; and presentations at our committee meetings. We extend our heartfelt thanks to Mary Dilworth for her diligence and patience in responding to our exten- sive inquiries. Mary gave generously of her time, and we thank her for all the information she provided in response to the committee’s questions. We also thank NBPTS president Joseph Aguerrebere and staff members Lillie Saunders and Joan Auchter for the valuable information they provided. vii 

viii ACKNOWLEDGMENTS During the course of this evaluation, the Educational Testing Service (ETS) served as the test development contractor to NBPTS, and we wish to acknowledge the assistance of several ETS staff members. We are grateful to Mari Pearlman for recapturing for us the history of the development of the assessment program and to Drew Gitomer for providing his review of research on the NBPTS program. We especially thank Steve Schreiner for providing an overview of the assessment scoring process and for guiding us through a review of sample portfolio materials and videotapes submit- ted by teachers. This review contributed greatly to our understanding of the assessment and the requirements that teachers must meet to earn board certification. Our evaluation drew on work conducted for us by several researchers. Our initial review of the research base revealed a large number of studies focused on the impact of board-certified teachers on their students’ achieve- ment. The findings from these studies presented a complex set of somewhat conflicting results. We thank Henry Braun and Paul Holland, both at ETS, for their insights about these studies, which helped us to sort out details of research methodologies and statistical analyses. We decided to conduct a full-scale review of these studies combined with work to identify and conduct additional analyses to help resolve the differing findings. The re- searchers who assisted us in this endeavor greatly contributed to our evalu- ation, and we are indebted to each of them, including Doug Harris at the University of Wisconsin–Madison, Helen Ladd at Duke University, Daniel McCaffrey at the RAND Corporation, Steven Rivkin at Amherst College, and Tim Sass at Florida State University. We are especially grateful to Tim Sass for the many additional analyses he conducted for us. We also thank Jon Fullerton and Tom Kane at Harvard University, Steve Raudenbush at the University of Chicago, and Bill Sanders at the SAS Institute, Inc., for their presentations at our third committee meeting and their insights about this body of work. We also commissioned a psychometric review of the NBPTS assess- ments. This was an extensive task, given that NBPTS awards certification in 25 areas, each using different assessments. We are indebted to Teresa Russell at the Human Resources Research Organization (HumRRO), who led this work for us. Her thorough and painstaking efforts to review a multitude of documents and summarize the psychometric characteristics of the assessments were a tremendous resource to the committee. In addition, we commissioned analyses of NBPTS participants and comparisons of their characteristics with those of nonparticipating teachers. David Perda, a doctoral student at the University of Pennsylvania, assisted us with these analyses. We sincerely appreciate his thorough and careful 

ACKNOWLEDGMENTS ix analyses and are particularly grateful for his responsiveness to our numer- ous follow-up requests for additional analyses. To learn more about the history of the NBPTS program, we conducted interviews of a number of key people involved with its early development. We are very grateful for the time these individuals spent with us and the wealth of information they provided. We thank former NBPTS staff m ­ embers Joan Baratz-Snowden at the American Federation of Teachers (former vice president of assessment and research); Chuck Cascio at ETS (former director of test development); Ann Harman at Harman and Asso­ ciates (former director of research); Jim Kelly, retired, who served as the first president of NBPTS; and David Mandel at Carnegie-IAS Commission on Mathematics and Science Education (former vice president for policy development at NBPTS). We are also grateful for the information provided by Sally Mernissi (former vice president and corporate secretary), who died in January 2006. We also thank the following individuals who assisted the NBPTS staff with development of the program: Lloyd Bond at the ­Carnegie Foundation (former director of the NBPTS Technical Analysis Group); Emerson ­Elliott at the National Council for Accreditation of Teacher Edu- cation; Mary Futrell at George Washington University (former president of the ­National Education Association and member of the NBPTS board of directors); Lee Shulman at the Carnegie Foundation (former consultant to the NBPTS board of directors); Gary Sykes at Michigan State University (former consultant to the NBPTS board of directors); and Suzanne Wilson at Michigan State University (former consultant to the NBPTS board of directors). We also thank Joshua Boots at the American Board for Cer- tification of Teacher Excellence (ABCTE) and Kathy Madigan, formerly at the ABCTE, for the information they provided about their assessment program. We sought to gather information from teachers and teacher educa- tors with regard to their experiences with and perceptions of the NBPTS program. Members of the Teacher Advisory Council of the National Research Council (NRC) spoke with us on numerous occasions. In addi­ tion four teachers and teacher educators attended our third meeting: Sara Eisenhardt with the Cincinnati school system, Maxine Freund and Mary Futrell at George Washington University, and Carol Matern with the Indianapolis public schools and Indiana University–Purdue University, Indianapolis. We thank these individuals for their insightful comments about the program. During the course of this project, we spoke with and heard presenta- tions from numerous individuals who conducted research on the NBPTS program. We are particularly grateful to the following researchers for their 

 ACKNOWLEDGMENTS responsiveness to our questions and the time they took to help us better understand their findings: Linda Cavaluzzo at CNA Corporation, Carol Cohen at the Finance Project, Dan Goldhaber at the University of Wash- ington, Michael Hansen at the University of Washington, Dan Humphrey at SRI International, Julia Koppich at J. Koppich Associates, David Lustick at Michigan State University, Jennifer King Rice at the University of Mary- land, and Jannese Woodward Moore at East Tennessee State University. Senior staff members of the NRC’s Division of Behavioral and Social Sciences and Education helped the committee move this project forward. Michael Feuer, executive director, enthusiastically backed the project and lent his wisdom and advice at key stages. Patricia Morison, associate ex- ecutive director and acting director of the Center for Education, provided sage advice throughout this project. Eugenia Grohman, associate executive director, lent her deep knowledge and experience with NRC procedures and the committee process. Christine McShane, senior editor, provided expert editing assistance. Kirsten Sampson Snyder, senior report review officer, ably guided the report through the NRC review process. Special thanks are due to Teresia Wilmore, senior project assistant, for her masterful handling of the logistical aspects of this project. Teresia very capably managed all of the committee meetings and is now pursuing a graduate degree in nursing—we wish her great success. We also thank Viola Horek and Dorothy Majewski, who ably stepped in to assist at various stages of the project. We are grateful to Lisa Alston, who provided support throughout the project and to Monica Ulewicz for her assistance with the literature reviews during the early stage of this project. On behalf of the committee and as its chair, I wish to recognize three members of the NRC staff who did everything needed to ensure the qual- ity and timeliness of this project. First, we are grateful to senior program officer, Alix Beatty, for her expert research, writing, and critical thinking skills. Alix provided initial drafts for several portions of the report, a ser- vice that was invaluable because it immediately propelled the committee’s deliberations. Second, Stuart Elliott, director of the Board on Testing and Assessment, provided calm and steady input at every stage of our delibera- tions. Stuart was particularly adept at helping us as individuals to under- stand ideas from others’ disciplines and to establish shared interpretations of particular research findings. Third, and deserving a solo bow, is Judith Koenig, our study director. Beginning with preparation of the committee proposal and then the vetting of nominees for membership on the committee through to the completion and publication of this report, Judy did it all. Although that is probably true for every NRC study director, what stands out about Judy are three attributes: she is very smart and learns quickly, she always listens and hears 

ACKNOWLEDGMENTS xi both what is said and what is not said, and she takes the initiative collab- oratively. While the committee could have conducted the evaluation and issued its report without her, our tasks would not have been completed as rapidly nor as well. Thank you, Judy, for your graceful leadership at the center of our efforts. Finally, I wish to thank the committee members for their dedication and outstanding contributions to this study. They drafted text, prepared background materials, reviewed numerous versions of this report, and gave generously of their time. In its first meeting, the committee adopted a pat- tern of collaborative inquiry and meticulous analysis that endured through the final signoff. Despite marked differences in our fields of expertise and differing viewpoints about what constitutes noteworthy evidence, the entire committee remained focused and engaged throughout. I am deeply grateful to each member. This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with pro- cedures approved by the NRC’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Lisa Barrow, Economic Research, Federal Reserve Bank of Chi- cago; Stephen B. Dunbar, College of Education, University of Iowa; David N. Figlio, Department of Economics, University of Florida; Robert E. Floden, Institute for Research on Teaching and Learning, College of Edu- cation, Michigan State University; Robert M. Hauser, Center for Demog- raphy of Health and Aging, University of Wisconsin–Madison; David H. Monk, Dean’s Office, College of Education, Penn State University; Gary J. Natriello, Teachers College, Columbia University; John J. Norcini, Office of the President, Foundation for Advancement of International Medical Edu- cation and Research, Philadelphia; Ruth J. Palmer, Educational Psychology, School of Education, College of New Jersey; and Mark R. Wilson, Gradu- ate School of Education, University of California, Berkeley. Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations nor did they see the final draft of the report before its release. The review of this report was overseen by Lauress Wise, HumRRO, Monterey, California, and Charles E. Phelps, university professor and pro- vost emeritus, University of Rochester. Appointed by the NRC, they were 

xii ACKNOWLEDGMENTS responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report, however, rests entirely with the authoring committee and the institution. Milton D. Hakel, Chair Committee on Evaluation of Teacher Certification by the National Board for Professional Teaching Standards 

Contents Summary 1   1 Introduction 15   2 The Evaluation Framework and Collection of Data 20   3 The Historical Context and Overview of the National Board 38   4 The Assessment Program 59   5 The Psychometric Quality of the Assessments 79   6 Teacher Participation in the Program 119   7 The Impact of Board-Certified Teachers on Student Outcomes 154   8 The Effects of the Certification Process on Practice 182   9 The Impact of Certification on Teachers’ Career Paths 198 10 The Effects of Certification on the Education System 210 xiii 

xiv CONTENTS 11  The Cost-Effectiveness of Certification as a Means of Improving Teacher Quality 223 12 Overall Evaluation 248 References 264 Appendixes A  Reviews of Studies That Provided Evidence for the Evaluation 275 B Biographical Sketches of Committee Members and Staff 315