Assessing Evaluation Studies

The Case of Bilingual Education Strategies

Michael M. Meyer and Stephen E. Fienberg, Editors

Panel to Review Evaluation Studies of Bilingual Education

Committee on National Statistics

Commission on Behavioral and Social Sciences and Education

National Research Council

NATIONAL ACADEMY PRESS
Washington, D.C.
1992



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
Assessing Evaluation Studies: The Case of Bilingual Education Strategies Assessing Evaluation Studies The Case of Bilingual Education Strategies Michael M. Meyer and Stephen E. Fienberg, Editors Panel to Review Evaluation Studies of Bilingual Education Committee on National Statistics Commission on Behavioral and Social Sciences and Education National Research Council NATIONAL ACADEMY PRESS Washington, D.C. 1992

OCR for page R1
Assessing Evaluation Studies: The Case of Bilingual Education Strategies NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competencies and with regard for appropriate balance. This report has been reviewed by a group other than the authors according to procedures approved by a Report Review Committee consisting of members of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Frank Press is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Robert M. White is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Frank Press and Dr. Robert M. White are chairman and vice chairman, respectively, of the National Research Council. This project was supported by funds from the U.S. Department of Education. Library of Congress Catalog Card No. 92-64011 International Standard Book Number 0-309-04728-5 Additional Copies of this report are available from: National Academy Press 2101 Consitution Avenue N.W. Washington, D.C. 20418 S621 Printed in the United States of America First Printing, June 1992 Second Printing, November 1992

OCR for page R1
Assessing Evaluation Studies: The Case of Bilingual Education Strategies Panel to Review Evaluation Studies of Bilingual Education STEPHEN E. FIENBERG (Chair), Department of Mathematics and Statistics and Osgoode Hall Law School, York University BARBARA F. FREED, College of Humanities and Social Sciences, Carnegie Mellon University KENJI HAKUTA, School of Education, Stanford University LYLE V. JONES, Department of Psychology, University of North Carolina KATHRYN B. LASKEY, School of Information Technology and Engineering, George Mason University LUIS C. MOLL, School of Education, University of Arizona P. DAVID PEARSON, College of Education, University of Illinois JOHN E. ROLPH, The RAND Corporation, Santa Monica, California PAUL R. ROSENBAUM, Department of Statistics, The Wharton School, University of Pennsylvania DONALD B. RUBIN, Department of Statistics, Harvard University KEITH F. RUST, Westat, Inc., Rockville, Maryland BURTON H. SINGER, Department of Epidemiology and Public Health, Yale University HERBERT L SMITH, Population Studies Center, University of Pennsylvania MICHAEL M. MEYER, Study Director (Carnegie Mellon University) MICHELE L. CONRAD, Senior Project Assistant

OCR for page R1
Assessing Evaluation Studies: The Case of Bilingual Education Strategies Committee on National Statistics 1991–1992 BURTON H. SINGER (Chair), Department of Epidemiology and Public Health, Yale University NORMAN M. BRADBURN, National Opinion Research Center, University of Chicago MARTIN H. DAVID, Department of Economics, University of Wisconsin ANGUS S. DEATON, Woodrow Wilson School of Public and International Affairs, Princeton University NOREEN GOLDMAN, Office of Population Research, Princeton University LOUIS GORDON, Department of Mathematics, University of Southern California JOEL B. GREENHOUSE, Department of Statistics, Carnegie Mellon University ROBERT M. HAUSER, Department of Sociology, University of Wisconsin GRAHAM KALTON, Westat, Inc., Rockville, Maryland WILLIAM A. MORRILL, Mathtech, Inc., Princeton, New Jersey JANET L. NORWOOD, The Urban Institute, Washington, D.C. DOROTHY P. RICE, Department of Social and Behavioral Sciences, School of Nursing University of California, San Francisco JOHN E. ROLPH, The RAND Corporation, Santa Monica, California DONALD B. RUBIN, Department of Statistics, Harvard University MIRON L. STRAF, Director

OCR for page R1
Assessing Evaluation Studies: The Case of Bilingual Education Strategies Contents     Preface   vii 1   Bilingual Education in the United States   1     Research Context   1     Policy Context   2     The Present Studies   6     Overview of the Panel's Report   8     References   10 2   Research Methodology and Bilingual Education   12     Determining Causes   12     Sources of Data   14     Treatments and Outcomes   20     Developing a Research and Evaluation Program   22     References   22 3   The Longitudinal Study   24     Overview   24     The Descriptive Phase   27     The Longitudinal Phase   37     Further Analyses   56     Prospects for the Future   59     Longitudinal Study Research Questions   62     References   65

OCR for page R1
Assessing Evaluation Studies: The Case of Bilingual Education Strategies 4   The Immersion Study   67     Overview   67     Data Analysis and Interpretation   74     Suggestions for the Future   84     References   88 5   Lessons for the Future   90     Specifying Objectives for Bilingual Education   91     Developing Interventions from Theory   92     Designing and Implementing Experiments   95     References   99 6   Conclusions and Recommendations   101     Summaries   101     The Panel's Conclusions and Recommendations   103     Appendices     A   Bilingual Education in the United States   107 B   Trajectory Analysis of Matched Percentiles   113 C   Acronyms   119 D   Biographical Sketches of Panel Members and Staff   120     References   124

OCR for page R1
Assessing Evaluation Studies: The Case of Bilingual Education Strategies Preface In October of 1990 the United States Department of Education requested the National Research Council to convene a panel to review two major evaluation studies of bilingual education. The panel was formed in early 1991 and given the following charge: The Panel to Review Evaluation Studies of Bilingual Education will review and assess the methodology of data collection and analysis of two major studies to evaluate bilingual education programs: a national longitudinal study of the effectiveness of instruction of limited-English-proficient students and a study to compare the effectiveness of three different instructional strategies for bilingual education for such students. The three strategies are immersion (teachers understand Spanish, but respond in English), early exit (students are placed in classes conducted in English as soon as possible), and late exit (both languages are maintained and a proficiency in English is developed over time). In the first phase of its study, the panel will: review the methods of data collection and analysis for potential sources of error and assess implications for some of the principal findings; assess whether additional analyses of the data from either study would strengthen or broaden the findings and, if so, recommend some analyses that could be carried out; and suggest alternative ways to compare the different instructional strategies and provide advice to the Department of Education Planning and Evaluation Service on commissioning and managing similar evaluation studies in the future.

OCR for page R1
Assessing Evaluation Studies: The Case of Bilingual Education Strategies If, in the panel's assessment, additional data analyses are warranted, and if the Department of Education and the panel agree, the panel would, in the second phase of its study, commission and review the analyses. The two studies reviewed are The National Longitudinal Study of the Evaluation of the Effectiveness of Services for Language-Minority Limited-English-Proficient Students and The Longitudinal Study of Immersion Strategy, Early-exit and Late-exit Transitional Bilingual Education Programs for Language-Minority Children. Throughout the report the panel has adopted the words Longitudinal Study to describe the first study and Immersion Study to describe the second study. These terms are misleading, but they were the language that the Department of Education adopted in its description of the studies. For the record, the panel notes that both of the studies are longitudinal and that the “immersion” study does not have a real immersion component. The panel's conclusions and recommendations are presented in Chapter 6. This is a very short chapter and readers who have a limited amount of time may wish to turn there first. The panel first met in March of 1991, shortly after the final report for the Immersion Study was made available. Subsequent meetings were held in June and December of 1991. This report is the result of the panel's deliberations. At the outset we thank the panel for its dedication to this project and its willingness to work in a very short time frame. Many people made generous contributions to the work of the panel. We thank the many staff members of the U.S. Department of Education who attended our meetings, answered our innumerable questions, and provided mountains of reports, interim reports and background material. In particular, the efforts of James English and David Moguel were invaluable. The contractors for the two studies, Development Associates and Research Triangle Institute for the Longitudinal Study and Aguirre International for the Immersion Study, attended our first meeting, provided us with background materials and responded to many questions. Graham Burkheimer of Research Triangle Institute, Malcolm Young of Development Associates, and David Ramirez of Aguirre International were especially helpful. Wallace Lambert of McGill University shared with the panel his experiences in the Canadian immersion studies; his discussions helped to set the stage for many of our own deliberations. The Departments of Statistics and of Academic Computing and Media at Carnegie Mellon University provided release time to allow Michael Meyer to direct the study and also provided excellent computing support. Miron Straf, director of Committee on National Statistics, was always available to answer our questions and to assist us through the operations of the National Research Council. The Committee on National Statistics provided valuable input into the report, and in particular encouraged us to expand the scope of our comments to the general domain of educational research.

OCR for page R1
Assessing Evaluation Studies: The Case of Bilingual Education Strategies Finally, we would like to thank the staff of the Committee on National Statistics, especially our senior project assistant, Michele Conrad. Michele kept the panel informed, organized meetings, provided editorial assistance, suffered our changing schedules, and still managed to keep a cheery disposition. The report would have taken a lot longer to complete without her assistance. Stephen E. Fienberg, Chair Michael M. Meyer, Study Director Panel to Review Evaluation Studies of Bilingual Education

OCR for page R1
Assessing Evaluation Studies: The Case of Bilingual Education Strategies This page in the original is blank.