Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
Improving Measurement of Productivity in Higher Education Panel on Measuring Higher Education Productivity: Conceptual Framework and Data Needs Teresa A. Sullivan, Christopher Mackie, William F. Massy, and Esha Sinha, Editors Committee on National Statistics Board on Testing and Assessment Division of Behavioral and Social Sciences and Education

OCR for page R1
THE NATIONAL ACADEMIES PRESS · 500 Fifth Street, NW · Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This study was supported by grant number 5793 between the National Academy of Sci- ences and Lumina Foundation. Support for the work of the Committee on National Sta- tistics is provided by a consortium of federal agencies through a grant from the National Science Foundation (award number SES-1024012). Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for this project. International Standard Book Number-13: 978-0-309-25774-9 International Standard Book Number-10: 0-309-25774-3 Library of Congress Cataloging-in-Publication data are available from the Library of Congress Additional copies of this report are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313, http://www.nap.edu. Copyright 2012 by the National Academy of Sciences. All rights reserved. Printed in the United States of America Suggested citation: National Research Council. (2012). Improving Measurement of Pro- ductivity in Higher Education. Panel on Measuring Higher Education Productivity: Con- ceptual Framework and Data Needs. Teresa A. Sullivan, Christopher Mackie, William F. Massy, and Esha Sinha, Editors. Committee on National Statistics and Board on Testing and Assessment, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

OCR for page R1
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examina- tion of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the Na- tional Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council. www.national-academies.org

OCR for page R1

OCR for page R1
PANEL ON MEASURING HIGHER EDUCATION PRODUCTIVITY: CONCEPTUAL FRAMEWORK AND DATA NEEDS TERESA A. SULLIVAN (Chair), Office of the President, University of Virginia THOMAS R. BAILEY, Institute on Education and the Economy and Community College Research Center, Teachers College, Columbia University BARRY P. BOSWORTH, Economic Studies Program, The Brookings Institution, Washington, DC DAVID W. BRENEMAN, Curry School of Education, University of Virginia RONALD G. EHRENBERG, Cornell Higher Education Research Institute, Cornell University PETER T. EWELL, National Center for Higher Education Management Systems, Boulder, CO IRWIN FELLER, Department of Economics (emeritus), Pennsylvania State University BARBARA FRAUMENI, Muskie School of Public Service, University of Southern Maine JULIET V. GARCIA, Office of the President, University of Texas at Brownsville and Texas Southmost College MICHAEL HOUT, Department of Sociology, University of California, Berkeley NATE JOHNSON, HCM Strategists, Washington, DC GEORGE D. KUH, Center for Postsecondary Research (emeritus), Indiana University WILLIAM F. MASSY, Independent Consultant, Florence, MA CAROL A. TWIGG, National Center for Academic Transformation, Saratoga Springs, NY DAVID J. ZIMMERMAN, Department of Economics, Williams College CHRISTOPHER D. MACKIE, Study Director STUART ELLIOTT, Senior Program Officer ESHA SINHA, Associate Program Officer MICHAEL SIRI, Program Associate v

OCR for page R1
COMMITTEE ON NATIONAL STATISTICS 2011-2012 LAWRENCE BROWN (Chair), Department of Statistics, The Wharton School, University of Pennsylvania JOHN M. ABOWD, School of Industrial and Labor Relations, Cornell University ALICIA CARRIQUIRY, Department of Statistics, Iowa State University WILLIAM DuMOUCHEL, Oracle Health Sciences, Waltham, MA V. JOSEPH HOTZ, Department of Economics, Duke University MICHAEL HOUT, Survey Research Center, University of California, Berkeley KAREN KAFADAR, Department of Statistics, Indiana University SALLIE KELLER, University of Waterloo, Ontario, Canada LISA LYNCH, Heller School for Social Policy and Management, Brandeis University SALLY MORTON, Graduate School of Public Health, University of Pittsburgh JOSEPH NEWHOUSE, Division of Health Policy Research and Education, Harvard University RUTH PETERSON, Criminal Justice Research Center, The Ohio State University HAL STERN, Department of Statistics, University of California, Irvine JOHN H. THOMPSON, NORC at the University of Chicago ROGER TOURANGEAU, Statistical Group, Westat, Rockville, MD ALAN ZASLAVSKY, Department of Health Care Policy, Harvard Medical School CONSTANCE F. CITRO, Director vi

OCR for page R1
BOARD ON TESTING AND ASSESSMENT EDWARD HAERTEL (Chair), Jacks Family Professor of Education and Associate Dean for Faculty Affairs, Stanford University GARY CHAMBERLAIN, Louis Berkman Professor of Economics, Harvard University MARK DYNARSKI, Researcher, Pemberton Research, LLC DAVID J. FRANCIS, Hugh Roy and Lillie Cranz Cullen Distinguished Professor, and Director, Texas Institute for Measurement, Statistics and Evaluation, University of Houston JOAN HERMAN, Director, National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles MICHAEL KANE, Messick Chair in Validity, Educational Testing Service SHARON LEWIS, Director of Research, Council of Great City Schools ROBERT MARE, Professor of Sociology, University of California, Los Angeles DIANA C. PULLIN, Professor, Boston College ANN MARIE RYAN, Professor of Psychology, Michigan State University BRIAN STECHER, Senior Social Scientist, Education Program, RAND Corporation JOHN ROBERT WARREN, Associate Professor of Sociology, University of Minnesota MARK WILSON, Professor of Policy, Organization, Measurement, and Evaluation Cognition and Development, University of California, Berkeley REBECCA ZWICK, Distinguished Presidential Appointee, Research and Development, Educational Testing Service STUART ELLIOTT, Director vii

OCR for page R1

OCR for page R1
Acknowledgments The work of this panel has been immeasurably assisted by the insight and counsel of numerous colleagues. In particular, we wish to acknowledge the reviewers. This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with pro- cedures approved by the Report Review Committee of the National Research Council (NRC). The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Julian Betts, Department of Economics, University of California, San Diego; William G. Bowen, President's Office, Andrew W. Mellon Foun- dation; Pat Callan, President's Office, Higher Education Policy Institute, San Jose, California; Charles T. Clotfelter, Center for the Study of Philanthropy and Voluntarism, Duke University; Don E. Detmer, University of Virginia School of Medicine; David N. Figlio, Institute for Policy Research, Northwestern Univer- sity; Brent R. Hickman, Economics Department, University of Chicago; Michael McPherson, President's Office, The Spencer Foundation; B. Don Russell, Jr., Department of Electrical Engineering, Texas A&M University; and Burton A. Weisbrod, Institute for Policy Research, Northwestern University. The review of this report was overseen by Greg Duncan, distinguished pro- fessor of education, University of California, Irvine, and Charles Manski, Board of Trustees professor in economics, Northwestern University. Appointed by the NRC's Report Review Committee, they were responsible for making certain that ix

OCR for page R1
x ACKNOWLEDGMENTS an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Although the reviewers have provided many constructive comments, and improved the content of the report a great deal, they were not asked to endorse the conclusions or recommendations; nor did they see the final draft of the report prior to its release. Responsibility for the final content of this report rests entirely with the authoring panel and the institution. Many others generously gave of their time in offering oral presentations at meetings and answering questions from panel members and staff, thereby helping us to develop a clearer understanding of key issues relevant to the measurement of higher education productivity and related issues. The panel thanks Lumina Foundation; they provided financial support for the project and, even more impor- tantly, helped shape the scope of the study. From Lumina, Jamie Merisotis, Kevin Corcoran, Suzanne Walsh (now with the Bill & Melinda Gates Foundation), and Charles (Chip) Hatcher provided insights and guidance in their roles as initiators of the project. Kristin Conklin (HCM Strategists, LLC) kept the panel informed about Lumina grantees' programs to increase productivity and proposals for measuring the effectiveness of those efforts. The panel benefited from the open discussion of these initiatives. During meetings and deliberations, the panel heard from a number of subject matter experts. Andrea Bonaccorsi, University of Pisa, Italy; Kevin Carey, New America Foundation; Hamish Coates, Australian Council for Education Research; Bo Hansson, OECD; Jorge Klor de Alva, University of Phoenix; and Donna Sundre, James Madison University informed the panel about efforts to measure higher education productivity, to design and implement accountability systems, and to improve input/output data at different levels of aggregation. The panel could not have conducted its work without an excellent and well- managed staff. Connie Citro, director of the Committee on National Statistics, and Stuart Elliott, director of the Board on Testing and Assessment, provided expert guidance to the panel about the NRC study process. Program associate Michael Siri provided excellent administrative, editorial, and research support. Esha Sinha, program officer, provided valuable research and analytic assistance with her understanding of higher education data sources. Her knowledge proved especially helpful as we worked through some of the thornier measurement issues. The panel also benefited from the work of Kirsten Sampson-Snyder, Division of Behavioral and Social Sciences and Education, who was responsible for overseeing the review process. Amy Smith provided able editing of numer- ous drafts. Christopher Mackie, the panel's study director, organized our meetings and facilitated communication among panel members, including a lengthy process of chapter revisions. His work required synthesizing and evaluating many disparate points of input, seeking what common ground could be found, and guiding the panel through careful discussion of the points of disagreement. He helped to

OCR for page R1
ACKNOWLEDGMENTS xi develop the structure for the panel's final report, and he shepherded the report through the final review process. Most importantly, the members of the panel deserve thanks for their patience, creativity, and hard work. There is a reason that higher education productivity is not currently reported in the national accounts. Reaching agreement on concep- tual and measurement issues was difficult work. This report reflects the collec- tive expertise and commitment of the individual members of the panel, each of whom brought a unique perspective based upon a scholarly discipline, research experience, and a lifetime of practice. Members were generous with their time and effort, and they struggled to understand and appropriately acknowledge the critical views of others. Our meetings provided many opportunities for panel members to learn from one another. Teresa A. Sullivan, Chair Panel on Measuring Higher Education Productivity: Conceptual Framework and Data Needs

OCR for page R1

OCR for page R1
Contents SUMMARY 1 Motivation and Panel Charge, 1 The Productivity Measure, 2 Measurement Limitations and Key Areas for Model Enhancement, 3 Joint Production, 3 Quality Variation and Change, 4 Nonmarket Production, 5 Segmentation by Institution Type, 6 Implications of Complexities for Measurement Prospects, 6 Developing the Data Infrastructure, 7 1THE IMPORTANCE OF MEASURING PRODUCTIVITY IN HIGHER EDUCATION 9 1.1.Social and Policy Context, 10 1.2.Charge to the Panel, 13 1.3.Audience and Report Structure, 17 2 DEFINING PRODUCTIVITY FOR HIGHER EDUCATION 19 2.1.Basic Concepts, 21 2.1.1. Outputs, 23 2.1.2. Inputs, 25 2.1.3. Instructional and Noninstructional Elements of the Higher Education Production Function, 29 2.2.Productivity Contrasted with Other Measurement Objectives, 31 2.2.1. Productivity and Cost, 31 2.2.2. Other Performance Metrics, 33 xiii

OCR for page R1
xiv CONTENTS 3WHY MEASUREMENT OF HIGHER EDUCATION PRODUCTIVITY IS DIFFICULT 37 3.1. Beyond the Degree Factory--Multiple Outputs and Joint Production, 38 3.2.Heterogeneity of Inputs and Outputs, 40 3.3.Nonmarket Variables and Externalities, 43 3.4.Quality Change and Variation, 44 3.4.1. Inputs, 45 3.4.2. Outputs (and Outcomes), 50 3.5. Measurement at Different Levels of Aggregation, 55 3.5.1. Course and Department Level, 55 3.5.2. Campus Level, 57 3.5.3. State or System Level, 58 3.6. Conclusion, 60 4 ADVANCING THE CONCEPTUAL FRAMEWORK 61 4.1. Chapter Overview, 61 4.2.A Baseline Multi-Factor Productivity Model for Higher Education, 63 4.2.1. Multi-Factor Productivity Indices, 64 4.2.2. Outputs, 65 4.2.3. Inputs, 67 4.2.4. Allocations to Education, 68 4.2.5. Illustrative Productivity Calculations, 69 4.3. Institutional Segmentation and Disaggregative Indices, 71 4.3.1. Institutional Segmentation, 72 4.3.2. State-Level and Single-Institution Indices, 73 4.4.Differentiating Labor Categories, 74 4.5.Differentiating Outputs, 78 4.6.Variations in Output Quality, 79 Technical Appendix: The Törnqvist Productivity Index, 82 5RECOMMENDATIONS FOR CREATING AND EXTENDING THE MEASUREMENT FRAMEWORK 87 5.1.The Basic Productivity Measure, 89 5.1.1. Instructional Outputs and Benefits, 90 5.1.2. Instructional Inputs and Costs, 93 5.2.Adjusting for Research Production, 95 5.2.1. Project-Driven Departmental Research, 97 5.2.2. Discretionary Departmental Research, 98 5.3.Dealing with Heterogeneity and Quality Issues, 99 5.3.1. Variation of Inputs, 100 5.3.2. Quality Variation and Change of Outputs, 103

OCR for page R1
CONTENTS xv 6 IMPLEMENTATION AND DATA RECOMMENDATIONS 107 6.1.General Strategies, 107 6.2.Recommendations for Improving the Data Infrastructure, 109 6.2.1. Data Demanded by the Conceptual Framework, 109 6.2.2. Envisioning the Next Generation IPEDS, 110 6.2.3. Administrative Data Sources, 114 6.2.4. Survey-Based Data Sources, 120 REFERENCES AND BIBLIOGRAPHY 125 APPENDIXES A Commonly Used Performance Metrics for Higher Education 137 B Methods for Measuring Comparative Quality and Cost Developed by the National Center for Academic Transformation 145 C Overview of Data Sources 151 D Estimating Project-Related Departmental Research 203 E Biographical Sketches of Panel Members 205

OCR for page R1