THE IMPORTANCE OF COMMON METRICS FOR ADVANCING SOCIAL SCIENCE THEORY AND RESEARCH

A Workshop Summary

Rose Maria Li, Rapporteur

Committee on Advancing Social Science Theory: The Importance of Common Metrics

Committee on Social Science Evidence for Use

Division of Behavioral and Social Sciences and Education

NATIONAL RESEARCH COUNCIL
OF THE NATIONAL ACADEMIES

THE NATIONAL ACADEMIES PRESS

Washington D.C.
www.nap.edu



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
THE IMPORTANCE OF COMMON METRICS FOR ADVANCING SOCIAL SCIENCE THEORY AND RESEARCH A Workshop Summary Rose Maria Li, Rapporteur Committee on Advancing Social Science Theory: The Importance of Common Metrics Committee on Social Science Evidence for Use Division of Behavioral and Social Sciences and Education

OCR for page R1
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Govern- ing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineer- ing, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropri- ate balance. This study was supported by contract number 2008-2146 between the National Academy of Sciences and the William and Flora Hewlett Foundation. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for this project. International Standard Book Number-13: 978-0-309-16300-2 International Standard Book Number-10: 0-309-16300-5 Additional copies of this report are available from National Academies Press, 500 Fifth Street, N.W., Lockbox 285, Washington, D.C. 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu. Copyright 2011 by the National Academy of Sciences. All rights reserved. Printed in the United States of America Suggested citation: National Research Council. (2011). The Importance of Common Metrics for Advancing Social Science Theory and Research: A Workshop Summary. Rose Maria Li, Rapporteur. Committee on Advancing Social Science Theory: The Importance of Common Metrics. Committee on Social Science Evidence for Use. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

OCR for page R1
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Acad- emy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding en- gineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineer- ing programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is presi- dent of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Insti- tute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sci- ences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Coun- cil is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council. www.national-academies.org

OCR for page R1

OCR for page R1
COMMITTEE ON ADVANCING SOCIAL SCIENCE THEORY: THE IMPORTANCE OF COMMON METRICS George W. Bohrnstedt (Chair), American Institutes for Research, Palo Alto, California Norman M. Bradburn, National Opinion Research Center and University of Chicago Nancy D. Cartwright, London School of Economics and Political Science and University of California, San Diego Harris Cooper, Department of Psychology and Neuroscience, Duke University Robert M. Hauser, Division of Behavioral and Social Sciences and Education, National Research Council, Washington, DC, and Vilas Research Professor, Emeritus, University of Wisconsin, Madison Robert A. Pollak, John M. Olin School of Business, Washington University, St. Louis, Missouri Barbara Schneider, College of Education, Michigan State University Miron L. Straf, Study Director Mary Ann Kasper, Senior Program Assistant Rose Maria Li, Rapporteur Dorothy Majewski, Administrative Assistant Christina Maranto, Mirzayan Fellow v

OCR for page R1
COMMITTEE ON SOCIAL SCIENCE EVIDENCE FOR USE Kenneth Prewitt (Chair), School of International and Public Affairs, Columbia University, New York George W. Bohrnstedt, American Institutes for Research, Palo Alto, California Norman M. Bradburn, National Opinion Research Center and University of Chicago Alicia Carriquiry, Department of Statistics, Iowa State University Nancy D. Cartwright, London School of Economics and Political Science and University of California, San Diego Harris Cooper, Department of Psychology and Neuroscience, Duke University Michael J. Farrell, New York City Police Department Stephen E. Fienberg, Department of Statistics, Carnegie Mellon University Sheila Jasanoff, Kennedy School of Government, Harvard University Robert L. Jervis, Columbia University, New York Robert E. Litan, Ewing Marion Kauffman Foundation Ann Morning, Department of Sociology, New York University Robert A. Pollak, John M. Olin School of Business, Washington University, St. Louis, Missouri Stephen H. Schneider (Deceased July 2010), Stanford University Thomas A. Schwandt, Department of Educational Psychology, University of Illinois, Urbana-Champaign Sidney Verba, Pforzheimer University Professor, Emeritus, Department of Government, Harvard University Miron L. Straf, Study Director vi

OCR for page R1
Preface In February 2010, the National Research Council (NRC) convened a workshop to investigate the feasibility of developing well-grounded com- mon metrics to advance behavioral and social science research, both in terms of advancing the development of theory and increasing the utility of research for policy and practice. A planning committee was appointed by the NRC’s Division of Behavioral and Social Sciences and Education (DBASSE) to organize the event, structure the sessions, select the partici- pants, and ensure that the workshop would address the variety of research methods and data sets. The workshop would not have been possible without the generous support and leadership provided by the William and Flora Hewlett Foun- dation. Marshall S. Smith, while at Hewlett, proposed the topic of com- mon metrics as one in need of exploration. We are especially grateful to the planning committee members and other experts who responded to our request for background papers: Norman M. Bradburn, National Opinion Research Center and the University of Chicago; Nancy D. Cartwright, Lon- don School of Economics and University of California, San Diego; Dennis Fryback, University of Wisconsin, Madison; David B. Grusky, Stanford University; Robert M. Hauser, Division of Behavioral and Social Sciences and Education, National Research Council, Washington, DC, and Vilas Re- search Professor, Emeritus, University of Wisconsin, Madison; Rick Hoyle, Duke University; Robert T. Michael, University of Chicago; Geoff Mulgan, The Young Foundation; Robert A. Pollak, Washington University, St. Louis, Missouri; C. Matthew Snipp, Stanford University; John Robert Warren, vii

OCR for page R1
viii PREFACE University of Minnesota; and Robert J. Willis, University of Michigan. Their papers provided a substantive context for the discussions that took place at the workshop. We also thank the many other people who participated as presenters, panelists, and discussants: Christine A. Bachrach, Duke University and University of Maryland; Kathleen A. Cagney, University of Chicago; Harris Cooper, Duke University; Sheila Jasanoff, Harvard University; Rebecca A. Maynard, University of Pennsylvania; Kenneth Prewitt, Columbia Univer- sity; Barbara Schneider, Michigan State University; and Jack E. Triplett, Brookings Institution. In the preparation of this workshop summary, we thank Rose Maria Li, who acted as rapporteur. In addition, Mary Lou Rife was helpful in the drafting of one of the chapters, and Christine McShane provided expert editing services for this report. For a fuller list of sources on the topic than is included in this report, see the papers presented at the workshop: http://www7.nationalacad- emies.org/dbasse/Workshop_on_Common_Metrics_Agenda.html. For later versions of the papers, readers should contact the authors or look for a separate volume of the papers that is in preparation for submission to a university press. This workshop summary has been reviewed in draft form by individu- als chosen for their diverse perspectives and technical expertise, in accor- dance with procedures approved by the NRC’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the charge. The review comments and draft manuscript remain confidential to protect the integ- rity of the process. We thank the following individuals for their review of this report: George W. Bohrnstedt, Research Division, American Institutes for Research; David S. Johnson, Social, Economic, and Housing Statistics Division, U.S. Census Bureau; and Howard J. Silver, Director’s Office, Con- sortium of Social Science Associations. Although the reviewers listed above provided many constructive com- ments and suggestions, they were not asked to endorse the content of the report, nor did they see the final draft of the report before its release. The review of this report was overseen by Cora B. Marrett, acting deputy direc- tor, National Science Foundation. Appointed by the NRC, she was respon- sible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review elements were carefully considered. Responsibility for the final content of this report rests entirely with the author and the institution.

OCR for page R1
ix PREFACE We also gratefully acknowledge the contributions of Dorothy Majewski, administrative assistant; Mary Ann Kasper, senior program as- sistant; Kirsten Sampson Snyder, senior report review officer; Christine Maranto, Christine Mirzayan Science and Technology Policy Graduate Fellow; Catherine Freeman, who, as initial study director, helped in the de- velopment of the workshop; and Michael J. Feuer, former executive director of the NRC’s DBASSE, for his leadership and support. George W. Bohrnstedt, Chair Miron L. Straf, Study Director Committee on Advancing Social Science Theory: The Importance of Common Metrics

OCR for page R1

OCR for page R1
Contents 1 Introduction 1 Workshop Goals and Issues, 1 About This Report, 5 2 Measurement in the Social Sciences 7 Measurement Standardization in the Physical Sciences, 7 Measurement Standardization in the Social Sciences, 8 Comparable Metrics: Some Examples, 9 Discussion, 15 What Can Be Learned from the Economic Sciences?, 18 Measuring Health-Related Quality of Life, 22 Discussion, 25 Open Discussion, 28 3 Indicators 31 The Standardization of Indicators Used in Policy, 31 Standardized Measurement, 36 High School Completion Rates, 39 A Common Metric for Race and Ethnicity?, 42 Discussion, 46 Open Discussion, 49 4 Social Science Constructs 53 The Theory of Measurement, 53 Measuring Poverty: The Question of Standardization, 56 xi

OCR for page R1
xii CONTENTS A National Protocol for Measuring Intergenerational Mobility?, 58 Discussion, 60 Measuring and Modeling of Self-Regulation: Is Standardization a Reasonable Goal?, 61 Discussion, 65 Open Discussion, 66 5 Final Comments 71 References 77 Appendixes A 81 Workshop Agenda and Participants B Biographical Sketches of Committee Members,Workshop 89 Speakers, and Workshop Discussants