QUANTITATIVE ASSESSMENTS OF THE PHYSICAL AND MATHEMATICAL SCIENCES

A Summary of Lessons Learned

Commission on Physical Sciences, Mathematics, and Applications

National Research Council

National Academy Press
Washington, D.C. 1994



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
QUANTITATIVE ASSESSMENTS OF THE PHYSICAL AND MATHEMATICAL SCIENCES: A Summary of Lessons Learned QUANTITATIVE ASSESSMENTS OF THE PHYSICAL AND MATHEMATICAL SCIENCES A Summary of Lessons Learned Commission on Physical Sciences, Mathematics, and Applications National Research Council National Academy Press Washington, D.C. 1994

OCR for page R1
QUANTITATIVE ASSESSMENTS OF THE PHYSICAL AND MATHEMATICAL SCIENCES: A Summary of Lessons Learned NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This report has been reviewed by a group other than the authors according to procedures approved by a Report Review Committee consisting of members of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce Alberts is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Robert M. White is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine. The National Research Council was established by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce Alberts and Dr. Robert M. White are chairman and vice chairman, respectively, of the National Research Council. Support for this project was provided by the Basic Science Fund of the National Academy of Sciences, whose contributors include AT& T Bell Laboratories, Atlantic Richfield Foundation, BP America, Inc., Dow Chemical Company, E.I. du Pont de Nemours and Company, IBM Corporation, Merck and Company, Inc., Monsanto Company, and Shell Oil Companies Foundation. Additional copies of this report are available from: Commission on Physical Sciences, Mathematics, and Applications National Research Council, NAS 285 2101 Constitution Avenue, NW Washington, DC 20418 202-334-3061 Copyright 1994 by the National Academy of Sciences. All rights reserved. Printed in the United States of America

OCR for page R1
QUANTITATIVE ASSESSMENTS OF THE PHYSICAL AND MATHEMATICAL SCIENCES: A Summary of Lessons Learned COMMISSION ON PHYSICAL SCIENCES, MATHEMATICS, AND APPLICATIONS RICHARD N. ZARE, Stanford University, Chair RICHARD S. NICHOLSON, American Association for the Advancement of Science, Vice Chair STEPHEN L. ADLER, Institute for Advanced Study JOHN A. ARMSTRONG, * IBM Corporation (retired) SYLVIA T. CEYER, Massachusetts Institute of Technology GEORGE W. CLARK, ** Massachusetts Institute of Technology AVNER FRIEDMAN, ** University of Minnesota SUSAN L. GRAHAM, University of California at Berkeley ROBERT J. HERMANN, United Technologies Incorporated RHONDA J. HUGHES, Bryn Mawr College SHIRLEY A. JACKSON, Rutgers University KENNETH I. KELLERMANN, National Radio Astronomy Observatory NEAL LANE, ** Rice University HANS MARK, University of Texas at Austin CLAIRE E. MAX, * Lawrence Livermore National Laboratory CHRISTOPHER F. McKEE, * University of California at Berkeley JAMES W. MITCHELL, * AT&T Bell Laboratories THOMAS A. PRINCE, California Institute of Technology JEROME SACKS, National Institute of Statistical Sciences L.E. SCRIVEN, University of Minnesota A. RICHARD SEEBASS III, University of Colorado LEON T. SILVER, California Institute of Technology CHARLES P. SLICHTER, University of Illinois at Urbana-Champaign ALVIN W. TRIVELPIECE, Oak Ridge National Laboratory SHMUEL WINOGRAD, IBM T.J. Watson Research Center CHARLES ZRAKET, The MITRE Corporation (retired) National Research Council Staff Norman Metzger, Executive Director Paul F. Uhlir, Associate Executive Director Julie M. Esanu, Administrative Assistant * Members of the Commission whose terms ended June 30, 1994. ** Members of the Commission who resigned during 1994.

OCR for page R1
QUANTITATIVE ASSESSMENTS OF THE PHYSICAL AND MATHEMATICAL SCIENCES: A Summary of Lessons Learned This page in the original is blank.

OCR for page R1
QUANTITATIVE ASSESSMENTS OF THE PHYSICAL AND MATHEMATICAL SCIENCES: A Summary of Lessons Learned Preface Calls for assessing the state of scientific fields are not new. Indeed, we are witnessing a period of increased introspection, especially with respect to evaluating the effectiveness of federal investments in research and development. Significant recent examples of this emphasis include the 1991 report of the Office of Technology Assessment, Federally Funded Research: Decisions for a Decade, and the Government Performance and Results Act of 1993 (PL 103-62). The quandaries are many. Who should do the assessing, and what criteria are to be used? What should be measured, and how do these measurements connect to desired outcomes? How are scientists to be held accountable for what they do? The scientific community, principally through the National Research Council, has issued major studies of the status and directions of scientific fields and subfields in the past. Priorities in terms of the most promising research opportunities have been identified and have led in many cases to enhanced federal investments in these areas and reductions in areas judged less promising. Left undone, however, is a response to a central issue of scientific policy, namely, the setting of priorities across fields. As yet, the scientific community has not found a satisfactory method for accomplishing this task, although cross-priorities are of course being set by the federal political and budgetary process. The Commission on Physical Sciences, Mathematics, and Applications confronted this issue of determining cross-priorities and judged that what seemed potentially promising was to assess the capacity of a field to retain its strength, adapt to new opportunities, connect with other fields, and the like—a set of criteria labeled as the “health” of a field or subfield. The hope was that if common criteria for assessment could be discovered and validated, then a comparison among different fields would guide the scientific community as well as the patrons of that community in setting priorities. As our study has demonstrated, however, “health” assessments are harder to make than we initially thought. For example, it is easy to posit that what is bigger is better, but when the bigger must become smaller, the affected community often will marshal measures to show that the health of the field is deteriorating when the contrary may actually be the case. Past experience teaches us, for example, that a

OCR for page R1
QUANTITATIVE ASSESSMENTS OF THE PHYSICAL AND MATHEMATICAL SCIENCES: A Summary of Lessons Learned small fraction of the workers in any given field produce the majority of output of value. Increasing the number of workers in a field may thus actually reduce the average output per worker. It is seductive to imagine that a set of metrics exist that, when tracked over time, will enable predicting with confidence the future prospects of a scientific field or subfield. Whereas some indicators may be useful for judging the normal pace of progress in a field as it follows its established paradigm, no evidence could be found that suggests that such statistical measures have long-range predictive power. How can we know whether some supposedly dull field in stasis or even in decline is on the verge of a remarkable rebirth that will result from an unexpected experimental discovery or from some new and vital idea coming from the work of just one individual? We must not let a cycle be created in which the need for accountability leads to the use of measurement standards, which leads to polishing existing paradigms, which leads to further demands for accountability, and so on, with the effect that this cycle drives out investment in possible paradigm shifts. Put another way, we must avoid “looking for the lost key where the light is brightest”: we must avoid allowing what can be measured to become what matters, rather than seeking to measure what matters—which frequently are attributes that cannot be quantified. A means must be found to judge when an incremental investment in a field is likely to yield only a minimal return in polishing the edges of an existing mode of research and when it will lead to ground-breaking advances. Thus, it must be granted that discipline assessments require both quantitative as well as qualitative measures, which the Commission believed when it set out to test its proposal for assessing the health of a field, and has now confirmed through this experiment. In initiating this study, the Commission proposed a set of candidate indicators of “health” and established three discipline panels to review them in the areas of mathematics, astronomy, and atomic, molecular, and optical science. This report summarizes the principal lessons learned from this experiment. I believe the study substantiates what many already knew intuitively. A large chasm exists between asking for performance measures or other indicators of the status of a discipline, and getting ones that are reliable and usable. Moreover, often unanticipated difficulties arise concerning the availability of data, the stability over time of ostensibly the same databases, efforts to compare what seem to be the same data, albeit collected by different groups or agencies, and efforts to interpret these data. In some ways this experiment may be regarded as reporting negative results. It also makes no recommendations. In other ways, however, the Commission hopes that this report will be regarded as positive in that it indicates that a premium should be placed on wise judgment by informed individuals. The Commission believes that the findings should be reported and discussed as both a warning and a guidepost for future efforts to judge the effectiveness of public investments in science. Richard N. Zare Chair, Commission on Physical Sciences, Mathematics, and Applications

OCR for page R1
QUANTITATIVE ASSESSMENTS OF THE PHYSICAL AND MATHEMATICAL SCIENCES: A Summary of Lessons Learned Acknowledgments The Commission on Physical Sciences, Mathematics, and Applications is grateful to the many individuals who participated in this pilot project, including the members of the three ad hoc panels that assessed the Commission's proposed indicators of the health of a discipline and related data sources; the participants in the Commission's August 1993 workshop, who helped integrate the results of the panels' work; and members of the National Research Council (NRC) staff who worked on various aspects of the study. The Mathematics Assessment Panel was chaired by Shmuel Winograd of the IBM T.J. Watson Research Center. The panel members were Michael Artin, Massachusetts Institute of Technology; Ramesh Gangolli, University of Washington; Ramanathan Gnanadesikan, Rutgers University; Deborah Tepper Haimo, University of Missouri at St. Louis; Richard Herman, University of Maryland; George Nemhauser, Georgia Institute of Technology; Bruce Spencer, Northwestern University; and Mary Wheeler, Rice University. Arthur Code, of the University of Wisconsin, chaired the Astronomy Assessment Panel. Peter Boyce, American Astronomical Society; Robert Brown, Space Telescope Science Institute; Claude Canizares, Massachusetts Institute of Technology; Frank Drake, University of California at Santa Cruz; Andrea Dupree, Harvard-Smithsonian Center for Astrophysics; Richard Lingenfelter, University of California at San Diego; Richard McCray, Joint Institute Laboratory for Astrophysics; Robert Rosner, University of Chicago; and Harry Shipman, University of Delaware, were members of this panel. The Atomic, Molecular, and Optical (AMO) Science Assessment Panel was chaired by F. Barrymore Dunning of Rice University. Panel members were Lloyd Armstrong, Jr., University of Southern California; Alexander Dalgarno, Harvard-Smithsonian Center for Astrophysics; Elsa Garmire, University of Southern California; Daniel Larson, University of Virginia; Paul Foo-Hung Liao, Bellcore; Thomas McIlrath, University of Maryland; Ronald Phaneuf, University of Nevada; and Roberta Saxon, SRI International. The Commission also would like to extend its thanks to the participants of the Workshop on the Assessment of the Health of the Physical and Mathematical Sci-

OCR for page R1
QUANTITATIVE ASSESSMENTS OF THE PHYSICAL AND MATHEMATICAL SCIENCES: A Summary of Lessons Learned ences, which was held in Washington, D.C., in August 1993: Richard Nicholson (Chair), American Association for the Advancement of Science; John Andelin, Office of Technology Assessment (retired); Darrell Beschen, Department of Energy; Jennifer Bond, National Science Foundation; Joseph Clark, Department of Commerce; Arthur Code, University of Wisconsin; Alan Fechter, NRC's Office of Scientific and Engineering Personnel; Barry Gold, House Committee on Science, Space, and Technology; Marcel LaFollette, George Washington University; Daniel Larson, University of Virginia; James McCullough, National Science Foundation; Jerome Sacks, National Institute of Statistical Sciences; and Shmuel Winograd, IBM T.J. Watson Research Center. Finally, the Commission is grateful to the staff of the National Research Council: Paul F. Uhlir, associate executive director of the Commission, who served as study director and workshop organizer; Hans Oser, Board on Mathematical Sciences, and Robert Riemer, Ronald Taylor, and Anne Simmons, Board on Physics and Astronomy, who provided staff support for the three panels; Julie Esanu and Jennifer Young, for the program assistance they provided to the panels and for the preparation of the panels' reports and final manuscript; Alan Fechter, Pamela Ebert Flattau, Susan Mitchell, Prudence Brown, Daniel Pasquini, George Orvis, and George Boyce, Office of Scientific and Engineering Personnel, who provided advice and support during the course of the study; and Susan Maurizi, who edited the final report.