NATIONAL ACADEMY PRESS
2101 Constitution Avenue, N.W. Washington, D.C. 20418
NOTICE: This volume was produced as part of a project approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. It is a result of work done by the Committee on Science, Engineering, and Public Policy (COSEPUP) as augmented, which has authorized its release to the public. This report has been reviewed by a group other than the authors according to procedures approved by COSEPUP and the Report Review Committee.
The National Academy of Sciences (NAS) is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Under the authority of the charter granted to it by Congress in 1863, the Academy has a working mandate that calls on it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the NAS.
The National Academy of Engineering (NAE) was established in 1964, under the charter of the NAS, as a parallel organization of distinguished engineers. It is autonomous in its administration and in the selection of members, sharing with the NAS its responsibilities for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. William A. Wulf is interim president of the NAE.
The Institute of Medicine (IOM) was established in 1970 by the NAS to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the NAS in its congressional charter to be an adviser to the federal government and, on its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the IOM.
The Committee on Science, Engineering, and Public Policy (COSEPUP) is a joint committee of the NAS, the NAE, and the IOM. It includes members of the councils of all three bodies.
This project has been funded with federal funds from the National Science Foundation under contract number STI-9523520. The contents of this publication do not necessarily reflect the views or policies of the National Science Foundation, nor does mention of trade names, commercial products or organizations imply endorsement by the U.S. Government.
Internet Access: This report is available on the National Academy of Sciences Internet host. It may be accessed via World Wide Web at http://www.nas.edu.
Copies of the final report are available from: National Academy Press
, 2101 Constitution Avenue, N.W., Box 285 , Washington, D.C. 20055 ; 1-800-624-6242 or 202-334-3313 (in Washington metropolitan area).
Copyright 1996 by the National Academy of Sciences . All rights reserved. This document may be reproduced solely for educational purposes without the written permission of the National Academy of Sciences.
Printed in the United States of America
PANEL TO EVALUATE THE NATIONAL SCIENCE FOUNDATION'S SCIENCE AND TECHNOLOGY CENTERS PROGRAM
WILLIAM F. BRINKMAN, Vice President,
Physical Sciences Research, Bell Laboratories, Lucent Technologies
MALCOLM R. BEASLEY, Professor of Applied Physics and Electrical Engineering,
Department of Applied Physics, Stanford University
RALPH J. CICERONE, Dean,
School of Physical Sciences, University of California at Irvine
GEORGE B. FIELD, Senior Physicist,
Smithsonian Astrophysical Observatory and
Robert W. Willson Professor of Applied Astronomy,
Harvard University
SCOTT E. FRASER, Anna L. Rosen Professor of Biology,
California Institute of Technology
ERNEST G. JAWORSKI, Distinguished Science Fellow,
Monsanto Company (retired), Ladue, Missouri
LYNN W. JELINSKI, Professor of Engineering and Director
of the Center for Advanced Technology in Biotechnology, Cornell University
A. FRANK MAYADAS, Program Manager,
Alfred P. Sloan Foundation
JOHN R. RICE, W. Brooks Fortune Professor of Computer Science and Professor of Mathematics,
Department of Computer Sciences, Purdue University
J. DAVID ROESSNER, Professor of Public Policy,
School of Public Policy, Georgia Institute of Technology
ROLAND W. SCHMITT, Emeritus President of
Rensselaer Polytechnic Institute, Rexford, New York
I.M. SINGER, Institute Professor,
Massachusetts Institute of Technology
JOHN C. WRIGHT, Professor of Science Education,
Institute for Science Education, University of Alabama at Huntsville
Principal Project Staff
DEBORAH D. STINE, Study Director and Associate Director,
COSEPUP
SCOTT WEIDMAN, Senior Program Officer
PATRICK P. SEVCIK, Program Assistant
NORMAN GROSSBLATT, Editor
COMMITTEE ON SCIENCE, ENGINEERING, AND PUBLIC POLICY
PHILLIP A. GRIFFITHS (Chair), Director,
Institute for Advanced Study
BRUCE M. ALBERTS, * President,
National Academy of Sciences
WILLIAM F. BRINKMAN, Vice President,
Physical Sciences Research, Bell Laboratories, Lucent Technologies
DAVID R. CHALLONER, Vice President of Health Affairs,
University of Florida
ELLIS B. COWLING,
University Distinguished Professor At-Large,
North Carolina State University
GERALD P. DINNEEN, Retired Vice President,
Science and Technology, Honeywell, Inc.
MILDRED S. DRESSELHAUS,
Institute Professor of Electrical Engineering and Physics, Massachusetts Institute of Technology
ALEXANDER H. FLAX, Consultant,
Potomac, Maryland
RALPH E. GOMORY, President,
Alfred P. Sloan Foundation
M.R.C. GREENWOOD, Chancellor,
University of California, Santa Cruz
RUBY P. HEARN, Vice President,
The Robert Wood Johnson Foundation
MARIAN KOSHLAND, Professor of Immunology,
Department of Molecular and Cell Biology, University of California, Berkeley
THOMAS D. LARSON, Consultant,
Lemont, Pennsylvania
WILLIAM A. WULF, * Interim President,
National Academy of Engineering
DANIEL L. McFADDEN, Director,
Department of Economics, University of California, Berkeley
MARY J. OSBORN, Head,
Department of Microbiology, University of Connecticut Health Center
KENNETH I. SHINE, * President,
Institute of Medicine
MORRIS TANENBAUM, Vice President,
National Academy of Engineering
WILLIAM JULIUS WILSON, Lucy Flower University Professor of Sociology and Public Policy,
University of Chicago
LAWRENCE E. McCRAY, Executive Director
* Ex officio member. |
Preface
The National Science Foundation (NSF) Science and Technology Centers (STCs) program supports 25 academic centers spanning a wide variety of topics, both disciplinary and multidisciplinary. NSF's total program budget for the STC program was some $64 million for fiscal year 1996. The STCs were meant to have an 11-year lifespan. Because the first group of STCs are in their eighth year and they are scheduled to phase out in the ninth year, NSF is now faced with the decision of whether this experimental program should be continued and, if so, in what form.
Thus, in the spring of 1995, NSF requested that the Committee on Science, Engineering, and Public Policy (COSEPUP) of the National Academy of Sciences (NAS), National Academy of Engineering (NAE), and Institute of Medicine (IOM) conduct a study of the STCs program. COSEPUP appointed a panel to carry out this study and this report is the result of the panel's work.
The panel was to review and interpret the data gathered by an outside contractor to NSF (Abt Associates), reach conclusions regarding the progress of the STCs program toward its goals, and make recommendations concerning NSF's future use of the STC mode of support. The study was not to critique the accomplishments of individual STCs except as necessary to draw more general conclusions.
The panel met three times. It heard from the director of NSF, the chair of the NSF advisory committee that oversaw the selection of the current STCs and several directors of the STCs, persons who question the STC approach (primarily because of the concern that it takes funding away from individual investigator awards), and other knowledgeable persons. Beyond that, the panel was able to draw on a number of reports that evaluated the STCs program (described in Chapter 1 )
and on the knowledge and expertise of the panel members themselves as well as site visit reports and materials provided to panel members by the centers. Each committee member was asked to become familiar with four of the centers so that the panel could have as much information as possible available to it.
The use of Abt Associates as well as the Academy was an experimental effort to have a contractor develop some of the empirical data desired by NSF while maintaining the role of expert guidance from an academy panel. In general, as described in Appendix A, the panel did not view the experiment as successful. Of particular concern was that Abt's initial design was developed without an advisory body and that, as events played out, this panel was brought into the process too late to influence Abt's data collection and analysis. That is of particular concern in that, as indicated by Abt (p. 1 –32), “there was no attempt to achieve either ‘balance' or an exhaustive range of opinion or judgment in the data collection phase.” Furthermore, “it was expected that in most cases, individuals associated with the centers would take the opportunity to ‘put their best foot forward,'” and “the survey findings reported . . . should be read with a critical eye, evaluating centers' claims concerning their most important goals, achievements, and impacts against the reader's own understanding of what should be expected of the STC program, given its context and stated purposes. Applying this judgment systematically and distilling the conclusions in the form of concise, definitive, indicator narratives is the role of an expert panel.”
In other words, Abt collected primarily positive data from the various participants in the program, as opposed to collecting information and guidance from those outside the program who might have a more balanced and more negative view (such as members of the site visiting committee). In addition, according to Abt, “there is no way to identify retrospectively a proper comparison group for a set of complex scientific enterprises elected in the first place because of their unique characteristics. ” So there was no time-series analysis, for example, to compare the scientific results obtained by the researchers before and after they became part of an STC. However, several contemporary NSF-funded evaluation studies are dealing with outcomes similar to those projected for the STC program and have managed to construct comparison groups (e.g., ongoing studies by Henderson et al. on academic patents and Feller et al. for COSMOS Corporation on citation measures of EPSCoR and non-EPSCoR researchers).
All these factors affected the panel's belief in and interpretation of the results of the Abt report. As a result, the panel does not recommend that this serial process whereby one contractor carries out data collection and another group reviews the results should be used in the future. The panel did make guarded and selected use of data from the Abt report, but always with supporting evidence (generally provided by the committees that visited the centers on a recurring basis).
Given this concern, it should not be assumed that the panel's work does not itself have flaws. As with most committee-style reports, this report rests primarily on the judgment of its panel–based on the members' own experiences as well as
their review and analysis of the visiting committee reports and many other sources of information (described in Chapter 1). Many panel members run centers themselves (although not STCs) and have a good understanding of the benefits and liabilities of such an approach. These judgments are subjective ones and often lack objective data and analysis. Indeed the panel questions whether quantitative measures alone are sufficient to judge whether or not a program is serving research and societal needs. As a result, the panel must rely on its own judgment using the limited valid qualitative and quantitative data that are available.
Although this report represents the work of the panel, it benefited greatly from the support of the staff of COSEPUP, specifically, Deborah Stine, study director and associate director, who managed the project and drafted the report based on panel input; Larry McCray, executive director; Scott Weidman, the initial project director; Norman Grossblatt, editor; and Patrick Sevcik and Dave Amber, who provided administrative support.
The panel also acknowledges with appreciation the presentations made at panel meetings by the following persons:
-
Neal Lane, Director, National Science Foundation
-
Frank Press, Cecil and Ida Green Senior Fellow, Carnegie Institution
-
William Harris, Assistant Director, NSF/Directorate for Mathematical and Physical Sciences
-
James Edwards, Executive Officer, NSF/Directorate for Biological Sciences
-
David Schindel, Senior Science Advisor, NSF/Office of Science and Technology Infrastructure
-
Nathaniel Pitts, Director, NSF/Office of Science and Technology Infrastructure
-
Susan Graham, Professor, University of California, Berkeley
-
Michael White, Senior Research Scientist, University of Texas, Austin; Director, Center for Synthesis, Growth, and Analysis of Electronic Structures
-
Timothy Pickering, Professor, Virginia Polytechnic Institute and State University; Industrial Liaison, Center for High Performance Polymeric Adhesives and Composites
-
Bruce Guile, Director, Program Office, National Academy of Engineering
-
Aravind Joshi, Professor, University of Pennsylvania; Co-Director and Co-Industrial Liaison, Center for Research in Cognitive Science
-
Bernard Sadoulet, Professor, University of California, Berkeley; Director, Center for Particle Astrophysics
-
Larry Forney, Professor, Michigan State University; Industrial Liaison, Center for Microbial Ecology
-
James Colvard, Visiting Professor, Virginia Polytechnic Institute and State University
-
Stephen Fitzsimmons, Vice President, International Operations; Director Center for Science and Technology Policy Studies, Abt Associates, Inc.