NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This report has been reviewed by a group other than the authors according to procedures approved by a Report Review Committee consisting of members of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine.
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce Alberts is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Harold Liebowitz is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine.
The National Research Council was established by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce Alberts and Dr. Harold Liebowitz are chairman and vice chairman, respectively, of the National Research Council.
Support for this project was provided by the National Science Foundation (under Contract No. MPS-952216). The contents of this publication do not necessarily reflect the views or policies of the National Science Foundation, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government.
Additional copies of this report are available from:
Commission on Physical Sciences, Mathematics, and Applications
2101 Constitution Avenue, NW
National Research Council
Washington, DC 20418
Copyright 1995 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
WORKSHOP STEERING COMMITTEE
W. CARL LINEBERGER,
University of Colorado,
Co-chair
CHARLES A. ZRAKET,
The MITRE Corporation,
Co-chair
SUSAN L. GRAHAM,
University of California at Berkeley
ROBERT J. HERMANN,
United Technologies Corporation
MARK B. MYERS,
Xerox Corporation
Staff, Commission on Physical Sciences, Mathematics, and Applications National Research Council
Paul F. Uhlir, Associate Executive Director
Cameron H. Fletcher, Executive Assistant
David J. Baskin, Project Assistant
COMMISSION ON PHYSICAL SCIENCES, MATHEMATICS, AND APPLICATIONS
ROBERT J. HERMANN,
United Technologies Corporation,
Chair
RICHARD N. ZARE,
Stanford University,
Chair (1992-1995)
RICHARD S. NICHOLSON,
American Association for the Advancement of Science,
Vice Chair (1992-1995)
STEPHEN L. ADLER,
Institute for Advanced Study
PETER BANKS,
Environmental Research Institute of Michigan
SYLVIA T. CEYER,
Massachusetts Institute of Technology
L. LOUIS HEGEDUS,
W.R. Grace and Co.
JOHN E. HOPCROFT,
Cornell University
RHONDA J. HUGHES,
Bryn Mawr College
SHIRLEY A. JACKSON,
U.S. Nuclear Regulatory Commission
KENNETH I. KELLERMANN,
National Radio Astronomy Observatory
KEN KENNEDY,
Rice University
HANS MARK,
University of Texas at Austin
THOMAS A. PRINCE,
California Institute of Technology
JEROME SACKS,
National Institute of Statistical Sciences
L.E. SCRIVEN,
University of Minnesota
LEON T. SILVER,
California Institute of Technology
CHARLES P. SLICHTER,
University of Illinois at Urbana-Champaign
ALVIN W. TRIVELPIECE,
Oak Ridge National Laboratory
SHMUEL WINOGRAD,
IBM T.J. Watson Research Center
CHARLES A. ZRAKET,
The MITRE Corporation (retired)
NORMAN METZGER, Executive Director
PAUL F. UHLIR, Associate Executive Director
Preface
In late 1994, the National Science Foundation (NSF) asked the National Research Council's (NRC's) Commission on Physical Sciences, Mathematics, and Applications to convene a one-day workshop focusing on the criteria or metrics used by corporations in restructuring their research, particularly in the physical and mathematical sciences. Several goals were envisioned: (1) to help the broader scientific community understand the factors that lead to restructuring of research; (2) to enable NSF, and perhaps other federal agencies, to determine whether the methods used by corporations to assess changes in their research capacities are in any way instructive for governmental agencies that must evaluate the results of their public research investments; and (3) to further the understanding in the academic research community of the role of research in corporations in maintaining their vitality.
In making this request, NSF was motivated by several factors. The most immediate reason was the requirements imposed by the Government Performance and Results Act, a largely unheralded piece of legislation passed by Congress in 1993. The act directed the administration to initiate a number of pilot projects to develop and test federal program performance metrics in order to improve the evaluation, understanding, and ultimately the accountability of all federal government activities. By October 1997, every agency must submit to the Office of Management and Budget a strategic plan that includes a mission statement, outcome-related goals, a plan for reaching those goals, and a description of how the agency will evaluate its performance against the goals. The goals themselves are supposed to be objective, quantifiable, and measurable.
In addition to the Government Performance and Results Act, the Clinton administration's National Performance Review, which also was initiated in 1993, has focused significant attention on the performance of all government programs, including the assessment of the government 's investments in research. Finally, as was made clear during the course of this workshop, NSF's assessment of its own programs should be undertaken in any case, in light of the inherent importance of such introspection to the Foundation and to the continuous improvement of its activities, and not because of some externally imposed mandates.
The National Science Foundation realized that the corporate experience in restructuring and evaluating research is only one of many that it ought to consider in developing its own assessment methods, and that, in any event, the conventional notion of “metrics” in the sense of using numerical scores may not apply; that is, the particular approaches used by any one corporation may not translate into criteria for judging public investments in research. Nevertheless, the Foundation believed that an examination of the approaches used by major corporations still could provide some valuable insights for it as well as for the broader research community, not only in responding to the requirements of the act, but also in better understanding and managing the current era of transition and change now faced by the sciences, especially the physical and mathematical sciences.
In convening this workshop, the National Research Council invited senior corporate research managers from IBM, AT&T, Ford, and Xerox to discuss their experiences in this area, specifically addressing the following issues: (1) why they restructured their research activities; (2) the goals, principles, and criteria that they established; (3) the methods and measures by which they determined the need for restructuring and which were subsequently used in evaluating the success of that process; and (4) any lessons they believe would be of value to the Foundation and to the broader scientific community.
The morning session consisted of presentations by the corporate speakers. The remainder of the day was devoted to a discussion of the issues by the other invited participants. A list of all workshop participants is included in the appendix.
As preparation for their participation in the workshop, we asked the discussants to consider the following issues:
-
What kinds of performance measures might be useful for NSF to adopt in assessing the implementation of its strategic objectives and institutional mission, as established in its strategic plan?
-
In the event that NSF needs to either reduce or restructure its activities because of declining funding, what criteria or metrics should be used to evaluate success or failure?
-
What are the most important elements of NSF's relationship with the university research community? What is the best method for tracking the status of those elements, especially in determining the impact of reduced funding?
The four presentations by the corporate speakers and the follow-up discussions provided a number of insights of great value to NSF and to government research activities in general. We found that industry experience in research restructuring can be directly relevant to government, although industry has not yet fully solved the problem of performance metrics. More important, the issues in research restructuring are much deeper than just performance metrics. They involve behavioral and cultural changes in all the stakeholders that are affecting and are being affected by the research process. Research restructuring and evaluation is a continuous process; it must be done on a regular, long-term basis to match the evolving needs of society.
We extend our sincere thanks to NSF for its sponsorship of the workshop and to the speakers and discussants for the substance and openness of their contributions. Bill Harris, Anne Petersen, and Susan Cozzens of NSF defined the Foundation's objectives and needs articulately and effectively. The NRC staff under the leadership of Paul Uhlir did an outstanding job in organizing and executing the workshop and the proceedings.
We commend the workshop proceedings to your attention.
W. Carl Lineberger
Workshop Co-chair
Charles A. Zraket
Workshop Co-chair