Quantitative Assessments of the Physical and Mathematical Sciences: A Summary of the Lessons Learned (1994)

To date, assessments of research activities in major fields and their subdisciplines by boards and committees of the National Research Council's (NRC) Commission on Physical Sciences, Mathematics, and Applications (the Commission) have relied predominantly on the knowledge of experts who are intimately familiar with the major issues confronting their areas of scientific research. In the performance of such reviews, the broad input of other practitioners has usually been sought through “town meetings” at professional society conferences, through briefings by government managers and other experts, and through informal interviews and surveys. This process has always been augmented by a review of the relevant literature and, less frequently, by a statistical analysis of key indicators regarding the status of the discipline. Examples of recent Commission reports that have successfully used data regarding various aspects of a discipline or subdiscipline include The Decade of Discovery in Astronomy and Astrophysics (NRC, 1991)and Atomic, Molecular, and Optical Science: An Investment in the Future (NRC, 1994).

Although reliance on the peer review process has worked well and has produced effective reports, the Commission was interested in investigating the possibility of more systematically assaying the capacity of a field to sustain and evolve a vigorous program of research.

In initiating this study, the Commission's hypothesis was that the significantly increased use of quantitative analysis of the health or status of a discipline might be expected to provide a firmer foundation for discipline assessments performed by its own and other groups. By using a set of generic and field-specific measures, the Commission and its subsidiary bodies could thus make more pointed assessments of the health or status of a given discipline —whether on an individual or comparative basis—and suggest any remedies that might be necessary. The Commission viewed this approach as a promising alternative to the much called for, but notably unsuccessful, attempts by the scientific community to set priorities across disciplines. A collateral function of such an assessment methodology would be to assist in the

review of the progress of a field in response to past research strategies and recommendations published by the NRC.

The Commission initially identified five broad areas of potential metrics that could be indicative of the health or status of any given discipline over time:

1. Demographics: The success of a field in attracting and retaining high-quality students and professionals, including women, minorities, and other underrepresented groups.

2. Support: The capability of a field to generate funding and other types of support for graduate programs, research activities, and equipment and facilities consistent with the requirements established by the community.

3. Productivity: The quantitative output of a field in terms of knowledge and educated professionals; the effectiveness of a field in setting priorities and optimizing the use of resources.

4. Adaptability: The demonstrated capability of a field to adjust to changes in scientific opportunities, levels of support, and national needs.

5. Quality: The quality of the research output of a field in terms of scientific knowledge, educated professionals, and the impact of the field on world science and society, as well as the ability of a field to increase the quality of scientists, teachers, and students in higher education.

The Commission's initial assumption was that each of these areas could be tracked and evaluated using data from a variety of government and nongovernment databases, thereby enabling creation of a statistical portrait of any given field, which could then be combined with a more qualitative, peer review approach to assess a field's status. To test this assumption and its five proposed metrics, the Commission established three panels: the Mathematics Assessment Panel, the Astronomy Assessment Panel, and the Atomic, Molecular, and Optical (AMO) Science Assessment Panel. In addition, the work of the latter panel was complemented by a concurrent study in that area (NRC, 1994) performed under the auspices of the Commission's Board on Physics and Astronomy. These disciplines were chosen primarily for their diverse characteristics, so that a broad spectrum of research activities could be considered.

The mathematical sciences are the quintessential “small science”—largely theoretical, conducted by individual investigators, and generally not requiring major facilities or equipment. The research is both basic and applied, classified and open, publicly and privately funded, and historically international. A number of assessments and research strategies, developed by the NRC and others, have provided a baseline characterization of the past status of the field and where it should be headed (e.g., NRC, 1984, 1990).

Astronomy and astrophysics provide a mix of theoretical and observational research activities and rely heavily on large observational and computational facilities. The research is basic and unclassified, funded extensively by both the public and private sectors, and historically international in scope. This discipline also has been examined in a series of NRC surveys, the most recent one published in 1991 (NRC, 1991), which have described the existing status of the discipline and established research priorities.

Finally, AMO science comprises a relatively new and evolving combination of subdisciplines in physics, chemistry, and engineering. It presents a mix of theoretical and experimental research that is both basic and applied, small- and large-scale, classified and open, and privately and publicly funded. AMO science may be viewed