anomalies over time, making precise or detailed long-term comparisons difficult even within a single data set.
Some of the problems associated with relying on multiple sources of data can be alleviated by conducting a well-planned, comprehensive survey that addresses many of the major internal and external indicators of a discipline's status. Only the AMO Science Assessment Panel was able to have access to a specialized survey, which had been done in conjunction with a concurrent study (NRC, 1994). Nevertheless, obtaining full value from such a survey requires its periodic repetition, which makes it subject to some of the problems outlined above.
CONCLUDING OBSERVATIONS
Although the three panels' reviews of potential indicators of the status of their particular fields and of the related data were neither comprehensive nor conclusive, they were sufficiently informative to show that the Commission's initial optimism about the benefits of greater reliance on or systematic use of quantitative measures in the assessment of scientific disciplines was unfounded. A comprehensive statistical analysis of the status of a scientific discipline is difficult, expensive, and time consuming. Unfortunately, such an analysis also tends to be relatively easy to impugn, especially when it attempts to address larger issues.
Using existing information, it is usually possible to obtain detailed and accurate data about some discrete portion—a branch of inquiry, a type of activity, or a particular source of support—of the larger system. In trying to assemble the pieces of a statistical portrait of an entire discipline, however, one quickly finds mismatches and large gaps. The resulting picture is incomplete and potentially misleading.
Despite the inherent drawbacks of using quantitative measures, it is possible to gain some useful insights about certain aspects of discipline-related research, particularly in an historical context. Future discipline assessments that attempt to use such data, however, must take into account the full costs and drawbacks of this approach at the outset.
REFERENCES
National Research Council (NRC). 1984. Renewing U.S. Mathematics: Critical Resource for the Future, National Academy Press, Washington, D.C.
National Research Council (NRC). 1990. Renewing U.S. Mathematics: A Plan for the 1990s, National Academy Press, Washington, D.C.
National Research Council (NRC). 1991. The Decade of Discovery in Astronomy and Astrophysics, National Academy Press, Washington, D.C.
National Research Council (NRC). 1994. Atomic, Molecular, and Optical Science: An Investment in the Future , Panel on the Future of Atomic, Molecular, and Optical Sciences (FAMOS) , National Academy Press, Washington, D.C.