anomalies over time, making precise or detailed long-term comparisons difficult even within a single data set.

Some of the problems associated with relying on multiple sources of data can be alleviated by conducting a well-planned, comprehensive survey that addresses many of the major internal and external indicators of a discipline's status. Only the AMO Science Assessment Panel was able to have access to a specialized survey, which had been done in conjunction with a concurrent study (NRC, 1994). Nevertheless, obtaining full value from such a survey requires its periodic repetition, which makes it subject to some of the problems outlined above.

CONCLUDING OBSERVATIONS

Although the three panels' reviews of potential indicators of the status of their particular fields and of the related data were neither comprehensive nor conclusive, they were sufficiently informative to show that the Commission's initial optimism about the benefits of greater reliance on or systematic use of quantitative measures in the assessment of scientific disciplines was unfounded. A comprehensive statistical analysis of the status of a scientific discipline is difficult, expensive, and time consuming. Unfortunately, such an analysis also tends to be relatively easy to impugn, especially when it attempts to address larger issues.

Using existing information, it is usually possible to obtain detailed and accurate data about some discrete portion—a branch of inquiry, a type of activity, or a particular source of support—of the larger system. In trying to assemble the pieces of a statistical portrait of an entire discipline, however, one quickly finds mismatches and large gaps. The resulting picture is incomplete and potentially misleading.

Despite the inherent drawbacks of using quantitative measures, it is possible to gain some useful insights about certain aspects of discipline-related research, particularly in an historical context. Future discipline assessments that attempt to use such data, however, must take into account the full costs and drawbacks of this approach at the outset.

REFERENCES

National Research Council (NRC). 1984. Renewing U.S. Mathematics: Critical Resource for the Future, National Academy Press, Washington, D.C.

National Research Council (NRC). 1990. Renewing U.S. Mathematics: A Plan for the 1990s, National Academy Press, Washington, D.C.

National Research Council (NRC). 1991. The Decade of Discovery in Astronomy and Astrophysics, National Academy Press, Washington, D.C.

National Research Council (NRC). 1994. Atomic, Molecular, and Optical Science: An Investment in the Future , Panel on the Future of Atomic, Molecular, and Optical Sciences (FAMOS) , National Academy Press, Washington, D.C.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 13
QUANTITATIVE ASSESSMENTS OF THE PHYSICAL AND MATHEMATICAL SCIENCES: A Summary of Lessons Learned anomalies over time, making precise or detailed long-term comparisons difficult even within a single data set. Some of the problems associated with relying on multiple sources of data can be alleviated by conducting a well-planned, comprehensive survey that addresses many of the major internal and external indicators of a discipline's status. Only the AMO Science Assessment Panel was able to have access to a specialized survey, which had been done in conjunction with a concurrent study (NRC, 1994). Nevertheless, obtaining full value from such a survey requires its periodic repetition, which makes it subject to some of the problems outlined above. CONCLUDING OBSERVATIONS Although the three panels' reviews of potential indicators of the status of their particular fields and of the related data were neither comprehensive nor conclusive, they were sufficiently informative to show that the Commission's initial optimism about the benefits of greater reliance on or systematic use of quantitative measures in the assessment of scientific disciplines was unfounded. A comprehensive statistical analysis of the status of a scientific discipline is difficult, expensive, and time consuming. Unfortunately, such an analysis also tends to be relatively easy to impugn, especially when it attempts to address larger issues. Using existing information, it is usually possible to obtain detailed and accurate data about some discrete portion—a branch of inquiry, a type of activity, or a particular source of support—of the larger system. In trying to assemble the pieces of a statistical portrait of an entire discipline, however, one quickly finds mismatches and large gaps. The resulting picture is incomplete and potentially misleading. Despite the inherent drawbacks of using quantitative measures, it is possible to gain some useful insights about certain aspects of discipline-related research, particularly in an historical context. Future discipline assessments that attempt to use such data, however, must take into account the full costs and drawbacks of this approach at the outset. REFERENCES National Research Council (NRC). 1984. Renewing U.S. Mathematics: Critical Resource for the Future, National Academy Press, Washington, D.C. National Research Council (NRC). 1990. Renewing U.S. Mathematics: A Plan for the 1990s, National Academy Press, Washington, D.C. National Research Council (NRC). 1991. The Decade of Discovery in Astronomy and Astrophysics, National Academy Press, Washington, D.C. National Research Council (NRC). 1994. Atomic, Molecular, and Optical Science: An Investment in the Future , Panel on the Future of Atomic, Molecular, and Optical Sciences (FAMOS) , National Academy Press, Washington, D.C.