National Academies Press: OpenBook

Quantitative Assessments of the Physical and Mathematical Sciences: A Summary of the Lessons Learned (1994)

Chapter: Summary of General Findings on Potential Indicators and Related Data

« Previous: Purpose and Scope of Discipline Assessments
Suggested Citation:"Summary of General Findings on Potential Indicators and Related Data." National Research Council. 1994. Quantitative Assessments of the Physical and Mathematical Sciences: A Summary of the Lessons Learned. Washington, DC: The National Academies Press. doi: 10.17226/9199.
×

onomies exist and the major research issues and activities are reasonably homogeneous; it is much more difficult in large fields or in newly emerging or merging disciplines, such as AMO science. In either case, however, it is impossible to collect existing data that are consistent across all the different elements of the discipline, or even for any given element, especially over time. Even expert longitudinal surveys that strive to maintain internal consistency are difficult to compare over time because of the evolving nature of science.

One of the most significant problems with regard to definitions is encountered with data provided by the federal R&D agencies. Because every agency defines any given discipline in the context of its statutorily mandated mission, the boundaries accorded each discipline vary widely among the agencies and even among large research centers or laboratories within an agency. Moreover, those boundaries are skewed further over time as certain funding programs end and new ones begin, often based on external political considerations and opportunistic relabeling rather than on the exigencies of the evolving research.

Similar problems of definition are posed by industry, which of course is not organized according to discipline boundaries. Finally, these problems are multiplied when data from foreign countries are used. It is essential, therefore, to determine the scope of the assessment at the outset, since new methodological issues and resource needs will be raised with every additional research function, sector, and geographic region considered. Failure to define all the relevant boundaries and the assumptions underlying them will lead to incoherent results.

Closely related to the purpose and scope of any given assessment is its intended audience. Discipline assessments conducted by the Commission's boards and committees are typically directed to program managers and policymakers in the client agency(s); other decisionmakers in the administration and Congress; the scientific community; other professionals in related disciplines and sectors, both in the United States and abroad; and the general public. Identifying all end users of an assessment at the outset will help in determining the appropriate structure and focus for the study and in making it more effective.

SUMMARY OF GENERAL FINDINGS ON POTENTIAL INDICATORS AND RELATED DATA

Credible, statistically derived indicators can be used as diagnostic tools to help determine certain aspects of the state of a discipline, to identify strengths and weaknesses, and to augment the factual basis for the conclusions and recommendations arising out of the assessment. Moreover, in the case of a repeat assessment there is the additional potential to quantify progress made since the previous assessment, provided that the scope remains the same. Nevertheless, because of the amorphous and rapidly changing nature of any given discipline, and because of the numerous difficulties involved in obtaining and using putatively relevant data, the Commission has concluded that a significantly increased or systematic use of quantitative measures is unlikely to significantly enhance either the credibility or usefulness of its assessments of the health or status of disciplines.

Health is an organic or biological concept. Dictionaries in general define health as the overall condition of an organism at a given time, and as the freedom from disease or abnormality, or as a condition of optimal well-being. For an organism to be in good health requires not only that all elements of the internal system are

Suggested Citation:"Summary of General Findings on Potential Indicators and Related Data." National Research Council. 1994. Quantitative Assessments of the Physical and Mathematical Sciences: A Summary of the Lessons Learned. Washington, DC: The National Academies Press. doi: 10.17226/9199.
×

functioning properly, but also that the system is in a state of dynamic equilibrium or balance with its external environment.

It became quite clear from the panel discussions that each discipline has its own standards or vital signs for measuring its health and that these metrics, like those for assessing the well-being of different organisms, must be relative and not absolute. Moreover, even within a single discipline, the metrics and their interpretation will vary depending on who is doing the assessment and for what purpose.

In this regard, the Commission agreed with the Mathematics Assessment Panel, which rejected the notion that the health of a discipline can be measured based solely on numerical indicators. Statistical data can provide us with quantitative measures of some aspects of the state of a field, and while these measures can be used singly or in the aggregate as indicators, they do not provide direct measurements of its “health.”

For example, statistics can tell us the number of Ph.D. mathematicians produced each year, the level of federal funding and its distribution among areas of the mathematical sciences, or the number of graduate students supported as research or teaching assistants. These are measurements of some aspects of the pipeline and support—two areas important to the health of a field. These same statistics, however, are insufficient to determine the adequacy or appropriate balance of the funding or of the personnel levels. Different groups of discipline experts will reach different conclusions on what rate of Ph.D. production is desired, or on what is the optimum balance of support among the different areas of, for example, the mathematical sciences.

The discussion that follows lists potentially useful indicators of a discipline's state, suggests possible supporting data, and summarizes several significant issues regarding their application. It is important to emphasize that these indicators are not all of equal importance or effectiveness and that their applicability depends on the discipline being assessed and the purpose of the assessment.

Human Resources

Information on human resources—the people involved in the research activities of a particular discipline —is fundamental to determining the discipline's overall status. By far the most voluminous and widely available statistics for all disciplines are data on human resources, which provide us with information about various characteristics of the scientific work force.

Human resources data have been used extensively by government and academia to track the student population from grade school through graduate school and to monitor a variety of characteristics of the professional work force. Such data are collected regularly by several organizations, including the National Science Foundation (NSF), the Department of Education's National Center for Education Statistics, the Department of Labor's Bureau of Labor Statistics, the Commerce Department's Census Bureau, the NRC's Office of Scientific and Engineering Personnel, and the scientific professional societies. In addition, specialized surveys are conducted periodically by various research institutions and universities for certain segments of the student and professional populations. Taken together, these existing data sources provide a reasonably comprehensive and accurate portrayal of past demographic trends in distinct areas of scientific research.

In particular, many of these data can be used to characterize specific aspects of a discipline's supporting human infrastructure over time, as initially suggested by the Commission. The assessment panels divided the Commission's proposed “demo-

Suggested Citation:"Summary of General Findings on Potential Indicators and Related Data." National Research Council. 1994. Quantitative Assessments of the Physical and Mathematical Sciences: A Summary of the Lessons Learned. Washington, DC: The National Academies Press. doi: 10.17226/9199.
×

graphics” indicator into three parts. The first, “the success of a field in attracting and retaining . . . students and professionals,” has at least two major underlying issues—the sufficiency of the quantity of human resources produced and the mobility or employability of those people once they enter the profession. The second, more specific element, “attracting and retaining . . . women, minorities, and other underrepresented groups,” can be summarized as the issue of work force diversity. Finally, the important question of the “quality” of the work force needs to be addressed.

These three potential indicators of the state of the human resources in any given discipline and their supporting data can be summarized as follows:

  1. The success of a field in attracting and retaining students and professionals

    • Number of students in graduate schools

    • Number of M.S. and Ph.D. degree recipients

    • Number of postdoctoral appointments

    • Unemployment and underemployment figures

    • Employment distribution by sector (education, business/industry, government) or job function (e.g., research and development, management/administration, teaching, professional services, consulting)

  2. Work force diversity

    • Same statistics as for number one above divided according to sex, race, age, and national origin/citizenship

  3. Quality of undergraduate and graduate students (indicative of input to the work force)

    • Scores on standardized tests

    • Selectivity of graduate institutions attended by those who plan careers in science, as indicated by grade point averages of incoming students, both overall and in a major

Although these data can indicate important trends, even in such small and rapidly evolving fields as AMO science, they have several significant limitations generally shared by most of the other data reviewed in this study. There is a 1- to 3-year time lag in their application to answering questions about the current quantity of human resources. They also tell us little, if anything, about the quality of the people working currently or of their work. Moreover, because each data source uses a somewhat different definition of any given discipline, and because the boundaries of a discipline change over time, the comparability and accuracy of the data are necessarily questionable, impugning their validity even for purely retrospective, historical studies.

Support of a Discipline

It is important to obtain as detailed an understanding of financial support as possible in order to determine a discipline's potential problems, opportunities, and requirements. The panels modified the Commission's key measures of “support” somewhat to include the capability of a field to generate funding and other types of support for researchers, research activities, equipment and facilities, and international cooperative activities consistent with the requirements established by the community.

Suggested Citation:"Summary of General Findings on Potential Indicators and Related Data." National Research Council. 1994. Quantitative Assessments of the Physical and Mathematical Sciences: A Summary of the Lessons Learned. Washington, DC: The National Academies Press. doi: 10.17226/9199.
×

In addition, the panels added the research expenditures by other nations as an indicator of worldwide support for a particular discipline, as well as for purposes of comparison. Because “the requirements established by the community” vary from discipline to discipline, and within a single discipline over time, the discussion that follows focuses on the indicators of support generic to all disciplines and the sources of available data.

A number of different sources of data compilations on government funding of scientific research are available. Aggregated summaries of federal and nonfederal funding by agency or discipline are available from the NSF, the Congressional Research Service, the Congressional Budget Office, and the American Association for the Advancement of Science. The summaries are useful for macroscopic analyses but quickly lose much of their relevance in reviews of discipline-specific activities and issues, and they are useless for any studies of subdisciplines.

At the next level of disaggregation of federal government (and occasionally state government and private) expenditures are various specialized surveys published by the NSF and professional societies. These compilations document funding of certain elements of R&D, either across all disciplines or for one discipline in greater detail. Data are generally available from the NSF for support of undergraduate and graduate programs at universities, research activities by sector, facilities and equipment, and the like. These data can be useful for showing funding trends regarding specific R&D functions in certain disciplines. Unfortunately, the availability of these specialized surveys depends on the luck of the draw, that is, on the discipline being assessed and on whether it happens to match the definition, scope, and categorization of the existing surveys. In addition, the varying methodologies of the different survey sources make a comparative analysis difficult.

The most useful sources of data regarding federal research support are the annual congressional appropriations bills and the agency budgets. Despite the greater level of detail, most of these figures need to be disaggregated further and then reconstituted to obtain useful information on funding of discipline-specific activities. This can be an extremely labor-intensive and frustrating process, although it is the only means of acquiring a comprehensive understanding of federal government support of a discipline.

Potential indicators of the status of support for a discipline and their supporting data include the following:

  1. Funding of researchers and research activities

    • Government contracts and grants (award amounts and periods; success ratios in having proposals funded; subdiscipline categories; characterization of funded research as theory, experimentation, or observation; type of recipient institution; race, sex, and age of individual recipients; and new versus continuing programs)

    • Ratio of contracts to grants (grants are used more for basic research than for applied research)

  2. Funding of equipment and facilities

  3. Funding of international cooperative activities

    • U.S. contributions to foreign projects

    • Foreign contributions to U.S. projects

Suggested Citation:"Summary of General Findings on Potential Indicators and Related Data." National Research Council. 1994. Quantitative Assessments of the Physical and Mathematical Sciences: A Summary of the Lessons Learned. Washington, DC: The National Academies Press. doi: 10.17226/9199.
×
  1. Research expenditures by other nations (competitive and comparative)

  2. Other types of nonfinancial support (e.g., public opinion surveys)

With the exception of highly aggregated government agency budget figures, detailed data on expenditures are difficult to acquire and use, primarily because the agencies do not keep the data according to the specific categories that might be requested. In addition, the mission agencies generally organize their budgets according to applied programs rather than by discipline, making an accurate disaggregation of the figures exceedingly difficult, if not impossible. Finally, funding data from classified research programs, industry, and other nations are generally unavailable, significantly obscuring a large portion of the support for most of the physical and mathematical sciences.

Output and Impact of a Field

The panels combined the Commission's original “productivity” and “quality” metrics into one indicator—the output and impact of a field. They had the greatest difficulty in applying quantitative measures to this area. The quality of new ideas, the intellectual vitality of a field, and the impact of scientific discoveries over time are not particularly amenable to statistical assessments. In contrast to the more easily quantifiable infrastructure or support factors—head counts, dollar counts, pieces of equipment —output or impact elements reflect the often unique characteristics of a field or a particular type of research. Most important, the relationship of inputs to outputs is difficult to establish accurately, especially with respect to basic research, where the time lines of ultimate impact are very long and the relationships obscured by numerous intervening factors.

There are several ways to measure the output and related impact of a field introspectively. These include examining the quantitative and qualitative output of a field in terms of knowledge and educated professionals, the effectiveness of a field in setting priorities, and the effectiveness of managers within a field in their use of resources. The most widely used database regarding research output is kept by the Institute for Scientific Information for bibliometric and citation analyses. Because of its limited scope, however, this study did not examine the quantitative aspects of a discipline's external relationships with society.

With these caveats in mind, the potential indicators of a discipline 's output and impact and the supporting data can include the following:

  1. The quantitative and qualitative output of a field in terms of knowledge

    • Bibliometric analysis (quantitative only; e.g., the number of papers published in peer-reviewed publications over time in comparison to the number of professionals, and distribution of the output over time)

    • Citation analysis (indications of quality may be inferred; e.g., highly cited papers and “hot topics”)

  2. The quantitative and qualitative output of a field in terms of educated professionals

    • Demographic data (quantitative only)

    • tation analysis (indications of relative quality may be inferred by com-

Suggested Citation:"Summary of General Findings on Potential Indicators and Related Data." National Research Council. 1994. Quantitative Assessments of the Physical and Mathematical Sciences: A Summary of the Lessons Learned. Washington, DC: The National Academies Press. doi: 10.17226/9199.
×

paring numbers of citations of U.S. researchers with those of researchers in other countries)

  • Prizes awarded to researchers (national comparisons)

  • Invited papers at major international conferences or research at foreign laboratories

  • Number of publications in highly cited journals of international scope

  1. The effectiveness of a field in implementing any previously established research priorities

    • Patterns of funding by government agencies

    • Bibliometric and citation analysis

  2. The effectiveness or efficiency of practitioners of a field in their use of resources

    • Use of funding (allocation of grants according to categories of activities)

    • Use of time (surveys of researchers regarding the distribution of their time spent on “productive” and “nonproductive” tasks)

    • Use of facilities (statistics on usage patterns of major observational or experimental facilities)

    • Use of supporting equipment (e.g., time spent on various uses of computers and networks)

  3. The effectiveness of managers within a field in their use of resources

    • Funding of highest-priority research objectives and functions

    • On-time and on-budget procurements of major facilities

    • Comparisons with other similar disciplines or with other nations regarding the issues related to the preceding two bulleted items

Measuring the impact of a discipline on other disciplines would likely provide an important indication of the discipline's intellectual vigor and its value to the broader scientific community. The major sources of data for tracing the impact of a discipline on other disciplines (and vice versa) are bibliometric and citation data and the various demographic surveys. Potential indicators that may merit further investigation include:

  • National and international cross-discipline collaboration and co-authorship (citation analysis),

  • Employment outside the discipline in other scientific fields (demographic data),

  • Number of B.S./M.S. graduates in a discipline who earn M.S./Ph.D. degrees in other fields, and

  • Awards from or invitations to give lectures to groups in other disciplines.

Adaptability

“Adaptability” was defined by the Commission as the demonstrated capability of a field to adjust to changes in scientific opportunities, levels of support, and national needs. It certainly is possible to think of indicators based on demographic, support, and output data that might be useful in characterizing the collective response of members of a discipline to a particular change in one of these categories. For example, the number of proposals received in response to an initiative announced by a

Suggested Citation:"Summary of General Findings on Potential Indicators and Related Data." National Research Council. 1994. Quantitative Assessments of the Physical and Mathematical Sciences: A Summary of the Lessons Learned. Washington, DC: The National Academies Press. doi: 10.17226/9199.
×
Page 4
Suggested Citation:"Summary of General Findings on Potential Indicators and Related Data." National Research Council. 1994. Quantitative Assessments of the Physical and Mathematical Sciences: A Summary of the Lessons Learned. Washington, DC: The National Academies Press. doi: 10.17226/9199.
×
Page 5
Suggested Citation:"Summary of General Findings on Potential Indicators and Related Data." National Research Council. 1994. Quantitative Assessments of the Physical and Mathematical Sciences: A Summary of the Lessons Learned. Washington, DC: The National Academies Press. doi: 10.17226/9199.
×
Page 6
Suggested Citation:"Summary of General Findings on Potential Indicators and Related Data." National Research Council. 1994. Quantitative Assessments of the Physical and Mathematical Sciences: A Summary of the Lessons Learned. Washington, DC: The National Academies Press. doi: 10.17226/9199.
×
Page 7
Suggested Citation:"Summary of General Findings on Potential Indicators and Related Data." National Research Council. 1994. Quantitative Assessments of the Physical and Mathematical Sciences: A Summary of the Lessons Learned. Washington, DC: The National Academies Press. doi: 10.17226/9199.
×
Page 8
Suggested Citation:"Summary of General Findings on Potential Indicators and Related Data." National Research Council. 1994. Quantitative Assessments of the Physical and Mathematical Sciences: A Summary of the Lessons Learned. Washington, DC: The National Academies Press. doi: 10.17226/9199.
×
Page 9
Next: Addititional Issues on Conducting Quantitative Assessments »
Quantitative Assessments of the Physical and Mathematical Sciences: A Summary of the Lessons Learned Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!