Improving the Current System in the Short Term
Improving statistical practices takes time and money, but the Division of Science Resources Statistics (SRS) of the National Science Foundation (NSF), like other federal agencies, has a limited budget for its statistical activities, most of which must be devoted to accomplishing the day-to-day tasks associated with collecting, processing, and disseminating data to users. The panel therefore advises SRS to make only a few modest improvements to the current system of surveys in the short term (over the next four years) and to spend most of its resources for improvement following a longer term strategy of adopting new technology to extract data automatically from administrative databases that we expect to become available in the next few years.
Our short-term recommendations and the rationales for them are detailed in this chapter. Longer term recommendations follow in Chapter 4. In Chapter 6, the pathway for leading to these changes is described, beginning with the incremental improvements outlined here and progressing to the stage at which the current surveys can be replaced with a system based on administrative records.
Our recommendations for short-term improvements in the two surveys address five major areas: (1) reform of the taxonomies of fields of science and engineering (S&E), (2) SRS’s relationships with the responding agencies, (3) the adequacy of SRS’s data collection authorities, (4) the timeliness of data collection and reporting, and (5) survey technology. Each is discussed below.
REFORM OF THE TAXONOMIES OF FIELDS OF SCIENCE AND ENGINEERING
Practical Aspects of the Selection of a Taxonomy
For a taxonomy to be useful for research and development (R&D) data, it must not only include categories that are meaningful to users of the data but also be suitable for data collection purposes. Users should experience the categories as being reflective of their reality. At the same time, the taxonomy should reflect the way organizations providing the data are organized and staffed for the support and the performance of R&D.
The panel is aware that not all supporters and performers of R&D classify their activities in the same way and that this fact introduces a certain tension in deciding whether a taxonomy is “suitable.” For example, a mission agency might classify a bioinformatics research activity as life science, whereas a funded performer might report the same activity as computer science. For this reason, and others, no single taxonomy will satisfy all. However, for purposes of collecting data on research and development statistics in a consistent manner across federal government agencies, it is necessary to establish a common taxonomy that will be useful to the largest number of data providers and users. In the longer term, a provision can be made for tailoring structures that meet the specific needs of providers and users by flexibly categorizing administrative records as outlined in Chapter 5. Box 3-1 lays out some general issues in creating a taxonomy of activity in research and development.
SRS has made a good start on identifying and analyzing issues with the taxonomy now used. As described in Chapter 2, SRS has recently held workshops and conducted interviews with federal agency respondents to delve into issues surrounding the taxonomy used in the federal funds survey. In these workshops, the agencies have provided SRS with practical comments from their perspective as respondents. For example, some agencies have reported that their databases do not collect or store data in a way that would permit easy reporting by fields of S&E, and that the current taxonomy is not relevant to the way in which they manage and track their R&D programs (Macro International, 2008, p. 8).
Users of the federal funds data who participated in various discussions over the past few years, including the workshop held by our panel, seem nearly unanimous in their view that the current taxonomy has shortcomings. Among other criticisms, users point to the fact that new fields of science are not included, there is no way to report interdisciplinary research, and more detailed breakdowns are needed. These criticisms focus on the need for additional categories in the current taxonomy, rather than a wholesale change in the taxonomy itself. The same users caution against
Issues in Creating a Taxonomy of Activity in Research and Development
A taxonomy groups together entities according to their common characteristics. The English word derives from the Greek τάξις, taxis (meaning order or arrangement), and νόμoς, nomos (meaning law or science). Used originally for biological classification of living and extinct beings, the term now refers to any classification scheme. There is no perfect or ideal classification, Lenoir and Beghtol (2004) caution, only better or worse schemes for particular purposes, contents, users, and contexts. Units and levels of analysis also vary by specificity or “granularity,” from shallow analysis dividing entities into large aggregates to deep analysis subdividing them into smaller units. Moreover, taxonomies are never finished and so become outdated.
Since the late 19th and early 20th centuries, taxonomies of knowledge in the West have been dominated by a system of disciplinarity that distinguishes domains of specialized inquiry by their particular subjects and objects of study. The federal funds survey, for example, categorizes fields of science and engineering into eight major types: (1) life sciences, (2) psychology, (3) physical sciences, (4) environmental sciences, (5) mathematics and computer sciences, (6) engineering, (7) social sciences, and (8) other sciences “not elsewhere classified” (n.e.c.). Each major type, in turn, contains subtypes. Physical sciences, for example, encompass astronomy, chemistry, and physics. Over the course of the 20th century, the scope and size of disciplinary domains has expanded with the growing number of specialties and subfields, turning many of them into large groups of disciplines that encompass a broad range of identifiable and in some cases autonomous specialties.
The social sciences illustrate the challenge of classification. The mainstream disciplines of social sciences are anthropology, economics, political science, psychology, and sociology. Yet, Neil Smelser advises, describing social sciences solely with reference to the “big five” disciplines distorts reality in two ways (Smelser, 2003). First, under those headings, subareas of investigation rely on variables and explanations outside the commonly understood scope of social sciences. Geopolitics, sociobiology, behavioral genetics, and behavioral neuroscience all appeal to nonsocial and nonpsychological explanatory variables and explanations. Second, another range of disciplines could be labeled behavioral and social scientific, although not entirely so. Demography might be considered a separate social science or part of sociology, economics, and anthropology. Archaeology might be classed as part of anthropology or
an independent social science. Geography, history, psychiatry, law, and linguistics present similar complications for taxonomy. So do relations with the intersecting fields of genetics, behavior, and society; behavioral and cognitive neurosciences; psychiatry; health; gender studies; religious studies; expressive forms; environmental/ecological sciences and technology studies; area and international studies; and urban studies, planning, and public policy. Strict assignment to one category of inquiry or another would vary according to the criteria used (Smelser, 2003).
The growth of multidisciplinary and interdisciplinary modes of research has further complicated classification, a challenge amply evident in the n.e.c. category of the federal funds survey. The category of “not elsewhere classified” is large and amorphous and lumps together a plurality of developments, including new subfields, single-discipline projects for which a separate field has not been assigned, emergent fields, established interdisciplinary fields, cross-cutting initiatives, “problem-focus” areas of research, and miscellaneous “other.” It also fails to discriminate multidisciplinary juxtapositions of different disciplinary approaches from interdisciplinary approaches that integrate separate disciplinary data, methods, tools, concepts, and theories, as well as comprehensive transdisciplinary frameworks that posit a new conceptual synthesis or theoretical framework (Klein, 2009). Authors and users of taxonomies also have different views of how they should be constructed. Birger Hjørland distinguishes four fundamental methods of classification: empiricism (based on observations and inductions), rationalism (principles of pure reason, deductions), historicism (context and development), and pragmatism (analysis of goals, values, and consequences in both subject and object) (Hjørland, 2008). The greater plurality and complexity of knowledge today has resulted in three major views of current taxonomy. A first group continues to use standardized classification schemes based on a limited number of broadly aggregated categories, lumping together diverse practices. A second group advocates widening the broad categories, with the possibilities of adding a few more major categories and using a “hierarchy of preference” approach that allows splitting into highly aggregated (2-3 digit) levels and distributing percentages of emphasis and time into more than one discipline. A third group supports a more open, flexible, dynamic, and transactional approach, depicting research in a network representation that allows for greater granularity and employs techniques of semantic mapping, web- and text-mining, controlled thesauri, and tag clouds, and Internet-based, user generated taxonomies (“folksonomies”). The work of the second and third groups inform this study and its recommendations.
making significant changes to the taxonomy, for fear of introducing discontinuity in the historical data it provides.
Associating a government-wide taxonomy for R&D funding with the disciplinary structure of academia is no easy matter. The academic community is most often organized around departments for the purpose of instruction, but research activities are often multidisciplinary, engaging more than one discipline and department. Federal mission agencies tend to organize their applied R&D activities around broad national challenges, such as energy efficiency or space exploration, or around technology areas, such as nanotechnology, biotechnology, or information technology. Thus, updating the taxonomy to make it more relevant for communicating with the academic community might work for basic research, but it will not contribute much when classifying applied R&D programs and projects.
One promising approach is to use federal agency records to quantify the extent of interdisciplinary research. Under the auspices of the National Academies’ Keck Futures Initiative, a team from the Georgia Institute of Technology performed keyword searches of awards databases at NSF, the National Institutes of Health, and the Department of Energy to estimate the amount of interdisciplinary research being supported. Over the time period covered by the research (1999-2003), the team found apparent growth in interdisciplinary research (Yasitis et al., 2004). Using agency contract and award information as a source of information on fields of S&E is explored at greater length in Chapter 4.
SRS currently uses slightly different taxonomies for the federal funds survey and the academic R&D expenditures survey, and the taxonomies utilized in these surveys have differences from the standard taxonomy in OMB Directive 16 (see Table 2-1). Ideally, classification of R&D funding by disciplines as reported by the supporters of that R&D should be the same as reported by the recipients of those funds. Uniformity of definitions and classifications between surveys is one means of achieving that goal.
Practically, some differences in taxonomies are likely to persist because it may be difficult to collect the same level of detail from different respondent groups, or there may be differences in the uses to which the taxonomic information is put. That said, because of the need to compare the results of the two surveys, SRS would benefit from using the same taxonomy for the federal funds survey and the academic R&D expenditures survey.
The previous National Research Council (NRC) studies discussed in Chapter 2 have laid out the problems associated with an aging taxonomy of R&D spending. Our panel was specifically asked to make a recommendation about updating Directive 16 issued by the U.S. Office of Management
and Budget (OMB) in 1978—the taxonomy that was the original basis of the classification of fields of S&E for the federal funds survey.
Several alternative taxonomies could replace or supplement the OMB Directive 16 taxonomy. In considering options for a new taxonomy, the panel looked not only at OMB Directive 16, but also at taxonomies used internationally. Two relevant international standards are “Recommendations Concerning the International Standardization of Statistics on Science and Technology” of the United Nations Educational, Scientific, and Cultural Organization (UNESCO) (United Nations Educational, Scientific, and Cultural Organization, 1978) and “Proposed Standard Practice for Surveys on Research and Experimental Development,” called the Frascati manual, of the Organisation for Economic Co-operation and Development (OECD) (Organisation for Economic Co-operation and Development, 2002).
The UNESCO-recommended fields of S&E are shown in Box 3-2. Like OMB Directive 16, the UNESCO fields have not been updated since 1978. However, these old taxonomies were considered when the international community developed the more recent OECD Frascati manual classification, which uses the most current categories in its taxonomy of fields of S&E.
As shown in Box 3-3, the Frascati manual classification provides a useful system for organizing subactivities under the major fields. The Frascati manual recognizes that need for disaggregation within the major fields may differ from country to country, stating in the instructions, “While the major fields of science and technology are clearly defined, the level of disaggregation within each component field is left to each country” (Organisation for Economic Co-operation and Development, 2002, p. 66). Adoption of the Frascati taxonomy is not a full solution, however. Although newer than Directive 16, the Frascati manual suffers from some of the same limitations, in that it fails to accommodate multidisciplinary fields and has no procedures for periodic updating.
Two classification systems, one in widespread use and the other in development, have the advantage of permitting portrayal of multidisciplinary and interdisciplinary fields. Both the U.S. Department of Education’s Classification of Instructional Programs (CIP) and the NRC’s Taxonomy of Fields (see Box 3-4) are designed to support the collection of information from educational institutions, yet they may have wider application and attributes that commend them for consideration as alternative classification structures to OMB Directive 16.
The CIP was originally developed by the National Center for Education Statistics (NCES) in 1980 to provide a taxonomy to support the tracking,
UNESCO Fields of Science and Technology
Natural sciences, including astronomy, bacteriology, biochemistry, biology, botany, chemistry, computer sciences, entomology, geology, geophysics, mathematics, meteorology, mineralogy, physical geography, physics, zoology, other allied subjects.
Engineering and technology, including engineering proper, such as chemical, civil, electrical, and mechanical engineering, and specialized subdivisions of these; forest products; applied sciences such as geodesy, industrial chemistry, etc.; architecture; the science and technology of food production; specialized technologies or interdisciplinary fields, e.g., systems analysis, metallurgy, mining, textile technology, other allied subjects.
Medical sciences, including anatomy, dentistry, medicine, nursing, obstetrics, optometry, osteopathy, pharmacy, physiotherapy, public health, other allied subjects.
Agricultural sciences, including agronomy, animal husbandry, fisheries, forestry, horticulture, veterinary medicine, other allied subjects.
Social sciences, anthropology (social and cultural) and ethnology, demography, economics, education and training, geography (human, economic, and social), law, linguistics, management, political science, psychology, sociology, organization and methods, miscellaneous social sciences, and interdisciplinary, methodological, and historical science and technology activities relating to subjects in this group.
Humanities, arts (history of the arts and art criticism, excluding artistic research of any kind), languages (ancient and modern languages and literature), philosophy (including the history of science and technology), prehistory and history, together with auxiliary historical disciplines, such as archaeology, numismatics, paleography, etc., religion, other fields and subjects pertaining to the humanities, and interdisciplinary, methodological, historical, and other science and technology activities relating to the subjects in this group.
SOURCE: Adapted from United Nations Educational, Scientific, and Cultural Organization (1979).
4. Agricultural Sciences
5. Social Sciences
SOURCE: Organisation for Economic Co-operation and Development (2002, p. 67).
National Research Council Taxonomy of Fields
Biochemistry, Biophysics, and Structural Biology
Biology/Integrated Biology/Integrated Biomedical Sciences (Note: Use this field only if the degree field is not specialized.)
Cell and Developmental Biology
Ecology and Evolutionary Biology
Forestry and Forest Sciences
Genetics and Genomics
Immunology and Infectious Disease
Neuroscience and Neurobiology
Pharmacology, Toxicology, and Environmental Health
Physical Sciences, Mathematics, and Engineering
Astrophysics and Astronomy
Oceanography, Atmospheric Sciences, and Meteorology
Statistics and Probability
Biomedical Engineering and Bioengineering
Civil and Environmental Engineering
Electrical and Computer Engineering
Engineering Science and Materials (not elsewhere classified)
Materials Science and Engineering
Operations Research, Systems Engineering, and Industrial Engineering
Nanoscience and Nanotechnology
Social and Behavioral Sciences
Agricultural and Resource Economics
Public Affairs, Public Policy, and Public Administration
Criminology and Criminal Justice
Science and Technology Studies
Urban Studies and Planning
Arts and Humanities
English Language and Literature
French and Francophone Language and Literature
German Language and Literature
Language, Societies, and Cultures
History of Art, Architecture, and Archaeology
Music (except performance)
Spanish and Portuguese Language and Literature
Theatre and Performance Studies
Feminist, Gender, and Sexuality Studies
Race, Ethnicity, and Post-Colonial Studies
Rhetoric and Composition
SOURCE: Taxonomy of Fields (http://sites.nationalacademies.org/PGA/Resdoc/PGA_044521).
assessment, and reporting of fields of study and program completion.1 This tracking is accomplished through the Integrated Postsecondary Education Data System (IPEDS) completions survey, which is submitted by postsecondary institutions that receive Title IV federal funding. This survey summarizes the number of completions by field of study across the full spectrum of school offerings. The CIP provides the lists of the field of study.
Although the CIP includes fields other than S&E, one obvious advantage of associating the fields of S&E taxonomy for reporting R&D data to the CIP is that it is frequently updated to stay current with educational offerings. The CIP was updated in 1985, 1990, and 2000. NCES is currently updating the 2000 CIP with the goal of releasing an updated version in June 2009. These updates are based on rigorous procedures.2 A disad-
Memorandum, Michelle Coon, “Update of the Classification of Instructional Programs (CIP),” National Center for Education Statistics, August 11, 2008.
One process involves examining data from the IPEDS completion survey to identify institutions that produced the largest number of completions for a specific CIP code. The course catalogs for these institutions are then mapped onto the existing CIP and examined to find instructional programs that did not fit into the existing CIP. Stakeholders and coordinators are involved in this process to identify instructional programs that are not currently represented in
vantage of using the IPEDS process to drive updates of the taxonomy of fields of S&E in the SRS surveys is that it could lead to unanticipated and undesirable changes in the reporting of R&D support.
If OMB and SRS were to more closely align the federal funds fields with the CIP fields, some modifications of the latter would be in order. The CIP taxonomy includes a greater number of categories than are needed for the SRS surveys, so the SRS taxonomy might need to select and combine CIP categories. The burden of this task should be minimal, since SRS currently provides respondents to another of its R&D surveys—the academic R&D expenditures survey—with a cross-walk between the CIP taxonomy and the taxonomy of fields of S&E.
The NRC Taxonomy of Fields is the most recently developed taxonomy for classifying fields of S&E. Developed to support collection of data on research doctorate programs, it is based on the classification of fields used in the Doctorate Records File, which is also maintained by SRS. The criteria for inclusion of a field in this taxonomy are tied to doctoral program production—that is, fields are included that have produced a total of at least 500 Ph.Ds in the past 5 years with participation by at least 25 universities.
One advantage of the NRC classification structure is that it attempts, whenever possible, to specifically include multidisciplinary, cross-disciplinary, and transdisciplinary fields and to make provision for emerging fields exhibiting significant growth. Thus, interdisciplinary fields—such as neuroscience, biomedical engineering, and American studies—are included, and emerging fields—such as bioinformatics, biotechnology, systems biology, computational engineering, information science, science and technology studies, feminist studies, race and ethnic studies, and rhetoric and composition—also make the list (National Research Council, 2003, pp. 19-20; see also National Research Council, 2006).
The NRC system was designed to depict academic research programs, so it would fit well with the federal support survey. However, it was not designed to support collection of R&D data, so it would have to be modified to serve as a general-purpose R&D taxonomy.
Need for Historical Continuity
The need to modernize the taxonomy of fields of S&E must be balanced with the need for historical continuity of the data series that are based on the existing taxonomy. Data series based on consistent definitions of fields
of S&E go back many years: Several of the series published by field in Science and Engineering Indicators (National Science Board, 2006) have been consistently published since the early 1980s. Because of this consistency, it is possible to trace, for example, the dramatic increase in federal obligations for life science fields over the past four decades.
The need for historical continuity suggests caution in proposing changes in the fields of S&E taxonomy. Changes should be incremental, and care should be taken to either carry forward the current taxonomy, even as data are published using a new taxonomy, or to develop cross-walks between any new taxonomy and the old one so as to minimize disruption to the historical data series. Abrupt changes to the taxonomy could confuse data users.
Recommendation 3-1: The Division of Science Resources Statistics, in the near term, should make the changes necessary to improve the comparability of the federal funds taxonomy and the taxonomy for the academic research and development expenditures survey and should focus on the medium- and long-term changes the panel recommends.
SRS’S RELATIONSHIPS WITH RESPONDING AGENCIES
SRS collaborates with the responding agencies through regular workshops and has initiated several efforts to study reporting issues. The 2008 report by Macro International is the most recent example. In that report, Macro International staff contacted respondents by email to solicit information and schedule face-to-face meetings for in-depth interviews. Although the information provided in the report is of high quality and ultimately very useful, the contracted investigators faced many challenges with regard to access to the agency respondents. The Macro International staff reported difficulties in identifying appropriate high-level personnel in each agency and in obtaining cooperation in some. Furthermore, the panel observes that SRS would have benefited from the social professional capital that can be built through an ongoing program of having its own staff conduct structured in-depth interviews and discussions with agency staff about the surveys, data, and reporting issues. Building relationships between SRS staff and agency respondents is critical to ensuring goodwill, understanding how the agency internal reporting systems interface with the survey questionnaire, improving reporting, and ultimately obtaining high-quality data.
The panel concludes that SRS would benefit from building better direct relationships with agencies responding to the federal funds and federal support surveys. The core competency and management capacity to maintain relationships with other agencies is an inherent governmental
responsibility, and SRS loses an opportunity to improve practices when it relies on contractors to facilitate and manage these relationships. Senior-level SRS staff involvement with responding agencies should go a long way toward demonstrating to agencies that SRS considers the data to be important and that it values their input.
Recommendation 3-2: The Division of Science Resources Statistics should devote staff and resources to managing relationships with responding agencies directly, relying less on contractors to maintain those relationships.
ADEQUACY OF THE DATA COLLECTION AUTHORITIES
We considered whether SRS needs more explicit statutory authority to collect data from federal agencies in a way that would improve reporting. The panel notes that agencies are already required to respond to the federal funds and federal support surveys (Congressional Research Service, 2000, p. 1). The panel concludes that restating SRS’s statutory authority would be unlikely to affect how agencies maintain their data, so difficulties in responding to the questionnaire would continue.
Instead, SRS would benefit from pursuing better relationships with responding agencies, reminding them regularly of the importance of the survey and the usefulness of the data and working with them to make the survey forms as easy as possible to complete. This activity could be initiated as one of the major activities on the path toward modernized collection of R&D spending data. A time schedule for this activity is suggested in Chapter 6.
Some cosmetic changes in the collection forms might be helpful. For example, in contrast to its companion federal support survey and surveys that go out to the public, letters to respondents, survey forms, and instructions for the federal funds survey do not provide respondents with any background on the law under which the data are collected, nor do they address the important uses of the information. It is important for SRS to regularly remind agencies about the authority they have and importance of the survey with each call for data. To accomplish this end, information about authorities and uses could be featured prominently on the survey form, in the instructions, and on the associated website.
Recommendation 3-3: The Division of Science Resources Statistics should ensure that all questionnaires and email solicitations sent to respondents provide information on its data collection authority and on the important uses of the data.
TIMELINESS OF DATA COLLECTION AND REPORTING
The inability of SRS to release the data in a timely way limits the usefulness of the federal funds survey and the federal support survey data to policy makers and interest groups. As previously noted, the delays result largely from the current practice of delaying publication of results until reports are received from all reporting agencies and the long delays of some agencies in assembling and forwarding the data. Some of the agencies that have delayed transmittal of their data in the past are quite large, and the publication of the estimates without their contribution could severely distort the data. For example, the two slowest reporters in 2006 were the National Institutes of Health and the National Aeronautics and Space Administration, together accounting for about one-third of federal R&D expenditures.3
SRS could consider alternatives to this practice. Alternatives include reporting incomplete information earlier, providing a preliminary report with imputation for late respondents, or designing a simple schedule to the form that could be completed more easily and quickly as the basis for publishing a preliminary report.
The panel reluctantly concludes that SRS should stick with current practice. Each of these alternatives has pros and cons. Incomplete data might be misinterpreted, and preliminary totals would necessarily be on the low side because of missing data. Imputation for late respondents would introduce a new source of error and could make the data less accurate. Some data users hold that the potential for increased error due to omission or imputation is less desirable than more timely publication. Developing a new and simpler schedule to elicit early reporting would be a major new activity for SRS. In consideration of these issues, the best course of action at this time would be to continue the current policy of delaying publication until all reports are received, aggressively pursue better relationships with the agencies to encourage more timely response, and devote scarce resources to the improvements the panel recommends.
Recommendation 3-4: The Division of Science Resources Statistics (SRS) should maintain its current approach to data reporting, which is to wait for receipt of reports from all respondents before publishing the data. SRS should continue to report complete data without imputation for missing reports and data elements. The agency should focus on working directly with respondents to find ways to improve the timeliness of their response to the surveys.
Both the federal funds survey and the federal support survey have been early adopters of a web-based reporting system to facilitate response. However, not all respondents use the web to report, and the current web forms do not tap the full potential of current web-based survey methods. For example, at present the focus is on filling in the blanks in the reporting instrument. Little attention has been paid to developing an online survey instrument that is user-friendly and reduces respondent burden by using skip patterns and automatically populating responses. Some possibilities for process improvement include the automatic entry of zero values to subquestions when a response to one broad question is “no.” For example, if the agency respondent reports that it does not support research in non-profit institutions, the subsequent subquestions should automatically be populated with zero values, instead of requiring the respondent to enter a zero for each item.
At the same time, SRS could consider tailoring the web survey for each agency based on prior knowledge and prior reports and with extensive collaboration with the agency. One goal could be to include only those data items that could be expected to be included in the agency submissions with relative ease and accuracy and that would be familiar to the agencies. This tailoring of the collection of data does not have to be applied to all agencies at once; SRS could begin with the larger ones.
Recommendation 3-5: The Division of Science Resources Statistics should invest in creating more user-friendly web surveys, possibly tailored to each agency, to replace current web versions of the paper surveys.