National Academies Press: OpenBook

Data on Federal Research and Development Investments: A Pathway to Modernization (2010)

Chapter: 2 The Current Surveys: Challenges and Opportunities

« Previous: 1 Introduction
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

2
The Current Surveys: Challenges and Opportunities

The National Science Foundation Act of 1950 charges the National Science Foundation (NSF) with providing “a central clearinghouse for the collection, interpretation, and analysis of data on scientific and engineering resources, and to provide a source of information for policy formulation by other agencies of the Federal Government.” NSF has carried out that task for many years through its Division of Science Resources Statistics (SRS) and its predecessor, the Division of Science Resources Studies.

SRS manages a family of five surveys bundled into the Research and Development Statistics Program, two of which are specifically designed to elicit information about the scope of spending on research and development (R&D) by federal agencies: the Survey of Federal Funds for Research and Development (the federal funds survey) and the Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions (the federal support survey). Table 2-1 provides details on the five surveys in the Research and Development Statistics Program.

The federal funds survey asks all federal departments and agencies that conduct or support R&D for information about their financial investments in R&D, and the federal support survey asks those same agencies to report on their financial investments for R&D in individual institutions as well as their spending on a range of educational and student assistance programs related to science and engineering (S&E). Both surveys are administered annually.

The National Research Council (NRC) has reviewed these surveys in two prior studies (National Research Council, 2000, 2005b). These reviews

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

TABLE 2-1 The SRS Research and Development Statistics Program

Survey

Universe

Description

Survey of Federal Funds for Research and Development (federal funds survey)

Federal departments and agencies that conduct and/or support R&D programs

An annual census survey of the 15 federal departments, their 70 subagencies, and 15 independent agencies that conduct R&D programs. It collects information about the federal funding of R&D in the United States for three fiscal years. The survey measures federal support of, and participation in, national scientific activities in terms of obligations and outlays. Conducted since the early 1950s.

Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions (federal support survey)

Federal departments and agencies that conduct and/or support R&D programs

An annual census survey of the same agencies that respond to the federal funds survey, which report only if they had obligations for science and engineering to universities, colleges, or nonprofit institutions during the past fiscal year. Agencies report obligations separately for each academic institution and nonprofit organization they fund.

Survey of Research and Development Expenditures at Universities and Colleges (academic R&D expenditures survey)

Academic institutions with greater than $150,000 in R&D expenditures in the previous year or doctoral programs in R&D

An annual census survey that collects information on separately budgeted R&D expenditures by academic institutions in the United States and outlying areas. Conducted since FY 1972, it collects information on R&D expenditures by source of funds and by academic field using a taxonomy similar (but not identical) to the one used in the federal funds survey.

Survey of Industrial Research and Development (industrial R&D survey)

For-profit R&D-performing companies, whether publicly or privately held

An annual sample survey that intends to include or represent all for-profit R&D-performing companies, either publicly or privately held. It is the primary source of information on R&D performed by industry in the 50 states and the District of Columbia. Data are collected in the technology categories of biotechnology, software, materials synthesis and processing, and others. Research in nanotechnology is separately identified and asked for in the 4 categories above. Respondents are asked to report energy R&D in areas of fossil fuels, geothermal and solar, nuclear, and all other energy sources.

Survey of State Research and Development Expenditures

State organizations and agencies that sponsor research and development

An annual census survey of 423 state organizations and state agencies that sponsor research and development. Begun in 2006, the survey measures the extent of R&D activity performed and funded by the 50 states, the District of Columbia, and Puerto Rico. Data are not collected by fields of science and engineering.

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

focused broadly on the entire portfolio of SRS R&D surveys, devoting only minimal attention to the federal funds and federal support surveys. The 2005 report, however, did examine whether the data collected in the federal funds survey were relevant and adequate for their intended uses, and it recommended that SRS reconsider several aspects of the survey operation in order to modernize it. The report concluded that SRS could improve its operations by collecting the information needed to complete the federal science and technology budget framework recommended in an earlier NRC report, Allocating Federal Funds for Science and Technology (National Research Council, 1995). The report also urged SRS to begin to work with the U.S. Office of Management and Budget (OMB), under the auspices of the E-Government Act of 2002, which had been recently enacted to develop guidance for standardizing the development and dissemination of R&D project-level data as part of an upgraded administrative records–based system.

The 2005 NRC report also recommended improvements in the federal support survey. It found the survey to be a useful supplement to the federal funds survey—but also concluded that data collection was cumbersome and time-consuming and needed significant modernization. One option offered for modernizing the survey is to more intensively use microdata in administrative records that are part of the standardized, automated reporting systems in the key federal agencies that provide the bulk of federal support to academic and nonprofit institutions. The 2005 report also briefly considered issues regarding the taxonomy of fields of S&E used by these surveys, recommending that OMB initiate a review of its Classification of Fields of Science and Engineering, last published in 1978 as Directive 16, and that OMB appoint the SRS division of NSF as the lead agency for the effort.

Since these earlier studies, the environment of the surveys on federal R&D spending has changed dramatically. Under the guidance of the E-Government Act and subsequent legislation, OMB has made progress in developing administrative databases for contracts and grants, establishing new standards for the quality of the data, and establishing guidance to making the new databases available to the public in a timely manner. In addition, SRS commissioned a major study by Macro International to assess the quality of the information provided in the surveys, particularly with regard to the classification and reporting of fields of S&E (Macro International, 2008). OMB also assigned responsibility to SRS for reviewing and upgrading the fields of S&E classification, as recommended by the 2005 NRC panel.

In this chapter, our panel examines the federal funds survey and the federal support survey in order to provide the basis for updated and more

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

focused advice on collecting more accurate, timely, and useful data of the type now collected by these surveys.1

SURVEY OF FEDERAL FUNDS FOR RESEARCH AND DEVELOPMENT

Since the early 1950s, SRS has conducted the federal funds survey as an annual census survey of federal science and technology agencies. There are now 15 federal departments, 70 subagencies, and 15 independent agencies that conduct R&D programs.2 The survey is conducted for SRS by a private-sector contractor (Macro International).

The survey has a response rate of 100 percent for both reporting units and survey items. This high rate of survey completion comes at a high price, as discussed later in this chapter and addressed more fully in Chapter 3, because the release of the survey results is often delayed for a year or more awaiting the receipt of reports from all of the reporting departments and agencies.

Years Covered

Data collected on the federal funds survey cover three federal fiscal years: actual (final) expenditures for the most recently completed fiscal year, preliminary estimates of actual expenditures for the current fiscal year, and projected expenditures for the forthcoming fiscal year based on the president’s budget submission to Congress. The best-documented data are those for the completed fiscal year, which are based on actual R&D outlays made or R&D obligations entered into by federal entities. The data for the current fiscal year are a stew of amounts in the congressional appropriations, obligation actions to date, and apportionment and reprogramming decisions

1

Information in this chapter is based on SRS’s online survey descriptions, documentation provided to the panel by SRS, the findings of a quality profile on the Research and Development Statistics Program (Bailar, 2004), presentations made at the panel workshop (see Appendix C), and additional information provided by SRS at the panel’s request.

2

Departments: Agriculture (10 subagencies); Commerce (5 subagencies); Defense (15 subagencies); Education; Energy; Health and Human Services (11 subagencies); Homeland Security (4 subagencies); Housing and Urban Development; Interior (4 subagencies); Justice (3 subagencies); Labor (6 subagencies); State; Transportation (9 subagencies); Treasury (3 subagencies); and Veterans Affairs. Independent agencies: Agency for International Development; Appalachian Regional Commission; Broadcasting Board of Governors; Consumer Product Safety Commission; Environmental Protection Agency; Federal Communications Commission; Federal Trade Commission; General Services Administration; Library of Congress; National Aeronautics and Space Administration; National Archives and Records Administration; National Science Foundation; Nuclear Regulatory Commission; Smithsonian Institution; Social Security Administration.

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

made and documented during the year. The least reliable data are those for the president’s budget year. These projections represent the amounts in the administration’s budget proposals not yet passed by Congress. The overall amounts in the preliminary and projected estimates are subject to revision as the president’s budget is executed and authorization, appropriation, deferral, and apportionment actions completed after these data were collected are realized over time.

There is little hard information on which to base estimates of expenditures by fields of S&E for the preliminary and projected years. The Congressional Research Service (CRS) reported that these estimates are computed as a percentage of current-year spending. Agency representatives told CRS that projecting by broad fields of S&E is, at best, an “educated guess” (Congressional Research Service, 2000, p. 10).

These uncertainties in reporting and the inevitable discrepancies among budgets, plans, and actual expenditures contribute to fairly large revisions in the data as the agencies accumulate information on their actual expenditures over time. As Table 2-2 indicates, some of these revisions can be quite significant. During periods of budget uncertainty, as when there is a change in administration, these revisions are even more marked. In 2002, for example, the projected level of federal R&D spending was about 6 percent lower than finally reported; the year after, the total of agency projections was about 5 percent greater than the total of final estimates made when actual spending data became available. Therefore, generally the projected data are thought to be useful only in suggesting broad trends.

Variables Collected

Data collected for each year include outlays and obligations. The key variables collected and their definitions (taken from the survey questionnaire) are found in Table 2-3. Only the 10 largest agencies report obligations for plant by performer and state. They account for about 97 percent of total R&D and R&D plant obligations each year. Only the six largest agencies report the data for obligations to colleges and universities by field of science.

Reporting agencies are challenged by the fact that their internal records do not use the categories requested by SRS. For example, internal records do not separate expenditures on R&D plant from expenditures for the conduct of R&D. This is one example of the kind of measurement error that arises because of the difficulty agencies have in translating the data from the categories in which they are maintained on agency records into the categories that are requested by SRS.

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

TABLE 2-2 Federal Obligations for R&D and R&D Plant: Projected, Preliminary, and Final, 1990-2008 (in millions of dollars)

Year

Projected

Preliminary

Final

Percentage Change, Projected to Preliminary

Percentage Change, Projected to Final

1990

68,524

63,353

65,831

−7.5

−3.9

1991

66,690

66,227

64,148

−0.6

−3.8

1992

73,484

75,586

68,577

+2.9

−6.7

1993

75,045

75,303

70,415

+0.3

−6.2

1994

74,436

72,818

69,451

−2.2

−6.7

1995

71,746

73,029

70,443

+1.8

−1.8

1996

70,906

71,048

69,399

+0.2

−2.1

1997

70,149

71,996

71,753

+2.6

+2.3

1998

71,593

74,202

73,914

+3.6

+3.2

1999

75,330

77,650

77,386

+3.1

+2.7

2000

77,186

81,772

77,356

+5.9

+0.2

2001

83,609

85,452

84,003

+2.2

+0.5

2002

84,938

97,465

90,158

+14.7

+6.1

2003

103,114

101,008

97,928

−2.0

−5.0

2004

105,220

106,488

105,371

+1.2

+0.1

2005

110,193

113,118

112,995

+2.7

+2.5

2006

112,076

116,873

112,271

+4.3

+0.1

2007

116,417

116,700

 

+0.2

 

2008

113,213

 

 

 

 

SOURCE: National Science Foundation (2009a).

TABLE 2-3 Variables and Definitions of Items Collected on the Survey of Federal Funds for Research and Development

Variable

Definition

Outlays for R&D and R&D plant by year

Outlays (expenditures) represent the amounts for checks issued and cash payments made for research and development activities and plant (facilities and fixed equipment) during the fiscal year.

Obligations for R&D and R&D plant by year

Obligations are the amounts of orders placed, contracts and sub-grants awarded, goods and services received, and similar transactions during a given period that will require payment by the grantee during the same or a future period for research and development activities and plant (facilities and fixed equipment) during the fiscal year.

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

Variable

Definition

Obligations for basic, applied, and total research by field of S&E (detailed field for past fiscal year, broad field for current and next years)

Basic research is systematic study directed toward fuller knowledge or understanding of the fundamental aspects of phenomena and of observable facts without specific applications towards processes or products in mind.

Applied research is systematic study to gain knowledge or understanding necessary to determine the means by which a recognized or specific need may be met.

Development is systematic application of knowledge or understanding directed toward the production of useful materials, devices, and systems or methods, including design, development, and improvement of prototypes and new processes to meet specific requirements.

A field of S&E is a recognized category of specialized expertise within S&E as defined by OMB Directive No. 16.

Obligations for basic and applied R&D by performer by year

A performer is either an intramural group or organization carrying out an operation or function or an extramural organization or person receiving support or providing services under a contract or grant.

Obligations to individual federally funded R&D centers for past year

Federally funded R&D centers are S&E performing organizations that are exclusively or substantially funded by the federal government, administered by an industrial firm, a college or university, or another nonprofit institution.

Obligations for R&D for foreign performers for past year

Foreign performers include foreign citizens, organizations, universities and colleges, or governments as well as international organizations (such as United Nations organizations).

Obligations for R&D plant by performer by year

See above.

Obligations for R&D and R&D plant by state for the past year

States include state and local government agencies, excluding colleges and universities. State data are reported only by the 10 agencies with largest R&D obligations.

Obligations for R&D to universities and colleges for field of S&E (detailed field for past fiscal year, broad field for current and next years)

Universities and colleges are institutions engaged in providing resident and/or accredited instruction for at least a 2-year program above the secondary school level. These data are reported only by the six agencies with the largest R&D obligations.

SOURCE: NSF, Survey of Federal Funds for Research and Development, Questionnaire. Available: http://www.nsf.gov/statistics/srvyfedfunds/surveys/srvynondod_fy03-05.pdf.

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

For example, actual data on outlays and obligations are available for only the past year, and agencies are asked to make their best estimates for both outlays and obligations for the current and future years. R&D plant data are underreported because of difficulties encountered by the Department of Defense (DoD), the National Aeronautics and Space Administration (NASA), and others in identifying and reporting them. DoD reports obligations for R&D plant funded under the agency’s appropriation for construction, but it is able to identify only a small portion of the R&D plant support from R&D contracts funded from DoD’s appropriation for research, development, testing, and evaluation. Similarly, NASA cannot separately identify the portions of industrial R&D contracts that apply to R&D plant, since these data are subsumed in the R&D data covering industrial performance. NASA R&D plant data for other performing sectors are reported separately.

Categorization of Research and Development Activities

A major issue affecting data accuracy is discrepancies in how federal agency staff assign their research spending into the basic research, applied research, and development categories, which may result in a lack of comparability in reporting the categories among federal agencies. In its 2000 study, the Congressional Research Service observed that “while SRS provides agencies and survey respondents with definitions of each data item collected on its surveys (e.g., basic research, applied research, and development, etc.), agencies, as well as individuals within agencies, interpret the definitions differently” (Congressional Research Service, 2000, p. 9).

The CRS study further observed that some agencies found it difficult to determine whether certain activities, such as training, program evaluation, and construction of facilities, met the criteria for inclusion in the categories (p. 9). There is little doubt that individual staff members sometimes make arbitrary judgments in order to report in the requested categories. Quite often, these staff members work in agency budget offices that have been assigned the task of completing the SRS reports, rather than in the scientific and technical program offices, where staff might more readily distinguish among the categories.

The panel is aware that the practice of characterizing scientific and technical inquiry as basic or applied research or development has long been controversial. The so-called linear model underlying these categories (basic research → applied research → development) has been met with some skepticism (Godin, 2005), and is widely understood to be a gross oversimplification. This linear model often fails to describe the actual processes of moving from inquiry to application. Furthermore, whether a particular project or program falls into one or another of the categories depends, to some degree,

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

on the intentions and perceptions of those making the categorization. The panel chose not to delve into this subject or to make recommendations about it, but we recognize that it is one of the issues that will need to be addressed in order for SRS to act on our recommendations to move ahead with deriving the federal funding data reports from administrative databases, as this report recommends.

Timeliness of Release

Although the federal funds survey enjoys 100 percent response rates, this is achieved, in part, because of the SRS policy that the data are not released until all agencies have reported. This policy puts SRS at the mercy of the slowest respondent and can severely delay publication of the data.

The delay in publication can be measured by the lag from the period of reference for the data to the time of publication. For example, the data for fiscal years 2006-2008 were solicited in mid-February 2007, with a due date in mid-April 2007. The last of the agency reports was not received until some eight months later, in mid-December 2007. Adding in the time for the lengthy process at SRS to aggregate and analyze the data as well as the agency inputs for release, publication of the data for these years did not occur until January 2009 (National Science Foundation, 2009a, p. 2). In this illustration, the “most recent” year data were actually being reported in January 2009 for outlays that had been completed by September 2006, some 28 months earlier. This was actually an improvement from prior years, when the final agency submissions were received as late as June of the year following the year of solicitation.

Clearly, time lags of this duration, resulting from the failure of agencies to submit timely reports as well as from the lengthy procedures at SRS, reduce the usability of the data for real-time planning and policy-making purposes. Although SRS is able to publish the federal funding data without introducing error-prone imputation for missing agency reports, it does so at a high price: the data are usually well out of date by the time they are finally published.

Collection Technology

On the positive side, SRS has made significant enhancements in recent years to the procedures and technology used for collecting these data from federal agencies. FEDWeb, a web-based data collection system, is used to collect and manage data for the federal funds survey. Most data collection efforts, data imports, data editing, and trend checking are accomplished using FEDWeb. This web-based system helps improve survey reporting and reduce data collection and processing costs by offering respondents

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

direct online reporting and editing. The web-based option has proven popular with most agency reporters. In the 2006-2008 survey cycle, 89 of the 100 reporting agencies submitted their data via FEDWeb. Three others submitted at least part of their data in Microsoft Excel files, which were then entered into FEDWeb by the SRS contractor. However, while modernization of the data collection method might have been expected to assist in improving the consistency and completeness of the data, the transition to web-based collection has not dramatically reduced the delays in data collection and reporting.

TAXONOMY OF FIELDS OF SCIENCE AND ENGINEERING

A key component of the federal funds survey is the taxonomy it uses to classify spending by fields of S&E. The taxonomy is anchored in a classification system that was last updated by OMB in 1978 in its Directive 16 (U.S. Office of Management and Budget, 1978). A specific part of the panel’s charge is to assess this classification structure.

The Challenge of Interdisciplinarity

Several challenges are associated with the current taxonomy of fields of S&E used in the federal funds survey. A significant one is that it does not account for new and emerging science or the more recent phenomenal growth in interdisciplinary, multidisciplinary, and transdisciplinary research.

In this report, multidisciplinary approaches are defined as when researchers maintain their disciplinary and professional perspectives but add breadth and available knowledge, information, and methods from other disciplines. Interdisciplinary approaches integrate separate disciplinary data, methods, tools, concepts, and theories in order to create a holistic view or common understanding of a complex issue, question, or problem. In transdisciplinary approaches, researchers develop comprehensive frameworks through an overarching synthesis, such as general systems, policy sciences, feminism, ecology, and sociobiology, in which the disciplines lose their identity.

The importance of updating the taxonomy to better incorporate interdisciplinary research is widely recognized by policy makers, funding agencies, professional organizations, and across academia. The growing role of research involving more than one discipline is a serious challenge to any taxonomy of fields and therefore to gathering, analyzing, and using federal funds data based on a single-field taxonomy.

There are four primary drivers of interdisciplinary research—the inherent complexity of nature and society, the desire to explore problems and

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

questions that are not confined to a single discipline, the need to solve societal problems, and the power of new technologies (National Research Council, 2005a, p. 40).

The heightened momentum for interdisciplinary research dates from the late 1970s, roughly the same time as when the current taxonomy used in the federal funds survey was developed. International competition in science-based fields of high technology propelled greater involvement and investment in interdisciplinary research in engineering and manufacturing, computers, biotechnology, and biomedicine. Breakthroughs in scientific research were also occurring increasingly at the interstices of established fields of inquiry. In 1986, a report from the National Research Council assessing major fields of physics noted “rapidly emerging interdisciplinary advances—which are enriching all of science” (National Research Council, 1986a, p. 15).

Subsequent reports have continued to document these trends. A 1990 NRC report on interdisciplinary research called attention to new intellectual understandings of biological systems, problem complexity, the costs of instrumentation and facilities, and increased collaborations between the life sciences and medicine as well as the physical sciences and engineering (National Research Council, 1990). More recently, an NRC report on the NSF Materials Research Science and Engineering Centers Program affirmed that “frequently, the most exciting and important advances in materials science and engineering occur at the interfaces between, or by unconventional combinations of, traditional disciplines” (National Research Council, 2007a, p. 8).

Funding agencies have responded to these developments with new structures and strategies. The National Institutes of Health Roadmap, for example, recognizes that new combinations of analytical skills and disciplines and new technologies are needed to deal with complex challenges of human health and well-being. New technologies of molecular imaging, nanomedicine, and bioinformatics are prominent in the life sciences. New tools of quantitative and computer-assisted mathematical analysis and advanced computing power are also facilitating the sharing of large quantities data across disciplinary boundaries in areas as diverse as medicine and the geosciences (National Institutes of Health, 2009). The implications may not merely be technical. In the journal Science, Alan Leshner observed that “new technologies are driving scientific advances as much as the other way around,” allowing new approaches to older questions and posing new ones (Leshner, 2004, p. 729).

The Fluid Boundaries of Traditional Disciplines

In addition to the challenge of interdisciplinary research, the changing nature of disciplines has contributed to making the 1978-era taxonomy less

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

relevant to today’s world. The current interface between physics and chemistry, for example, has been crossed so often in both directions that authors of Scientific Interfaces and Technological Applications reported “its exact location is obscure” and “its passage is signaled more by gradual changes in language and approach than by any sharp demarcation in content” (National Research Council, 1986b, p. 53). Interactions and cross-fertilizations that characterize the interface have been sources of continual advances in concepts and applications across the science of molecules and atoms, surfaces and interfaces, and fluids and solids. “Thirty years ago,” Norman Burkhard observed, “the difference between a physicist and a chemist was obvious. Now we have chemists who are doing quantum-level, fundamental studies of material properties, just like solid-state physicists. There’s almost no difference” (National Research Council, 2005a, p. 54).

Inadequacies of the Current Taxonomy

The taxonomy of fields of S&E used in the federal funds survey takes little notice of the trends in the conduct and organization of R&D discussed above. The current taxonomy includes eight broad field categories, each including a number of detailed fields. The broad fields are life sciences, psychology, physical sciences, environmental sciences, mathematics and computer sciences, engineering, social sciences, and other sciences. Each broad field includes a subfield of “not elsewhere classified” (n.e.c.). SRS provides illustrative disciplines for each detailed field in its guidance documentation for survey respondents.

The n.e.c. category is used for both multidisciplinary projects involving more than one field and single-discipline projects for which a separate field is not part of the taxonomy. The lack of up-to-date categories has resulted in overuse of the n.e.c. category. “Not elsewhere classified” has become an amorphous category that lumps together many unrelated types of research, including work in new subfields, emergent fields, established interdisciplinary fields, cross-cutting initiatives, “problem-focus” areas of research, and miscellaneous “other.” Moreover, it fails to discriminate multidisciplinary and genuinely integrated interdisciplinary activities. Furthermore, there is no specific way to report interdisciplinary research.

A further problem with the R&D taxonomy is that, at the detailed level, SRS uses somewhat different classifications structures for the federal funds survey and its other surveys. Table 2-4 compares the various taxonomies used by SRS for the federal funds survey with another of its surveys—the Survey of Research and Development Expenditures at Colleges and Universities (known as the academic R&D expenditures survey)—and the “official” taxonomy from OMB Directive 16. There are potentially confusing differences among the three.

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

TABLE 2-4 Comparison of Taxonomies

Federal Funds Survey

Academic R&D Survey

OMB Directive 16

Life Sciences

Life Sciences

Life Sciences

Biological sciences (excluding environmental)

Biological sciences

Biological

Environmental biology

 

 

Agricultural science

Agricultural sciences

 

Medical science

Medical

Clinical medical

Other medical

Life sciences, n.e.c.

Other

 

Psychology

Psychology (total)

Psychology

Biological aspects

 

Biological aspects

Social aspects

 

Social aspects

Psychological sciences, n.e.c.

Other

Psychological sciences, n.e.c.

Physical sciences

Physical sciences (total)

Physical sciences

Astronomy

 

Astronomy

Chemistry

 

Chemistry

Physics

 

Physics

Physical sciences, n.e.c.

Other

Physical sciences, n.e.c.

Environmental sciences

Environmental sciences (total)

Environmental sciences—terrestrial and extraterrestrial

Atmospheric sciences

Atmospheric

Atmospheric sciences

Geological sciences

Earth

Geological sciences

Oceanography

 

Oceanography

Environmental sciences, n.e.c.

Other

Environmental sciences, n.e.c.

Mathematics

Mathematical sciences (total)

Mathematics

Computer sciences

Computer sciences (total)

 

Mathematics and computer sciences, n.e.c.

 

 

Engineering

Engineering (total)

Engineering

Aeronautical

Aeronautical and astronautical

Aeronautical

Astronautical

 

Astronautical

 

Bioengineering/biomedical engineering

 

Chemical

Chemical

Chemical

Civil

Civil

Civil

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

Federal Funds Survey

Academic R&D Survey

OMB Directive 16

Electrical

Electrical

Electrical

Mechanical

Mechanical

Mechanical

Metallurgical and material

Metallurgical and material

Metallurgy and materials

Engineering, n.e.c.

Other

Engineering, n.e.c.

Social sciences

Social sciences (total)

Social sciences

Anthropology

 

Anthropology

Economics

Economics

Economics

 

 

History

 

 

Linguistics

Political science

Political science

Political science

Sociology

Sociology

Sociology

Social sciences, n.e.c.

Other

Social sciences, n.e.c.

Other sciences, n.e.c.

Other sciences (total)

Other sciences, n.e.c.

NOTE: n.e.c. = not elsewhere classified.

SOURCE: National Science Foundation, Survey of Federal Funds for Research and Development, Questionnaire. Available: http://www.nsf.gov/statistics/srvyfedfunds/surveys/srvynondod_fy03-05.pdf; National Science Foundation, Survey of Research and Development Expenditures at Universities and Colleges, FY 2008 Questionnaire. Available: http://www.nsf.gov/statistics/srvyrdexpenditures/surveys/srvyrdexpenditures_2008.pdf; U.S. Office of Management and Budget (1978).

Recent SRS Efforts to Address Problems

The SRS is well aware of the challenges faced by its current taxonomy of fields of S&E. As noted above, at the direction of OMB it has undertaken a project that is to lead to recommendations on how to revise and update OMB Directive 16, on which the taxonomy is based. SRS has commissioned or conducted several recent studies and workshops to address these issues. SRS convened an interagency working panel meeting (hosted at SRI International on October 21-22, 2004), sponsored a study of agency reporting practices, and commissioned a detailed internal review of a proposed social sciences taxonomy. In NSF, a process was initiated to add a data element to its internal R&D project reporting system (called the “E-jacket”) to capture fields of science codes. In this section, we summarize the findings of the interagency working meeting and the study of agency reporting practices.

The October 2004 SRI workshop focused on means of updating the fields of S&E taxonomy. This workshop included discussions about the needs of the users of SRS data; the nature of S&E disciplines and multidisciplinary, interdisciplinary, and transdisciplinary work; criteria for good

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

classification systems; data mapping and mining; international issues; and approaches to improving classification systems (Cheney and Park, 2005, p. 2). The workshop participants included the government officials who provide the data and two major groups of users: representatives of policy communities, who use the information to make policy decisions about S&E investments, and representatives of the research community, who use the data to study the characteristics of the S&E enterprise.

The workshop identified three options for updating the taxonomy. One is to make only minor revisions to the taxonomy in order to maintain continuity in the data, particularly at the higher levels of aggregation. A second is to expand the taxonomy to increase the level of detail, although that might compound the difficulty of reporting the fields. A third is to restructure reporting to permit respondents to report more than one field, enabling analysis of multidisciplinary, interdisciplinary, and transdisciplinary research. The workshop participants emphasized the importance of consulting with users, representatives of the disciplines, survey respondents, and others as SRS moves forward with any revision of the fields of science and engineering, as well as to consider international, educational, and employment taxonomies in the process (Cheney and Park, 2005, p. 7).

Reaching out to other federal agencies, SRS organized and hosted an interagency working group on June 16, 2006, consisting of representatives from OMB and the six largest research-supporting federal agencies: NSF, the National Institutes of Health (NIH), DoD, the Department of Energy (DoE), NASA, and the U.S. Department of Agriculture (USDA). Its aim was to lay the groundwork for the review and revision of OMB Directive 16, identify the primary uses of R&D data organized using this classification, and acknowledge the problems associated with implementing new classification schemes across agencies. Importantly, SRS commissioned Macro International to undertake a study to document (1) the taxonomy each agency uses for analyzing or reporting its research funds (or combined R&D totals, if appropriate); (2) the purposes for which the taxonomy is used; (3) how each agency actually goes about classifying its research/R&D totals into this taxonomy; and (4) how each agency uses the fields of science and engineering taxonomy reported to the federal funds survey (Macro International, 2008, p. 4).

The study of agency reporting practices by Macro International documented interviews with representatives of the 16 major agencies that report on the federal funds survey (Macro International, 2008). The top five sponsors of R&D (NIH, DoD, DoE, NSF, and NASA), accounting for 86 percent of all federal research obligations in FY 2005, stated that they do not use the federal funds taxonomy for their own program management or budgeting. As a result, they consider the data they submit to the federal funds survey to be of questionable value (to them) because of the

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

methods they must use to classify and report them (Macro International, 2008, p. 5).

For example, NIH respondents reported that the agency has no use for the taxonomy because it does not reflect the current state of science and does not include reporting for interdisciplinary research. The DoD respondents reported similar concerns, noting that much of their current cutting-edge research overlaps two or more of the disciplines in the current taxonomy. The DoE respondents reported that the current taxonomy may limit how agencies report their funding. Furthermore, DoE reported confusion about how to classify its own programs, noting that it could more accurately report research activities by theme (e.g., energy, environment, national security). For internal operations, the NSF research program staff (not to be confused with the SRS staff) reported that the current taxonomy is adequate but recognized the need for a taxonomy that reflects new scientific areas and interdisciplinary research. Although the National Institute of Standards and Technology, NSF, DoE, and DoD reported that the resulting data are of some benefit to them, they concurred that the data have limited value for the reasons noted above.

Some interesting uses of the taxonomy were reported. For example, the DoD participant reported using the federal funds taxonomy to compare DoD research funding with funding for the education of graduate students by the U.S. Department of Education. The Macro International report recommended revising the taxonomy to be useful to the reporting agencies, warning that methods used to collect and report data, as well as the quality of the resulting information, are otherwise unlikely to improve.

SURVEY OF FEDERAL SCIENCE AND ENGINEERING SUPPORT TO UNIVERSITIES, COLLEGES, AND NONPROFIT INSTITUTIONS

The federal support survey is an annual census survey of the same agencies that respond to the federal funds survey, but they report only if they had obligations for S&E to universities, colleges, or nonprofit institutions during the past fiscal year (the time frame of this survey). The data that are collected from the 19 agencies (in FY 2006) that made such obligations include federal obligations for R&D; R&D plant; facilities and equipment for S&E instruction; S&E fellowships, traineeships, and training grants; general support for S&E; and other S&E activities.

Like the federal funds survey, this survey has a response rate of 100 percent (both unit and item) because of the SRS policy to withhold release of the data until all agencies have reported. Also like the federal funds survey,

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

the policy of withholding release of the federal support data until all agencies have reported is a contributing factor to excessive lags in publication of the survey results. For example, the data for FY 2006, which ended on September 30, 2006, were solicited in mid-February 2007 with a requested due date of mid-April 2007. The data from the last 2 of the 19 reporting agencies were not received by SRS until November 2007. The brief analysis and abbreviated set of summary tables for FY 2006 were finally released by SRS in October 2008 (National Science Foundation, 2008b), and the full set of detailed information by agency with institutional rankings was not published until March 2009 (National Science Foundation, 2009b).

In this survey, on behalf of SRS, a contractor asks departments and agencies to complete a web-based survey instrument for each university and college for which they obligated R&D funding during the previous fiscal year.3 There are approximately 1,200 such colleges and universities. In FY 2006, the reporting agencies obligated over $28 billion to those institutions, most of it going to the top 100 universities and colleges (National Science Foundation, 2009b, Table 17).

Agencies also complete a survey instrument for each of the approximately 1,300 nonprofit institutions to which they obligated funds. The list of nonprofit institutions receiving federal R&D funds has been growing. Obligations to universities and colleges are reported in six categories, whereas the obligations to nonprofit institutions are reported only for R&D and R&D plant (see Table 2-5). In FY 2006, over $6.5 billion was obligated to nonprofit institutions, about 80 percent of those funds going to the top 100 recipient nonprofits (National Science Foundation, 2009b, Table 29).

Respondents are instructed that totals for R&D transfers to universities should be similar to those reported by them for R&D to academic institutions in the federal funds survey. If differences exist, respondents are asked to explain. One reason for differences in totals is that methods differ for reporting funds that are transferred from one agency to another before being sent to an institution. For example, the federal support survey asks the agency that distributes the funds to a university to report the transaction, whereas the federal funds survey asks the agency transferring the funds to the agency that ultimately sent them on to report the transaction.

This survey also uses a web-based collection tool akin to the FEDWeb system described above. The FSSWeb system, like the FEDWeb system, is not universally used by the agencies. A few agencies submit their data in alternative formats (e.g., ASCII text files, Excel spreadsheets). In the 2006-2008 survey cycle, 47 agencies reported via FSSWeb, 13 agencies or subagencies reported using Excel files, and, ironically, NSF itself provided an ASCII file.

3

Note that, to be included in the survey, an institution must perform R&D, even though data on funding for topics other than R&D are included in the survey and the report.

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×

TABLE 2-5 Categories for Reporting Federal Science and Engineering (S&E) Support to Universities, Colleges, and Nonprofit Institutions

Category of Support

Academic Institutions

Nonprofit Institutions

R&D

X

X

Fellowships, traineeships, and training grants

X

 

R&D plant and equipment

X

X

Facilities and equipment for instruction in S&E

X

 

General support of S&E

X

 

Other activities related to S&E

X

 

All other activities

X

 

SOURCE: National Science Foundation, Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions, Overview (FY 2007 cycle). Available: http://www.nsf.gov/statistics/srvyfedsupport/.

The contractor that carries out the survey for SRS must convert these multiple formats into the FSSWeb data system.

Since this survey does not ask for a breakdown of R&D funding by fields of S&E, the problems with completing it are different from those for the federal funds survey. Agencies find it difficult to report funding for each college or university in the six categories requested.

SUMMARY

The federal funds survey and the federal support survey are challenged and fragile, but they are not broken. With its limited resources, SRS has been able to implement a patchwork of improvements and to fund a series of studies that together provide a basis for continuing to provide an indication of the size and direction of federal R&D spending while laying the groundwork for making improvements in the future.

The surveys have a number of problems. The panel judges that two of them—delays in the assembly of data from the reporting agencies and in the publication of results and problems with the taxonomy of fields of S&E—are not solvable in the near term.

If conducted using the current methodology into the future, the surveys are likely to remain fragile, subject as they are to the reporting decisions made by agencies that do not always accord them a high priority and in an environment in which good practices are not always guaranteed and good documentation is not always available. In the next chapter, the panel turns to a discussion of short-term possibilities for improvement. We lay out a series of recommendations for the short term that will serve to keep the surveys viable while establishing the basis for major improvements in both the process of gathering data and the way they are reported.

Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 13
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 14
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 15
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 16
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 17
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 18
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 19
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 20
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 21
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 22
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 23
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 24
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 25
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 26
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 27
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 28
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 29
Suggested Citation:"2 The Current Surveys: Challenges and Opportunities." National Research Council. 2010. Data on Federal Research and Development Investments: A Pathway to Modernization. Washington, DC: The National Academies Press. doi: 10.17226/12772.
×
Page 30
Next: 3 Improving the Current System in the Short Term »
Data on Federal Research and Development Investments: A Pathway to Modernization Get This Book
×
Buy Paperback | $42.00 Buy Ebook | $33.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Two surveys of the National Science Foundation's Division of Science Resources Statistics (SRS) provide some of the most significant data available to understand research and development spending and policy in the United States. These are the Survey of Federal Funds for Research and Development and the Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions. These surveys help reach conclusions about fundamental policy questions, such as whether a given field of research is adequately funded, whether funding is balanced among fields, and whether deficiencies in funding may be contributing to a loss of U.S. scientific or economic competitiveness.

However, the survey data are of insufficient quality and timeliness to support many of the demands put on them. In addition the surveys are increasingly difficult to conduct in times of constrained resources, and their technological, procedural, and conceptual infrastructure has not been modernized for procedure or content.

Data on Federal Research and Development Investments reviews the uses and collection of data on federal funds and federal support for science and technology and recommends future directions for the program based on an assessment of these uses and the adequacy of the surveys. The book also considers the classification structure, or taxonomy, for the fields of science and engineering.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!