2
The Current Surveys: Challenges and Opportunities

The National Science Foundation Act of 1950 charges the National Science Foundation (NSF) with providing “a central clearinghouse for the collection, interpretation, and analysis of data on scientific and engineering resources, and to provide a source of information for policy formulation by other agencies of the Federal Government.” NSF has carried out that task for many years through its Division of Science Resources Statistics (SRS) and its predecessor, the Division of Science Resources Studies.

SRS manages a family of five surveys bundled into the Research and Development Statistics Program, two of which are specifically designed to elicit information about the scope of spending on research and development (R&D) by federal agencies: the Survey of Federal Funds for Research and Development (the federal funds survey) and the Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions (the federal support survey). Table 2-1 provides details on the five surveys in the Research and Development Statistics Program.

The federal funds survey asks all federal departments and agencies that conduct or support R&D for information about their financial investments in R&D, and the federal support survey asks those same agencies to report on their financial investments for R&D in individual institutions as well as their spending on a range of educational and student assistance programs related to science and engineering (S&E). Both surveys are administered annually.

The National Research Council (NRC) has reviewed these surveys in two prior studies (National Research Council, 2000, 2005b). These reviews



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 13
2 The Current Surveys: Challenges and Opportunities T he National Science Foundation Act of 1950 charges the National Science Foundation (NSF) with providing “a central clearinghouse for the collection, interpretation, and analysis of data on scien- tific and engineering resources, and to provide a source of information for policy formulation by other agencies of the Federal Government.” NSF has carried out that task for many years through its Division of Sci- ence Resources Statistics (SRS) and its predecessor, the Division of Science Resources Studies. SRS manages a family of five surveys bundled into the Research and Development Statistics Program, two of which are specifically designed to elicit information about the scope of spending on research and development (R&D) by federal agencies: the Survey of Federal Funds for Research and Development (the federal funds survey) and the Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institu- tions (the federal support survey). Table 2-1 provides details on the five surveys in the Research and Development Statistics Program. The federal funds survey asks all federal departments and agencies that conduct or support R&D for information about their financial investments in R&D, and the federal support survey asks those same agencies to report on their financial investments for R&D in individual institutions as well as their spending on a range of educational and student assistance programs related to science and engineering (S&E). Both surveys are administered annually. The National Research Council (NRC) has reviewed these surveys in two prior studies (National Research Council, 2000, 2005b). These reviews 

OCR for page 13
 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS TABLE 2-1 The SRS Research and Development Statistics Program Survey Universe Description Survey of Federal An annual census survey of the 15 federal Federal Funds departments and departments, their 70 subagencies, and 15 for Research and agencies that independent agencies that conduct R&D Development conduct and/or programs. It collects information about the (federal funds support R&D federal funding of R&D in the United States survey) programs for three fiscal years. The survey measures federal support of, and participation in, national scientific activities in terms of obligations and outlays. Conducted since the early 1950s. Survey of Federal Federal An annual census survey of the same agencies Science and departments and that respond to the federal funds survey, which Engineering agencies that report only if they had obligations for science Support to conduct and/or and engineering to universities, colleges, or Universities, support R&D nonprofit institutions during the past fiscal year. Colleges, and programs Agencies report obligations separately for each Nonprofit academic institution and nonprofit organization Institutions they fund. (federal support survey) Survey of Academic An annual census survey that collects Research and institutions with information on separately budgeted R&D Development greater than expenditures by academic institutions in the Expenditures $150,000 in R&D United States and outlying areas. Conducted at Universities expenditures since FY 1972, it collects information on R&D and Colleges in the previous expenditures by source of funds and by academic (academic R&D year or doctoral field using a taxonomy similar (but not identical) expenditures programs in R&D to the one used in the federal funds survey. survey) Survey of For-profit R&D- An annual sample survey that intends to include Industrial performing or represent all for-profit R&D-performing Research and companies, companies, either publicly or privately held. It Development whether publicly is the primary source of information on R&D (industrial R&D or privately held performed by industry in the 50 states and survey) the District of Columbia. Data are collected in the technology categories of biotechnology, software, materials synthesis and processing, and others. Research in nanotechnology is separately identified and asked for in the 4 categories above. Respondents are asked to report energy R&D in areas of fossil fuels, geothermal and solar, nuclear, and all other energy sources. Survey of State State An annual census survey of 423 state Research and organizations organizations and state agencies that sponsor Development and agencies that research and development. Begun in 2006, the Expenditures sponsor research survey measures the extent of R&D activity and development performed and funded by the 50 states, the District of Columbia, and Puerto Rico. Data are not collected by fields of science and engineering.

OCR for page 13
 THE CURRENT SURVEYS: CHALLENGES AND OPPORTUNITIES focused broadly on the entire portfolio of SRS R&D surveys, devoting only minimal attention to the federal funds and federal support surveys. The 2005 report, however, did examine whether the data collected in the federal funds survey were relevant and adequate for their intended uses, and it recommended that SRS reconsider several aspects of the survey operation in order to modernize it. The report concluded that SRS could improve its operations by collecting the information needed to complete the federal science and technology budget framework recommended in an earlier NRC report, Allocating Federal Funds for Science and Technology (National Research Council, 1995). The report also urged SRS to begin to work with the U.S. Office of Management and Budget (OMB), under the auspices of the E-Government Act of 2002, which had been recently enacted to develop guidance for standardizing the development and dis- semination of R&D project-level data as part of an upgraded administrative records–based system. The 2005 NRC report also recommended improvements in the fed- eral support survey. It found the survey to be a useful supplement to the federal funds survey—but also concluded that data collection was cumbersome and time-consuming and needed significant modernization. One option offered for modernizing the survey is to more intensively use microdata in administrative records that are part of the standardized, automated reporting systems in the key federal agencies that provide the bulk of federal support to academic and nonprofit institutions. The 2005 report also briefly considered issues regarding the taxonomy of fields of S&E used by these surveys, recommending that OMB initiate a review of its Classification of Fields of Science and Engineering, last published in 1978 as Directive 16, and that OMB appoint the SRS division of NSF as the lead agency for the effort. Since these earlier studies, the environment of the surveys on fed- eral R&D spending has changed dramatically. Under the guidance of the E-Government Act and subsequent legislation, OMB has made progress in developing administrative databases for contracts and grants, establish- ing new standards for the quality of the data, and establishing guidance to making the new databases available to the public in a timely manner. In addition, SRS commissioned a major study by Macro International to assess the quality of the information provided in the surveys, particularly with regard to the classification and reporting of fields of S&E (Macro International, 2008). OMB also assigned responsibility to SRS for review- ing and upgrading the fields of S&E classification, as recommended by the 2005 NRC panel. In this chapter, our panel examines the federal funds survey and the federal support survey in order to provide the basis for updated and more

OCR for page 13
 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS focused advice on collecting more accurate, timely, and useful data of the type now collected by these surveys.1 SURVEY OF FEDERAL FUNDS FOR RESEARCH AND DEVELOPMENT Since the early 1950s, SRS has conducted the federal funds survey as an annual census survey of federal science and technology agencies. There are now 15 federal departments, 70 subagencies, and 15 independent agen- cies that conduct R&D programs.2 The survey is conducted for SRS by a private-sector contractor (Macro International). The survey has a response rate of 100 percent for both reporting units and survey items. This high rate of survey completion comes at a high price, as discussed later in this chapter and addressed more fully in Chapter 3, because the release of the survey results is often delayed for a year or more awaiting the receipt of reports from all of the reporting departments and agencies. Years Covered Data collected on the federal funds survey cover three federal fiscal years: actual (final) expenditures for the most recently completed fiscal year, preliminary estimates of actual expenditures for the current fiscal year, and projected expenditures for the forthcoming fiscal year based on the presi- dent’s budget submission to Congress. The best-documented data are those for the completed fiscal year, which are based on actual R&D outlays made or R&D obligations entered into by federal entities. The data for the cur- rent fiscal year are a stew of amounts in the congressional appropriations, obligation actions to date, and apportionment and reprogramming decisions 1Information in this chapter is based on SRS’s online survey descriptions, documentation provided to the panel by SRS, the findings of a quality profile on the Research and Develop- ment Statistics Program (Bailar, 2004), presentations made at the panel workshop (see Ap- pendix C), and additional information provided by SRS at the panel’s request. 2 Departments: Agriculture (10 subagencies); Commerce (5 subagencies); Defense (15 subagencies); Education; Energy; Health and Human Services (11 subagencies); Home- land Security (4 subagencies); Housing and Urban Development; Interior (4 subagencies); Justice (3 subagencies); Labor (6 subagencies); State; Transportation (9 subagencies); Treasury (3 subagencies); and Veterans Affairs. Independent agencies: Agency for International Devel- opment; Appalachian Regional Commission; Broadcasting Board of Governors; Consumer Product Safety Commission; Environmental Protection Agency; Federal Communications Commission; Federal Trade Commission; General Services Administration; Library of Congress; National Aeronautics and Space Administration; National Archives and Records Administration; National Science Foundation; Nuclear Regulatory Commission; Smithsonian Institution; Social Security Administration.

OCR for page 13
 THE CURRENT SURVEYS: CHALLENGES AND OPPORTUNITIES made and documented during the year. The least reliable data are those for the president’s budget year. These projections represent the amounts in the administration’s budget proposals not yet passed by Congress. The overall amounts in the preliminary and projected estimates are subject to revision as the president’s budget is executed and authorization, appropriation, defer- ral, and apportionment actions completed after these data were collected are realized over time. There is little hard information on which to base estimates of expen- ditures by fields of S&E for the preliminary and projected years. The Congressional Research Service (CRS) reported that these estimates are computed as a percentage of current-year spending. Agency representatives told CRS that projecting by broad fields of S&E is, at best, an “educated guess” (Congressional Research Service, 2000, p. 10). These uncertainties in reporting and the inevitable discrepancies among budgets, plans, and actual expenditures contribute to fairly large revisions in the data as the agencies accumulate information on their actual expenditures over time. As Table 2-2 indicates, some of these revi- sions can be quite significant. During periods of budget uncertainty, as when there is a change in administration, these revisions are even more marked. In 2002, for example, the projected level of federal R&D spend- ing was about 6 percent lower than finally reported; the year after, the total of agency projections was about 5 percent greater than the total of final estimates made when actual spending data became available. Therefore, generally the projected data are thought to be useful only in suggesting broad trends. Variables Collected Data collected for each year include outlays and obligations. The key variables collected and their definitions (taken from the survey question- naire) are found in Table 2-3. Only the 10 largest agencies report obliga- tions for plant by performer and state. They account for about 97 percent of total R&D and R&D plant obligations each year. Only the six largest agencies report the data for obligations to colleges and universities by field of science. Reporting agencies are challenged by the fact that their internal records do not use the categories requested by SRS. For example, internal records do not separate expenditures on R&D plant from expenditures for the conduct of R&D. This is one example of the kind of measurement error that arises because of the difficulty agencies have in translating the data from the cat- egories in which they are maintained on agency records into the categories that are requested by SRS.

OCR for page 13
 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS TABLE 2-2 Federal Obligations for R&D and R&D Plant: Projected, Preliminary, and Final, 1990-2008 (in millions of dollars) Percentage Change, Percentage Change, Projected to Projected to Year Projected Preliminary Final Preliminary Final 1990 68,524 63,353 65,831 –7.5 –3.9 1991 66,690 66,227 64,148 –0.6 –3.8 1992 73,484 75,586 68,577 +2.9 –6.7 1993 75,045 75,303 70,415 +0.3 –6.2 1994 74,436 72,818 69,451 –2.2 –6.7 1995 71,746 73,029 70,443 +1.8 –1.8 1996 70,906 71,048 69,399 +0.2 –2.1 1997 70,149 71,996 71,753 +2.6 +2.3 1998 71,593 74,202 73,914 +3.6 +3.2 1999 75,330 77,650 77,386 +3.1 +2.7 2000 77,186 81,772 77,356 +5.9 +0.2 2001 83,609 85,452 84,003 +2.2 +0.5 2002 84,938 97,465 90,158 +14.7 +6.1 2003 103,114 101,008 97,928 –2.0 –5.0 2004 105,220 106,488 105,371 +1.2 +0.1 2005 110,193 113,118 112,995 +2.7 +2.5 2006 112,076 116,873 112,271 +4.3 +0.1 2007 116,417 116,700 +0.2 2008 113,213 SOURCE: National Science Foundation (2009a). TABLE 2-3 Variables and Definitions of Items Collected on the Survey of Federal Funds for Research and Development Variable Definition Outlays for R&D and R&D Outlays (expenditures) represent the amounts for plant by year checks issued and cash payments made for research and development activities and plant (facilities and fixed equipment) during the fiscal year. Obligations for R&D and Obligations are the amounts of orders placed, contracts R&D plant by year and sub-grants awarded, goods and services received, and similar transactions during a given period that will require payment by the grantee during the same or a future period for research and development activities and plant (facilities and fixed equipment) during the fiscal year.

OCR for page 13
 THE CURRENT SURVEYS: CHALLENGES AND OPPORTUNITIES TABLE 2-3 Continued Variable Definition Obligations for basic, Basic research is systematic study directed toward fuller applied, and total research by knowledge or understanding of the fundamental aspects field of S&E (detailed field of phenomena and of observable facts without specific for past fiscal year, broad applications towards processes or products in mind. field for current and next years) Applied research is systematic study to gain knowledge or understanding necessary to determine the means by which a recognized or specific need may be met. Development is systematic application of knowledge or understanding directed toward the production of useful materials, devices, and systems or methods, including design, development, and improvement of prototypes and new processes to meet specific requirements. A field of S&E is a recognized category of specialized expertise within S&E as defined by OMB Directive No. 16. Obligations for basic and A performer is either an intramural group or organization applied R&D by performer carrying out an operation or function or an extramural by year organization or person receiving support or providing services under a contract or grant. Obligations to individual Federally funded R&D centers are S&E performing federally funded R&D organizations that are exclusively or substantially funded by centers for past year the federal government, administered by an industrial firm, a college or university, or another nonprofit institution. Obligations for R&D for Foreign performers include foreign citizens, organizations, foreign performers for past universities and colleges, or governments as well as year international organizations (such as United Nations organizations). Obligations for R&D plant See above. by performer by year Obligations for R&D and States include state and local government agencies, excluding R&D plant by state for the colleges and universities. State data are reported only by the past year 10 agencies with largest R&D obligations. Obligations for R&D to Universities and colleges are institutions engaged in universities and colleges for providing resident and/or accredited instruction for at least field of S&E (detailed field for a 2-year program above the secondary school level. These past fiscal year, broad field for data are reported only by the six agencies with the largest current and next years) R&D obligations. SOURCE: NSF, Survey of Federal Funds for Research and Development, Questionnaire. Avail- able: http://www.nsf.gov/statistics/srvyfedfunds/surveys/srvynondod_fy03-05.pdf.

OCR for page 13
0 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS For example, actual data on outlays and obligations are available for only the past year, and agencies are asked to make their best esti- mates for both outlays and obligations for the current and future years. R&D plant data are underreported because of difficulties encountered by the Department of Defense (DoD), the National Aeronautics and Space Administration (NASA), and others in identifying and reporting them. DoD reports obligations for R&D plant funded under the agency’s appropriation for construction, but it is able to identify only a small portion of the R&D plant support from R&D contracts funded from DoD’s appropriation for research, development, testing, and evaluation. Similarly, NASA cannot separately identify the portions of industrial R&D contracts that apply to R&D plant, since these data are subsumed in the R&D data covering industrial performance. NASA R&D plant data for other performing sec- tors are reported separately. Categorization of Research and Development Activities A major issue affecting data accuracy is discrepancies in how federal agency staff assign their research spending into the basic research, applied research, and development categories, which may result in a lack of com- parability in reporting the categories among federal agencies. In its 2000 study, the Congressional Research Service observed that “while SRS pro- vides agencies and survey respondents with definitions of each data item collected on its surveys (e.g., basic research, applied research, and develop- ment, etc.), agencies, as well as individuals within agencies, interpret the definitions differently” (Congressional Research Service, 2000, p. 9). The CRS study further observed that some agencies found it difficult to determine whether certain activities, such as training, program evaluation, and construction of facilities, met the criteria for inclusion in the categories (p. 9). There is little doubt that individual staff members sometimes make arbitrary judgments in order to report in the requested categories. Quite often, these staff members work in agency budget offices that have been assigned the task of completing the SRS reports, rather than in the scientific and technical program offices, where staff might more readily distinguish among the categories. The panel is aware that the practice of characterizing scientific and technical inquiry as basic or applied research or development has long been controversial. The so-called linear model underlying these categories (basic research → applied research → development) has been met with some skep- ticism (Godin, 2005), and is widely understood to be a gross oversimplifica- tion. This linear model often fails to describe the actual processes of moving from inquiry to application. Furthermore, whether a particular project or program falls into one or another of the categories depends, to some degree,

OCR for page 13
 THE CURRENT SURVEYS: CHALLENGES AND OPPORTUNITIES on the intentions and perceptions of those making the categorization. The panel chose not to delve into this subject or to make recommendations about it, but we recognize that it is one of the issues that will need to be addressed in order for SRS to act on our recommendations to move ahead with deriving the federal funding data reports from administrative data- bases, as this report recommends. Timeliness of Release Although the federal funds survey enjoys 100 percent response rates, this is achieved, in part, because of the SRS policy that the data are not released until all agencies have reported. This policy puts SRS at the mercy of the slowest respondent and can severely delay publication of the data. The delay in publication can be measured by the lag from the period of reference for the data to the time of publication. For example, the data for fiscal years 2006-2008 were solicited in mid-February 2007, with a due date in mid-April 2007. The last of the agency reports was not received until some eight months later, in mid-December 2007. Adding in the time for the lengthy process at SRS to aggregate and analyze the data as well as the agency inputs for release, publication of the data for these years did not occur until January 2009 (National Science Foundation, 2009a, p. 2). In this illustration, the “most recent” year data were actually being reported in January 2009 for outlays that had been completed by September 2006, some 28 months earlier. This was actually an improvement from prior years, when the final agency submissions were received as late as June of the year following the year of solicitation. Clearly, time lags of this duration, resulting from the failure of agen- cies to submit timely reports as well as from the lengthy procedures at SRS, reduce the usability of the data for real-time planning and policy-making purposes. Although SRS is able to publish the federal funding data without introducing error-prone imputation for missing agency reports, it does so at a high price: the data are usually well out of date by the time they are finally published. Collection Technology On the positive side, SRS has made significant enhancements in recent years to the procedures and technology used for collecting these data from federal agencies. FEDWeb, a web-based data collection system, is used to collect and manage data for the federal funds survey. Most data collection efforts, data imports, data editing, and trend checking are accomplished using FEDWeb. This web-based system helps improve survey reporting and reduce data collection and processing costs by offering respondents

OCR for page 13
 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS direct online reporting and editing. The web-based option has proven popular with most agency reporters. In the 2006-2008 survey cycle, 89 of the 100 reporting agencies submitted their data via FEDWeb. Three others submitted at least part of their data in Microsoft Excel files, which were then entered into FEDWeb by the SRS contractor. However, while modern- ization of the data collection method might have been expected to assist in improving the consistency and completeness of the data, the transition to web-based collection has not dramatically reduced the delays in data col- lection and reporting. TAXONOMY OF FIELDS OF SCIENCE AND ENGINEERING A key component of the federal funds survey is the taxonomy it uses to classify spending by fields of S&E. The taxonomy is anchored in a clas- sification system that was last updated by OMB in 1978 in its Directive 16 (U.S. Office of Management and Budget, 1978). A specific part of the panel’s charge is to assess this classification structure. The Challenge of Interdisciplinarity Several challenges are associated with the current taxonomy of fields of S&E used in the federal funds survey. A significant one is that it does not account for new and emerging science or the more recent phenom- enal growth in interdisciplinary, multidisciplinary, and transdisciplinary research. In this report, multidisciplinary approaches are defined as when researchers maintain their disciplinary and professional perspectives but add breadth and available knowledge, information, and methods from other dis- ciplines. Interdisciplinary approaches integrate separate disciplinary data, methods, tools, concepts, and theories in order to create a holistic view or common understanding of a complex issue, question, or problem. In trans­ disciplinary approaches, researchers develop comprehensive frameworks through an overarching synthesis, such as general systems, policy sciences, feminism, ecology, and sociobiology, in which the disciplines lose their identity. The importance of updating the taxonomy to better incorporate inter- disciplinary research is widely recognized by policy makers, funding agen- cies, professional organizations, and across academia. The growing role of research involving more than one discipline is a serious challenge to any taxonomy of fields and therefore to gathering, analyzing, and using federal funds data based on a single-field taxonomy. There are four primary drivers of interdisciplinary research—the inher- ent complexity of nature and society, the desire to explore problems and

OCR for page 13
 THE CURRENT SURVEYS: CHALLENGES AND OPPORTUNITIES questions that are not confined to a single discipline, the need to solve societal problems, and the power of new technologies (National Research Council, 2005a, p. 40). The heightened momentum for interdisciplinary research dates from the late 1970s, roughly the same time as when the current taxonomy used in the federal funds survey was developed. International competition in science- based fields of high technology propelled greater involvement and investment in interdisciplinary research in engineering and manufacturing, computers, biotechnology, and biomedicine. Breakthroughs in scientific research were also occurring increasingly at the interstices of established fields of inquiry. In 1986, a report from the National Research Council assessing major fields of physics noted “rapidly emerging interdisciplinary advances—which are enriching all of science” (National Research Council, 1986a, p. 15). Subsequent reports have continued to document these trends. A 1990 NRC report on interdisciplinary research called attention to new intellec- tual understandings of biological systems, problem complexity, the costs of instrumentation and facilities, and increased collaborations between the life sciences and medicine as well as the physical sciences and engineering (National Research Council, 1990). More recently, an NRC report on the NSF Materials Research Science and Engineering Centers Program affirmed that “frequently, the most exciting and important advances in materials science and engineering occur at the interfaces between, or by unconven- tional combinations of, traditional disciplines” (National Research Council, 2007a, p. 8). Funding agencies have responded to these developments with new structures and strategies. The National Institutes of Health Roadmap, for example, recognizes that new combinations of analytical skills and disci- plines and new technologies are needed to deal with complex challenges of human health and well-being. New technologies of molecular imag- ing, nanomedicine, and bioinformatics are prominent in the life sciences. New tools of quantitative and computer-assisted mathematical analysis and advanced computing power are also facilitating the sharing of large quanti- ties data across disciplinary boundaries in areas as diverse as medicine and the geosciences (National Institutes of Health, 2009). The implications may not merely be technical. In the journal Science, Alan Leshner observed that “new technologies are driving scientific advances as much as the other way around,” allowing new approaches to older questions and posing new ones (Leshner, 2004, p. 729). The Fluid Boundaries of Traditional Disciplines In addition to the challenge of interdisciplinary research, the changing nature of disciplines has contributed to making the 1978-era taxonomy less

OCR for page 13
 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS relevant to today’s world. The current interface between physics and chemis- try, for example, has been crossed so often in both directions that authors of Scientific Interfaces and Technological Applications reported “its exact loca- tion is obscure” and “its passage is signaled more by gradual changes in lan- guage and approach than by any sharp demarcation in content” (National Research Council, 1986b, p. 53). Interactions and cross-fertilizations that characterize the interface have been sources of continual advances in con- cepts and applications across the science of molecules and atoms, surfaces and interfaces, and fluids and solids. “Thirty years ago,” Norman Burkhard observed, “the difference between a physicist and a chemist was obvious. Now we have chemists who are doing quantum-level, fundamental studies of material properties, just like solid-state physicists. There’s almost no dif- ference” (National Research Council, 2005a, p. 54). Inadequacies of the Current Taxonomy The taxonomy of fields of S&E used in the federal funds survey takes little notice of the trends in the conduct and organization of R&D discussed above. The current taxonomy includes eight broad field categories, each including a number of detailed fields. The broad fields are life sciences, psy- chology, physical sciences, environmental sciences, mathematics and com- puter sciences, engineering, social sciences, and other sciences. Each broad field includes a subfield of “not elsewhere classified” (n.e.c.). SRS provides illustrative disciplines for each detailed field in its guidance documentation for survey respondents. The n.e.c. category is used for both multidisciplinary projects involving more than one field and single-discipline projects for which a separate field is not part of the taxonomy. The lack of up-to-date categories has resulted in overuse of the n.e.c. category. “Not elsewhere classified” has become an amorphous category that lumps together many unrelated types of research, including work in new subfields, emergent fields, established interdisciplin- ary fields, cross-cutting initiatives, “problem-focus” areas of research, and miscellaneous “other.” Moreover, it fails to discriminate multidisciplinary and genuinely integrated interdisciplinary activities. Furthermore, there is no specific way to report interdisciplinary research. A further problem with the R&D taxonomy is that, at the detailed level, SRS uses somewhat different classifications structures for the fed- eral funds survey and its other surveys. Table 2-4 compares the various taxonomies used by SRS for the federal funds survey with another of its surveys—the Survey of Research and Development Expenditures at Colleges and Universities (known as the academic R&D expenditures survey)—and the “official” taxonomy from OMB Directive 16. There are potentially confusing differences among the three.

OCR for page 13
 THE CURRENT SURVEYS: CHALLENGES AND OPPORTUNITIES TABLE 2-4 Comparison of Taxonomies Federal Funds Survey Academic R&D Survey OMB Directive 16 Life Sciences Life Sciences Life Sciences Biological sciences Biological sciences Biological (excluding environmental) Environmental biology Agricultural science Agricultural sciences Medical science Medical Clinical medical Other medical Life sciences, n.e.c. Other Psychology Psychology (total) Psychology Biological aspects Biological aspects Social aspects Social aspects Psychological sciences, Other Psychological sciences, n.e.c. n.e.c. Physical sciences Physical sciences (total) Physical sciences Astronomy Astronomy Chemistry Chemistry Physics Physics Physical sciences, n.e.c. Other Physical sciences, n.e.c. Environmental sciences Environmental sciences Environmental sciences— (total) terrestrial and extraterrestrial Atmospheric sciences Atmospheric Atmospheric sciences Geological sciences Earth Geological sciences Oceanography Oceanography Environmental sciences, Other Environmental sciences, n.e.c. n.e.c. Mathematics Mathematical sciences Mathematics (total) Computer sciences Computer sciences (total) Mathematics and computer sciences, n.e.c. Engineering Engineering (total) Engineering Aeronautical Aeronautical and Aeronautical astronautical Astronautical Astronautical Bioengineering/ biomedical engineering Chemical Chemical Chemical Civil Civil Civil continued

OCR for page 13
 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS TABLE 2-4 Continued Federal Funds Survey Academic R&D Survey OMB Directive 16 Electrical Electrical Electrical Mechanical Mechanical Mechanical Metallurgical and Metallurgical and Metallurgy and materials material material Engineering, n.e.c. Other Engineering, n.e.c. Social sciences Social sciences (total) Social sciences Anthropology Anthropology Economics Economics Economics History Linguistics Political science Political science Political science Sociology Sociology Sociology Social sciences, n.e.c. Other Social sciences, n.e.c. Other sciences, n.e.c. Other sciences (total) Other sciences, n.e.c. NOTE: n.e.c. = not elsewhere classified. SOURCE: National Science Foundation, Survey of Federal Funds for Research and Devel- opment, Questionnaire. Available: http://www.nsf.gov/statistics/srvyfedfunds/surveys/ srvynondod_fy03-05.pdf; National Science Foundation, Survey of Research and Development Expenditures at Universities and Colleges, FY 2008 Questionnaire. Available: http://www. nsf.gov/statistics/srvyrdexpenditures/surveys/srvyrdexpenditures_2008.pdf; U.S. Office of Management and Budget (1978). Recent SRS Efforts to Address Problems The SRS is well aware of the challenges faced by its current taxonomy of fields of S&E. As noted above, at the direction of OMB it has undertaken a project that is to lead to recommendations on how to revise and update OMB Directive 16, on which the taxonomy is based. SRS has commissioned or conducted several recent studies and workshops to address these issues. SRS convened an interagency working panel meeting (hosted at SRI Inter- national on October 21-22, 2004), sponsored a study of agency reporting practices, and commissioned a detailed internal review of a proposed social sciences taxonomy. In NSF, a process was initiated to add a data element to its internal R&D project reporting system (called the “E-jacket”) to capture fields of science codes. In this section, we summarize the findings of the interagency working meeting and the study of agency reporting practices. The October 2004 SRI workshop focused on means of updating the fields of S&E taxonomy. This workshop included discussions about the needs of the users of SRS data; the nature of S&E disciplines and multi- disciplinary, interdisciplinary, and transdisciplinary work; criteria for good

OCR for page 13
 THE CURRENT SURVEYS: CHALLENGES AND OPPORTUNITIES classification systems; data mapping and mining; international issues; and approaches to improving classification systems (Cheney and Park, 2005, p. 2). The workshop participants included the government officials who provide the data and two major groups of users: representatives of policy communities, who use the information to make policy decisions about S&E investments, and representatives of the research community, who use the data to study the characteristics of the S&E enterprise. The workshop identified three options for updating the taxonomy. One is to make only minor revisions to the taxonomy in order to maintain continuity in the data, particularly at the higher levels of aggregation. A second is to expand the taxonomy to increase the level of detail, although that might compound the difficulty of reporting the fields. A third is to restructure reporting to permit respondents to report more than one field, enabling analysis of multidisciplinary, interdisciplinary, and transdisciplinary research. The workshop participants emphasized the importance of consult- ing with users, representatives of the disciplines, survey respondents, and others as SRS moves forward with any revision of the fields of science and engineering, as well as to consider international, educational, and employ- ment taxonomies in the process (Cheney and Park, 2005, p. 7). Reaching out to other federal agencies, SRS organized and hosted an interagency working group on June 16, 2006, consisting of representatives from OMB and the six largest research-supporting federal agencies: NSF, the National Institutes of Health (NIH), DoD, the Department of Energy (DoE), NASA, and the U.S. Department of Agriculture (USDA). Its aim was to lay the groundwork for the review and revision of OMB Directive 16, identify the primary uses of R&D data organized using this classification, and acknowledge the problems associated with implementing new classification schemes across agencies. Importantly, SRS commissioned Macro International to undertake a study to document (1) the taxonomy each agency uses for analyzing or reporting its research funds (or combined R&D totals, if appro- priate); (2) the purposes for which the taxonomy is used; (3) how each agency actually goes about classifying its research/R&D totals into this taxonomy; and (4) how each agency uses the fields of science and engineering taxonomy reported to the federal funds survey (Macro International, 2008, p. 4). The study of agency reporting practices by Macro International docu- mented interviews with representatives of the 16 major agencies that report on the federal funds survey (Macro International, 2008). The top five sponsors of R&D (NIH, DoD, DoE, NSF, and NASA), accounting for 86 percent of all federal research obligations in FY 2005, stated that they do not use the federal funds taxonomy for their own program manage- ment or budgeting. As a result, they consider the data they submit to the federal funds survey to be of questionable value (to them) because of the

OCR for page 13
 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS methods they must use to classify and report them (Macro International, 2008, p. 5). For example, NIH respondents reported that the agency has no use for the taxonomy because it does not reflect the current state of science and does not include reporting for interdisciplinary research. The DoD respondents reported similar concerns, noting that much of their current cutting-edge research overlaps two or more of the disciplines in the cur- rent taxonomy. The DoE respondents reported that the current taxonomy may limit how agencies report their funding. Furthermore, DoE reported confusion about how to classify its own programs, noting that it could more accurately report research activities by theme (e.g., energy, envi- ronment, national security). For internal operations, the NSF research program staff (not to be confused with the SRS staff) reported that the current taxonomy is adequate but recognized the need for a taxonomy that reflects new scientific areas and interdisciplinary research. Although the National Institute of Standards and Technology, NSF, DoE, and DoD reported that the resulting data are of some benefit to them, they con- curred that the data have limited value for the reasons noted above. Some interesting uses of the taxonomy were reported. For example, the DoD participant reported using the federal funds taxonomy to compare DoD research funding with funding for the education of graduate students by the U.S. Department of Education. The Macro International report rec- ommended revising the taxonomy to be useful to the reporting agencies, warning that methods used to collect and report data, as well as the quality of the resulting information, are otherwise unlikely to improve. SURVEY OF FEDERAL SCIENCE AND ENGINEERING SUPPORT TO UNIVERSITIES, COLLEGES, AND NONPROFIT INSTITUTIONS The federal support survey is an annual census survey of the same agen- cies that respond to the federal funds survey, but they report only if they had obligations for S&E to universities, colleges, or nonprofit institutions during the past fiscal year (the time frame of this survey). The data that are collected from the 19 agencies (in FY 2006) that made such obligations include federal obligations for R&D; R&D plant; facilities and equipment for S&E instruction; S&E fellowships, traineeships, and training grants; general support for S&E; and other S&E activities. Like the federal funds survey, this survey has a response rate of 100 per- cent (both unit and item) because of the SRS policy to withhold release of the data until all agencies have reported. Also like the federal funds survey,

OCR for page 13
 THE CURRENT SURVEYS: CHALLENGES AND OPPORTUNITIES the policy of withholding release of the federal support data until all agen- cies have reported is a contributing factor to excessive lags in publication of the survey results. For example, the data for FY 2006, which ended on September 30, 2006, were solicited in mid-February 2007 with a requested due date of mid-April 2007. The data from the last 2 of the 19 reporting agencies were not received by SRS until November 2007. The brief analysis and abbreviated set of summary tables for FY 2006 were finally released by SRS in October 2008 (National Science Foundation, 2008b), and the full set of detailed information by agency with institutional rankings was not published until March 2009 (National Science Foundation, 2009b). In this survey, on behalf of SRS, a contractor asks departments and agencies to complete a web-based survey instrument for each university and college for which they obligated R&D funding during the previous fiscal year.3 There are approximately 1,200 such colleges and universities. In FY 2006, the reporting agencies obligated over $28 billion to those insti- tutions, most of it going to the top 100 universities and colleges (National Science Foundation, 2009b, Table 17). Agencies also complete a survey instrument for each of the approxi- mately 1,300 nonprofit institutions to which they obligated funds. The list of nonprofit institutions receiving federal R&D funds has been grow- ing. Obligations to universities and colleges are reported in six categories, whereas the obligations to nonprofit institutions are reported only for R&D and R&D plant (see Table 2-5). In FY 2006, over $6.5 billion was obligated to nonprofit institutions, about 80 percent of those funds going to the top 100 recipient nonprofits (National Science Foundation, 2009b, Table 29). Respondents are instructed that totals for R&D transfers to universi- ties should be similar to those reported by them for R&D to academic institutions in the federal funds survey. If differences exist, respondents are asked to explain. One reason for differences in totals is that methods differ for reporting funds that are transferred from one agency to another before being sent to an institution. For example, the federal support survey asks the agency that distributes the funds to a university to report the transac- tion, whereas the federal funds survey asks the agency transferring the funds to the agency that ultimately sent them on to report the transaction. This survey also uses a web-based collection tool akin to the FEDWeb system described above. The FSSWeb system, like the FEDWeb system, is not universally used by the agencies. A few agencies submit their data in alter- native formats (e.g., ASCII text files, Excel spreadsheets). In the 2006-2008 survey cycle, 47 agencies reported via FSSWeb, 13 agencies or subagencies reported using Excel files, and, ironically, NSF itself provided an ASCII file. 3 Notethat, to be included in the survey, an institution must perform R&D, even though data on funding for topics other than R&D are included in the survey and the report.

OCR for page 13
0 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS TABLE 2-5 Categories for Reporting Federal Science and Engineering (S&E) Support to Universities, Colleges, and Nonprofit Institutions Academic Nonprofit Category of Support Institutions Institutions R&D X X Fellowships, traineeships, and training grants X R&D plant and equipment X X Facilities and equipment for instruction in S&E X General support of S&E X Other activities related to S&E X All other activities X SOURCE: National Science Foundation, Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions, Overview (FY 2007 cycle). Available: http://www.nsf.gov/statistics/srvyfedsupport/. The contractor that carries out the survey for SRS must convert these multiple formats into the FSSWeb data system. Since this survey does not ask for a breakdown of R&D funding by fields of S&E, the problems with completing it are different from those for the federal funds survey. Agencies find it difficult to report funding for each college or university in the six categories requested. SUMMARY The federal funds survey and the federal support survey are challenged and fragile, but they are not broken. With its limited resources, SRS has been able to implement a patchwork of improvements and to fund a series of studies that together provide a basis for continuing to provide an indi- cation of the size and direction of federal R&D spending while laying the groundwork for making improvements in the future. The surveys have a number of problems. The panel judges that two of them—delays in the assembly of data from the reporting agencies and in the publication of results and problems with the taxonomy of fields of S&E—are not solvable in the near term. If conducted using the current methodology into the future, the surveys are likely to remain fragile, subject as they are to the reporting decisions made by agencies that do not always accord them a high priority and in an environment in which good practices are not always guaranteed and good documentation is not always available. In the next chapter, the panel turns to a discussion of short-term possibilities for improvement. We lay out a series of recommendations for the short term that will serve to keep the surveys viable while establishing the basis for major improvements in both the process of gathering data and the way they are reported.