5
Measuring R&D Spending in the Federal Government

The question of how much the federal government is spending on research and development should be answerable in a fairly straight forward manner. One would hope for a tally that includes all federal government agencies that engage in research and development; that there would be a consistent definition of R&D applied across the government; and that the question could be answered in a standard manner with recurrent collections associated with continuing processes that would minimize the burden of collecting the information and enhance data quality. However, none of these expectations are met in today’s Survey of Federal Funds for Research and Development. The survey does not have a comprehensive list of all activities; it is not consistently defined and applied across the government; and it is not associated with ongoing budget and financial management processes in many agencies. Most importantly, it is not completed in a timely manner, so its worth is somewhat diluted by the age of the data when they finally become available.

It is important to get this survey right. It is widely considered to be a primary source of information about federal funding for R&D in the United States by government, academia, and the science community. Data from the survey are expected to take their place alongside the U.S. Office of Management and Budget’s (OMB’s) budget authority information that underscores decisions on the federal Science and Engineering (S&E) budget and serve as the basis for the extensive program of analysis of the federal S&E budget by the American Association for the Advancement of Science and other organizations. The possible shortcomings in the survey are highlighted in recent studies that have examined the source of the discrepancy between reports of



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 102
Measuring Research and Development Expenditures in the U.S. Economy 5 Measuring R&D Spending in the Federal Government The question of how much the federal government is spending on research and development should be answerable in a fairly straight forward manner. One would hope for a tally that includes all federal government agencies that engage in research and development; that there would be a consistent definition of R&D applied across the government; and that the question could be answered in a standard manner with recurrent collections associated with continuing processes that would minimize the burden of collecting the information and enhance data quality. However, none of these expectations are met in today’s Survey of Federal Funds for Research and Development. The survey does not have a comprehensive list of all activities; it is not consistently defined and applied across the government; and it is not associated with ongoing budget and financial management processes in many agencies. Most importantly, it is not completed in a timely manner, so its worth is somewhat diluted by the age of the data when they finally become available. It is important to get this survey right. It is widely considered to be a primary source of information about federal funding for R&D in the United States by government, academia, and the science community. Data from the survey are expected to take their place alongside the U.S. Office of Management and Budget’s (OMB’s) budget authority information that underscores decisions on the federal Science and Engineering (S&E) budget and serve as the basis for the extensive program of analysis of the federal S&E budget by the American Association for the Advancement of Science and other organizations. The possible shortcomings in the survey are highlighted in recent studies that have examined the source of the discrepancy between reports of

OCR for page 102
Measuring Research and Development Expenditures in the U.S. Economy the level of federal spending for R&D as reported in the federal funds survey and the amounts that performers in industry and academia report as receiving from the federal government. (These issues are addressed in Chapter 7). When data collections fail to fully meet the needs of users, substitutes spring up. As might be expected, there is a growing, new competition to the survey in the form of a microdata-based collection known as the RaDiUS database, also funded by NSF. In this chapter, we focus mainly on the current status of the survey of federal funds and its deployment within the federal government. In view of the uses put to the data as addressed in Chapter 2, we examine the adequacy and relevance of this data collection. Finally, we consider the RaDiUS database to determine the extent to which this data source compliments or competes with the federal funds survey. We recommend a reconsideration of several aspects of this survey operation to modernize it and improve its operations. Attention is also paid to examining these issues as they pertain to the annual, congressionally mandated survey of federal obligations to academic institutions. Is the accounting framework appropriate, that is, the use of obligations rather than authorizations or outlays? Can federal agency staffs that are asked each year to provide information about their R&D obligations be motivated to regard this task as a benefit rather than a burden? Are the federal funds collections maximizing the ability to report data from agency accounting systems in the form requested by NSF? Can the collection be placed on a microdata level, that is, comparable to Level 5 in RaDiUS? FEDERAL FUNDS SURVEY The federal funds survey collects data on federal support of national scientific activities in terms of budget obligations and outlays. For each year, survey data are to be provided for three fiscal years (FY)—the FY just past, the current FY, and the president’s budget year. Actual data are collected for the year just completed, while estimates are obtained for the current year and the budget year. Agencies are asked to submit the survey data on the same basis as the budget authority figures submitted to the OMB in January. Definitions of obligations and outlays are purposely the same as those in the U.S. budget—obligations represent the amount for orders placed, contracts awarded, services received, and similar transactions during a given period, regardless of when the funds were appropriated and when future payment of money is required, and outlays represent the amounts of checks issued and cash

OCR for page 102
Measuring Research and Development Expenditures in the U.S. Economy payments made during a given period, regardless of when the funds were appropriated or when the obligations were incurred. The reporting unit for the survey is the subagency or agency to whom the survey materials are sent. The survey is sponsored and funded by the National Science Foundation (NSF) and is carried out under contract by QRC Macro. The scope of the survey has changed over the years, reflecting the number of subagencies that now fund R&D, the kinds of places in which R&D is conducted, and the kinds of questions agencies have the resources to answer. Federal obligations for research to universities and colleges by agency and detailed science and engineering field were added to the survey in 1973. Federally Funded Research and Development Centers (FFRDCs) are also included. The Central Intelligence Agency and other security-related agencies are not included. In order to reduce respondent burden, this survey is tailored to the respondents: All are asked for information on total obligations and outlays for R&D and R&D plant; obligations for basic, applied, and total research, by field of S&E and by performer; and obligations to R&D plant, by performer. Some data are collected only for the immediate past year. These include obligations to individual FFRDCs and to foreign performers. Only the 10 largest R&D funding agencies are asked about the geographical distribution of obligations for R&D and the equipment and facilities where the R&D takes place. (These agencies account for about 97 percent of total R&D and R&D plant obligations each year.) Six agencies are asked to report on the distribution of R&D to universities by field of science and engineering and character of research. Only the Department of Defense (DoD) is asked to break out state obligations by research and development separately. Furthermore, NSF has further been sensitive to the ability of federal agencies to provide good data and has removed items, as well as added items, over the years. Among the data removed from collection are data for the special foreign currency program and detailed field of S&E data for estimated out-years. Some data were removed but later reinstated by popular demand. Data on foreign performers by region, country, and agency are an example of this reconsideration. To align with the president’s budget process, the frame for the survey is the list of federal agencies that report funding R&D obtained from information in the president’s budget submitted to Congress. In FY 2003, the 29

OCR for page 102
Measuring Research and Development Expenditures in the U.S. Economy agencies and 73 subagencies that have reported R&D data in OMB budget documents comprised the census for the data collection. The collection process focuses on key respondents in the agencies. For the most part, the persons assigned by agencies to complete these forms are budget analysts, not experts in the conduct of R&D. In this way, they are much like the normal respondents in the industry survey, who tend to be in the comptroller and accounting end of corporate operations. The respondents are identified by the contractor, QRC Macro, which keeps a master list of all respondents by agency and subagency, updating that list throughout the survey process. Quality and timeliness begin with these respondents. They are counted on to know their agencies, develop some expertise in concepts and definitions, doggedly followup to obtain responses from across the agency, summarize a variety of responses into an agency total, and report to the NSF contractor via the Internet. This process renews itself about the same time each year, when packets are mailed and respondents are granted access to the system to begin data entry. This is usually around the first week of March. The agencies are asked to return the completed questionnaires by April 15. Few are able to meet this deadline. The responses and subsequent processing drag on to the extent that data on federal obligations and outlays with a reference period of September 2001 were not available in brief form until June 2003, or in full detail until April of 2004—some 2 1/2 years later. (Issues of timeliness for all of the NSF surveys are addressed in Chapter 8.) The discussion of measurement problems in the survey reports is limited. Some agencies find it difficult to report certain items. Some agencies, such as the Department of Defense, do not include some relatively minor headquarters planning and administering of R&D programs in the full cost of R&D. R&D plant data are also underreported because of difficulties encountered by some agencies, particularly DoD and the National Aeronautics and Space Agency (NASA), in identifying, compiling, and reporting these data. It is not surprising that federal agencies should report difficulties in reporting accurate and consistent data over time to NSF. In a presentation to the 1998 NSF workshop on federal R&D data, Robert Tuohy, director of program analysis in the Office of the Director, Defense Research and Engineering, stated that the collection of the data is intensive and difficult to administer, and that the benefits barely justify the high cost to DoD in labor and resources (Touhy, 1998). He identified antiquated software, inflexible collections, and the fact that DoD divisions frequently take “shortcuts” as problems those compiling the data face; he recommended no less than a reengineering of the process of federal funds data collection as the solution to these problems.

OCR for page 102
Measuring Research and Development Expenditures in the U.S. Economy The reporting of federal funds is governed by common concepts and definitions but anchored in agency contract management and classifications systems that may generate results that are at variance with the overall definitions. Interviews conducted for a previous National Research Council study determined that agency respondents were working “with contract classification systems that evolved long ago and may no longer fit current classifications of fields of science and engineering” (National Research Council, 2000:95). Some agencies report difficulties in distinguishing activities between the categories of basic research, applied research, and development. In the 1998 NSF workshop, it was pointed out that some research may be fundamental (basic) but have a strong relevance to the agency mission, so it could be classified as applied (National Science Foundation, 1998). In some agencies, however, the classification systems are much more robust, creating a rich analytical database that is unfortunately lost to that purpose when forced into the three basic classifications of R&D. Such is the case with the DoD science and technology (S&T) financial reporting structure. The financial reporting framework breaks S&T spending into nine classifications, most of which illuminate “development” but these distinctions are lost when aggregated into the three standard classifications of R&D (see Table 5-1). Accounting Framework for Federal Funds Over the years, there have been several efforts to supplement the federal budget classifications that underpin the federal funds survey with more practical or illustrative classification structures. One such proposal, which has been partially adopted, is the federal science and technology (FS&T) budget framework recommended in Allocating Federal Funds for Science and Technology (National Research Council, 1995). This framework would focus on investment in the creation of new knowledge and technologies and exclude activities not involving the creation of new knowledge or technologies, such as the testing and evaluation of new weapons systems. Specifically, the FS&T budget would exclude nongeneric technology development at DoD and the Department of Energy (DoE). The FS&T budget was reexamined in a 2002 National Research Council report and was adopted by OMB in the annual budget process. It has provided an annual tabulation of the results in the “Analytical Perspectives: Research and Development” section of the president’s budget, explicitly labeling the presentation as the “Federal Science and Technology Budget” beginning with the FY 2002 budget proposal (U.S. Office of Management and Budget, 2004). While it is certainly useful and feasible to present the budget in this format, the classification requires numerous judgments about

OCR for page 102
Measuring Research and Development Expenditures in the U.S. Economy activities to include and exclude. The FS&T budget also includes all costs associated with these programs, including staff salaries, in contrast to the federal funds data. Other features include the incorporation of key science and engineering education programs at the National Science Foundation that are not considered R&D but are critical investments in science and technology, as well as the presentation of identifiable line items in the budget to permit easy tracking through the Congressional appropriations process (National Research Council, 2002). Although it is generally agreed that the FS&T budget is useful as a tool to analyze the federal budget, there is less agreement that the federal funds survey should collect and publish an FS&T obligations (and outlays) series. The inability by users to extract the entire FS&T budget detail from the federal funds survey was reported as a disadvantage of the current provision of FS&T in the 1998 federal funds workshop (National Science Foundation, 1998). To generate such a series would require the collection of more explicit and disaggregated reports by activity than are currently collected by NSF. In order to replicate the FS&T budget authority information published by OMB, the federal funds survey would need to collect programmatic information not now collected that would extend the data the collection beyond the normal R&D, particularly in the civilian agencies. Nonetheless, to meet this demand, NSF has collected breakouts of DoD major weapons systems developments from DoD for the past four years and has recently begun breaking out atomic energy defense program spending at DOE. In view of the important uses of the federal science and technology budget, the panel recommends that NSF continue to collect those additional data items that are readily available in the defense agencies and expand collection of expenditures for those activities in the civilian agencies that would permit users to construct data series on FS&T expenditures in the same manner as the FS&T presentation in the president’s budget documentation (Recommendation 5.1). Enhancing Information About the Federal R&D Portfolio In view of the differences in interpretation and application of the concepts and definitions of R&D among the agencies, as well as the growing desire to publish different “cuts” of the data, such as the FS&T reporting of obligations and outlays, the panel considered whether the federal funds survey data could be collected and preserved at a highly disaggregated (microdata) level with detail at the contract, project, program, and activity levels. However, an attempt to do just that in the construction of the RaDiUS database by the Science and Technology Policy Institute of RAND has identified complexities of the task that need to be taken into account

OCR for page 102
Measuring Research and Development Expenditures in the U.S. Economy TABLE 5-1 Definitions of R&D   Government-Wide OMB Circular No. A-11 (1998) Conduct of R&Db Basic Research Systematic study directed toward greater knowledge or understanding of the fundamental aspects of phenomena and or observable facts without specific applications towards processes or products in mind. Applied Research Systematic study to gain knowledge or understanding necessary to determine the means by which a recognized and specific need may be met. Development Systematic application of knowledge toward the production of useful materials, devices, and systems or methods, including design, development, and improvement of prototypes and new processes to meet specific requirements.

OCR for page 102
Measuring Research and Development Expenditures in the U.S. Economy   DOD-Uniquea DOD Financial Management Regulation (Volume 2B, Chapter 5) S&T Activitiesc Basic Research (6.1) Systematic study directed toward greater knowledge or understanding of the fundamental aspects of phenomena and or observable facts without specific applications towards processes or products in mind. Applied Research (6.2) Systematic study to gain knowledge or understanding necessary to determine the means by which a recognized and specific need may be met. Advanced Technology Development (6.3) Includes all efforts that have moved into the development and integration of hardware for field experiments and tests. Demonstration and Validation (6.4) Includes all efforts necessary to evaluate integrated technologies in as realistic an operating environment as possible to assess the performance or cost reduction potential of advanced technology. Engineering and Manufacturing Development (6.5) Includes those projects in engineering and manufacturing development for service use but which have not received approval for full-rate production. Research, Development, Test and Evaluation (RDT&E) Management Support (6.6) Includes R&D efforts directed toward support of installations or operations required for general R&D use. Included would be test ranges, military construction, maintenance support of laboratories, operations and maintenance of test aircraft and ships, and studies and analyses in support of R&D programs.

OCR for page 102
Measuring Research and Development Expenditures in the U.S. Economy   Government-Wide OMB Circular No. A-11 (1998) R&D Equipment The acquisition of major equipment for R&D. Includes expendable or moveable equipment (e.g., spectrometers, microscopes) and office furniture and equipment. Routine purchases of ordinary office equipment or furniture and fixtures are normally excluded. R&D Facilities The construction and rehabilitation of R&D facilities. Includes the acquisition, design, and construction of, or major repairs or alterations to all physical facilities for use in R&D activities. Facilities include land, buildings, and fixed capital equipment, regardless of whether the facilities are to be used by the government or by a private organization, and regardless of where title to the property may rest. Includes such fixed facilities as reactors, wind tunnels, and particle reactors. Excludes movable R&D equipment. aDoes not pertain to the Corps of Engineers. bIncludes administative expenses. Excludes routine product testing, quality control, mapping, collection of general-purpose statistics, experimental production, routine monitoring and evaluation of an operational program, and the training of scientific and technical personnel.

OCR for page 102
Measuring Research and Development Expenditures in the U.S. Economy   DOD-Uniquea DOD Financial Management Regulation (Volume 2B, Chapter 5)   Operational System Development (6.7) Includes those development projects in support of development acquisition programs or upgrades still in engineering and manufacturing development, but which have received Defense Acquisition Board (DAB) or other approval for production, or production funds have been included in the DoD budget submission for the budget or subsequent fiscal year. Developmental Test & Evaluation Efforts associated with engineering or support activities to determine the acceptability of a system, subsystem, or component. Operational Test & Evaluation Efforts associated with engineering or support activities to determine the acceptability of a system, subsystem, or component. No separate definition Major equipment dollars are mixed with the dollars for the “Conduct of R&D” and carried in the RDT&E accounts (i.e., 6.1 through 6.7) listed above. In FY 1998, DoD requested a total of $68M for major R&D equipment. No separate definition In FY 1998, close to 90 percent of the $67M requested by DoD for R&D facilities was carried separately in Military Construction accounts. The rest were included in the costs of major development programs and are mixed with the dollars for the “Conduct of R&D”c carried in the RDT&T accounts (i.e., 6.1 throgh 6.7) listed above. cIncludes costs of laboratory personnel, either in-house or contractor operated. SOURCE: Fossum et al. (2000:615).

OCR for page 102
Measuring Research and Development Expenditures in the U.S. Economy should the federal funds survey be conducted based on disaggregated data. Indeed, the ability to distinguish R&D activities from science and technology activities of the federal government was described as the biggest challenge to creating the RaDiUS database (Fossum et al., 2000). This pioneering database development project, sponsored by the National Science Foundation, illuminates us on the promise and pitfalls of building depth into the view of federal R&D investments. The RaDiUS database is designed to connect the aggregated data on authority, obligations, and outlays with disaggregated data on actual spending (purchases) to complete the picture of federal R&D activities. With the exception of funds spent on the construction and rehabilitation of federal R&D facilities and the purchase of major R&D equipment, the RaDiUS database explicitly utilizes the published federal information for the top 24 agencies that spend R&D dollars. The contribution of the RaDiUS project is that it then disaggregates the data by subagencies and, finally, into the records, of actual obligational transactions to their final destination at universities, laboratories, and centers. The database contains over 500,000 records, which are said to provide details on close to 80 percent of the activities in the federal R&D portfolio. The database is constructed largely by harvesting data from various transactional administrative data sources throughout the federal government. These sources usually exist for other purposes and were not designed with the objective of identifying R&D activities with precision.1 For each transactional record culled from these databases, it is necessary for the RaDiUS builders to construct a record that depicts the hierarchial levels that permits aggregation back into agency and subagency totals. Depending on the data source, the detailed records contain (in addition to agency-identifying information) identifying numbers, type of funding, and estimated start/end dates; performer, performer type, and performer location; contact information, place of performance, funding information, and an abstract. These records are updated on schedules specific to each input source. For example, DoD records are updated 2-3 times a year, and NASA and DoE records are updated annually. 1   Among the sources included in the RaDiUS data files are the Catalogue of Federal Domestic Assistance; the U.S. Department of Agriculture’s Current Research Information System; the U.S. Department of Health and Human Service’s Computer Retrieval of Information on Scientific Projects and the Information for Management, Planning, Analysis, and Coordination system; the DoD’s R-1 and R-2 Budget Exhibits and Technical Effort and Management System; the DoE’s Laboratory Information System; the Federal Assistance Awards Data System; the Federal Procurement Data System; the OMB’s MAX System; the Department of Veterans Affairs’ R&D Information System; the NSF’s Science and Technology System (STIS); and NASA’s RAMIS System.

OCR for page 102
Measuring Research and Development Expenditures in the U.S. Economy The RaDiUS database is an ambitious undertaking, and it has been utilized to support analysis of sectoral and geographic investment in R&D. Most recently, the database served as the source of information for a report on federal research and development funds provided to colleges and universities (Fossum et al., 2004). The database is not immune from the vagarities of federal administrative classification systems, however. RAND reports that federal procurement officials decide which activities are defined as R&D on a daily basis, with reference to federal procurement regulations. The dividing line between R&D and “studies and analysis” is particularly blurry, and it is possible that there is some misclassification of activities at their procurement source that would affect the data. As sponsor of the RaDiUS project, NSF has conducted several external reviews of the system to provide an independent review of the database. In a 1998 staff review, NSF concluded that the R&D project database was only 65 percent complete, with large parts of the DoD and DoE development investment portfolio not accounted for (National Science Foundation, 1999). The coverage was estimated by RAND at “close to 80 percent of the activities in the R&D portfolio” in its 2000 report (Fossum et al., 2000:613). Likewise, the quality of some of the input data is uneven. It was noted that some agencies, such as the National Institutes of Health (NIH) and NSF, provide very polished and comprehensive project descriptions, while other agencies provide project titles or minimal descriptions. The passage of the E-Government Act of 2002 has added a new sense of urgency to considering the proper level at which to tally federal R&D activities. The act calls on NSF, in conjunction with OMB and the Office of Science and Technology Policy, to create a database and web site to track federally funded research and development (U.S. Office of Management and Budget, 2003b). The panel is encouraged by the initiative to provide detailed and additive information on federal R&D activities that is represented by the RaDiUS database. Although it has several drawbacks, including the inability to extract important field-of-science data from the administrative data sources, the use of administrative records represents a valuable addition to the stock of information on the federal R&D portfolio. The committee does not suggest that an administrative database will ever supplant the collection of federal funds data on a regular basis. For example, RaDiUS data is not available by field of science. However, the RaDiUS-type data can improve the timeliness of the federal statistics, and they may allow less frequent collection of the greater detail in the federal funds survey. The promise and possibilities of a microdatabase on federal R&D activities will not be fully realized until the heterogeneous data formats and R&D procurement practices of the several key federal R&D agencies are brought into consistency. The panel urges NSF, under the auspices of the E-Government Act of 2002, to begin to work with the Office of Management and Budget to

OCR for page 102
Measuring Research and Development Expenditures in the U.S. Economy develop guidance for standardizing the development and dissemination of R&D project data as part of an upgraded administrative records-based data system (Recommendation 5.2). Statistical Methodology Issues The Survey of Federal Funds for Research and Development is a virtual census of federal agencies that support national scientific activities. It consists of collection of some data that have rigid definition and strict accounting measures. However, others, such as current and future year obligations, are best estimates on the part of agency reporters. The trigger for inclusion of an agency in the annual data collection round is reporting of R&D activities in OMB budget documents or in the media. Rather intense effort is given to identifying respondents in the selected agencies and contacting respondents at multiple intervals—before, during, and after data collection. This is not a full-time job for respondents, and many of them are replaced from year to year, so reeducation is a constant challenge. Data collection is tiered, in that larger R&D agencies are asked to provide data on obligations to states and to colleges and universities by field of science. The response rate for the census is an enviable 100 percent, and there is no item nonresponse, since the agency respondents must answer all questions before the data can be submitted. Lack of timeliness is a continuing issue. In his remarks at the panel’s July 2003 workshop, Kei Koizumi pointed out that data for FY 2001, which ended on September 30, 2001, were just becoming available in July 2003, over a year and a half later. This delay in assembly and transmittal of the data is particularly troublesome because reporting offices have the data at the end of the fiscal year (interview with Mark Herbst, Office of the Secretary of Defense, March 26, 2003). The lack of timeliness is such a severe deterrent to utility of the data that most organizations that assess R&D spending trends turn to budget authority data rather than obligations and outlays, limiting timely application of some very useful R&D analysis tools, such as the Federal Science and Technology Budget. Although several steps are being taken by the contractor to enhance cooperation and speed data processing, the list of problems inhibiting on-time reporting is a constant challenge to the staff. Speaking at an agency workshop, Ronald Meeks of NSF identified as timeliness issues the lack of support from senior officials in some agencies, the need for constant reeducation of reporters, problems with completion of the automated form, delays from internal review and controls, and timing conflicts with the higher priority president’s budget (Quantum Research Corporation, 1999).

OCR for page 102
Measuring Research and Development Expenditures in the U.S. Economy SURVEY OF FEDERAL SCIENCE AND ENGINEERING SUPPORT TO UNIVERSITIES, COLLEGES, AND NONPROFIT INSTITUTIONS NSF has considerable experience in collecting microdata from federal government agencies, primarily from contract award and grants files, and has done so annually since 1965. It also has experience in gathering data that are more encompassing of the scope of science and engineering than the more restrictive traditional R&D definitions that underscore the federal funds survey. These innovations in data collection are part of the annual Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions. This survey is, in a manner of speaking, a precursor for the kind of data collection that could significantly enrich the understanding of the nature of the federal government’s investment in science and engineering. Supplement to the Federal Funds Survey As it stands, this congressionally mandated survey is a useful supplement to the federal funds survey. It is the only source of comprehensive data on federal S&E support to individual academic and nonprofit institutions. Federal policy makers, state and local government officials, university policy analysts, R&D managers, and nonprofit institution administrators use it to assess S&E investment trends. NSF and other federal agencies also use it for internal administrative purposes. The federal support survey asks agency representative responders from the 21 federal agencies that incur nearly all of the obligations for federal academic R&D to rake through their contract and grant files to compile several pieces of information for each academic institution and each nonprofit institution for which they have records. The agencies are asked to compile, for each institution, the following information: Academic institution Geographic location (within the United States) Highest degree granted Historically black colleges and universities Obligations Performer (type of organization doing work) R&D plant Type of academic institution (historically black and others) Type of activity (e.g., R&D, S&E instructional facilities) Type of institutional control (public or private) Individual contract and grant record data are aggregated to institution totals, since just one survey instrument is to be filled out for each of the over

OCR for page 102
Measuring Research and Development Expenditures in the U.S. Economy 1,000 universities and colleges for which an agency obligated R&D funding during the previous fiscal year. When reports are gathered from several subagencies in large R&D organizations like NIH and DoD, the reports for each institution must be aggregated to funding agency totals, coded, and entered into the NSF collection formats. 2 Needless to say, this is a cumbersome and time-consuming process even when agencies have automated systems that compile the basic information.3 In order to align with the OMB budget information, the data are to be collected on a fiscal year basis (October 1 through September 30), with collection to start in the following February and to be submitted by April 15. NSF has made extensive changes in recent years to streamline the collection, submission, and editing pieces of the survey operation. This was one of the first data collections converted to web-based systems and automated editing procedures. The contractor, QRC Macro, has an extensive program of outreach and education for the agency reporters. Despite the attempts to streamline the processes over which NSF and its contractor have some control, the burdensome nature of this collection is reflected in the lack of timeliness. The data for the most recent available fiscal year, 2002, were not published in full detail until 18 months after the reference period. Data are to be provided in fairly extensive categories: Research and development Fellowships, traineeships, and training grants R&D plant (facilities and equipment) Facilities and equipment for instruction in S&E General support of S&E Other activities related to S&E All other activities Aside from the category of “research and development” for which a definition is well codified and fairly widely understood, there is the possibil- 2   Agency reporters are responsible for finding and entering a 6-digit Federal Interagency Committee on Education (FICE) code on each record. This requires reference to a 400-page Code Book for Compatible Statistical Reporting of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions. This process may lead to coding errors, and perhaps some loss of coverage, particularly among nonprofit institutions, when the institutions do not get added to the list of “new” institutions in the code book. 3   The automated systems may ease data collection, but also they may lead to errors in the process of identifying and tabulating the individual record data. Changes in software have been known to have an influence on the level of federal support reported. For example, NSF was forced to cease publication of non-S&E support estimates in FY 1993 when the Department of Education revamped codes in a major software modification.

OCR for page 102
Measuring Research and Development Expenditures in the U.S. Economy ity that rather ambiguous definitions of the other categories, particularly those pertaining to non-R&D S&E, may lead to agency difficulties in matching program descriptions to the proper funding category. For example, NASA has placed increased emphasis on including educational components to projects, and education is always reported as “other S&E.” NSF warns data users that categories of “general support for S&E” and “other S&E activities” are catchall categories (National Science Foundation, 2001). Data on over 1,000 nonprofit institutions, each to be submitted on a separate form, are also collected, but they cover only the activities of research and development and R&D plant. Thus it is not possible to develop an estimate of S&E spending for nonprofit institutions. The variables in this survey use definitions comparable to those used by OMB and the federal funds survey. Respondents are told that the totals reported in this survey and the federal funds survey for R&D and R&D plant obligations should be in close agreement. If differences exist, respondents should include an explanation. Not every federal agency is in the survey every year, so, for some, there is no opportunity to build experience in compiling the data from year to year. The number of agencies included in the survey can vary from year to year depending on their activity reflected in the prior year’s federal funds survey for academic and nonprofit data, so some may not have anything to report in the year of interest. Since not all federal agencies are surveyed, some funding could be missed. The omissions are believed to have little impact on understanding the total funding picture, but NSF warns users that the impact could be significant for understanding the funding for some institutions. The recent publication of estimates of federal R&D spending in universities and colleges based on the RaDiUS database provides a glimpse at the potential for utilizing the individual record files from contracts and grants for identifying federal support for the colleges and universities, excluding FFRDCs (Fossum et al., 2004). The RaDiUS database is enriched with information from sources that are secondary to agency sources: the Federal Procurement Data System, which tracks all federal contracts; the Federal Assistance Awards Data System, operated by the Census Bureau which tracks federal grants and cooperative agreements; and the Catalogue of Federal Domestic Assistance, compiled by the General Services Administration, of all programs, projects, services, and other types of assistance provided to nonfederal entities. Again, the approach taken by the RaDiUS database project to aggregate national R&D totals from individual record data have several shortcomings in terms of coverage and comprehensiveness, representing, as they do, a view of R&D limited by the coverage of the input data sources. Nonprofit institutions are not covered. Nonetheless, the publication of the

OCR for page 102
Measuring Research and Development Expenditures in the U.S. Economy extensive detail on universities and colleges for FY 1996 to FY 2002, in a favorably timely manner compared with the publication of NSF’s federal support data, suggests the potential of data systems based on individual records of contracts, grants, and cooperative agreements to produce useful estimates. The panel recommends that NSF devote attention to further researching the issues involved with converting the federal support survey into a system that aggregates microdata records taken from standardized, automated reporting systems in the key federal agencies that provide federal support to academic and nonprofit institutions. In connection with this investigation, NSF should evaluate the possibility of collecting for nonprofit organizations the same science and engineering variables that pertain to academia (Recommendation 5.3). Statistical Methodology Issues There are several difficult aspects of the survey methodology. For example, the frame for the survey is the list of all federal agencies that sponsor R&D, obtained from the president’s budget submission to Congress. In practical terms, the survey covers respondents to the federal funds survey, focusing on only the largest of the agencies (in FY 2000, 18 agencies were in the target population). Unlike the federal funds survey, this survey is not a census of science and engineering support. Not all agencies are surveyed, and some funding can be missed. While the overall amount of missed funding is not significant, the patterns of funding by agency and recipient may be somewhat distorted by these omissions. The data collection is web-based, with automated functions supporting all data collection, data imports, data editing, and trend checks. There is no nonresponse from agencies and no item nonresponse, since the forms must be completed prior to transmittal, raising issues that need to be studied, as discussed before. It is possible that coding errors, such as an incorrect institutional code or incorrect branch of a multiunit institution, could lead to errors in the estimates of funding by institution. Matching program descriptions to proper funding categories may cause some confusion on the part of respondents.