National Academies Press: OpenBook

Measuring Research and Development Expenditures in the U.S. Economy (2005)

Chapter: 7 Analysis of the R&D Data Funding Discrepancies

« Previous: 6 Measuring R&D Activity in Academic Institutions
Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×

7
Analysis of the R&D Data Funding Discrepancies

One characteristic of the portfolio of R&D expenditure surveys is that each survey was designed to meet a specific goal or set of goals. A previous National Research Council study concluded that each survey was developed to address a narrow topic rather than serve as an integrated part of a comprehensive system of R&D expenditure data (National Research Council, 2000). The nonintegrated nature of the portfolio came about as the survey managers scrambled to meet priority needs and initiated collections as resources became available, not necessarily as part of an overall master plan.

Despite much attention on the part of NSF to a concordance of concepts and definitions across the data collections, the independent development of the pieces of the R&D expenditure portfolio has led to gaps and anomalies that are of some trouble to data users. Perhaps no gap is more troublesome over time than the apparent discrepancy in funding and performance estimates among the surveys.

The topside measures of the apparent discrepancy illustrate this problem. When NSF collects data from the federal government on the amount that Congress authorizes to be spent on R&D (budget authority) that goes to academia and industry, it reports that the federal government has $86.8 billion to spend on R&D in fiscal year 2001. But when NSF tallies reports from academia and industry on funding received from federal sources during that same fiscal year, it only comes up with $72.6 billion. This discrepancy of $14.1 billion between budget authority and receipts is one depiction of the size of the discrepancy. Often, a comparison is made between obligations (which are tracked in detail in the federal funds survey) and

Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×

performer receipts. The discrepancy between obligations and performer receipts in fiscal year (FY) 2001 amounted to a hefty $7.3 billion—smaller than the discrepancy between authority and receipts, but still worthy of attention. In an ideal world, the reports of spending and receipts would match, perhaps with some adjustment for lags in the transfer of funds.

There have been several attempts to disentangle this apparent discrepancy over the past several years (Congressional Research Service, 2000). This is considered an important issue by many. The discrepancy is annoying as an accounting anomaly to analysts who seek to trace the funding streams and answer the question “Where did the money go?” (Koizumi, 2003). It is also of concern to members of Congress and others who have oversight and legislative responsibilities, because the discrepancy casts a shadow over the accuracy of the data that are used to gauge the overall health and vitality of the nation’s R&D enterprise, as well as answer the question “Are spending and receipt estimates accurately depicting how the money is flowing, and are the funds flowing in amounts and directions prescribed by legislation?” (U.S. General Accounting Office, 2001).

The discrepancy is of special concern to the panel and others who have studied it because its existence may signal a systemic quality problem in one or more of the NSF datasets. One litmus test of the quality of any statistical series is to compare the series with comparable data collected by other means. It is common statistical practice for federal statistical agencies to use administrative data sources to benchmark the totals derived from surveys. Measurement of undercoverage and overcoverage error (the failure to include or exclude some population units in the frame used for sample selection) often rely on comparing survey estimates to independent sources (U.S. Office of Management and Budget, 2001).

NATURE OF THE DISCREPANCY

As shown in Table 7-1, the discrepancy as measured by both obligations and outlays (see Box 2-1) has appeared to change course three times over the past three decades.

  • Prior to 1980, the discrepancy was quite small and generally indicated that federal obligations exceeded performer spending.

  • In the 1980s, the discrepancy became noticeable and changed, and for nearly a decade, spending appeared to significantly exceed obligations.

  • Around 1990, the trend shifted again, and spending once again has lagged obligations and outlays, only now by fairly substantial amounts.

The timing of these swings in the data series is in close alignment with changes in the design and operation of the NSF data collections, so it is

Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×

TABLE 7-1 Comparisons of Federal- and Performer-Reported Expenditures for Federal R&D ($millions)

 

Budget Authority (FY)

Total Obligations (FY)

Total Outlays (FY)

Total Performer Reported (CY)

Expenditures Less Budget Authority

Expenditures Less Obligations

Expenditures Less Outlays

1970

14,911

15,339

15,734

14,993

82

−346

−741

1975

19,039

19,039

19,551

18,615

−424

−424

−936

1980

29,739

29,830

29,154

30,002

263

172

848

1985

49,887

48,360

44,171

52,662

2,775

4,302

8,491

1990

63,781

63,559

62,135

61,596

−2,185

−1,963

−539

1991

65,898

61,295

61,130

60,799

−5,099

−496

−331

1992

68,398

65,593

62,935

60,815

−7,583

−4,778

−2,120

1993

69,884

67,314

65,241

60,581

−9,303

−6,733

−4,660

1994

68,331

67,235

66,151

60,787

−7,544

−6,448

−5,364

1995

68,791

68,187

66,662

62,965

−5,826

−5,222

−3,697

1996

69,049

67,653

66,142

63,341

−5,708

−4,312

−2,801

1997

71,653

69,827

68,898

64,548

−7,105

−5,279

−4,350

1998

73,569

72,101

70,632

66,340

−7,229

−5,761

−4,292

1999

77,637

75,341

70,585

67,015

−10,622

−8,326

−3,570

2000

78,664

72,863

69,807

66,371

−12,293

−6,492

−3,436

2001

86,756

79,933

75,336

72,637

−14,119

−7,296

−2,699

NOTE: Adjustments have been made to university reported R&D expenditures for 1998 and later years to eliminate double-counting of funds passed through one academic institution to another. Data for 1998 and later years are thus not directly comparable to data for 1997 an d earlier years. For FY 1998, $479 million in pass-through funds were reported. For FY 2002, $865 million in pass-through funds were estimated.

Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×

useful to examine the possible relationship of these shifts with changes in survey design and implementation.

As the nature of the discrepancy suggests, there is probably no single point of origin of the difference between the data series. The discrepancy could arise from differing concepts, definitions, and interpretations in the data sources; from errors in the administrative reporting of federal funds; from measurement errors in the survey of R&D expenditures at universities and colleges; from sampling and measurement errors in the survey of industrial research and development; or from combinations of these sources of error. It has also been postulated that some of the discrepancy may be attributed to systemic reporting problems arising from the classified nature of some Department of Defense (DoD) R&D spending. In this section, we examine each of these potential sources of error.

Differing Concepts, Definitions, and Interpretations

In order for the data on federal obligations and performer expenditures to add up to the same total, they must be based on the same definitional criteria and have the same time horizon. However, federal obligations and performer expenditures are not definitionally the same and they do not necessarily pertain to the same point in time. In its report on the discrepancy, the General Accounting Office (GAO) pointed out that obligations are estimates of payments to be made by federal agencies without regard to when the payments may (or may not) ultimately be made. These obligations are made on the basis of the availability of appropriations and on the assumption that the performers will meet their expectations. Expenditures represent actual performer cost or expense data without regard to when the federal obligation was made. These expenses can occur years after the obligation (U.S. General Accounting Office, 2001).

Obligations are tied to orders placed, contracts awarded, services received, and similar transactions during a given period, regardless of when the future payment of money is actually required. The period for the reporting of obligations is standardized on the federal fiscal year: October 1 to September 30. Expenditures are reported when a reimbursement is made and in reference to a calendar year. These basic differences are exacerbated by federal contract payment practices. It was pointed out at the panel’s workshop that no money comes to an academic contactor as a result of an obligation until the work is actually performed and billed, with a wait of 30-90 days for payment. Thus, at the performing institution, reported federal expenditures lag actual spending of money by several months. Since the average award is made toward the end of the fiscal year, this dynamic could lead to a natural delay between an obligation and an expenditure of 18 months to 2 years.

Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×

FIGURE 7-1 Comparison of annual federal and performer-related expenditures, 1970-2001.

NOTE: Data are shown as a percentage of annual obligations in order to adjust for inflation and changing level of federal support.

Considering these definitional differences, it is not surprising that there should be a discrepancy. The GAO report concluded that the discrepancy results primarily from annually comparing two separate and distinct types of financial data—federal obligations and performer expenditures—that are not comparable (U.S. General Accounting Office, 2001).

The extent of the contribution of this difference in definition and time horizon to explaining the alleged discrepancy can be seen when the comparison is made using outlays rather than obligations as the basis of comparison with expenditures. Outlays are generally considered to be closely akin to spending. They represent the amounts for checks issued and cash payments made during a given period, regardless of when the funds were appropriated or obligated. When these more comparable parameters are used as the basis for the reconciliation, a good part of the apparent discrepancy disappears. In 2001, when obligations exceeded expenditures by $7.3 billion, the difference between outlays and expenditures was just $2.7 billion (see Figure 7-1).

Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×

Although the size of this discrepancy varies by year, it could be said that the more valid comparison of federal outlays and performer expenditures reduces the discrepancy by one-half to two-thirds. On this basis, the panel recommends that future comparisons of federal funding and performer expenditures be based on outlays versus expenditures, not obligations versus expenditures (Recommendation 7.1). However, this more proximate alignment of definitions and time horizons still does not explain away the gap. Something else is apparently affecting the comparison.

Sampling and Measurement Errors in the Industrial R&D Survey

The most significant discrepancy resides between data reported by the federal government as obligated to R&D in industry and that reported by industry as expenditures in the industrial research and development survey. In this section, we examine the size and direction of this gap, summarize previous investigations into the source of this difference, and introduce discussion of a potential cause of the growing discrepancy over the 1990s.

As previously discussed, most of the overall discrepancy between federal obligations and reported expenditures has been between federally reported obligations to industry and the amounts reported as being received and expended by industry. The survey of federal funds reported $27.0 billion in obligations to industry in 2001, while the survey of industrial R&D reported that the federal government was the source of just $16.9 billion in industry expenditures. This gap of $10.1 billion, or nearly one-third, could be more than explained by the discrepancy between what DoD reports as R&D funding (obligations) and what industry reports it receives from DoD, a gap that amounted to $11.4 billion that year. Other agency-related gaps were smaller or offsetting.

A Congressional Research Service (CRS) study discussed several potential reasons for the phase shift in the gap in industry R&D numbers in the 1990s: acquisitions, mergers, and other structural changes in defense and aerospace industries in the decade; differing interpretations of R&D in the federal government and industry; and partial reporting by industry (Congressional Research Service, 1999, 2000).

When companies merge and acquire other companies, the quality and consistency of data reporting may suffer. In the 1990s there was an aggressive round of mergers, acquisitions, and other structural changes in the defense and aerospace industry that may have resulted in decreased awareness of the nature and scope of R&D in the new company. In some cases, smaller firms that were R&D intensive and contributed significantly to the estimates simply went away. Their R&D activities may or may not have been identified in the new reports of R&D data to the government provided by the acquiring company. This led to the conclusion that consolidation of

Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×

the defense and aerospace industry contributed to the R&D data reporting problem (Quantum Research Corporation, 1999). Further on this issue, according to the CRS, DoD officials postulate that corporate mergers have affected continuity of reporting because job losses that accompany the consolidations have often pared individuals in front offices who had competently completed these surveys in the past (Congressional Research Service, 1999).

Although the definitions of R&D are reasonably clear and have been consistent over time, we have observed that there can be different interpretations of what constitutes R&D between a government agency and the companies that receive the funds at the individual project level. The Quantum Research Corporation (QRC) study compared a set of examples of project awards classified as research, development, test, and evaluation (RDT&E) by DoD with the categorizations provided by the recipient companies. The results of this small test were striking. Two-thirds of the companies did not consider most of the activities selected by DoD to meet their definition of traditional, reportable R&D work. A Census Bureau study of defense subcontractors came up with the same conclusion. Many of the subcontractors reported work as general R&D without ascribing it to an agency, often because the agency detail was not mandated by law (U.S. Census Bureau, 1997). This observation may well account for the fact that DoD expenditures are underreported by industry, while industry reports more “other” sources of funding than does the federal government. However, even if all of the gap in “other” were ascribed to DoD, it would account for only a minority of the DoD discrepancy.

Recalling that one of the so-called phase shifts in the gap between federal obligations for industry R&D and the amount reported by industry performers took place between 1991 and 1992, the panel examined results of the large-scale revision of the industry R&D survey that was introduced in 1991 to determine if that might have had an effect on the gap. Prior to the 1992 survey, the sample of firms surveyed was selected at irregular intervals, roughly associated with the economic census updates.1 In the intervening years, only a panel of the largest firms known to perform R&D was surveyed. In latter years of the 1980s, for example, the sample was refreshed in 1987, when 14,000 firms were surveyed. From 1988 to 1991, just 1,700 of these firms were resurveyed, and the R&D data for the rest were estimated. This survey design biased the survey results in favor of large manufacturing firms, when it was increasingly apparent that there was substantial growth in R&D performance in the

1  

Prior to 1967, samples were selected every 5 years. Samples were selected then for 1967, 1971, 1976, 1981, and 1987.

Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×

nonmanufacturing sector and among smaller firms. Consequently, in 1991, the survey was redesigned to draw new samples with broader coverage annually and increase the sample size to approximately 23,000 firms each year. The impact of this change in the sample design is seen in the revisions to the data for the immediately prior year, 1990. The new sample design accounted for $13.7 billion or about 93 percent of the total $14.7 billion revision to total R&D, with most of that revision ($11.4 billion) attributed to better coverage of nonmanufacturing industries. In keeping with standard practice when dealing with large revisions, the new levels were wedged back in time, in this case to 1987, the year in which the last full sample was surveyed.

Although new estimates were prepared for 1991, it is obvious that a good part of the jump of nearly $5 billion in the size of the discrepancy between agency R&D spending and industry performer reports was attributable to changes in survey design. Some of the increase that year may be due to a one-time dip in DoD R&D spending, probably associated with diversion of R&D funds to support the Persian Gulf war effort, but the fact that the discrepancy continued to grow over the next couple of years, when DoD spending returned to prewar levels, indicates that a long-term shift in the data series indeed occurred.

The impact of the 1991-1992 survey redesign is readily apparent, but the question remains, why? In this matter, we can only speculate that the larger representation of small firms and nonmanufacturing industries changed reporting patterns. Outside the manufacturing sector, where R&D is conducted in a very formalized setting usually involving laboratories and R&D management structures, there may be less knowledge of the source of origination of R&D funds and more frequent pass-through and collaborative arrangements, which might well confuse respondents when asked to report on funding from federal sources. This may lead to underreporting among these firms. When the results of the underreported federal R&D receipts in the nonmanufacturing sector are weighted by the employment in that sector, the resulting estimate will fall short of an estimate derived largely from a survey that previously covered mainly the manufacturing sector. This downward bias in reporting of federal receipts appears to date back to the 1991 redesign of the industrial R&D survey. As the weight of the manufacturing sector in the survey has continued to decline with the growth in nonmanufacturing, this effect may well be the source of a large part of the growing reported discrepancy.

Errors in the Administrative Reporting of Federal Funds

In Chapter 5, we discuss several sources of potential error in the administrative reporting of federal funds in the annual Survey of Federal Funds

Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×

for Research and Development. In this section, we highlight several of those sources of potential error, culled largely from previous studies by CRS and GAO as a means of pinpointing possible reasons for the discrepancy.

There is considerable evidence that timely and accurate completion of this survey is not a high priority for the federal agencies that annually provide the data. Top agency officials do not seem to understand the purpose and importance of collecting obligation data, in addition to the authority and outlay data provided regularly on a priority basis to the U.S. Office of Management and Budget (OMB) and to Congress on a “real-time” basis. Consequently, agency reporters may not receive sufficient resources to fully provide complete and timely data to NSF.

The lack of timeliness of data submission is indicative of the problem. NSF broadcasts its call for data in February of each year, closely on the heels of the release of the president’s budget, requesting that agencies submit the data by May 15. Only about half of agencies meet that deadline, most blaming the requirements for extraordinary data compilation to levels not called for in managing the R&D programs nor in reporting to OMB. Labor-intensive and time-consuming processes without direct payoff to the agency are likely to be error-ridden, no matter how conscientious the agency reporters who have responsibility for furnishing the data.

Further complicating the ability to provide data to NSF have been dramatic changes in procurement practices in the federal government over the past decade. The shift away from specific task-oriented R&D contracts with the ultimate performer to larger contract awards which tend to be multiyear, multiagency, “blanket” indefinite order-type contracts and a trend toward “bunching” smaller contracts into large vehicles may have caused a difference in how R&D activities are categorized by respondents. These shifts in the composition of RDT&E procurements during the past 10 years were labeled the “most likely cause of divergence between the NSF reports” in a study by the Quantum Research Corporation (1999). However, a subsequent CRS study reported that DoD officials did not agree with the assessment that this was a primary cause, since many of the procurement reforms proceeded the growth in the gap (Congressional Research Service, 1999).

There is some speculation that agencies may not be consistently reporting R&D, despite the fact that NSF provides survey respondents with definitions of each data item collected on the surveys and stands ready to provide extensive hands-on assistance in ensuring that the right data are collected. Many of the agency reporters are budget analysts without detailed knowledge of the R&D enterprise and procurement activities in their agency. When confronted with judgment calls (e.g., should training, program evaluation, and construction be included?) they may not be as qualified to make the judgment as scientists and engineers and procurement

Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×

officials (Congressional Research Service, 1999, 2000). Furthermore, even when agencies agree on definitions, they may be at variance with performer practices. Federal agencies often consider program management as an R&D activity, but performers may not (U.S. General Accounting Office, 2001).

Another source of potential error identified by participants in the 1998 NSF workshop on federal R&D was the challenge of determining the ultimate performer of R&D, particularly in larger R&D agencies with many of the aforementioned multitask, multiyear procurement vehicles. Even when the agency is able to identify the ultimate performer, the multitask contracts may mingle procurement and R&D activities in ways that make it difficult for the ultimate performers to appropriately break down the contract into its procurement and R&D pieces to prepare their NSF report. It has been observed that agencies tend to overreport intramural funding for federal laboratories, while underreporting R&D funding spent at an industrial or academic institution (Congressional Research Service, 1999, 2000). In addition, the visibility of some R&D funding appears to disappear when sent to another agency for dispersal. There may be inadequate mechanisms for interagency exchange of information that allows tracing of R&D data to the ultimate performer.

The changing nature of federal programs as they move through time is a further potential complication. An emerging program often moves through a progression from basic research to applied research to development to implementation, and the reporting system may not catch up with that progression. An example used quite often in the literature is the ambitious National Aeronautic and Space Administration (NASA) space station program, which saw significantly increased reported R&D obligations during the 1990s and sharply reduced reported R&D in 2000 when the program matured from an R&D-intensive activity to other categories of reporting. Industry-reported NASA R&D expenditures did not correspondingly reflect this upsurge and reduction in R&D, so the gap widened over the 1990s to the extent that, in 1998, industry reported a level of spending only one-third that reported by NASA (see Table 7-2).

Measurement Errors in the Survey of Academic R&D Expenditures

Compared with the size of the gap in the reporting of federal R&D outlays to industry and industry-reported R&D expenditures, the gap between reports of the federal government and academic institutions is minuscule (Table 7-3). In 2001, the difference between federal R&D obligations and reported university R&D expenditures amounted to only $397,000, or 2.0 percent of the total.

When adjusted to eliminate double-counting of funds passed through from one academic institution to another, the difference was a bit larger, totaling $1.1 million, or 5.6 percent. This adjustment for double-counting

Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×

TABLE 7-2 Selected Differences in Agency Reported R&D Funding to Industry and Industry-Reported R&D Expenditures ($millions)

 

Total Federal R&D Obligations to Industry (FY)

DoD R&D Obligations to Industry (FY)

Industry- Reported DoD R&D Expenditures (CY)

DoE R&D Obligations to Industry (FY)

Industry- Reported DoE R&D Expenditures (CY)

NASA R&D Obligations to Industry (FY)

Industry- Reported NASA R&D Expenditures (CY)

Other R&D Obligations to Industry (FY)

Industry- Reported Other R&D Expenditures (CY)

Industry- Total R&D Less Agency Total R&D

1981

16,282

10,931

10,540

2,486

1,976

2,096

2,306

769

1,560

100

1983

18,521

14,670

14,571

2,263

1,949

902

2,150

686

2,010

2,159

1985

23,496

19,069

20,948

2,348

2,901

1,312

NA

767

NA

NA

1987

28,628

24,258

22,252

2,128

2,968

1,479

2,805

763

2,732

2,129

1993

31,670

23,865

15,044

2,540

2,961

4,112

435

1,153

4,369

−8,861

1995

31,438

22,409

13,867

2,586

3,547

4,687

2,054

1,756

3,974

−7,996

1997

32,548

24,190

12,603

2,083

2,505

4,770

2,022

1,505

6,798

−8,620

1999

33,230

24,700

11,650

2,052

2,209

4,566

1,469

1,912

7,207

−10,695

2000

28,836

22,345

11,142

1,678

1,455

2,645

1,328

2,168

5,193

−9,718

2001

27,006

21,386

9,959

1,041

1,253

2,659

1,071

1,920

4,616

−10,107

NOTE: All years but 2001 include industry Federally Funded Research and Development Center (FFDRC) R&D. Beginning in 2001 these FFRDCs were no longer included in the industry R&D survey frame. CY = Calendar Year, FY = Fiscal Year, DoE = Department of Energy, NASA = National Aeronautics and Space Administration.

SOURCES: Data were derived from National Science Foundation, Division of Science Resources Statistics (NSF/SRS), Survey of Research and Development in Industry; and NSF/SRS, Survey of Federal Funds for Research and Development.

Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×

TABLE 7-3 Comparison of University Federal R&D Expenditures with Agency-Reported R&D Obligations ($millions)

 

University R&D Expenditures

Expenditures Less Obligations

Percent of Federal Obligations

Reported

Net of Pass-through R&D

Federal Obligations

Reported

Net of Pass-through R&D

Reported

Net of Pass-through R&D

1988

8,193

 

7,828

365

 

4.7

 

1989

8,991

 

8,672

319

 

3.7

 

1990

9,638

 

9,138

500

 

5.5

 

1991

10,234

 

10,169

65

 

0.6

 

1992

11,093

 

10,271

822

 

8.0

 

1993

11,957

 

11,208

749

 

6.7

 

1994

12,646

 

11,797

849

 

7.2

 

1995

13,328

 

11,928

1,400

 

11.7

 

1996

13,836

 

11,978

1,858

 

15.5

 

1997

14,309

 

12,559

1,750

 

13.9

 

1998

15,145

14,724

13,381

1,764

1,343

13.2

10.0

1999

16,071

15,569

14,959

1,112

610

7.4

4.1

2000

17,508

16,878

16,815

693

63

4.1

0.4

2001

19,191

18,484

19,588

−397

−1,104

−2.0

−5.6

NOTE: Adjustments have been made to university-reported R&D expenditures for 1998 and later years to eliminate double-counting of funds passed through from one academic institution to another. For the purpose of comparison, both the reported federal R&D and the federal R&D net of federal R&D reported as passed through to another academic institution are shown.

SOURCES: Data were derived from National Science Foundation, Division of Science Resources Statistics (NSF/SRS), Survey of Academic Research and Development Expenditures; and NSF/SRS, Survey of Federal Funds for Research and Development.

Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×

has been made to university R&D since 1998 and was enabled by the addition of questions to the data collection form that break out how much of the R&D expenditures were passed through the institution to subrecipients, and how much were received by the institution as a subrecipient. These questions were added when NSF determined, as the result of an investigation, that at least $350 million of the $1.9 million difference in 1996 was due to double-counting (National Research Council, 2000). The effect of this correction was dramatic, although not necessarily consistent in direction (see Figure 7-1).

RECONCILING THE ESTIMATES

It is tempting to suggest that differences in definitions, time horizons, data collection methodology, and the effect of sampling and measurement errors simply invalidate a direct comparison of federal spending and performer expenditures. The estimates of federal funding for R&D in academia and industry, academic R&D expenditures, and industry R&D expenditures are independently derived from very different sources and cannot be expected to add up. However, it is essential to continue to peel away these potential sources of the discrepancy as a means of aiding in the interpretation of the NSF data, as well as a means of assisting in identifying sources of error in the estimates.

Thus, the panel’s recommendation is that a reconciliation of the estimates of federal outlays for R&D and performer expenditures be conducted by NSF on an annual basis (Recommendation 7.2). This reconciliation should be published and widely disseminated by NSF as an aid to data users and as a blueprint for modifying the structure and implementation of the data collections to improve their concordance over time. The recent decision on the part of NSF to attempt to reduce a source of noncomparatibility by collecting data on potential double-counting of university R&D subcontracting was in direct response to attention to the size of the discrepancy. This change in data collection indicates that shining the spotlight on these differences can lead to improvements in data collection that, in turn, will reduce the gap over time.

Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×
Page 139
Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×
Page 140
Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×
Page 141
Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×
Page 142
Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×
Page 143
Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×
Page 144
Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×
Page 145
Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×
Page 146
Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×
Page 147
Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×
Page 148
Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×
Page 149
Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×
Page 150
Suggested Citation:"7 Analysis of the R&D Data Funding Discrepancies." National Research Council. 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington, DC: The National Academies Press. doi: 10.17226/11111.
×
Page 151
Next: 8 Survey Management and Adminstrative Issues »
Measuring Research and Development Expenditures in the U.S. Economy Get This Book
×
Buy Paperback | $57.00 Buy Ebook | $45.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This report examines the portfolio of research and development (R&D) expenditure surveys at the National Science Foundation (NSF), identifying gaps and weaknesses and areas of missing coverage. The report takes an in-depth look at the definition of R&D, the needs and potential uses of NSF’s R&D data by a variety of users, the goals of an integrated system of surveys and other data collection activities, and the quality of the data collected in the existing Science Resources Statistics surveys.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!