3
Measuring R&D in Business and Industry

The annual survey of R&D in industry is one of the most important products of the NSF’s Science Resources Statistics (SRS) division. It measures a sector that accounts for the bulk of R&D activity in the economy, accounting for 66 percent of funding for R&D by source and more than 70 percent of funds spent on R&D.

Because of the predominance of the industrial sector in R&D, the industrial R&D expenditure data are closely watched as an indicator of the health of the R&D enterprise in the United States. For example, a recent downturn in real spending on industrial R&D has given rise to some concern over the health of that enterprise.1 The decline in industrial R&D spending between 2001 and 2002, coinciding with the overall economic slowdown, was identified by NSF as the largest single-year absolute and percentage reduction in current-dollar industry R&D spending since NSF initiated the survey in 1953 (National Science Foundation, 2004). This downturn was identified as the leading cause of the downtick in the ratio of R&D spending to gross domestic product from 2000 to 2002.

This most important R&D survey is also the most problematic. No survey in the NSF portfolio of R&D expenditure surveys has been as impacted by changes in the R&D environment (discussed in Chapter 1) as the Survey of Industrial Research and Development.

This survey is also the most sensitive to changes in the procedures for statistical measurement. Little is known about respondents, what they

1  

Industrial R&D failed to keep pace with inflation and experienced its first decline in real terms after 1994 (National Science Foundation, 2002).



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy 3 Measuring R&D in Business and Industry The annual survey of R&D in industry is one of the most important products of the NSF’s Science Resources Statistics (SRS) division. It measures a sector that accounts for the bulk of R&D activity in the economy, accounting for 66 percent of funding for R&D by source and more than 70 percent of funds spent on R&D. Because of the predominance of the industrial sector in R&D, the industrial R&D expenditure data are closely watched as an indicator of the health of the R&D enterprise in the United States. For example, a recent downturn in real spending on industrial R&D has given rise to some concern over the health of that enterprise.1 The decline in industrial R&D spending between 2001 and 2002, coinciding with the overall economic slowdown, was identified by NSF as the largest single-year absolute and percentage reduction in current-dollar industry R&D spending since NSF initiated the survey in 1953 (National Science Foundation, 2004). This downturn was identified as the leading cause of the downtick in the ratio of R&D spending to gross domestic product from 2000 to 2002. This most important R&D survey is also the most problematic. No survey in the NSF portfolio of R&D expenditure surveys has been as impacted by changes in the R&D environment (discussed in Chapter 1) as the Survey of Industrial Research and Development. This survey is also the most sensitive to changes in the procedures for statistical measurement. Little is known about respondents, what they 1   Industrial R&D failed to keep pace with inflation and experienced its first decline in real terms after 1994 (National Science Foundation, 2002).

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy know, and how they treat this survey. We do not know who is responding in each firm, or what their position or capabilities are. Nor do we know how the data each firm reports are gathered within the firm, or their relationship to similar data reported to the Securities and Exchange Commission (SEC).2 Since its inception in 1953, the survey has experienced several focused efforts to modernize and strengthen its conceptual and technical foundation. The most ambitious modernization effort took place a little over a decade ago, in 1992, when the sample design was substantially modified to better represent the growing service sector, extend coverage to smaller firms, and collect additional information for understanding R&D outsourcing and that performed by foreign subsidiaries. The 1992 redesign effort updated the operations of the survey to reflect many aspects of the changed environment and correct some of the most severe deficiencies, which had rendered the survey results quite misleading over time. At the conclusion of that redesign, however, there was still much to do to meet a growing need for data to measure the emerging realities in the conduct and measurement of R&D in the United States. Not much progress has been made since that redesign; in fact, the survey changes that have been introduced since 1992 have been almost cosmetic in their application. Minimal revisions have been implemented to maintain currency, such as the introduction of the North American Industry Classification System (NAICS) in 1997 and the more recent changes in questionnaire wording and collection procedures (see Box 3-1). Thus, this decade-old redesign of the survey has left several issues unresolved. A major unresolved issue is the failure to implement collection of R&D data at the product or line-of-business level of detail; another is the failure to learn more about respondents and to sharpen concepts and definitions to more adequately reflect business organization for R&D; and a third is the inability to speed the production and release of the estimates to data users. With the passage of time, the need to consider further revisions has accelerated. The R&D environment has changed even further since the early 1990s. As discussed in Chapter 1, organizational arrangements for the conduct of R&D have continued to change apace. New forms of R&D in 2   Hall and Long (1999) compared data from the 1991 and 1992 Survey of Industrial Research and Development to the SEC 10K reports and found a number of discrepancies of three basic types: differences between fiscal and calendar year reporting, differences in coverage (whether foreign-performed R&D was included), and differences in definition, either intended or unintended. They pursued the source of the definitional differences in interviews with a small number of firms. Given the confidential nature of the RD-1 data, detailed tables of the discrepancies by industry and reason could not be produced, so that a precise picture of the relationship between SEC and NSF data was not obtained.

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy BOX 3-1 Thumbnail History of the Industrial R&D Survey 1953 First Survey of Industrial Research and Development is conducted by the Bureau of Labor Statistics. 1957 The Bureau of the Census assumes responsibility for the survey. 1967 The sample design called for the selection of a new sample (about 14,000 companies) every 5 years with estimates updated from a subset or panel of (about 1,700) companies in the intervening years. The current-year panel was a subset of the sample drawn for 1967. 1976 Sample wedging was introduced to update the sample between the irregular interval sample selections. 1992 This was the first year of “annual” sampling. Beginning with 1992, a new sample of 23,000-25,000 firms was selected annually. Previously, the sample frame was limited to companies above certain size criteria based on number of employees that varied by industry; for 1992 and beyond these size criteria were dropped. Sampling error maximums were established, 2 percent for industries in which there was indication of a high level of R&D activity, 5 percent for all others. And 25 new nonmanufacturing industry groups were added to the sampling frame. 1994 Criteria for predetermination of “certainty” selections for the sample were changed. The predetermination was limited to companies with reported or estimated R&D expenditures of $1 million or with 1,000 or more employees. Also, the sampling frame for each industry sampling recode stratum was partitioned into large and small companies based on payroll, thereby expanding the use of the more efficient simple random sampling technique for the majority of companies. The probability proportionate to size sampling technique was used for the balance of the frame. 1995 Sampling strata were redefined to correspond to the industry or groups of industries for which statistics are published. the service sector (particularly in the biomedical fields), the expanding role of small businesses, the geographic clustering of R&D, cooperation, collaboration, alliances, “open innovation,” partnerships, globalization, government programs to promote advanced technology demonstrations and nanotechnology—all have impacted the relevance and quality of the data. The list of structural and organizational changes in the business world that impact on the ability to understand the extent and impact of R&D grows daily. Despite NSF staff efforts, the survey is not keeping up. The panel concludes that it is time to implement another major redesign of this survey (Conclusion 3.1). The redesign would take a four-pronged approach: The redesign would begin with a reassessment of the U.S. survey against the “standard,” that is, the international definitions as promulgated

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy 1996 The predetermination of certainty selections for the sample was limited to companies with reported or estimated R&D expenditures of $5 million or more. Previously, the criteria were R&D expenditures of $1 million or more or 1,000 or more employees. Instead of being based on payroll, partitioning of the sample was based on employment. Manufacturing (< 50 employees) and nonmanufacturing (< 15 employees) small company totals were collected based on simple random sampling and reported with the “other industries” residual. A “zero industry” designation was established for those industries that reported no R&D expenditures in 1992-1994 but may report R&D in the future. 1999 The North American Industry Classification System (NAICS) was introduced into the survey, and industry codes assigned during the sampling process were retained for publication of the resulting statistics. No other major revisions to the sample design were made since the previous survey cycle. This was the first year that the small company totals were reported as a separate “small companies” category. 2002 For the first time in the survey’s 50-year history, the full survey was mandatory for this single survey cycle. The sample size was expanded to improve state estimates. Sampling, estimation, and reporting of small company strata were eliminated since analytical interpretation of these cells was problematic. Industry Federally Funded Research and Development Centers were not included in the survey estimates. 2003 An R&D check-off item was placed on the Census Bureau’s Company Organization Survey, an annual survey sent to all multi-unit companies that is used to update the Standard Statistical Establishment List (SSEL)—the frame for the industry R&D survey in the 2003 survey statistical period. The inclusion of this question was tested as a way to create an R&D registrar. 2004 A redesigned questionnaire was introduced based on cognitive research. The major impact was to put open-ended items into question format. through the Frascati Manual (Organisation for Economic Co-operation and Development, 2002a), thereby adding some data items. It would benchmark U.S. survey methodology against best practices in other countries, many of which appear to be producing data of better quality and with more relevance than NSF. In order to sharpen the focus of the survey and fix problems further identified in this report, the redesign would update the questionnaire to facilitate an understanding of new and emerging R&D issues. In particular, it would test and implement the collection of data on R&D funds from abroad, from affiliate firms, and from independent firms and other institutions for the performance of R&D in the United States. It would sharpen the question on the outsourcing of R&D to distinguish between payments to affiliated firms, to independent firms, and to other institutions abroad. It would make more extensive use of web-based

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy collection technology after appropriate cognitive and methodological research. The redesign would enhance the program of data analysis and publication that would facilitate additional respondent cooperation, enhance the understanding of the industrial R&D enterprise in the United States, and provide feedback on the quality of the data to permit updating the survey methodology on an ongoing basis. The redesign would be supported by an extensive program of research, testing, and evaluation so as to resolve issues regarding the appropriate level at which to measure R&D, particularly to answer, once and for all, the question about the collectability of product or line-of-business detail. The redesign would revise the sample to enhance coverage of growing sectors and the collection procedures to better nurture, involve, and educate respondents and to improve relevance and timeliness. To assist NSF in identifying these venues for improvement and in prioritizing these tasks, this chapter addresses each of these areas. INTERNATIONAL STANDARDS The concepts and definitions of R&D used in the United States today are generally in keeping with international standards. This is not surprising. After all, the science of measuring R&D was pioneered in the United States, and it has proceeded ahead by virtue of an international effort marked by remarkable comity and underscored by collaborative development. This collaboration among the national experts in member countries of the Organisation for Economic Co-operation and Development (OECD) who collect and issue national R&D data has been ongoing for about five decades, under OECD auspices. The collaboration has been codified in the Frascati Manual, which was initially issued about 40 years ago, with several updates over the years, the most recent in 2002. Nonetheless, there is increasing evidence that the United States is departing somewhat from the reporting standards used internationally. For example, the OECD publication Main Science and Technology Indicators (Organisation for Economic Co-operation and Development, 2004) provides an indication that some data are not comparable within OECD guidelines. Although not of serious consequence, these departures indicate that some catch-up work may be needed in order to conform with international standards.3 3   The R&D expenditure data for the United States are underestimates for a number of reasons: (1) R&D performed in the government sector covers only federal government activities. State and local government establishments are excluded; (2) In the higher education sector, R&D in the humanities is excluded, as are capital expenditures; and (3) R&D expenditure in the private nonprofit (PNP) sector covers only current expenditures.

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy Progress in Other Countries In spite of a long history of U.S. leadership in measuring R&D, there is evidence that other countries may have caught up with and moved ahead of the United States in the measurement of industrial R&D. To the extent that this may have happened, there are reasons for the trend. Other countries generally have more access to administrative sources of data, or they have traditions in statistical collection that allow (or prescribe) more intrusive collection of data in greater depth than is considered feasible in the United States. The experience of countries in the European Community with the development of the round of Community Innovation Surveys (CIS) is instructive in this regard. The fact that other countries have developed more information on foreign funding of domestic R&D is an indicator of how they have extended knowledge of the R&D enterprise beyond that available to U.S. policy makers. One country that has made recent strides in the measurement of industrial R&D, which could well be emulated in many respects in the United States, is Canada. It has developed a robust R&D expenditure measurement program. The Industrial Research and Development Survey in Canada A survey of research and development performance in commercial enterprises (privately or publicly owned) and of industrial nonprofit organizations has been conducted in Canada since 1955. The survey has changed in frequency, in detail, and in use of administrative data over the years. Now it is an annual survey of enterprises that perform $1 million (Canadian) worth or more of R&D, complemented by the use of administrative data from the tax authority in order to eliminate the reporting burden for small R&D performers. The tax data arise because of the federal Scientific Research and Experimental Development program, which, in some form, is     Depreciation is reported in place of gross capital expenditures in the business enterprise sector. Higher education (and national total) data were revised back to 1998 due to an improved methodology that corrects for double-counting of R&D funds passed between institutions. Breakdown by type of R&D (basic research, applied research, etc.) was also revised back to 1998 in the business enterprise and higher education sectors due to improved estimation procedures. Beginning with the 2000 Government Budget Appropriations or Outlays for R&D (GBAORD) data, budgets for capital expenditure, “R&D plant” in national terminology, are included. GBAORD data for earlier years relate to budgets for current costs only. The United States Technological Balance of Payments (TBP) data cover only “royalties and license fees,” which are internationally more comparable. Other transactions, notably “other private services,” have been excluded.

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy available to all firms that perform R&D. The data are available to the statistical office under a provision of the Statistics Act. Not only are the tax files a source of data for R&D performing enterprises, but they also provide the list of enterprises in Canada that perform R&D. Of course, not all firms apply for the tax benefit, as some regard the administrative demand and the risk of audit as outweighing the benefits. However, these firms tend to be small performers of R&D with few staff committed to the activity. With the survey frame given by administrative records, the statistical office is able to take data directly from the appropriate tax form for about 85 percent of the population of R&D performers. For the rest, those that perform $1 million or more of R&D, a more elaborate questionnaire is mailed to the enterprise, once a contact person is identified as a result of an initial telephone contact. It is felt that this initial contact is essential to the success of the survey. The information collected includes the expenditure on the performance of R&D, including current and capital costs, and the full-time-equivalent number of personnel engaged in the activity. As well as collecting data on the performance of R&D, questions are asked about the source of funds. There are five sources: governments (federal or provincial); business, including the firm itself; higher education; private nonprofit organizations; and foreign firms and other institutions. About 20 percent of funding for industrial R&D in Canada comes from abroad, as measured in the Canadian R&D survey. In contrast, the United States does not collect this type of information on the industrial R&D survey, missing an opportunity to illuminate this important aspect of the globalization of R&D. Although data exist on foreign sources of R&D funding for other countries, there are no data on foreign funding sources of U.S. R&D performance (National Science Board, 2004). The country of control of the R&D-performing firm in the Canadian survey is derived from administrative data, which are used to examine the different characteristics of foreign-controlled and domestic-controlled enterprises in relation to their performance of R&D. About 30 percent of the expenditures on industrial R&D in Canada are made by foreign-controlled firms. This spending emanates from within Canada; it is separately identified although counted as domestic R&D spending, in keeping with international practice. Most R&D performers in Canada have a single head office with one geographical location for both their production and R&D units. However, there are large firms that have production units classified to different NAICS codes, and the production units may be in different provinces. For these firms, special follow-up is required to identify the geographical and industrial allocation of the R&D performed in Canada. Once this is done, the data can be presented by geographical region.

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy The survey is also a vehicle for questions about new or growing activities, such as R&D in software, biotechnology, new materials, and the environment. It collects data on payments and receipts for R&D and other technological services. And it supports international comparison of Canada’s R&D performance (for more details, see Statistics Canada, 2003). ADEQUACY OF CONCEPTS AND DEFINITIONS Mainly for purposes of historical and cross-sectional comparisons and continuity, the definitions that underscore NSF’s collection of R&D expenditure data from industry have followed a one-size-fits-all philosophy. This approach impedes understanding of the scope and nature of industrial R&D. The division of industrial R&D activities into the standard categories of basic research, applied research, and development has characterized the industrial R&D survey since its inception. In an effort to maintain a concordance with the definitions collected in federal and academic surveys, as well as with international sources, NSF defines industrial basic research as the pursuit of new scientific knowledge that does not have specific immediate commercial objective, although it may be in fields of present or potential commercial interest; industrial applied research is investigation that may use findings of basic research toward discovering new scientific knowledge that has specific commercial objectives with respect to new products, services, processes, and methods; and industrial development is the systematic use of the knowledge and understanding gained from research or practical experience directed toward the production or significant improvement of useful products, services, processes, or methods, including the design and development of useful products, materials, devices, and systems (U.S. Census Bureau, 2004). The reporting of data by these basic categories is only as sound as the understanding of the meaning of those categories. In an important 1993 study, Link asked research directors from three research-intensive industries about the accuracy of the categories for describing the scope of R&D that is financed by their companies. Most of the industrial firms reported that the categories described their scope, but there were disagreements among the largest firms. The disagreement had to do with the narrowness of the definition of development. The researcher’s conclusion was that the NSF definition understated the amount of development, from the perspective of the firms (Link, 1996). This problem at the outer edge of the definition of research and development is not surprising since, with some notable exceptions like pharmaceuticals and semiconductors, most corporate investment into R&D exploits science-based inventions that are beyond the research and

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy development phase. The development of new goods and services is often based on costs and specifications defined by marketing opportunities. Branscomb and Auerswald (2002) point out that blurred distinctions between these traditional categories complicate accurate analysis of existing data. Ambiguous usage of these common categories leaves the door open for variation in interpretation by survey respondents, especially across different firms and industries. Jankowski (2002) points out that the service sector (finance, communications, transportation, and trade) creates value and competes by buying products and assembling them into a system or network, efficiently running or operating the system, and providing services for customers who are often members of the public. It is an open question whether the system design, system operation, and service design and delivery functions can be readily classified into the traditional components of R&D. An example suggested to the panel during its workshop would be a major retailing company that has invested in innovative supply chain management techniques. Such a company would quite possibly not recognize and report this activity as R&D. It was observed that at least one large retailer with widely recognized leadership in distribution management reports no R&D program in its 10K filings to the SEC. The distinction between applied research and the two other components of R&D is especially problematic in the industrial sector. Applied research could well involve original research believed to have commercial applications, and it could include research that applies knowledge to the solution of practical problems (Branscomb and Auerswald, 2002). Although, the share of R&D represented by applied research has been fairly steady at about 20 percent of the total since the industry data were first published in 1953, the possible blurring of the lines between the basic and applied research and between applied research and development has the potential to distort analysis of the role of R&D in the generation of innovation and growth. These questions are important for focusing and assessing such federal investment programs as the Advanced Technology Program and the Small Business Innovation Research programs, in which the federal government provides project-level support of early-stage commercial technological development. In assessing the impact of R&D on industries, these distinctions are also important. Branscomb and Auerswald observe that mature industries, such as the automotive sector, tend to invest a smaller percentage of R&D into earlier stages of technological development than do industries at an earlier stage of evolution, such as biotechnology. They make the case for this evolutionary shift in R&D investment strategy by tracing the emerging emphasis on maximizing the “yield from R&D.” This has been accomplished by reducing the amount of basic research accomplished in corporate laboratories, outsourcing, collaborations with universities, and developing corporate venture capital organizations that spin off from the core business.

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy In contrast, the overwhelming focus of U.S. industry on development activities more than on pure research means that three-fourths of R&D activity is lumped into these categories that are closer to the customer. These are less likely to be carried out in formal R&D facilities and thus less likely to be classified as R&D. Especially in the service sector, the failure to recognize these customer-driven activities as development would tend to bias the estimates of R&D expenditures downward. At the same time, there are classification issues in the development category. Today, companies classify activities that range from advanced technology development to operational system development in the same general category. There is evidence that companies are having difficulty in separating development from engineering as technology advances (McGuckin, 2004). The same issue exists for “technical service,” which has certain elements of development embedded. In consequence, the lumping of these activities into one aggregated “development” category tends to obscure important shifts in R&D emphasis over time. Definitional quirks lead to other anomalies. Software development simulating product performance in lieu of product tests is reported as R&D, but other software development, such as software to predict the nature of consumer demand, is not considered R&D. Heavy investments in technical service developments in quality control and in artificial intelligence and expert systems are similarly uncounted in the R&D expenditure estimates. The panel learned that companies may be able to provide additional useful breakouts of more detailed information on the components of R&D if the categories are carefully constructed for ease of collection. For example, one large manufacturing corporation maintains detailed records on investment in developmental functions and could report using a classification structure akin to that adopted in the Department of Defense Uniform Budget and Fiscal Accounting Classifications for RDT&E (research, development, test, and evaluation) Budget Activities.4 It is not certain whether other companies could be similarly accommodating to providing additional detail to help further define the boundaries and details of the R&D classification structure. There is a need for additional classification detail to better describe differences between the industrial sectors and to better focus on the sources of innovative activity. The potential lack of understanding of the concepts and definitions in the Survey of Industrial Research and Development (often referred to as RD-1, the number on the survey form) is especially troubling. A Census 4   These activities are (1) basic research, (2) applied research, (3) advanced technology development, (4) advanced component development and prototypes, (5) system development and demonstration, (6) RDT&E management support, (7) and operational system development.

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy Bureau study found that R&D professionals and support personnel have a good understanding of these concepts, even though they do not necessarily classify their own work that way (U.S. Census Bureau, 1995). However, the persons most often charged with responding to the questionnaire are in the financial or government relations offices of the companies, and their understanding is less likely to be accurate. This finding was confirmed in panel discussions with representatives of reporting companies and covered industries. Although definitions of these terms are included in the instructions transmitted with the forms, prior to the 2003 survey they were not included on the form and thus may have been overlooked. Abbreviated instructions have been included in a revised 2003 questionnaire introduced in March 2004. NSF should conduct research into recordkeeping practices of reporting establishments by industry and size of company to determine if they can report by more specific categories that further elaborate applied research and development, such as the categories utilized by the Department of Defense (Recommendation 3.1). ALIGNMENT OF THE SURVEY WITH TAX AND ACCOUNTING PRACTICES R&D data have never existed in isolation. From their beginning, they were molded and shaped in a larger context. The work that went into developing the collection of information on federal R&D emanated from the concern of Vannevar Bush and others over the lagging role of the U.S. R&D enterprise. Similarly, the concern over appropriate measurement of industrial R&D arose from a concern over the impact of R&D on productivity and economic growth. By the same token, the conceptual foundation of industrial R&D and the basis for the R&D measures have, for two decades, been closely associated with U.S. tax and accounting policies. Along with international standards and policies promulgated by the U.S. Office of Management and Budget, the tax code and accounting rules have shaped and defined R&D. A study by Hall (2001) for the European Union finds that there are several features of federal corporate tax law that have an effect on the amount and type of R&D: expensing of R&D; the research and experimentation (R&E) tax credit; foreign source allocation rules for R&D spending; the preferential capital gains tax rate; accelerated depreciation and investment tax credits for capital equipment; and the treatment of acquisitions, especially as it relates to the valuation of intangibles. The foremost provision of the tax code that defines R&D is the federal research and experimentation tax credit. The credit is designed to stimulate research investment by virtue of a tax credit for incremental research ex-

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy TABLE 3-5 Sample Size and Response Rates, 1990-2002, Industry R&D Survey Year Sample Size Manufacturers Nonmanufacturers Response Rate 1990 1,696 about 1,400 about 300 79.3% 1991 1,648 about 1,400 about 300 85.2 1992 23,376 11,818 11,558 84.0 1993 23,923 15,018 8,905 81.8 1994 23,543 2,939 10,604 84.8 1995 23,809 7,595 16,214 85.2 1996 24,964 4,776 20,188 83.9 1997 23,327 4,655 18,672 84.7 1998 24,809 4,836 19,973 82.7 1999 24,231 4,933 19,498 83.2 2000 24,844 4,808 20,036 81.1 2001 24,809 4,505 20,604 83.2 2002 30,999 10,920 20,079 80.9 error constraints relating to individual industry estimates. The threshold was raised to $5 million in 2001 in order to reduce reporting burden on the basis that, historically, R&D costs between $1 and $5 million account for only 2 percent of total R&D expenditures. When the changes in coverage were introduced a decade ago, they were accompanied by several other changes designed to better focus and simplify the survey. Only companies identified in the Census Bureau’s Business Register as having five or more paid employees are asked to participate in the survey. Furthermore, extensive use is made of information from the Business Register to supplement reporting and reduce redundant data collection. Despite this increase in sample size in the early 1990s, response rates stayed relatively high because, in addition to expanding the survey, the survey was bifurcated into two distinct operations, nearly unique among federal surveys (Table 3-5). A new form, the RD-1A form, was introduced to simplify collection from the expanded base of noncertainty companies. This form, which is sent to nearly all first-time reporters, collects a shorter list of information than the RD-1 form, waiving the five items asterisked below. Research, applied research, development Company-sponsored R&D expenditures in foreign countries R&D performed under contract by others Federally funded R&D by contracting agency*

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy R&D costs by type of expense* Domestic R&D expenditures by state* Energy-related R&D expenditures* Foreign R&D expenditures by country* The RD-1A form also makes it easy for firms that do not perform R&D to quickly report that fact. The form has a screening item for respondents to indicate that they do no R&D. Those firms that do no R&D are able to complete their reporting requirements by using a toll-free touch-tone data entry (TDE) system that consists of a series of voicemail options directing respondents according to the nature of their request. Nearly 90 percent of companies responding to the short form use the TDE system. While revisions in survey operations have done much to enrich the coverage of the survey and reduce the burden on those companies that do little or no R&D, the ease of reporting no R&D raises a new concern: Does the relative simplicity of reporting no R&D encourage underreporting? This is an issue that could benefit from research through the program of continuous record-keeping practice surveys recommended later in this chapter. There is also an attempt to simplify the data-gathering chore for the larger respondents. The instruction package for the longer RD-1 provides definitions and item-by-item instructions. These item-by-item instructions give methods of estimating expenditures if the company does not keep records that give exact allocations. For example, methods were given to estimate basic, applied, and development expenditures. On the RD-1 form, companies are informed that they may report on a diskette rather than on paper. In addition, in 2004, a more extensive use of the Internet for reporting is envisioned with a new version of a web-based form being made available for respondents to use. Despite these measures, identifying companies that conduct R&D, aside from the very largest, is a difficult matter. Companies that perform reportable R&D are rare as a proportion of all companies. The Census Bureau’s sample frame is comprised of over 1.8 million companies, of which only 35,000 are estimated to be actively engaged in R&D in any given year. In the 2001 survey, forms were mailed to about 25,000 companies, and only about 3,400 of them reported that they actually conducted R&D (Bostic, 2003). Survey managers at NSF have apparently done a good job of reducing unnecessary burden on respondents while obtaining some coverage of noncertainty firms. However, the question remains: Has enough been done to produce fully representative data? Without an independent, corroborating census of R&D, it is difficult to make a judgment as to the adequacy of survey coverage. The panel urges NSF to continue to seek an answer to

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy whether the data are representative. This will be partially achieved through analysis of the results of collection of R&D information on the Company Organization Survey, continued matches to outside administrative sources such as the SEC 10K reports, reference to patent data to identify innovating firms, comparisons with patterns and trends obtained in R&D surveys in other countries, and, under the auspices of new legislation, matching data files with surveys conducted by fellow statistical agencies (Recommendation 3.7). Need for a Consultative Sample During the course of its deliberations, the panel found the interaction with senior staff of a large industrial R&D performer and the membership of the Industrial Research Institute’s Financial Network and the Research on Research Committee to be invaluable as sources of practical information and advice. We note that fairly elaborate consultation takes place with advisory bodies and workshops assembled in support of the federal funds and academic spending surveys. There is no such standing mechanism for the industrial R&D survey. Prior to 1990, NSF had an active and productive Industrial Panel on Science and Technology, to which an annual inquiry was sent. This panel consisted of company officials in major R&D performing industries, including the 20 top R&D spending companies as identified by research expenditures reported on the RD-1 form. In the final survey, conducted from April through September 1989, responses were received from 72 of the 83 surveyed R&D officials. Their companies represented more than one-half of all U.S. company-funded R&D expenditures. The primary purpose of the panel was to project the growth in R&D expenditures for the current and subsequent year, and the major factors responsible for projected changes. This projection of spending by the largest R&D companies was used to provide a timely estimate of R&D spending, given the lags in the release of results from the R&D survey. The data were benchmarked to the results of the RD-1 survey for 1988, and the estimates for the 72 respondents to the 1989 survey were then used to estimate R&D spending for industrial R&D throughout the economy. Moreover, respondents were asked to complete special inquiries from time to time. In the final survey, for example, respondents were asked about possible government incentives, the effect of cooperative R&D ventures, and whether leading foreign-born R&D personnel were being lured back to their homelands. The results of this survey were released by NSF fairly quickly after the period of collection in a “Highlights” report. The survey was curtailed in 1990 because of resource shortfalls and concerns over the representativeness of the panel. Over time, the inability of

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy a panel of the largest manufacturing industry reporters to represent the totality of R&D activity in the business community became even more evident. This same shortcoming in the overall RD-1 led to the survey redesign in 1991 to be more inclusive of the range of industries performing R&D. The panel concludes that there is value in having a regular sounding board of R&D professionals from the firms. As mentioned earlier, the RD-1 survey goes primarily to financial officials in companies who are charged with responding to government questionnaires and appropriately are responsible for keeping track of R&D expenditures. In consequence, there is a low awareness of the national R&D survey’s existence among the R&D executives, who are most aware of trends in the structure and performance of R&D (McGuckin, 2004). When NSF needs information on emerging trends, concepts, definitions, and the like, it is best to go to a panel of R&D officials. Therefore, we recommend that NSF again develop a panel of R&D experts, broadly representative of the R&D performing and R&D data-using communities, to serve as a feedback mechanism to provide advice on trends and issues of importance to maintaining the relevance of the R&D data (Recommendation 3.8). Microdata Analysis The opportunity for improving analysis of the scope and impact of R&D on the economy afforded by linking R&D data with data from other collections was touched on in the earlier discussion of the impact of tax and accounting definitions. This, along with longitudinally matching responses from companies to trace the path of R&D within the firm, is a highly promising area of endeavor, and some work has been done in it. Starting in 1965, Griliches (1980) and colleagues began to develop and use the industry R&D survey data at the Census Bureau for studies of the relationship between R&D and productivity. They then linked firm-level R&D data with production establishment data in order to assess the (private) returns to R&D. This work was extended by Lichtenberg and Siegel (1991) and by Adams and Jaffe (1996). However, it has not received much attention at NSF or the Census Bureau until recently. For some time, there has been a recognized need by the Census Bureau, the Bureau of Economic Analysis (BEA), and the National Science Board to link Census Bureau R&D data to BEA data to improve understanding of the impact of foreign investment in R&D conducted in the United States and by U.S. firms abroad. This unfilled need surfaced again in 2003 when Congress passed the Confidential Information Protection and Statistical Efficiency Act, often referred to as data sharing legislation. This law permits the Census Bureau, BEA, and Bureau of Labor Statistics to share

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy business data for statistical purposes only. Shortly after the legislation was passed, the R&D Link Project was developed under the sponsorship of NSF. The R&D Link Project involves linking the data from the 1997 and 1999 R&D surveys to BEA’s 1997 Foreign Direct Investment in the U.S. and the 1999 U.S. Direct Investment Abroad surveys. The feasibility study match is envisioned as a win-win for all three agencies. The Census Bureau will identify unmatched companies to the BEA files that conduct research and development activities, adding them to the R&D survey sample to improve coverage. The unmatched cases are expected, at a minimum, to yield additional information on foreign companies conducting R&D in the United States and the location of R&D activities conducted by U.S. companies abroad. BEA will augment its existing R&D-related data, identify quality issues arising from reporting differences in the BEA and Census Bureau surveys, and improve its sample frames. NSF will obtain aggregate data, which will provide a more integrated dataset on R&D performance and funding with domestic and foreign ownership detail. This new dataset is expected to lead to a more complete account of international aspects of R&D performance and funding. If the project is deemed successful, based on the data quality, the benefits derived, and the utility of the data, it is expected to form the basis for an annual linking project. The panel commends the three agencies for this initiative and encourages this and other opportunities to extend the usefulness of the R&D data collected by enhancing them through matching with like datasets. We urge that the data files that result from these ongoing matching operations be made available, under the protections afforded in the Census Bureau’s Center for Economic Studies, for the conduct of individual research and analytical studies (Recommendation 3.9). Collecting Data on the Nonprofit Sector It has been recognized for many years that a major shortfall in the collection of data from R&D performers has been the lack of coverage of the nonprofit sector. Nonprofit organizations are excluded by definition from the industry survey and are not included in the Census Bureau’s frame from which the sample is drawn. Some are included in the survey of R&D expenditures at colleges and universities, but the vast majority are not covered. This failure to obtain information on nonprofit performers means that the national data are incomplete for a fairly significant (4 percent) part of national R&D performance. On an ongoing basis, NSF must estimate this important part of the total national R&D effort based on reports from

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy federal funding sources and extrapolations of information obtained in the academic performers survey. Collection of data from this sector is exceedingly difficult, hence NSF has mounted special efforts to collect these data just twice in the last 30 years. The first collection was in 1973; the most recent covered fiscal years (FY) 1996-1997 and was conducted over the period April 1998 to the spring 1999 (The Gallup Organization, 2000). The difficulties arise from several factors. There is no list of nonprofit institutions, much less those involved in R&D activities, so the first task for each study must be to compile a list of potentially eligible nonprofit organizations, and that requires refining a number of list sources, each with problems of coverage and completeness. There is little information on nonprofit organizations, and what there is appears scattered among several sources, so classifying them into strata for sampling purposes is quite cumbersome and judgmental. Finally, periodic surveys of this sort often are plagued by low response rates because the collections are unfamiliar to the respondents and many fail to appreciate the usefulness of the data that result from the collection. The final response rate for the 1996-1997 survey exemplifies these problems: only 40 percent (adjusted for out-of-business and nontraceable nonrespondents) of the sample responded. There was considerable confusion over the questions and, as a result, a large item nonresponse. Over 45 percent of the questionnaires of R&D performers required some type of data cleaning or hot-deck imputation. The low response rate means that the analytical possibilities for the data set were severely limited. NSF could not publish state-level estimates, for example. Nonetheless, there were several important lessons learned in the conduct of the FY 1996-1997 survey that can instruct a future effort of this kind: Infrequent surveys require long periods of rather expensive development and pretesting of the questionnaire and field procedures. The cost of this rather large investment should be included in the cost estimate for the survey. The process of developing the questionnaire should include involvement of focus groups and cognitive interviews. Much attention should be paid to developing, refining, and unduplicating the frame for the survey. The survey operators should focus on obtaining buy-in from survey respondents and organizations representing groups of respondents prior to the survey and devote sufficient resources to follow-up activities.

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy Despite these very evident problems that were well documented in the methodology report on this survey, the panel recommends that another attempt should be made to make a survey-based, independent estimate of the amount of R&D performed in the nonprofit sector (Recommendation 3.10). Taking into account the lessons learned in the FY 1996-1997 survey, NSF should devote resources to laying out the design for a new survey utilizing the input of modern cognitive science and sampling theory. If possible, the resurrected survey should be planned from the beginning as a continuing survey operation in order to build a stable frame, and to introduce certainty strata and other survey efficiency and quality improvement features that are not possible with periodic surveys. Statistical Methodology Issues Of several challenging issues in statistical methodology related to the survey, the major problem is that, following a major revision in the early 1990s, changes to the survey in the past decade have been piecemeal and incremental, impeded by the lack of resources to modernize the survey operations. Today’s industrial R&D survey stands in contrast to the other surveys in the NSF portfolio, in that the statistical methodologies and technologies employed in the survey are far from cutting-edge. Survey Design The basic sampling frame is the Business Register, previously known as the Standard Statistical Establishment List. This list, used since 1976, has problems in coverage and currency. The Business Register may have particular problems with completeness of coverage; this potential undercoverage may be an important issue for measurement of R&D expenditures by type or by state. In view of the growing recognition that small firms are increasing as a share of research and development spending, a problem of undercoverage of small firms may well lead to a growing problem of under-estimation (Acs and Audretsch, 2003). The sampling procedure has also evolved, while maintaining a constant focus on companies with the largest R&D expenditures (1,700 companies in 2000), with additional strata defined by industry classification and various expenditure-level cutoff schemes. The largest firms were first identified by the number of employees, and later by the total R&D expenditures based on the previous year’s survey. The industry strata are defined as manufacturing, nonmanufacturing, and unclassified, with 48 categories represented in the 2002 round. To improve state estimates, take advantage of historical data, and improve industry-level estimates, the sample was further divided in 2001

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy to represent “knowns” and “unknowns.” The known segment is comprised of companies that reported R&D expenditures at least once in the previous three survey cycles. These various sampling changes, each introduced to achieve a worthwhile objective, have made it very difficult to achieve control over sampling error in the survey estimates. The variability of the estimates has been a constant concern in the survey. Some of the sampling errors, as computed, remain very high, particularly for the nonmanufacturing universe. High sample errors and high imputation rates for some of the key companies mean that the quality of the published data are suspect, suggesting the need for evaluation of the statistical underpinnings of the survey. Data Collection The method of data collection currently relies on two forms: the RD-1, sent to known, large R&D performers, and the more limited RD-1A, sent to small R&D performers and companies that have not previously reported. Both surveys collect sales or receipts, total employment, employment of scientists and engineers, R&D expenditures information, character of the R&D (basic research, applied research, development), R&D expenditures in other countries, and R&D performed under contract by others. In addition, the RD-1 forms collect information on federally funded R&D by contracting agency, R&D costs by type of expense, domestic R&D expenditures by state, energy-related R&D expenditures, and foreign R&D expenditures by country. Although a diskette is provided and a web-based version of the form is available for completion and mail-in, the primary mode of data collection is by postal mail. Survey forms are mailed in March, with a requested completion date of 30 or 60 days later. Mail follow-up is fairly extensive, with a total of five mailings to delinquent Form RD-1A recipients, but telephone follow-up is limited by resource constraints to the 300 largest R&D performers. If no response was received and no current-year data reported, several data items are imputed. The data collection procedures employed by the Census Bureau for the industry survey stand in stark contrast to the more technologically advanced procedures employed in the smaller federal and academic surveys, which, for the most part, are web-based and have more intensive education, response control, and follow-up schemes. The printed questionnaire is in dire need of review and substantial revision. The last full-scale review of the cognitive aspects of the industry R&D survey was reported in 1995 by Davis and DeMaio (U.S. Census Bureau, 1995). This internal Census Bureau study identified a number of possible improvements in graphics and question wording in the questionnaire. Some of the graphical suggestions have been implemented, but a key

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy recommendation that all survey items be put into question format has not yet been acted on. The wide range of suggestions for possible question wording changes were based on the level of respondent understanding of the concepts and definitions implied in the questions. The issues developed in U.S. Census Bureau (1995) strongly support the need for an ongoing dialogue between the data collectors and data providers in the industry R&D survey. An active program of respondent contacts and record-keeping practice surveys that would have supported a dialogue was dropped in the early 1990s because of resource constraints. Such a program need not be expensive or overly intensive. Davis and DeMaio were able to obtain valuable insights into cognitive issues with just 11 company visits and a mail-out study of some 75 companies that incorporated cognitively oriented debriefing (COD) questions. There are opportunities for improving respondent contacts within the ongoing program. For example, the Census Bureau now conducts an intensive nonresponse follow-up program for the 300 largest R&D performers. Perhaps more useful information could be gleaned from these telephone contacts, which are designed to secure cooperation or to clarify information, if a cadre of telephone interviewers trained in cognitive aspects of telephone interviewing probed more deeply into areas that would improve the data. Supplementing the regular follow-up program could yield invaluable data on the firms with little cost or additional burden on the reporters. The panel strongly recommends that the National Science Foundation and the Census Bureau resume a program of field observation staff visits to a sampling of reporters to examine record-keeping practices and conduct research on how respondents fill out the forms (Recommendation 3.11). In this recommendation, the panel adds its voice in support of the OMB directive, which gave approval of the industry R&D survey in 2002 with the proviso that “a record-keeping study should be done to find out what information businesses have regarding the voluntary items and the reasons for nonresponse to those items” (U.S. Office of Management and Budget, 2001). The problem of nonresponse, however, does not apply solely to items collected on a voluntary basis. As with all surveys, some sample units do not respond at all, or they omit some items. The response rate from 1999 to 2001 was in the 83 to 85 percent range. When there is additional attention, as is the case with the largest 300 companies, response rates can be higher. The rate for these large companies was 90 percent in 2000. The impact of new rules requiring mandatory reporting in the 2003 survey cycle is not yet known. Item nonresponse is also a problem, for both the five mandatory data items and the voluntary items. It is difficult to assess the scope and impact of the problem of nonresponse, since no item nonresponse rates are given

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy for any items. Instead, as mentioned earlier, the Census Bureau publishes imputation rates, which can be quite large and serve as a poor proxy for item nonresponse rates because the imputation rates are weighted to proportion of the total contributed by imputation. In a skewed distribution, they will be very different from nonresponse rates. NSF could make a significant contribution to understanding the quality of the data by ensuring clear definitions and regular reporting of item nonresponse rates. Furthermore, the recent rise in the proxy measure—imputation rates—gives cause for concern and impetus to the need to understand the quality impact of item nonresponse. Survey Content Closely associated with issues of data collection and the cognitive aspects of survey design are issues of data content and related questions of usefulness to potential data users. One problematic aspect of the survey content is the questionable validity of data that break down R&D into the components of basic research, applied research, and development. This issue is discussed earlier in this chapter. A second content problem relates to estimates of the number of R&D scientists and engineers employed by the company, which derive from the definition: scientists and engineers with a 4-year degree or equivalent in the physical and life sciences, mathematics, or engineering. Aside from the obvious problem of transferring the form between the financial office of the company, where the form is completed, and the personnel department, where personnel records are kept, the definition is in some aspects quite vague. What does “equivalent” to a 4-year degree mean? Even if it is possible to identify field of educational specialization, what is a person’s field of employment in the company? When an employee works on multiple tasks, how should they be apportioned? Identification of the location of R&D activity by state is also problematic. Although there is intense interest in the location of R&D activity, there is anecdotal evidence that respondents have great difficulty in accurately allocating this activity to geographic areas, and consequently they have developed measures for geographic allocation that produce data of questionable quality. NSF needs to conduct a more intensive study to determine the quality of state breakdown of R&D activity and to implement changes if warranted. Processing Quality problems can crop up in a survey at the stages of data entry and editing. According to a series of memoranda by Douglas Bond that

OCR for page 48
Measuring Research and Development Expenditures in the U.S. Economy discuss research on various sources of processing errors (U.S. Census Bureau, 1994), data entry procedures produced little error, but the editing process was replete with potential for error. That potential starts with the observation that there is no written description of the editing process, including the process in which an analyst supplies data codes. This study is now a decade old. Although many of the sources of editing error may have been corrected, the absence of a more recent study of processing error and the lack of current documentation of the editing process cause concern over the impact of this source of error. The panel recommends that the indudtrial R&D editing system be redesigned so that the current problems of undocumented analyst judgment and other sources of potential error can be better understood and addressed (Recommendation 3.12). This redesign should be initiated as soon as possible, but it could later proceed in conjunction with the design of a web-based survey instrument and processing system. Imputation is an integral part of the survey operation, and the rates of imputation are high. Again, Bond’s studies found several sources of potential error in an imputation process that varies with the item being imputed. There should be an ability to clearly determine whether errors arise in the editing or the imputation processes. MANAGEMENT STRATEGIES The recommendations concerning the industrial R&D survey are the panel’s highest priorities. There is an urgent need for the survey to be better managed. This can be achieved in a number of ways, including: Finding a contact person to whom the survey is sent for each respondent. Assigning a person at the Census Bureau or NSF to each contact person to answer questions and discuss aspects of the survey. Creating a standing committee of contact people or high-level R&D people who could discuss all issues pertaining to the survey and who could be queried about the usefulness of potential changes in the survey. Increasing NSF involvement in the administration or implementation of the survey, whereby NSF more closely oversees the work done for it by the Census Bureau. Reporting and publication of the R&D data in a more timely manner. The panel discusses the industrial R&D survey and makes recommendations for redesign in Chapter 8.