Methodology Paper

1. Introduction

As the Small Business Innovation Research (SBIR) program approached its twentieth year of operation, the U.S. Congress requested that the National Research Council (NRC) conduct a “comprehensive study of how the SBIR program has stimulated technological innovation and used small businesses to meet federal research and development needs,” and to make recommendations on improvements to the program.1

Mandated as a part of SBIR’s renewal in 2000, the NRC study is to assess the SBIR program as administered at the five federal agencies that together make up 96 percent of SBIR program expenditures. The agencies, in order of program size, are DoD, NIH, NASA, DoE, and NSF.

The objective of the study is not to consider if SBIR should exist or not—Congress has already decided affirmatively on this question. Rather, the NRC Committee conducting this study is charged with providing assessment-based findings to improve public understanding of the program as well as recommendations to improve the program’s effectiveness.

In addition to setting out the study objectives, this report defines key concepts, identifies potential metrics and data sources, and describes the range of methodological approaches being developed by the NRC to assess the SBIR program. Following some historical background on the SBIR program, this introduction outlines the basic parameters of this NRC study.

A Brief History of the SBIR Program

In the 1980s, the country’s slow pace in commercializing new technologies—compared especially with the global manufacturing and marketing success of Japanese firms in autos, steel, and semiconductors—led to serious concern in the United States about the nation’s ability to compete. U.S. industrial competitiveness in the 1980s was frequently cast in terms of American industry’s failure “to translate its research prowess into commercial advantage.”2 The pessimism of some was reinforced by evidence of slowing growth at corporate research laboratories that had been leaders of American innovation in the postwar period and the apparent success of the cooperative model exemplified by some Japanese kieretsu.3

Yet, even as larger firms were downsizing to improve their competitive posture, a growing body of evidence, starting in the late 1970s and accelerating in the 1980s, began to indicate that small businesses were assuming an increasingly important role in both innovation and job creation.4 Research by David Birch and others suggested that national policies should promote and build on the competitive strength offered by small businesses.5 In addition to considerations of economic growth and competitiveness, SBIR was also motivated by concerns that small businesses were being disadvantaged vis-à-vis larger firms in competition for R&D contracts. Federal

1  

See Public Law 106-554, Appendix I – H.R. 5667, Section 108. Also Annex A in this volume.

2  

David C. Mowery, “America’s Industrial Resurgence (?): An Overview,” in David C. Mowery, ed., U.S. Indus yin 2000: Studies in Competitive Perfomance. Washington, D.C.: National Academy Press, 1999, p. 1. Mowery examines eleven economic sectors, contrasting the improved performance of many industries in the late 1990s with the apparent decline that was subject to much scrutiny in the 1980s. Among the studies highlighting poor economic performance in the 1980s are Dertouzos, et al. Made in America: The MIT Commission on Industrial Productivity, Cambridge, MA: The MIT Press, 1989 and Eckstein, et al. DRI Report on U.S. Manufacturing Industries, New York: McGraw Hill, 1984.

3  

Richard Rosenbloom and William Spencer, Engines of Innovation: US. Industrial Research at the End of an Era. Boston: Harvard Business Press, 1996.

4  

For an account of the growing importance of the small firm in employment and innovation, see Zoltan J. Acs and David B. Audretsch, Innovation and Small Business. Cambridge, Massachusetts: MIT Press, 1991, p. 4. For specifics on job growth, see Steven J. Davis, John Haltiwanger, and Scott Schuh, “Small Business and Job Creation: Dissecting the Myth and Reassessing the Facts,” Business Economics, vol. 29, no. 3, 1994, pp. 113-22. More recently, a report by the Organisation for Economic Cooperation and Development (OECD) notes that small and medium-sized enterprises are attracting the attention of policy makers, not least because they are seen as major sources of economic vitality, flexibility, and employment. Small business is especially important as a source of new employment, accounting for a disproportionate share of job creation. See OECD, Small Business Job Creation and Growth: Facts, Obstacles, and Best Practices, Paris, 1997.

5  

David L. Birch, “Who Creates Jobs?” The Public Interest. Vol. 65, 1981, pp. 3-14



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 4
Methodology Paper 1. Introduction As the Small Business Innovation Research (SBIR) program approached its twentieth year of operation, the U.S. Congress requested that the National Research Council (NRC) conduct a “comprehensive study of how the SBIR program has stimulated technological innovation and used small businesses to meet federal research and 1 development needs,” and to make recommendations on improvements to the program. Mandated as a part of SBIR’s renewal in 2000, the NRC study is to assess the SBIR program as administered at the five federal agencies that together make up 96 percent of SBIR program expenditures. The agencies, in order of program size, are DoD, NIH, NASA, DoE, and NSF. The objective of the study is not to consider if SBIR should exist or not—Congress has already decided affirmatively on this question. Rather, the NRC Committee conducting this study is charged with providing assessment-based findings to improve public understanding of the program as well as recommendations to improve the program’s effectiveness. In addition to setting out the study objectives, this report defines key concepts, identifies potential metrics and data sources, and describes the range of methodological approaches being developed by the NRC to assess the SBIR program. Following some historical background on the SBIR program, this introduction outlines the basic parameters of this NRC study. A Brief History of the SBIR Program In the 1980s, the country’s slow pace in commercializing new technologies—compared especially with the global manufacturing and marketing success of Japanese firms in autos, steel, and semiconductors—led to serious concern in the United States about the nation’s ability to compete. U.S. industrial competitiveness in the 1980s was frequently cast in terms of American industry’s failure “to translate its research prowess into commercial advantage.”2 The pessimism of some was reinforced by evidence of slowing growth at corporate research laboratories that had been leaders of American innovation in the postwar period and the apparent success of the cooperative model exemplified by some Japanese kieretsu.3 Yet, even as larger firms were downsizing to improve their competitive posture, a growing body of evidence, starting in the late 1970s and accelerating in the 1980s, began to indicate that small businesses were assuming an increasingly important role in both innovation and job creation.4 Research by David Birch and others suggested that national policies should promote and build on the competitive strength offered by small businesses.5 In addition to considerations of economic growth and competitiveness, SBIR was also motivated by concerns that small businesses were being disadvantaged vis-à-vis larger firms in competition for R&D contracts. Federal 1 See Public Law 106-554, Appendix I – H.R. 5667, Section 108. Also Annex A in this volume. 2 David C. Mowery, “America’s Industrial Resurgence (?): An Overview,” in David C. Mowery, ed., U.S. Industry in 2000: Studies in Competitive Performance. Washington, D.C.: National Academy Press, 1999, p. 1. Mowery examines eleven economic sectors, contrasting the improved performance of many industries in the late 1990s with the apparent decline that was subject to much scrutiny in the 1980s. Among the studies highlighting poor economic performance in the 1980s are Dertouzos, et al. Made in America: The MIT Commission on Industrial Productivity, Cambridge, MA: The MIT Press, 1989 and Eckstein, et al. DRI Report on U.S. Manufacturing Industries, New York: McGraw Hill, 1984. 3 Richard Rosenbloom and William Spencer, Engines of Innovation: U.S. Industrial Research at the End of an Era. Boston: Harvard Business Press, 1996. 4 For an account of the growing importance of the small firm in employment and innovation, see Zoltan J. Acs and David B. Audretsch, Innovation and Small Business. Cambridge, Massachusetts: MIT Press, 1991, p. 4. For specifics on job growth, see Steven J. Davis, John Haltiwanger, and Scott Schuh, “Small Business and Job Creation: Dissecting the Myth and Reassessing the Facts,” Business Economics, vol. 29, no. 3, 1994, pp. 113-22. More recently, a report by the Organisation for Economic Co- operation and Development (OECD) notes that small and medium-sized enterprises are attracting the attention of policy makers, not least because they are seen as major sources of economic vitality, flexibility, and employment. Small business is especially important as a source of new employment, accounting for a disproportionate share of job creation. See OECD, Small Business Job Creation and Growth: Facts, Obstacles, and Best Practices, Paris, 1997. 5 David L. Birch, “Who Creates Jobs?” The Public Interest. Vol. 65, 1981, pp. 3-14 4

OCR for page 4
commissions from as early as the 1960s had recommended the direction of R&D funds toward small businesses.6 These recommendations, however, were opposed by competing recipients of R&D funding. Although small businesses were beginning to be recognized by the late-1970s as a potentially fruitful source of innovation, some in government remained wary of funding small firms focused on high-risk technologies with commercial promise. The concept of early-stage financial support for high-risk technologies with commercial promise was first advanced by Roland Tibbetts at the National Science Foundation (NSF). As early as 1976, Mr. Tibbetts advocated that the NSF should increase the share of its funds going to small business. When NSF adopted this initiative, small firms were enthused and proceeded to lobby other agencies to follow NSF’s lead. When there was no immediate response to these efforts, small businesses took their case to Congress and higher levels of the Executive branch.7 In response, a White House Conference on Small Business was held in January 1980 under the Carter Administration. The conference’s recommendation to proceed with a program for small business innovation research was grounded in: Evidence that a declining share of federal R&D was going to small businesses; Broader difficulties among small businesses in raising capital in a period of historically high interest rates; and Research suggesting that small businesses were fertile sources of job creation. Congress responded under the Reagan Administration with the passage of the Small Business Innovation Research Development Act of 1982, which established the SBIR program.8 The SBIR Development Act of 1982 The new SBIR program initially required agencies with R&D budgets in excess of $100 million to set aside 0.2 percent of their funds for SBIR. This amount totaled $45 million in 1983, the program’s first year of operation. Over the next 6 years, the set-aside grew to 1.25 percent.9 The legislation authorizing SBIR had two broad goals: “to more effectively meet R&D needs brought on by the utilization of small innovative firms (which have been consistently shown to be the most prolific sources of new technologies) and to attract private capital to commercialize the results of federal research.” SBIR’s Structure and Role As conceived in the 1982 Act, SBIR’s grant-making process is structured in three phases: Phase I is essentially a feasibility study in which award winners undertake a limited amount of research aimed at establishing an idea’s scientific and commercial promise. Today, the legislation anticipates Phase I grants as high as $100,000.10 Phase II grants are larger – normally $750,000 – and fund more extensive R&D to further develop the scientific and technical merit and the feasibility of research ideas. Phase III. This phase normally does not involve SBIR funds, but is the stage at which grant recipients should be obtaining additional funds either from a procurement program at the agency that made the award, from private investors, or from the capital markets. The objective of this phase is to move the technology to the prototype stage and into the marketplace. Phase III of the program is often fraught with difficulty for new firms. In practice, agencies have developed different approaches to facilitating this transition to commercial viability; not least among them are additional SBIR awards.11 6 For an overview of the origins and history of the SBIR program, see James Turner and George Brown, “The Federal Role in Small Business Research,” Issues in Science and Technology, Summer 1999, pp. 51-58. 7 Ibid. 8 Additional information regarding SBIR’s legislative history can be accessed from the Library of Congress. See http://thomas.loc.gov/cgi-bin/bdquery/z?d097:SN00881:@@@L 9 Today, the set aside is fixed at 2.5 percent. 10 With the accord of the Small Business Administration, which plays an oversight role for the program, this amount can be higher in certain circumstances; e.g., drug development at NIH, and is often lower with smaller SBIR programs, e.g., EPA or the Department of Agriculture. 5

OCR for page 4
Some firms with more experience with the program have become skilled in obtaining additional awards. Previous NRC research showed that different firms have quite different objectives in applying to the program. Some seek to demonstrate the potential of promising research. Others seek to fulfill agency research requirements on a cost- effective basis. Still others seek a certification of quality (and the additional awards that can come from such recognition) as they push science-based products toward commercialization.12 Given this variation and the fact that agencies do not maintain data on Phase III, quantifying the contribution of Phase III is difficult. The 1992 and 2000 SBIR Reauthorizations The SBIR program approached reauthorization in 1992 amidst continued worries about the U.S. economy’s capacity to commercialize inventions. Finding that “U.S. technological performance is challenged less in the creation of new technologies than in their commercialization and adoption,” the National Academy of Sciences at the time recommended an increase in SBIR funding as a means to improve the economy’s ability to adopt and commercialize new technologies.13 Accordingly, the Small Business Research and Development Enhancement Act (P.L. 102-564), which reauthorized the program until September 30, 2000, doubled the set-aside rate to 2.5 percent.14 This increase in the percentage of R&D funds allocated to the program was accompanied by a stronger emphasis on encouraging the commercialization of SBIR-funded technologies.15 Legislative language explicitly highlighted commercial potential as a criterion for awarding SBIR grants. For Phase I awards, Congress directed program administrators to assess whether projects have “commercial potential” in addition to scientific and technical merit when evaluating SBIR applications. With respect to Phase II, evaluation of a project’s commercial potential was to consider, additionally, the existence of second-phase funding commitments from the private sector or other non-SBIR sources. Evidence of third-phase follow-on commitments, along with other indicators of commercial potential, was also sought. Moreover, the 1992 reauthorization directed that a small business’ record of commercialization be taken into account when considering the Phase II application.16 The Small Business Reauthorization Act of 2000 (P.L. 106-554) again extended SBIR until September 30, 2008. It also called for an assessment by the National Research Council of the broader impacts of the program, including those on employment, health, national security, and national competitiveness.17 Previous NRC Assessments of SBIR Despite its size and tenure, the SBIR program has not been comprehensively examined. There have been some previous studies focusing on specific aspects or components of the program—notably by the General Accounting Office and the Small Business Administration.18 There are, as well, a limited number of internal assessments of 11 NSF, for example, has what is called a Phase II-B program that allocates additional funding to help potentially promising technology develop further and attract private matching funds. 12 See Reid Cramer, “Patterns of Firm Participation in the Small Business Innovation Research Program in Southwestern and Mountain States,” in National Research Council, The Small Business Innovation Research Program, An Assessment of the Department of Defense Fast Track Initiative, op. cit. In this report, we use the term “product” to refer to goods and services produced by the SBIR firm. 13 See National Research Council, The Government Role in Civilian Technology: Building a New Alliance, Washington, D.C.: National Academy Press, 1992, pp. 29. 14 For fiscal year 2003, this has resulted in a program budget of approximately $1.6 billion across all federal agencies, with the Department of Defense having the largest SBIR program at $834 million, followed by the National Institutes of Health (NIH) at $525 million. The DoD SBIR program, is made up of 10 participating components: (see Figure 1): Army, Navy, Air Force, Missile Defense Agency (MDA), Defense Advanced Research Projects Agency (DARPA), Chemical Biological Defense (CBD), Special Operations Command (SOCOM), Defense Threat Reduction Agency (DTRA), National Imagery and Mapping Agency (NIMA), and the Office of Secretary of Defense (OSD). NIH counts 23 institutes and agencies making SBIR awards. 15 See Robert Archibald and David Finifter, “Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach,” op. cit. pp. 211-250. 16 A GAO report had found that agencies had not adopted a uniform method for weighing commercial potential in SBIR applications. See U.S. General Accounting Office, 1999, Federal Research: Evaluations of Small Business Innovation Research Can Be Strengthened, AO/RCED-99-114, Washington, D.C.: United States General Accounting Office. 17 The current assessment is congruent with the Government Performance and Results Act (GPRA) of 1993: http://govinfo.library.unt.edu/npr/library/misc/s20.html. As characterized by the GAO, GPRA seeks to shift the focus of government decision-making and accountability away from a preoccupation with the activities that are undertaken - such as grants dispensed or inspections made - to a focus on the results of those activities. See http://www.gao.gov/new.items/gpra/gpra.htm 18 An important step in the evaluation of SBIR will be to identify existing evaluations of SBIR. See for example, GAO, “Federal Research: Small Business Innovation Research shows success but can be strengthened. Washington, D.C.: U.S. General Accounting Office, 1992; and GAO, “Evaluation of Small Business Innovation can be Strengthened,” Washington, D.C.: U.S. General Accounting Office, 1999. There is also a 1999 unpublished SBA study on the commercialization of SBIR surveys Phase II awards from 1983 to 1993 among non-DoD agencies. 6

OCR for page 4
agency programs.19 The academic literature on SBIR is also limited.20 Annex E provides a bibliography of SBIR as well as more general references of interest. Against this background, the National Academies’ Committee for Government-Industry Partnerships for the Development of New Technologies—under the leadership of its chairman, Gordon Moore—undertook a review of the SBIR program, its operation, and current challenges. The Committee convened government policy makers, academic researchers, and representatives of small business on February 28, 1998 for the first comprehensive discussion of the SBIR program’s history and rationale, review existing research, and identify areas for further research and program improvements.21 The Moore Committee reported that: SBIR enjoyed strong support both within and outside the Beltway. At the same time, the size and significance of SBIR underscored the need for more research on how well it is working and how its operations might be optimized. There should be additional clarification about the primary emphasis on commercialization within SBIR, and about how commercialization is defined. There should also be clarification on how to evaluate SBIR as a single program that is applied by different agencies in different ways.22 Subsequently, at the request of the DoD, the Moore Committee was asked to review the operation of the SBIR program at Defense, and in particular the role played by the Fast Track Initiative. This resulted in the largest and most thorough review of an SBIR program to date. The review involved substantial original field research, with 55 case studies, as well as a large survey of award recipients. It found that the SBIR program at Defense was contributing to the achievement of mission goals—funding valuable innovative projects—and that a significant portion of these projects would not have been undertaken in the absence of the SBIR funding. The Moore Committee’s assessment also found that the Fast Track Program increases the efficiency of the DoD SBIR program by encouraging the commercialization of new technologies and the entry of new firms to the program. More broadly, the Moore Committee found that SBIR facilitates the development and utilization of human capital and technological knowledge. Case studies have shown that the knowledge and human capital generated by the SBIR program has economic value, and can be applied by other firms. And through the certification function, it noted, SBIR awards encourage further private sector investment in the firm’s technology. Based on this and other assessments of public private partnerships, the Moore Committee’s Summary Report on U.S. Government-Industry Partnerships recommended that “regular and rigorous program-based evaluations and feedback is essential for effective partnerships and should be a standard feature,” adding that “greater policy attention and resources to the systematic evaluation of U.S. and foreign partnerships should be encouraged.”23 Preparing the Current Assessment of SBIR As noted, the legislation mandating the current assessment of the nation’s SBIR program focuses on the five agencies that account for 96 percent of program expenditures (although the National Research Council is seeking to learn of the views and practices of other agencies administering the program as well.) The mandated agencies, in order of program size, are the Department of Defense, the National Institutes of Health, the National Aeronautics and Space Administration, the Department of Energy, and the National Science Foundation. Following the passage of H.R. 5667 in December 2000, extensive discussions were held between the NRC and the responsible agencies on the scope and nature of the mandated study. Agreement on the terms of the study, formalized in a Memorandum of Understanding, was reached in December 2001 (See Annex B), and the funding necessary for the Academies to begin the study was received in September 2002. The study was officially launched on 1 October 2002. The study will be conducted within the framework provided by the legislation and the NRC’s contracts with the five agencies. These contracts identify the following principal tasks: 19 Agency reports include an unpublished 1997 DoD study on the commercialization of DoD SBIR. Following the authorizing legislation for the NRC study, NIH launched a major review of the achievements of its SBIR program. NASA has also completed several reports on its SBIR program. See Annex C for a list of agency reports. 20 See the attached bibliography. 21 See National Research Council, Small Business Innovation Research: Challenges and Opportunities, C. Wessner, ed., Washington, D.C.: National Academy Press, 1999. 22 Ibid. 23 See National Research Council, Government-Industry Partnerships for the Development of New Technologies, Summary Report, C. Wessner, ed., Washington, D.C.: National Academies Press, 2002. 7

OCR for page 4
Collection and analysis of agency databases and studies, o Survey of firms and agencies, o Conduct of case studies organized around a common template, and; o Review and analysis of survey and case study results and program accomplishments. o As per the Memorandum of Understanding between the NRC and the agencies, the study is structured in two-phases. Phase I of the study, beginning on October 2002, focuses on identifying data collection needs and the development of a research methodology. Phase II of the study, anticipated to start in 2004, will implement the research methodology developed in Phase I of the study. This document outlines the methodological approach being developed under Phase I of the study. It introduces many of the methodological questions to be encountered during Phase II of the NRC study. Finally, it outlines strategies for resolving these questions, recognizing that some issues can only be resolved in the context of the study itself. Given that agencies covered in this study differ in their objectives and goals, the assessment will necessarily be agency-specific.24 As appropriate, the Committee will draw useful inter-agency comparisons and multiyear comparisons. In this regard, a table with the agencies in one dimension and all of the identified SBIR objectives in the other may be a useful expository tool. The study will build on the methodological models developed for the 1999 NRC study of the DoD’s Fast Track initiative, as appropriate, clearly recognizing that the broader and different scope of the current study will require some adjustments.25 Additional areas of interest, as recognized by the Committee, may also be pursued as time and resources permit. 24 Particularly, with respect to DoD, methodological comparability will be sought to enable multiyear comparisons. Where possible and appropriate, tracking of progress of previously surveyed/interviewed firms will be considered as well. 25 In particular, the objective of the Fast Track study was to compare Fast Track awards and non-Fast Track awards within the DoD SBIR program, in order to determine the efficacy of Fast Track. See National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, C. Wessner, ed., National Academy Press, Washington, D.C., 2000. 8