I
INTRODUCTION



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium I INTRODUCTION

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium This page intentionally left blank.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium Introduction As the Small Business Innovation Research (SBIR) program approached its twentieth year of operation, the U.S. Congress asked the National Research Council (NRC) to carry out a “comprehensive study of how the SBIR program has stimulated technological innovation and used small businesses to meet federal research and development needs” and make recommendations on improvements to the program.1 The NRC’s charge is to examine SBIR’s contributions and how it can be improved; not to debate its rationale.2 An initial conference to launch this assessment was convened in Washington, D.C. on 24 October 2002. It provided an opportunity for agency officials from each of the five departments and agencies accounting for 96 percent of SBIR program funds to provide an overview of their goals, operations, and challenges while including input from smaller agencies. As the first comprehensive perspective on the SBIR program, the conference captured new information and understanding of its operation and potential. It also reviewed the key challenges involved in measuring and assessing the impact of this varied and complex program. 1   See Public Law 106-554, Appendix I–H.R. 5667, Section 108. 2   The nature of this charge was emphasized at the conference launching the study by Committee Chairman, Jacques Gansler, as well as by Kenneth Flamm and James Turner. See the Proceedings section of this volume for summaries of their remarks. Indeed, Mr. Turner noted that the study by the NRC is not expected to question whether the program should exist. “We’re 20 years into the SBIR now,” he said. “It is a proven entity; it’s going to be with us.” He suggested that the appropriate goals for the study would be to look ahead and craft a series of sound suggestions on how to improve the program and to give good advice to Congress on what legislative changes, if any, are necessary.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium This volume, which reports on the conference of October 24, 2002, is the first in a series to be published by the National Academies in response to the Congressional mandate. This introduction provides a summary of the program’s history leading up to the current assessment, a précis of SBIR’s role in the nation’s innovation system, an overview of SBIR’s operations at different agencies, and the methodological issues and challenges facing the current NRC assessment—the themes of the launch conference. A BRIEF HISTORY OF THE SBIR PROGRAM The Founding In the 1980s, the country’s slow pace in commercializing new technologies—compared especially with the global manufacturing and marketing success of Japanese firms in autos, steel, and semiconductors—led to serious concern in the United States about the nation’s ability to compete. U.S. industrial competitiveness in the 1980s was frequently cast in terms of American industry’s failure “to translate its research prowess into commercial advantage.”3 The pessimism of some was reinforced by evidence of slowing growth at corporate research laboratories that had been leaders of American innovation in the postwar period and the apparent success of the cooperative model exemplified by some Japanese kieretsu.4 Yet, even as larger firms were downsizing to better compete, a growing body of evidence, starting in the late 1970s and accelerating in the 1980s, began to indicate that small businesses were assuming an increasingly important role in both innovation and job creation. Research by David Birch and others suggested that national policies should promote and build on the competitive strength offered by small businesses.5 3   David C. Mowery, “America’s Industrial Resurgence (?): An Overview,” National Research Council, U.S. Industry in 2000: Studies in Competitive Performance. David C. Mowery, ed. Washington, D.C.: National Academy Press, 1999, p. 1. Mowery examines eleven economic sectors, contrasting the improved performance of many industries in the late 1990s with the apparent decline that was subject to much scrutiny in the 1980s. Among the studies highlighting poor economic performance in the 1980s are Dertouzos, et al., Made in America: The MIT Commission on Industrial Productivity, Cambridge, MA: The MIT Press, 1989 and Eckstein, et al., DRI Report on U.S. Manufacturing Industries, New York: McGraw Hill, 1984. 4   Richard Rosenbloom and William Spencer, Engines of Innovation: U.S. Industrial Research at the End of an Era. Boston: Harvard Business Press, 1996. 5   David L. Birch, “Who Creates Jobs?” The Public Interest. 65(1981):3-14. Birch’s work exercised major influence on the perception of the role of small firms. Over the last twenty years, it has been carefully scrutinized, leading to the discovery of some methodological flaws, namely making dynamic inferences from static comparisons, confusing gross and net job creation, and admitting biases from chosen regression techniques. See S. J. Davis, J. Haltiwanger, and S. Schuh, “Small Business and Job Creation: Dissecting the Myth and Reassessing the Facts, Working Paper No. 4492, Cam-

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium Meanwhile, federal commissions from as early as the 1960s had recommended the direction of R&D funds toward small businesses.6 These recommendations, however, were opposed by traditional recipients of government R&D funding.7 Although small businesses were beginning to be recognized by the late-1970s as a potentially fruitful source of innovation, some in government remained wary of funding small firms focused on high-risk technologies with commercial promise. The concept of early-stage financial support for high-risk technologies with commercial promise was first advanced by Roland Tibbetts at the National Science Foundation (NSF). As early as 1976, Mr. Tibbetts advocated that the NSF should increase the share of its funds going to small business. When NSF adopted this initiative, small firms were enthused and proceeded to lobby other agencies to follow NSF’s lead. When there was no immediate response to these efforts, small businesses took their case to Congress and higher levels of the Executive branch.8 In response, a White House Conference on Small Business was held in January 1980 under the Carter Administration. The conference’s recommendation to proceed with a program for small business innovation research was grounded in: Evidence that a declining share of federal R&D was going to small businesses; Broader difficulties among small businesses in raising capital in a period of historically high interest rates; and Research suggesting that small businesses were fertile sources of job creation. In addition to these policy rationales, some would suggest (e.g., see Kenneth Flamm) that there was political appeal in seeing R&D dollars “spread a little more widely than they were being spread before.” Congress responded under the Reagan Administration with the passage of the Small Business Innovation Research Development Act of 1982, which established the SBIR program.9     bridge, MA: National Bureau of Economic Research, 1993. At the same time, these methodological fallacies “have not had a major influence on the empirically based conclusion that small firms are over-represented in job creation,” according to Per Davidsson. See Per Davidsson, “Methodological concerns in the estimation of job creation in different firm size classes.” Working Paper, Jönköping International Business School, 1996. 6   For an overview of the origins and history of the SBIR program, see James Turner and George Brown, “The Federal Role in Small Business Research,” Issues in Science and Technology, Summer 1999, pp. 51-58. 7   See Roland Tibbetts, “The Role of Small Firms in Developing and Commercializing New Scientific Instrumentation: Lessons from the U.S. Small Business Innovation Research Program,” in Equipping Science for the 21st Century, John Irvine, Ben Martin, Dorothy Griffiths, and Roel Gathier, eds. Cheltenham, UK: Edward Elgar Press, 1997. For a summary of some of the critiques of SBIR, see “Assessing SBIR” in this Introduction. 8   Ibid. 9   Additional information regarding SBIR’s legislative history can be accessed from the Library of Congress. See http://thomas.loc.gov/cgi-bin/bdquery/z?d097:SN00881:@@@L.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium Box A: Small Businesses as Engines of Growth and Job Creation Confirming Birch’s initial insight, small businesses are now widely recognized as the engines of growth and job creation. A recent report by the Organisation for Economic Cooperation and Development (OECD) notes that small and medium-sized enterprises are attracting the attention of policy makers because they are seen as major sources of economic vitality, flexibility, and employment. Small business is seen as especially important as a source of new employment, accounting for a disproportionate share of job creation.a This perception is reflected in Congressman Roscoe Bartlett’s introduction to the conference, where he credited small businesses with creating more than 90 percent of new jobs in the recovery that followed the recession of the early 1990s.b Recent analysis by the Census Bureau notes that “new firm start-ups play a far more important role in the economy than has previously been recognized. According to recent U.S. Census data (1999-2000), small businesses created three-quarters of U.S. net new jobs (2.5 million of the 3.4 million total). This small business share varies from year to year, reflecting economic trends. Over the decade of the 1990s, the U.S. Census reports that small business net job creation fluctuated between 60 and 80 percent.c a   For an account of the growing importance of the small firm in employment and innovation, see Zoltan J. Acs and David B. Audretsch, Innovation and Small Business, Cambridge, Massachusetts: MIT Press, 1991, p. 4. For specifics on job growth, see Steven J. Davis, John Haltiwanger, and Scott Schuh, “Small Business and Job Creation: Dissecting the Myth and Reassessing the Facts,” Business Economics 29(3):113-122. See OECD, Small Business Job Creation and Growth: Facts, Obstacles, and Best Practices, Paris, 1997. b   See the presentation by Congressman Roscoe G. Bartlett in the Proceedings section of this volume. c   See Small Business Administration, “Small Business by the Numbers,” SBA Office of Advocacy, May 2003. See also “David Birch,” Fortune Small Business, December 1, 2002. See U.S. Bureau of the Census; Administrative Office of the U.S. Courts; Endogenous Growth and Entrepreneurial Activity in Cities by Zoltan Acs and Catherine Armington, Center for Economic Studies, U.S. Bureau of the Census, Working Paper #CES-WP-03-2, January 2003. The SBIR Development Act of 1982 The new SBIR program initially required agencies with R&D budgets in excess of $100 million to set aside 0.2 percent of their funds for SBIR. This amount totaled $45 million in 1983, the program’s first year of operation. Over the next six years, the set-aside grew to 1.25 percent.10 10   The set-aside is currently 2.5 percent of an agency’s extramural R&D budget.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium The legislation authorizing SBIR had two broad goals: “To more effectively meet R&D needs brought on by the utilization of small innovative firms (which have been consistently shown to be the most prolific sources of new technologies) and To attract private capital to commercialize the results of federal research.” SBIR’s Structure and Role As conceived in the 1982 Act, SBIR’s grant-making process is structured in three phases: Phase I is essentially a feasibility study in which award winners undertake a limited amount of research aimed at establishing an idea’s scientific and commercial promise. Today, the legislation anticipates Phase I grants as high as $100,000.11 Phase II grants are larger—normally up to $750,000—and fund more extensive R&D to further develop the scientific and technical merit and the feasibility of research ideas. Phase III. This phase normally does not involve SBIR funds, but is the stage at which grant recipients should be obtaining additional funds either from a procurement program at the agency that made the award, from private investors, or from the capital markets. The objective of this phase is to move the technology to the prototype stage and into the marketplace. Phase III of the program is often fraught with difficulty for new firms. In practice, agencies have developed different approaches to facilitating this transition to commercial viability; not least among them are additional SBIR awards.12 Some firms with more experience with the program have become skilled in obtaining additional awards. Previous NRC research showed that different firms have quite different objectives in applying to the program. Some seek to demonstrate the potential of promising research. Others seek to fulfill agency research requirements on a cost-effective basis. Still others seek a certification of quality (and the 11   With the accord of the Small Business Administration, which plays an oversight role for the program, this amount can be higher in certain circumstances; e.g., drug development at NIH, and is often lower with smaller SBIR programs, e.g., EPA or the Department of Agriculture. 12   NSF, for example, has what is called a Phase II-B program that allocates additional funding to help potentially promising technology develop further and attract private matching funds. As with venture-funded firms, Phase III is likely to include some mix of economically viable and non-viable products, ultimately to be determined by the relevant agency mission requirements or private markets.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium investments that can come from such recognition) as they push science-based products towards commercialization.13 Features that make SBIR grants attractive from the firm’s perspective include the fact that there is no dilution of ownership or repayment required. Importantly, grant recipients retain rights to intellectual property developed using the SBIR award, with no royalties owed to the government, though the government retains royalty free use for a period. Selection to receive SBIR grants also confer a certification effect—a signal to private investors of the technical and commercial promise of the technology.14 From the perspective of the government, the SBIR program helps achieve agency missions as well as encourage knowledge-based economic growth.15 By providing a bridge between small companies and the federal agencies, especially for procurement, SBIR serves as a catalyst for the development of new ideas and new technologies to meet federal missions in health, transport, the environment, and defense.16 It also provides a bridge between universities and the marketplace, thereby encouraging local and regional growth.17 Finally, by addressing gaps in early-stage funding for promising technologies, the program helps the nation capitalize on its substantial investments in research and development.18 While SBIR operations and accomplishments are sometimes discussed in general terms, the actual implementation of the program is carried out in agencies with quite distinct missions and interests. There is, therefore, significant variation in objectives and mechanisms. Box B: SBIR—Addressing Small Business Concerns In addition to its legislative goals, SBIR can also help address three concerns of small businesses, as highlighted by Kenneth Flamm in his conference remarks: First, it addresses imperfections in the capital markets. For structural and institutional reasons, small businesses may have difficulty accessing the capital markets, and this could be a handicap in commercializing their innova 13   See Reid Cramer, “Patterns of Firm Participation in the Small Business Innovation Research Program in Southwestern and Mountain States,” in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, C. Wessner, ed., Washington, D.C.: National Academy Press, 2000. 14   This certification effect was initially identified by Josh Lerner, “Public Venture Capital,” in National Research Council, The Small Business Innovation Program: Challenges and Opportunities, C. Wessner, ed. Washington, D.C.: National Academy Press, 1999. 15   See, for example, the presentation of Robert Norwood of NASA, summarized in the Proceedings. 16   See the presentation of Kenneth Flamm, summarized in the Proceedings. 17   See the presentation of Christina Gabriel, summarized in the Proceedings. 18   See the presentation by Joseph Bordogna, summarized in the Proceedings.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium tions. Dr. Flamm cited a recent National Academies study to point out that the SBIR program could play some role in rebalancing specific kinds of capital market imperfections in some documented instances.a Second, SBIR addresses “contracting overhead” asymmetries. Small companies must learn to deal with a complex and sometimes “arcane” contracting system characterized by many rules and procedures when dealing with agencies such as the Department of Defense. SBIR can assist small firms who lack the financial resources to invest in “contracting overhead.” The program effectively defrays some of the fixed costs of accessing the government procurement system.b Third, SBIR helps overcome small firm disadvantages in access to the procurement process.. The major defense contractors, notably, have active Washington operations that are skilled at lobbying on Capitol Hill and at the Pentagon. An outsider, without the overhead to maintain a well-staffed Washington office and cultivate appropriate connections at the Pentagon on a regular basis, is likely to enter the contracting system at a disadvantage. Small firms cannot afford the expense of traditional lobbying; the SBIR affords a route for small firms to enter the procurement system directly. a   See the conference presentation by Professor Flamm, summarized in the Proceedings section. See also Robert Archibald and David Finifter, “Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach,” in National Research Council, SBIR: An Assessment of the Department of Defense Fast Track Initiative, op. cit., pp. 211-250. b   Charles Kolb of Aerodyne Research makes the related point that SBIR helps small businesses penetrate large agencies: “If I want my technology to go to NASA, I have to have a champion inside NASA, who’s going to push it.” This official can go to an internal NASA meeting and argue that this technology is a better way to do a part of NASA’s mission, he noted. See the Proceedings section of this volume for a summary of Charles Kolb’s conference presentation. SBIR Reauthorizations The SBIR program approached reauthorization in 1992 amidst continued worries about the U.S. economy’s capacity to commercialize inventions. Finding that “U.S. technological performance is challenged less in the creation of new technologies than in their commercialization and adoption,” the National Academy of Sciences at the time recommended an increase in SBIR funding as a means to improve the economy’s ability to adopt and commercialize new technologies.19 Following this report, the Small Business Research and Development Enhancement Act (P.L. 102-564), which reauthorized the program until 19   See National Research Council, The Government Role in Civilian Technology: Building a New Alliance, Washington, D.C.: National Academy Press, 1992, p. 29.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium September 30, 2000, doubled the set-aside rate to 2.5 percent.20 This increase in the percentage of R&D funds allocated to the program was accompanied by a stronger emphasis on encouraging the commercialization of SBIR-funded technologies.21 Legislative language explicitly highlighted commercial potential as a criterion for awarding SBIR grants. For Phase I awards, Congress directed program administrators to assess whether projects have “commercial potential” in addition to scientific and technical merit when evaluating SBIR applications. With respect to Phase II, evaluation of a project’s commercial potential was to consider, additionally, the existence of second-phase funding commitments from the private sector or other non-SBIR sources. Evidence of third-phase follow-on commitments, along with other indicators of commercial potential, was also sought. Moreover, the 1992 reauthorization directed that a small business’ record of commercialization be taken into account when considering the Phase II application.22 The Small Business Reauthorization Act of 2000 (P.L. 106-554) again extended SBIR until September 30, 2008. It also called for an assessment by the National Research Council of the broader impacts of the program, including those on employment, health, national security, and national competitiveness.23 The NRC Assessment The NRC assessment represents a significant opportunity to gain a better understanding one of the largest of the nation’s early-stage finance programs. Despite its size and 20-year history, the SBIR program has not been comprehensively examined. There have been some previous studies focusing on specific 20   For fiscal year 2003, this has resulted in a program budget of approximately $1.6 billion across all federal agencies, with the Department of Defense having the largest SBIR program at $834 million, followed by the National Institutes of Health (NIH) at $525 million. The DoD SBIR program, is made up of 10 participating components: (see Figure 1 in the Proceedings): Army, Navy, Air Force, Missile Defense Agency (MDA), Defense Advanced Research Projects Agency (DARPA), Chemical Biological Defense (CBD), Special Operations Command (SOCOM), Defense Threat Reduction Agency (DTRA), National Imagery and Mapping Agency (NIMA), and the Office of Secretary of Defense (OSD). NIH counts 23 institutes and agencies making SBIR awards. 21   See Robert Archibald and David Finifter, “Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach,” op. cit. pp. 211-250. 22   A GAO report had found that agencies had not adopted a uniform method for weighing commercial potential in SBIR applications. See U.S. General Accounting Office, 1999, Federal Research: Evaluations of Small Business Innovation Research Can Be Strengthened, AO/RCED-99-114, Washington, D.C.: U.S. General Accounting Office. 23   The current assessment is congruent with the Government Performance and Results Act (GPRA) of 1993: http://govinfo.library.unt.edu/npr/library/misc/s20.html. As characterized by the GAO, GPRA seeks to shift the focus of government decision making and accountability away from a preoccupation with the activities that are undertaken—such as grants dispensed or inspections made—to a focus on the results of those activities. See http://www.gao.gov/new.items/gpra/gpra.htm.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium aspects or components of the program—notably by the General Accounting Office and the Small Business Administration.24 There are, as well, a limited number of internal assessments of agency programs.25 The academic literature on SBIR is also limited.26 Writing in the 1990s, Joshua Lerner positively assessed the program, finding “that SBIR awardees grew significantly faster than a matched set of firms over a 10-year period.” 27 Underscoring the importance of local infrastructure and cluster activity, Lerner’s work also showed that the “positive effects of SBIR awards were confined to firms based in zip codes with substantial venture capital activity.” These findings were consistent with both the corporate finance literature on capital constraints and the growth literature on the importance of localization effects.28 To help fill this assessment gap, and to learn about a large, relatively under-evaluated program, the National Academies’ Committee for Government-Industry Partnerships for the Development of New Technologies was asked to review the SBIR program, its operation, and current challenges. Under its chairman, Gordon Moore, the Committee convened government policy makers, academic researchers, and representatives of small business for the first comprehensive discussion of the SBIR program’s history and rationale, review existing research, and identify areas for further research and program improvements.29 The Committee chaired by Moore reported that: SBIR enjoyed strong support in parts of the federal government as well as in the country at large. At the same time, the size and significance of SBIR underscored the need for more research on how well it is working and how its operations might be optimized. 24   An important step in the evaluation of SBIR will be to identify existing evaluations of SBIR. See for example, GAO, Federal Research: Small Business Innovation Research Shows Success But Can Be Strengthened. Washington, D.C.: U.S. General Accounting Office, 1992; and GAO, Evaluation of Small Business Innovation Can Be Strengthened, Washington, D.C.: U.S. General Accounting Office, 1999. There is also a 1999 unpublished SBA study on the commercialization of SBIR surveys Phase II awards from 1983 to 1993 among non-DoD agencies. 25   Agency reports include an unpublished 1997 DoD study on the commercialization of DoD SBIR. NASA has also completed several reports on its SBIR program. Following the authorizing legislation for the NRC study, NIH launched a major review of the achievements of its SBIR program. See Appendix C for a list of agency reports. 26   See the bibliography in Appendix C. 27   See Joshua Lerner, “The Government as Venture Capitalist: The Long-Term Effects of the SBIR Program,” Journal of Business 72(2) July 1999. 28   See Michael Porter, “Clusters and Competition: New Agendas for Government and Institutions,” in On Competition, Boston: Harvard Business School Press, 1998. 29   See National Research Council, The Small Business Innovation Research Program: Challenges and Opportunities, op. cit.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium award winners, or SBIR “mills,” receive many awards yet generate little in product commercialization.68 Some multiple award winners are largely contract research organizations, meeting the research requirements of agencies, yet not focusing on the commercial goals of the program. Given the diversity of the SBIR program’s goals, the main question is simply whether the quality and value of this research is in alignment with agency needs and priorities—essentially a question of management rather than program concept. The actual operation of the program in this respect and its relevance to agency missions needs to be clarified—one of the goals of the current study. Some have also suggested that the failure rate of SBIR awards is too high, which suggests, in turn, that the program funds R&D of marginal value. This is a challenging point. Measuring the impact and results of an R&D program is intrinsically difficult.69 What constitutes an acceptable failure rate for a program designed to make high-risk, potentially high payoff investments is, of course, a central question—one that is especially difficult for those with a fiduciary responsibility for public funds. High-risk R&D investments are, indeed, high-risk—project failures in such initiatives are inevitable and not necessarily indicative of program failure. Still, the question of what an appropriate return on investment in new technologies remains. One benchmark may be the venture capital market, where only about 10 percent of investments in new firms succeed. A key question in assessing SBIR is whether this comparison is appropriate.70 Another recurrent question is whether a project or firm failure is indicative of a complete loss on federal investment—as it sometimes is—or if the loss is mitigated by knowledge generated by the SBIR grant that is then transmitted through less direct ways to the overall benefit of society. This second scenario takes into account potential indirect knowledge spillovers that were not a part of the original research design or intent. Consider, for example, the case of a principal investigator who takes the knowledge gained from work at a “failed” firm, and uses it at a new firm to guide product development in an entirely new market.71 68   Kenneth Flamm refers to this issue in his presentation. See a summary of his remarks in the Proceedings section of this volume. The Department of Defense Fast Track Initiative addresses this issue. It attracts significantly more new, first-time applicants to the program. See National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, op. cit. 69   See National Research Council, Capitalizing on the Results of Scientific Research, op. cit., 1999. 70   Despite the growing popularity of the idea of “public venture capital” programs, SBIR cannot be considered a venture capital program because awards do not involve equity ownership, management input, or an exit strategy, involving sale of the firm. For a description of a public venture initiative, see the presentation of the CIA’s In-Q-Tel by Gilman G. Louie, “In-Q-Tel A “Nonprofit Venture Capital Fund,” in National Research Council, A Review of the New Initiatives at the NASA Ames Research Center, C. Wessner, ed., Washington, D.C.: National Academy Press, 2001. 71   Relatedly, see the discussion of David Audretsch, Duncan Moore, and Paula Stephan on the challenges of “Gauging Commercial Success,” summarized in the Proceedings and highlighted in this Introduction.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium An additional concern is that SBIR awards might “crowd out” or replace private capital. While theoretically possible, recent work by Bronwyn Hall, Paul David, and Andrew Toole suggests that the overall empirical evidence for “crowding out” is at least equivocal.72 Interestingly, there is some positive evidence that programs like SBIR can prompt “crowding in” of private capital. Awards have a “halo effect” that attracts private investors, who see the awards as a certification of technical quality, reducing the uncertainty inherent in early-stage investment.73 Finally, some object to the SBIR program more broadly as an unwarranted and unnecessary intervention in capital markets.74 Yet, as noted above, it is widely recognized that capital markets are imperfect with significant gaps (or asymmetries) in information between the potential investor and the prospective entrepreneur.75 Venture capital markets, in particular, tend to focus on later stages of technology development than SBIR—though venture-based companies can and do obtain SBIR awards—and venture funds in the aggregate seem to be prone to herding tendencies. In particular, the attention of private investors does not necessarily extend to all areas of socially valuable innovation.76 Perhaps the most significant point to retain from these various perspectives on SBIR is how much uncertainty surrounds early-stage finance in the U.S. economy. As noted, some recent work suggests that the federal role in early-stage firm development is more significant than commonly believed, while also affirming the analytical uncertainty surrounding the funding and development of early-stage firms. Strong affirmations about the “appropriate” role of government support for innovation are not borne out by the history of innovation and industrial development in the United States or, indeed, recent experience.77 Indeed, while the appropriateness of the government’s role in fostering new industry has been debated since the origins of the republic, American policy has tended to be fairly pragmatic in practice, meeting national needs from the tele- 72   See Paul A. David, Bronwyn H. Hall and Andrew A. Toole. “Is Public R&D a Complement or Substitute for Private R&D? A Review of the Econometric Evidence,” No 7373, NBER Working Papers, 1999. 73   See Maryann P. Feldman and Maryellen R. Kelley, “Leveraging Research and Development: The impact of the Advanced Technology Program,” in National Research Council, The Advanced Technology Program: Assessing Outcomes, C. Wessner, ed., Washington, D.C.: National Academy Press, 2001. 74   See, for example, Scott Wallsten, “Rethinking the Small Business Innovation Research Program,” in Branscomb and Keller, eds., Investing in Innovation. Cambridge, MA: The MIT Press, 1998, pp. 194-220. 75   See Michael Spence, Market Signaling: Informational Transfer in Hiring and Related Processes, op. cit. 1974. 76   See case studies in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense’s Fast Track Initiative, op. cit. 77   See the discussion of this question in the Introduction to the review of the Advanced Technology Program in National Research Council, The Advanced Technology Program: Assessing Outcomes, op. cit., and National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, op. cit.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium graph to railroads to radio to the Internet.78 Likewise, debates about public-private partnerships, especially the type that emerged in the 1980s and 1990s, have sometimes been more ideological than analytical in nature. Yet, as Berglund and Coburn have noted, “the [policy] debate should address not whether these [partnership] programs will endure, but whether they are shaped properly—at the program and aggregate levels—to achieve the desired benefits.”79 This is essentially the approach taken in the legislation that launched this review of SBIR.80 The task of the Committee is, thus, not to determine if the nation should have an SBIR program; that decision has repeatedly been taken by the Congress. Rather, the Committee’s charge is to assess its operations and accomplishments, and to consider how it might be improved. KEY ISSUES FROM THE CONFERENCE To address this task, the Committee convened a conference on October 24, 2002 to gather new information and agency perspectives on the operation and potential of the SBIR program. Participants at this conference discussed the diverse goals of SBIR awards and highlighted key issues to be addressed and the challenges faced by the NRC’s assessment of the SBIR program. The presentations by senior officials from the participating agencies drew attention to the administrative flexibility and operational diversity that characterizes this innovation program. Administrative Flexibility Does the statutory language establishing SBIR provide the government agencies implementing the program the flexibility needed to play an effective role within a complex innovation system? In reviewing the meaning of the original two-page statute that Congress issued in 1982, James Turner of the House Science Committee noted in his conference comments that many of SBIR’s features—particularly the three-phase structure—were based on an assumption of a linear model of research and development.81 In the linear model, innovation begins with basic research supplying a steady stream of fresh and new ideas. Among these ideas, those that promise technical and economic feasibility become innovations. Such innova- 78   See LeBow, op. cit. For a more general discussion, see David M. Hart, Forged Consensus: Science, Technology and Economic Policy in the United States, 1921-1953, Princeton: Princeton University Press, 1998. 79   See Dan Berglund and Christopher Coburn, Partnerships, A Compendium of State and Federal Cooperative Technology Programs, Columbus, OH: Battelle Press, 1995. 80   SBIR has been reauthorized until the end of fiscal year 2007. The SBIR legislation, contained within H.R. 5667, can be viewed on-line at: http://thomas.loc.gov/home/omni2000/omni2000.html. 81   This view was echoed by Duncan Moore: “Innovation does not follow a linear model. It stops and starts.” See the summary of comments by Dr. Moore in the Proceedings section of this volume.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium FIGURE 3 The linear model of innovation. tions, when further developed by firms can become marketable products driving growth. The schematic in Figure 3 depicts this linear model of innovation.82 While it undoubtedly has conceptual value, it substantially simplifies reality. Indeed, as the National Science Foundation’s Joseph Bordogna observed, innovation almost never takes place through a protracted linear progression from research to development to market. Research and development can drive technological innovation, but it can also happen in the reverse direction. True innovation, he noted, can spur the search for new knowledge and create the context in which the next generation of research identifies new frontiers. Taking this point, Mr. Turner remarked that we now understand that knowledge moves in more complex and circular fashion, with numerous feedback loops between the marketplace and the laboratory.83 As Figure 4 illustrates, the innovation process is neither linear nor unidirectional.84 Applied research can be a source FIGURE 4 A non-linear model of innovation. 82   For a discussion of the limits of this model and its potential impact on policy, see Donald E. Stokes, Pasteur’s Quadrant, Basic Science and Technological Innovation, Washington, D.C.: Brookings Institution Press, 1997, p. 10. 83   See George E. Brown Jr. and James Turner, op. cit. 84   The schematic in Figure 4 was developed by Adam K. Korobow. The model captures added features, but not all of the complexities, of the innovation process.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium of knowledge as well as technological advance. The progression of a SBIR project from Phase I to II to III is often marked with discoveries or questions that may stimulate new or additional research, with new research sometimes suggesting new applications. It is also littered with conceptual and practical dead-ends. The nature of the research process and the unique missions of diverse agencies underscore the advantages of a program that can be flexibly implemented. Christina Gabriel of Carnegie Mellon University noted that when she worked at Bell Laboratories, she and her colleagues implicitly believed in the linear model. “We didn’t think it was important for the researchers to talk to the rest of the company,” she said. “My personal opinion is that that’s why we don’t have Bell Labs and the original research labs any more. Some of the large companies never understood how to connect their research to the rest of the company in a productive way. So small businesses and Japanese companies and others outside the labs were the ones who commercialized almost all the innovations that came out of our labs.” However, she thought it was remarkable that the SBIR, even though it was founded in an earlier era, continued to serve the nation in an effective way—not least because it has been able to maintain great flexibility. As Mr. Turner concluded, the challenge for the NRC Committee would be to recommend ways to amend some of the “statutory impediments” that reflected the earlier, linear model of innovation, that we now know impacts overall program effectiveness. Such impediments, he noted, include narrow time-frames, fixed amounts of money, and a “one-size-fits-all” approach to the program. How can we amend the rules, he asked, to nurture the best ideas and make award winners into market winners?85 Although formally structured on a common three-phase template, SBIR operations have come to reflect in many ways the non-linearity that characterizes real research and development as well as the diversity of missions and cultures of the federal agencies that administer the program. Agencies administering SBIR are coping with these exigencies by decentralizing important aspects of SBIR’s administration. As noted at the conference, the need to deal with operational complexity is particularly acute in larger and more diverse agencies. To illustrate the need for flexibility in an environment characterized by diversity, Jo Anne Goodnight of the National Institutes of Health pointed out in her conference presentation that time and funding requirements to develop new products can vary significantly among firms in different technology sectors. While NIH-supported projects to develop pharmaceuticals require an average of 12 years and hundreds of millions of dollars to complete, some Department of Defense 85   In fact, anecdotal evidence suggests that the program is evolving—at least incrementally. To meet this need for greater flexibility, agencies can and do obtain permission from the Small Business Administration to modify standard practice to meet particular research requirements, e.g., larger awards for drug development.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium supported military products (e.g., software) are typically developed for more immediate use.86 Similarly, the development time-line for NSF-supported information technology products is relatively short, while time taken to plant and grow crops under a USDA-supported SBIR can be considerably longer. The nature of the product and the pace of market competition (such as in the information technology sector) can, in addition, leave some firms more vulnerable to lags in funding between SBIR Phase I and Phase II.87 These realities preclude a “one-size-fits-all” approach to administering the SBIR program. While SBIR’s overall three-phase approach is designed to implement national policy goals across federal agencies and sub-agencies, the program must be implemented taking into account the multiple agency goals and operational requirements to meet their diverse objectives. Capturing this point, David Finifter of the College of William and Mary observed that “one must understand programs at the agency level in order to understand the policy effects of the program at the national level.” Operational Diversity SBIR’s operational diversity reflects its adaptation to the non-linear realities of science, technology, and agency practice. Charles Holland noted that officials with the Army, Navy, Air Force, Missile Defense Agency, DARPA, and other agencies that participate in SBIR at the Department of Defense independently develop topics that address their own agency’s strategic efforts to develop the nation’s war-fighting capabilities. At the same time, he noted that this decentralization was coordinated through his office to ensure that they are in line with the department’s overall Defense Technology Area Plan. Similarly, Jo Anne Goodnight noted that while the operation of NIH’s SBIR program was coordinated through her office, each of the 23 individual institutes and centers that administer SBIR at NIH develop topics that relate to own specialized missions. These missions, she said, range from promoting public health to the investigation of particular diseases such as cancer to addressing broader areas of concern such as aging. Describing the case for NASA, Robert Norwood noted that while SBIR is closely tied to the agency’s overall missions in space science, earth science, and aerospace technology (among others), officials at each of NASA’s 10 major centers are responsible for writing SBIR proposal solicitations. This diversity, by more accurately reflecting the interests of NASA’s component enterprises, increased the likelihood that SBIR-supported technologies would realize commercial success. 86   NIH also supports a broad range of other applications from diagnostic software, to audiovisual material, to biosensors with shorter lead times. 87   See comments by Linda Powers in the summary of Proceedings in this volume.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium Recognizing this need for operational flexibility, Ms. Goodnight said, the Small Business Administration—which oversees the administration of the SBIR program—allows for different ways of supporting agency missions. Rather than limiting ideas to those that can be conducted under prescribed budgets and timelines, this latitude allows companies to propose research and development in fields that have the most commercial promise. In the context of the current assessment, the presence of such operational diversity among agencies in program implementation provides opportunities for identifying best practices—some of which might be replicated in other contexts.88 At the same time, recognition of the diversity of the program’s goals and administration cautions against a “one-size-fits-all” approach to a complex program that is focused on a critical phase of the innovation process. Measurement and Assessment Challenges The program diversity in goals and practice must be taken into account as the program evaluation is developed. How can the impact of innovation awards to small businesses be measured? What metrics are most useful in this regard? And is there a need to look beyond these indicators to achieve a balanced understanding of the limits and contributions of SBIR? In keeping with the conference theme, “Measuring Outcomes,” conference participants considered the possibilities and limitations of available measurement tools. In addition, they assessed to the broader dimensions of the program’s impact. Tracking Awards One way to measure the value of SBIR is to track awards to see if they have resulted (variously) in new publications, citations, patents, products, licensing, sales, and increased employment for firms receiving awards. While data relating to these metrics promise to give information of some significance on SBIR’s impact, conference participants noted that accurate data might be difficult to procure for a variety of reasons. For instance, program realities can complicate the task of tracking data. As NIH’s Jo Anne Goodnight pointed out, SBIR recipients can get, in some cases, a Phase I award from one agency and a Phase II award from another. She noted that additional assessment challenges could also arise in cases where multiple SBIR awards are provided to bring a particular product to market, or when the SBIR award is one among other federal and non-federal sources of funding. 88   The Committee subsequently held a public conference on May 28, 2003 at the National Academies, entitled, “Identifying Best Practice.” Given that the variety in implementation across agencies affords an opportunity to identify best practices, conference participants had an opportunity to consider which aspects can be successfully replicated by other agencies.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium The issue of causality becomes more complex. Awards are given to firms for projects; yet, specific projects funded by SBIR can be difficult to track. MaryAnn Feldman of Johns Hopkins University observed that firms that are granted SBIR awards can merge, fail, and change their name before a product reaches the market. In addition, key individuals can change firms, carrying their knowledge of the project with them. In this way, an SBIR investment in one firm may translate into a valuable product from another firm. Especially when the process from discovery to market is long, as is the case for drug development, these transitions are difficult to track. Measuring Indirect Impacts The indirect effects of SBIR awards are also difficult to measure and assess. Rosalie Ruegg of TIA Consulting suggested, however, that evaluation techniques, such as those developed for the assessment of NIST’s Advanced Technology Program, can help measure these impacts: She cited newly developed econometric techniques could help gauge the impact of halo effects. (The halo effect refers to the ability of a firm that has received a federal grant to attract new private or public funding by signaling the technical and commercial potential of its product.) She made note that recent models based on “fuzzy logic” that appear to improve assessments of knowledge spillovers. Finally, she referred to a new index method that promises to provide better estimates of market spillovers. While Ms. Ruegg conceded that “we cannot hope to do a comprehensive measure of all indirect effects,” she expressed optimism that new assessment techniques will increasingly provide the means to capture snapshots and indicators of the full range of SBIR’s impacts. Gauging Commercial Success Even the apparently straightforward task of assessing commercial success can be elusive. For example, David Audretsch of Indiana University pointed out that research enabled by a particular SBIR award may take on commercial relevance in new unanticipated contexts. Illustrating this point, Duncan Moore of the University of Rochester noted how his SBIR-funded research in gradient index optics was initially a commercial failure when an anticipated market for its application (35mm SLR camera lenses) did not emerge. However, the technology later found substantial commercial success in the boroscope, a device used to look inside materials and structures. As Paula Stephan of Georgia State University concluded, “today’s dead end often can be a key to tomorrow’s success, or what is seen as a failure today can be seen as a success tomorrow.” In the case of public procurement, the challenge is one of developing a satisfactory measure of how useful an SBIR-funded innovation has been to an agency mission. Relatedly, it is important to assess just how compatible success in meeting an agency mission has or has not been with commercial success for an SBIR

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium company. Success in such cases varies depending on the nature of the product, the type of research, and its utility for the agency mission. In some cases, the appropriate metric is likely to vary with the specific mission of the agency or subunit. For the Department of Defense, Charles Holland noted that one way of measuring commercialization success would be to count the products procured by the agency—although, as he acknowledged, large procurements from major suppliers are more easily tracked than products from small suppliers such as SBIR firms. Gauging commercial success accurately is made more challenging since successful development of a technology or product does not always translate into successful “uptake” by the procuring agency, often for reasons having little to do with product quality or potential contribution. Indeed, the conference highlighted varying approaches to the concept of commercialization. While the concept of “commercial” at the Department of Defense most often relates to the use of a new product or process by government, the concept more conventionally refers to the means by which a new product or process—provided by a viable business enterprise—enters the market on an independent, third party, competitive basis. These differing interpretations also reveal the differing pathways to commercialization. For some products, this path is akin to a long, complex, winding, and uncertain road. For others, the pathway is more immediate with visible linkages to mission, industrial and commercial applications. Program Effectiveness The value of the SBIR also derives from how well it serves firms seeking to cross the Valley of Death. Efficient evaluation of proposals and the prompt disbursement of funds can be important to small businesses in this phase of development. Jon Baron of the Coalition for Evidence Based Policy noted that SBIR’s reputation for accurately gauging the technical quality and commercial potential of a proposal is key to an award’s usefulness as a signal to capital markets. Linda Powers of Toucan Capital observed that long time lags in program award cycles could be fatal for small firms facing capital shortages: “The reason why so many companies die trying to cross the Valley of Death,” she noted, “is that it is so wide and takes companies so long to cross it.” She noted that firms can ill-afford to spend long periods waiting for grant cycles and other agency procedures. A Phase II SBIR award of $750 thousand is expected to last for two years, but such grants, she said, are too small to support a company during delays, adding that, in some cases, “even in an early stage, companies are burning that much per month.” Milton Johnson of the Department of Energy noted that despite resource constraints, his agency has been able to provide on-time service with modest time lags between Phase I and Phase II funding. Robert Norwood of NASA reported that his agency had recently implemented an Electronic Handbook System that integrated all

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium elements of SBIR administration. The system is designed to speed the processing of grant applications, improve transparency, and generate data for future evaluation. Program effectiveness is also affected by how well state and local governments leverage SBIR to foster local development. Noting that some states were better at using the SBIR program than others, John Williams of the U.S. Navy wondered how state outreach programs run by SBIR agency managers could best be evaluated. Realistic Expectations An assessment of SBIR must be based on an understanding of the realities of the distribution of successes and failures in early-stage finance. Jon Baron noted that the program is characterized by a highly skewed distribution of successes. This includes a few genuinely large successes that generate returns that would cover, in themselves, the cost of the entire program. He cited two examples of such successes—Science Research Laboratory Inc., which had reported over a billion dollars in sales from a new technology that increased the number of circuits on a computer chip by thirty percent, and Digital System Resources, whose technology had improved the computing power of sonar technology by twenty percent leading to its adoption across the U.S. submarine fleet. Below these star performers were a number of more modest successes, followed by a large number of awards that had produced few or no results, he said. Given this skew, he noted that a purely random sample of individual project outcomes is likely to yield an imbalanced assessment of the SBIR program. A related issue concerns the appropriate expectation for success or, indeed, failure. Providing a comparison from the private sector, Gail Cassell of Eli Lilly noted that the failure rate for biotechnology industry—from target identification to product launch—was about ninety percent. This held true even in the best of circumstances and even for large companies that had invested billions of dollars in research and development. Thus, in setting metrics for SBIR projects, she said, it is important to give a realistic expectation of success, especially for small firms investing in biotechnology. In his remarks, David Finifter of the College of William and Mary expanded on this point, noting that measures of success vary greatly by area of discipline and by federal policy objective. Professor Finifter also noted that while a high success rate was gratifying, it could imply that the SBIR program does not have a sufficiently risky portfolio: The question of “how high is high” is therefore important to the program and to the study, he concluded. Similarly, Greg Millman of the National Institutes of Health noted that success alone would be a dangerous metric for assessing SBIR, arguing that if there were no risk taking, there would be no need for the SBIR program.

OCR for page 3
SBIR Program Diversity and Assessment Challenges: Report of a Symposium ADDRESSING TOMORROW’S CHALLENGES While it is important to know whether firms receiving SBIR awards are successful or not, David Audretsch reminded the conference audience that the larger purpose of the SBIR program is to improve the efficiency of the American economy “to make it more innovative than it would be otherwise.” The creation of innovation firms, as a result of SBIR awards, is an important contribution of the program. Thus, a balanced assessment of SBIR has to go beyond measuring the operational impact of individual awards. It is also to consider the broader institutional role that SBIR plays in the United States economy. Given the need for administrative flexibility and the considerable operational diversity of SBIR, highlighted in the conference, an important question to ask is how agency-specific implementation practices support or detract from these broader program goals.89 Carnegie Mellon University’s Christina Gabriel noted in this regard that SBIR plays an important role in bringing the contribution of university research to market. Pointing to the role that technology transfer from Carnegie Mellon has played in the revitalization of Pittsburgh’s economy, she noted that job gains and regional economic growth could be realized by exploring ways by which SBIR can be better linked to the national innovation system. Further to this point, Joseph Bordogna of the National Science Foundation observed that in addition to investing in the nation’s scientists and engineers, SBIR also serves as a key institutional facilitator in the integrative research that increasingly characterizes innovation-led growth. This distinctive role for SBIR, he argued, is best understood as one element in the large-scale transformations taking place in the nation’s research and innovation enterprise. As the nation’s innovation system continues to evolve towards greater collaboration and multi-disciplinary research, public-private partnerships such as SBIR may play an increasingly important role in bringing together the expertise from business, academia, and government, as well as from across disciplinary boundaries. As the innovation system changes, the ability of firms to traverse the Valley of Death also grows increasingly important, highlighting the role that SBIR can play in facilitating the transition of new ideas to commercial application. In these respects, as Dr. Bordogna concluded, an SBIR assessment that takes the long view could well serve as a “revolutionary chart of the new paths we will follow in the twenty-first century.” The conference launching the NRC assessment of SBIR is a first step in charting this ambitious path. 89   Understanding how the program, in its various agency contexts, supports the overall goals of the program is a key objective of the NRC study. The NRC’s evaluations of SBIR at individual agencies are expected to contribute to an overall assessment of the program’s effectiveness.