4
SBIR Program Outputs

4.1
INTRODUCTION

Congress has tasked the National Academies to assess whether and to what extent the Small Business Innovation Research (SBIR) program has met the congressionally-mandated objectives for the program, and to suggest possible areas for improvement in program operations. Congress has, over the years, identified a number of objectives for the program, and these mandated objectives can be summarized as follows:

  • Supporting the commercialization of federally funded research.

  • Supporting the agency’s mission.1

  • Supporting small business and, in particular, woman- and minority-owned businesses.

  • Expanding the knowledge base.

Congress has not prioritized among the four objectives, although report language and discussions with congressional staff suggest that commercialization has become increasingly important to Congress. Still, it remains important to assess each of the four objectives, and each should therefore be taken as equally important in evaluating the achievements and challenges of the SBIR program. These four objectives help to define the structure and content of this chapter.

1

Each of the SBIR-funding agencies has a different mission, described in the agency volumes of this study. For a review of the different ways the program is operated at the five agencies under review, see National Research Council, SBIR: Program Diversity and Assessment Challenges, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2004.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 108
4 SBIR Program Outputs 4.1 INTRODUCTION Congress has tasked the National Academies to assess whether and to what extent the Small Business Innovation Research (SBIR) program has met the congressionally-mandated objectives for the program, and to suggest possible areas for improvement in program operations. Congress has, over the years, identified a number of objectives for the program, and these mandated objectives can be summarized as follows: • Supporting the commercialization of federally funded research. • Supporting the agency’s mission.1 • Supporting small business and, in particular, woman- and minority-owned businesses. • Expanding the knowledge base. Congress has not prioritized among the four objectives, although report lan- guage and discussions with congressional staff suggest that commercialization has become increasingly important to Congress. Still, it remains important to assess each of the four objectives, and each should therefore be taken as equally important in evaluating the achievements and challenges of the SBIR program. These four objectives help to define the structure and content of this chapter. 1 Each of the SBIR-funding agencies has a different mission, described in the agency volumes of this study. For a review of the different ways the program is operated at the five agencies under review, see National Research Council, SBIR: Program Diersity and Assessment Challenges, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2004. 0

OCR for page 108
09 SBIR PROGRAM OUTPUTS Assessing program outcomes against these four objectives entails numer- ous methodological challenges. These challenges are discussed in detail in the National Research Council’s (NRC) Methodology Report.2 4.1.1 Compared to What? Assessment usually involves comparison—comparing programs and activi- ties, in this case. Three kinds of comparison seem possible: with other programs at each agency, between SBIR programs at the various agencies, and with early- stage technology development funding in the private sector, such as venture capital activities. Yet, the utility of each of these three types of comparison is limited. Other award programs at the agencies have fundamentally different objec- tives, such as promoting basic research through grant programs (at the National Institutes of Health [NIH] and the National Science Foundation [NSF]), develop- ing capacity (awards for university infrastructure), or training. There are often no other dedicated programs for innovative small businesses. And no other programs for small businesses have as a primary goal the commercial exploitation of re- search. This fundamental difference in objectives makes it difficult to usefully compare an SBIR program with other programs at the relevant agency. Comparisons between SBIR programs at different agencies appear superfi- cially more useful, but must be regarded with considerable caution. As discus- sions in Chapter 1 of this volume and in the separate agency volumes indicate, the widely differing agency missions have shaped the agency SBIR programs, focus- ing them on different objectives and on different mechanisms and approaches. Agencies whose mission is to develop technologies for internal agency use via procurement—notably the Department of Defense (DoD) and the National Aeronautics and Space Administration (NASA)—have a quite different orienta- tion from agencies that do not procure technology and are instead focused on developing technologies for use outside the agency. Finally, SBIR might be compared with venture capital (VC) activities, but there are important differences. VC funding is typically supplied later in the development cycle when innovations are in, or close to, market—most venture investments are made with the expectation of an exit from the company within three years. VC investments are typically larger than SBIR awards—the aver- age investment made by VC firms in a company was over $7 million in 2005, 2 National Research Council, An Assessment of the Small Business Innoation Research Program: Project Methodology, Charles W. Wessner, ed. Washington, DC: The National Academies Press, 2004, pp. 20-21. For a broader discussion of the scope and limitations of surveys by the University of Michi- gan Survey Research Center, see Robert M. Groves et al., Surey Methodology, Wiley-IEEE, 2004.

OCR for page 108
0 AN ASSESSMENT OF THE SBIR PROGRAM BOX 4-1 Multiple Sources of Bias in Survey Response Large innovation surveys involve multiple sources of bias that can skew the results in both directions. Some common survey biases are noted below.a • uccessful and more recently funded firms are more likely to respond. Re- S search by Link and Scott demonstrates that the probability of obtaining research project information by survey decreases for less recently funded projects and it increased the greater the award amount.b Nearly 40 percent of respondents in the NRC Phase II Survey began Phase I efforts after 1998, partly because the number of Phase I awards increased, starting in the mid-1990s, and partly because winners from more distant years are harder to reach. They are harder to reach as time goes on because small businesses regularly cease operations, are acquired, merge, or lose staff with knowledge of SBIR awards. • uccess is self reported. Self-reporting can be a source of bias, although the S dimensions and direction of that bias are not necessarily clear. In any case, policy analysis has a long history of relying on self-reported performance measures to represent market-based performance measures. Participants in such retrospectively analyses are believed to be able to consider a broader set of allocation options, thus making the evaluation more realistic than data based on third party observation.c In short, company founders and/or principal investigators are in many cases simply the best source of information available. • urvey sampled projects at firms with multiple awards. Projects from firms S with multiple awards were underrepresented in the sample, because they could not be expected to complete a questionnaire for each of dozens or even hundreds of awards. • ailed firms are difficult to contact. Survey experts point to an “asymmetry” in F their ability to include failed firms for follow-up surveys in cases where the firms no longer exist.d It is worth noting that one cannot necessarily infer that the SBIR project failed; what is known is only that the firm no longer exists. • ot all successful projects are captured. For similar reasons, the NRC Phase N II Survey could not include ongoing results from successful projects in firms that merged or were acquired before and/or after commercialization of the project’s technology. The survey also did not capture projects of firms that did not respond to the NRC invitation to participate in the assessment. • ome firms may not want to fully acknowledge SBIR contribution to project S success. Some firms may be unwilling to acknowledge that they received important benefits from participating in public programs for a variety of reasons. For example, some may understandably attribute success exclusively to their own efforts. • ommercialization lag. While the NRC Phase II Survey broke new ground in C data collection, the amount of sales made—and indeed the number of projects that generate sales—are inevitably undercounted in a snapshot survey taken at a single point in time. Based on successive data sets collected from NIH SBIR award recipients, it is estimated that total sales from all responding projects will likely be on the order of 50 percent greater than can be captured in a single survey.e This underscores the importance of follow-on research based on the now-established survey methodology.

OCR for page 108
 SBIR PROGRAM OUTPUTS 14 12 Sales (Millions of Dollars) 10 8 6 4 2 0 1 2 3 4 5 6 7 8 9 10 11 12 survey taken Years after Phase II Award FIGURE B-4-1 Survey bias due to commercialization lag. These sources of bias provide a context for understanding the response rates to the NRC Phase I and Phase II Surveys conducted for this study. For the NRC Phase II Survey, of the 4,523 firms that could be contacted out of a sample size of 6,408, 1,916 responded, representing a 42 percent response rate. The NRC Phase I Survey captured 10 percent of the 27,978 awards made by all five agencies over the period of 1992 to 2001. See Appendix A and B for additional information on the surveys. Figure B-A-1 aFor a technical explanation of the sample approaches and issues related to the NRC surveys, see Appendix A. bAlbert N. Link, and John T. Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Program’s Intramural Research Initiative, London: Routledge, 2005. cWhile economic theory is formulated on what is called “revealed preferences,” meaning individu- als and firms reveal how they value scarce resources by how they allocate those resources within a market framework, quite often expressed preferences are a better source of information especially from an evaluation perspective. Strict adherence to a revealed preference paradigm could lead to misguided policy conclusions because the paradigm assumes that all policy choices are known and understood at the time that an individual or firm reveals its preferences and that all relevant markets for such preferences are operational. See {1} Gregory G. Dess and Donald W. Beard, “Dimensions of Organizational Task Environments,” Administrative Science Quarterly, 29: 52-73, 1984; {2} Albert N. Link and John T. Scott, Public Accountability: Evaluating Technology-Based Institutions, Norwell, MA: Kluwer Academic Publishers, 1998. dAlbert N. Link, and John T. Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Program’s Intramural Research Initiative, op. cit. eData from NIH indicates that a subsequent survey taken two years later would reveal very substantial increases in both the percentage of firms reaching the market, and in the amount of sales per project. See National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, Charels W Wessner, ed., Washington, DC: The National Academies Press, 2009.

OCR for page 108
 AN ASSESSMENT OF THE SBIR PROGRAM compared to less than $1 million for SBIR over a two to three year cycle.3 VC investments are also focused on companies, not projects, and often come both with substantial management support and influence (e.g., through seats on the company’s board). None of this is true for SBIR. The lack of available comparators means that we must assess each program in terms of the benchmarks developed to review the program in the Methodology report described below.4 4.2 COMMERCIALIZATION 4.2.1 Challenges of Commercialization Commercialization of the technologies developed under the research sup- ported by SBIR awards has been a central objective of the SBIR program since its inception. The program’s initiation in the early 1980s, in part, reflected a concern that American investment in research was not adequately deployed to the nation’s competitive advantage. Directing a portion of federal investment in R&D to small businesses was thus seen as a new means of meeting the mission needs of federal agencies, while increasing the participation of small business and thereby the proportion of innovation that would be commercially relevant.5 Congressional and Executive branch interest in the commercialization of SBIR research has increased over the life of the program. A 1992 GAO study6 focused on commercialization in the wake of con- gressional expansion of the SBIR program in 1986.7 The 1992 reauthorization specifically “emphasize[d] the program’s goal of increasing private sector com- 3 2005 saw some 3,000 deals worth an average of $7.35 million according to data from National Venture Capital Association. See National Venture Capital Association Web site, . 4 For a discussion of this and related methodological challenges, see, National Research Council, An Assessment of the Small Business Innoation Research Program: Project Methodology, op. cit. 5A growing body of evidence, starting in the late 1970s and accelerating in the 1980s indicates that small businesses were assuming an increasingly important role in both innovation and job creation. See, for example, J. O. Flender and R. S. Morse, The Role of New Technical Enterprise in the U.S. Economy, Cambridge, MA: MIT Development Foundation, 1975, and David L. Birch, “Who Creates Jobs?” The Public Interest, 65:3-14, 1981. Evidence about the role of small businesses in the U.S. economy gained new credibility with the empirical analysis by Zoltan Acs and David Audretsch of the U.S. Small Business Innovation Data Base, which confirmed the increased importance of small firms in generating technological innovations and their growing contribution to the U.S. economy. See Zoltan Acs and David Audretsch, “Innovation in Large and Small Firms: An Empirical Analysis,” The American Economic Reiew, 78(4):678-690, September 1988. See also Zoltan Acs and David Audretsch, Innoation and Small Firms, Cambridge, MA: The MIT Press, 1990. 6 U.S. General Accounting Office, Federal Research: Small Business Innoation Research Shows Success But Can be Strengthened, GAO/RCED-92-37, Washington, DC: U.S. General Accounting Office, 1992. 7 PL 99-443, October 6, 1986.

OCR for page 108
 SBIR PROGRAM OUTPUTS mercialization of technology developed through federal research and develop- ment.”8 The 1992 reauthorization also changed the order in which the program’s objectives are described, moving commercialization to the top of the list. 9 The term “commercialization” means “reaching the market,” which some agency managers interpret as “first sale”—that is, the first sale of a product in the market place, whether to public or private sector clients. This definition, how- ever, misses significant components of commercialization that do not result in a discrete sale. It also fails to provide any guidance on how to evaluate the scale of commercialization, an important element in assessing the degree to which SBIR programs successfully encourage commercialization. The metrics for assessing commercialization can also be elusive,10 and it is important to understand that it is not possible to completely quantify all commercialization from a research project: • The multiple steps needed after the research has been concluded mean that a single, direct line between research inputs and commercial outputs rarely exists in practice; cutting edge research is only one contribution among many leading to a successful commercial product. • Markets themselves have major imperfections, or information asymme- tries so high quality, even path-breaking research, does not always result in commensurate commercial returns. • The lags involved in the timeline between an early stage research project and a commercial outcome mean that for a significant number of the more recent SBIR projects, commercialization is still in process, and sales—often substantial sales—will be made in the future. The current “total” sales are in this case just a “snapshot half way through the race,” and will require updating as the full impact of the award becomes appar- ent in sales. • Yet the impact of SBIR awards also needs to be qualified. Research rarely results in stand-alone products. Often, the output from an SBIR project is combined with other technologies. The SBIR technology may provide a critical element in developing a winning solution, but that commercial impact—the sale of the larger combined product—is not captured in the data. 8 PL 102-564 October, 28, 1992. 9These changes are described by R. Archibald and D. Finifter in “Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach” in National Research Council, The Small Business Innoation Research Program: An As- sessment of the Department of Defense Fast Track Initiatie, Charles W. Wessner, ed., Washington, DC: National Academy Press, 2000. 10 See National Research Council, An Assessment of the Small Business Innoation Research Pro- gram: Project Methodology, op. cit.

OCR for page 108
 AN ASSESSMENT OF THE SBIR PROGRAM • In some cases, the full value of an “enabling technology” that can be used across industries is difficult to capture. All this is to say that commercialization results must be viewed with cau- tion, first because our ability to track them is limited (indeed it appears highly likely that our efforts at quantification of research awards may understate the true commercial impact of SBIR projects) and because an award, and a successful project, cannot lay claim to all subsequent commercial successes, though it may contribute to that success in a significant fashion. These caveats notwithstanding, it is possible to deploy a variety of assess- ment techniques to measure commercialization outcomes. 4.2.2 Commercialization Indicators and Benchmarks This report uses three sets of indicators to quantitatively assess commercial- ization success: (1) Sales and Licensing Revenues (“sales” hereafter, unless otherwise noted). Revenues flowing to a company from the commercial marketplace and/or through government procurement constitute the most obvious measure of commercial success. They are also an important indicator of uptake for the product or service. Sales indicate that the result of a project has been sufficiently positive to convince buyers that the product or service is the best available solution. Yet if there is general agreement that sales are a key benchmark, there is no such agreement on what constitutes “success.” Companies, naturally enough, focus on projects that contribute to the bottom line—that are profitable. Agency staff provide a much wider range of views. Some view any sales a substantial success for a program focused on such an early stage of the product and development cycle, while others seem more ambi- tious.11 Some senior executives in the private sector viewed only projects that generated cumulative revenues at $100 million or more as a complete commercial success.12 Rather than seeking to identify a single sales benchmark for “success,” it therefore seems more sensible to simply assess outcomes against a range of benchmarks reflecting these diverse views, with each marking the tran- sition to a greater level of commercial success: 11 Interviewswith SBIR program coordinators at DoD, NIH, NSF, and DoE. 12 PeteLinsert, CEO, Martek, Inc., meeting of the NRC Committee on Capitalizing on Science, Technology, and Innovation: An Assessment of the Small Business Innovation Research Program, June 5, 2005.

OCR for page 108
5 SBIR PROGRAM OUTPUTS (a) Reaching the market—a finished product or service has made it to the marketplace (b) Reaching $ million in cumulatie sales (beyond SBIR Phases I and II)—the approximate combined amount of standard DoD Phase I and Phase II awards (c) Reaching $5 million in cumulatie sales—a modest commercial suc- cess that may imply that a company has broken even on a project (d) Reaching $50 million in cumulatie sales—a full commercial success. (2) Phase III Activities Within DoD. As noted above, Phase III activities within DoD are a primary form of commercialization for DoD SBIR projects. These activities are considered in Section 4.3 (Agency Mission) of this report and Chapter 5 (Phase III Challenges and Opportunities) of the DoD report.13 (3) R&D Investments and Research Contracts. Further R&D investments and contracts are good evidence that the project has been successful in some significant sense. These investments and contracts may include partnerships, further grants and awards, or government contracts. The benchmarks for success at each of these levels should be the same as those above, namely: (a) Any R&D additional funding (b) Additional funding of $1 million or more (c) Additional funding of $5 million or more (d) Funding of $50 million or more (4) Sale of Equity. This is a clear-cut indicator of commercial success or market expectations of value. Key metrics include: (a) Equity investment in the company by an independent third party (b) Sale or merger of the entire company 4.2.3 Sales and Licensing Revenues The most basic of all questions on commercialization is whether results from a project reached the marketplace. The NRC Phase II Survey14 indicates that just under half (47 percent) of respondents had generated some sales, and that a further 18 percent still expected sales, though they had none at the time of the survey. In addition, 5 percent were still in the research stage of the project. 13 NationalResearch Council, An Assessment of the SBIR Program at the Department of Defense, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009. 14 Much of the primary data in this section of the report was derived from the NRC’s Phase II Survey. See Appendix A for additional information about the NRC’s Phase II Survey, including response rates.

OCR for page 108
 AN ASSESSMENT OF THE SBIR PROGRAM Ongoing research (5%) Discontinued/No sales (22%) Sales (47%) Sales expected (18%) Sales not expected (7%) FIGURE 4-1 Sales from Phase II projects. SOURCE: NRC Phase II Survey. 4-01 There is variation among the agencies, but these data are consistent with program objectives.15 4.2.3.1 Distribution of Sales Research on early-stage financing strongly suggests a pronounced skew to the results, and this turns out to be the case. Most projects that reach the market generate minimal revenues. A few awards generate substantial results, and a small number bring in large revenues. Of the 790 SBIR Phase II projects reporting sales greater than $0, average sales per project were $2,403,255. Over half of the total sales dollars were due to 26 projects (1.4 percent of the total), each of which had $15,000,000 or more in sales. The highest cumulative sales figure reported was $129,000,000. This distribution is reflected in Figure 4-2. Almost three quarters of the projects reporting sales greater than zero had $1 million or less in sales; two projects reported sales greater than $100 million. The latter by themselves accounted for 16.5 percent of all the revenues reported; together, the 1.7 percent of respondents reporting sales greater than $20 million accounted for 43.7 percent of all revenues reported. These distributions are similar to those reported from other SBIR data 15 See the section on venture capital in National Research Council, An Assessment of the SBIR Pro- gram at the National Institutes of Health, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009.

OCR for page 108
7 SBIR PROGRAM OUTPUTS >$20 million (1.0%) >$50 million >$10 million (0.4%) (2.8%) >$100 million >$5 million (0.3%) (2.3%) >$1 million (20.1%) $1-$1 million (73.2%) FIGURE 4-2 Distribution of projects with sales >$0. SOURCE: NRC Phase II Survey. 4-2 $0-$1 million (8.7%) >$100 million (16.5%) $1-5 million (21.6%) $50-100 million (12.6%) $5-10 million (7.0%) $20-50 million (14.6%) $10-20 million (18.9%) FIGURE 4-3 Distribution of sales, by total sales (percent of total sales dollars). SOURCE: NRC Phase II Survey.

OCR for page 108
 AN ASSESSMENT OF THE SBIR PROGRAM 100 90 Percent of Reporting Projects 80 70 60 50 40 30 20 10 0 >0 and <$1M $1M to <$5M $5M to <$50M $50M + Sales (Dollars) NIH DoD NRC FIGURE 4-4 NIH sales by sales range (percent of projects in each range). Total for 1992-2002. SOURCE: NIH data, NRC Phase II Survey, and DoD Commercialization database. sources.16 For example, at NIH, sales data are available from the NRC survey, a 4-4 previous NIH survey, and the DoD commercialization database.17 These data in Figure 4-4 show that about 10 percent of NIH projects that report any revenues report more than $5 million. 4.2.3.2 Sales Expectations Because it may take years after the end of a Phase II for a commercial prod- uct to reach the market, many projects that do not yet report sales still expect them eventually. About 36 percent of NRC survey respondents with no sales (19 percent of all projects) still expected sales in the future. Analysis elsewhere sug- gests that these expectations are often optimistic, and that a considerably smaller number of these projects will, in the end, reach the market. However, it is equally important to note that a complete accounting of all sales from the projects funded during 1992-2001 (the focus of the NRC survey) will be possible only some years in the future. Many projects have only recently reached the market, so the bulk of their sales will be made in the future and are 16 See National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit., Chapter 4, where data from the DoD Commercialization database and the NIH Phase II Survey are compared with the NRC data. 17The DoD database captures commercialization data from NIH projects where the firm subse- quently applied for a DoD SBIR award.

OCR for page 108
59 SBIR PROGRAM OUTPUTS 4.5.3.2 National Institutes of Health NIH is unique among the agencies in providing data on new applicants— companies not previously funded at NIH—and on new awards to those companies. The trendline shows that the share of applications from previously unfunded companies has actually increased somewhat, and, in any event, remains within fairly narrow boundaries (58-63.5 percent of applications come from previously unfunded companies). Given that previously funded companies have already passed a quality screen, it is not surprising that the share of awards going to previously unfunded companies is lower. Nonetheless, at least 35 percent of awards have gone to previous nonwinners in each year since 2000, although that share has declined from 47 percent in 2000 to just over 35 percent in 2005. This decline might partly reflect the fact that the number of previous winners has increased sharply during this period, and that many of these new “previous winner” firms continue to apply for more awards. 4.5.3.3 Department of Defense At Defense, the data appear to show similar award patterns for Phase II awards. According to the SBIR program office at DoD, 37 percent of FY2005 Phase II awards went to companies that had not previously won a Phase II SBIR award from DoD. This steady infusion of new firms constitutes a major strength of the pro- gram. It underscores the positive impact of SBIR awards, as they encourage in- novation activity across a broad spectrum of firms, create additional competition among suppliers for the procurement agencies, and provide agencies with new mission-oriented research and solutions. 4.6 SBIR AND THE EXPANSION OF KNOWLEDGE Quantitative metrics for assessing knowledge outputs from research pro- grams are well-known, but far from comprehensive. Patents, peer-reviewed pub- lications, and, to a lesser extent, copyrights and trademarks, are all widely-used metrics, and are discussed in detail below. However, these metrics do not capture the entire transfer of knowledge involved in programs such as SBIR. Michael Squillante, Vice-President for Re- search at Radiation Monitoring Devices, Inc., points out that there may be ben- efits from the development and diffusion of knowledge that are not reflected in any quantitative metric: For example, our research led to a reduction in the incidence of stroke following open-heart surgery. Under an NIH SBIR grant we developed a tool for medical

OCR for page 108
0 AN ASSESSMENT OF THE SBIR PROGRAM 70 60 50 40 Percent 30 20 10 0 1996 1997 1998 1999 2000 2001 2002 2003 Fiscal Year FIGURE 4-21 Percentage of Phase I awards to companies new to the NSF SBIR program, 1996-2003. SOURCE: National Science Foundation. new 4-21 65 64 63 62 61 Percent 60 59 58 57 56 55 2000 2001 2002 2003 2004 2005 Fiscal Year FIGURE 4-22 “New” Phase I applicants (percent of all applicants) at NIH, FY2000-2005. SOURCE: National Institutes of Health data. 4-22

OCR for page 108
 SBIR PROGRAM OUTPUTS 50 45 40 35 30 Percent 25 20 15 10 5 0 2000 2001 2002 2003 2004 2005 Fiscal Year FIGURE 4-23 Percentage of Phase I awards to companies new to the NIH SBIR program, FY2000-2005. SOURCE: National Institutes of Health data. 4-23 60 50 40 Percent 30 20 10 0 6 or More Prior Phase II 1-5 Prior Phase II New Awardees Awards Awards FIGURE 4-24 New Phase II winners in the DoD SBIR program, FY2005. NOTE: Firm data as of March 2006. The “New Awardees” category includes prior Phase I award winners that have not won a Phase II award. SOURCE: Michael Caccuitto, DoD SBIR/STTR Program Administrator, and Carol Van Wyk, DoD CPP Coordinator, Presentation to SBTC “SBIR in Rapid Transition Confer- ence,” Washington, DC, September,4-24 new 27 2006.

OCR for page 108
 AN ASSESSMENT OF THE SBIR PROGRAM researchers who were examining the causes of minor and major post-operative stroke occurring after open-heart surgery.47 This was not a project whose success could be measured in commercial terms; the market (stroke researchers) is far too small to generate large revenues. However, the impact of the research was profound. It is, therefore, quite important to understand that the quantitative metrics discussed below are an indicator of the expansion of knowledge; they reflect that expansion but do not fully capture it. In particular, they say little about the im- pact of that knowledge. In the case illustrated above, for example, there was no patent, no commercial sales, no impact on the company’s bottom line—but the research, nevertheless, made a remarkable improvement in outcomes for stroke patients undergoing surgery. 4.6.1 Patents According to the Small Business Administration, small businesses produce 13 to 14 times more patents per employee than do large patenting firms. These patents are twice as likely as are large firm patents to be among the one percent most cited.48 With respect to SBIR, the NRC survey data indicate that about 30 percent of respondents received patents related to their SBIR research. About two-thirds of projects generated at least one patent application, and as with most other SBIR metrics, the distribution of outcomes is highly skewed. The data show that three companies account for almost 20 percent of all patent applications related to the surveyed SBIR award. However, they were not especially successful, as no company reported more than 20 successful patents granted. Overall, 2.2 percent of the responding companies applied for more than five patents, and 1.3 percent received more than five. More detailed case study analysis also indicates the importance of intel- lectual property to firm strategies. At NSF, a number of the case studies showed this clearly. 47 Presentation to NRC Research Team by Michael Squillante, vice president, Radiation Monitoring Devices, Inc. June 11, 2004. 48Accessed on May 16, 2007 at . Drawing on seminal empirical research, Acs and Audretsch found that small business have comparatively higher rates of innovation—specifically, that “the number of innovation increases with increased industry R & D expenditures but at a decreasing rate. Similarly, while the literature has found a somewhat ambiguous relationship between concentration and various measures of technical change, our re- sults are unequivocal—industry innovation tends to decrease as the level of concentration rises.” See Zoltan J. Acs and David B. Audretsch, “Innovation in Large and Small Firms: An Empirical Analysis,” op. cit.

OCR for page 108
 SBIR PROGRAM OUTPUTS 4.6.2 Scientific Publications The NRC Phase II Survey determined that slightly more than half of the re- spondents had published at least one related scientific paper (45.4 percent). About one-third of those with publications had published only a single paper, but one company had published 165 papers on the basis of its SBIR research, and several others had published at least 50 papers. These data fit well with case studies and interviews, which suggested that SBIR companies are proud of the quality of their research. Publications are featured prominently on many Web sites, and companies like Advanced Brain Monitoring, SAM Technologies, and Polymer Research all made the point dur- ing interviews that their work was of the highest technical quality as measured in the single measure that counts most in the scientific community—peer-reviewed publication. Publications therefore fill two important roles in the study of SBIR programs: • First, they provide an indication of the quality of the research being conducted with program funds. In this case, more than half of the funded projects were of sufficient value to generate at least one peer-reviewed publication. • Second, publications are themselves the primary mechanism thorough which knowledge is transmitted within the scientific community. The ex- istence of the articles based on SBIR projects is therefore direct evidence that the results of these projects are being disseminated widely, which in turn means that the congressional mandate to support the creation and dissemination of scientific knowledge is being met. We note that no agency seems to have in place evaluation programs for deter- mining whether similar knowledge effects are being generated at the same or at a different rate outside the SBIR program. Likewise, there is insufficient tracking to determine citation rates for these articles. 4.6.3 SBIR and Universities According to agency and congressional staff, one of the implicit, though not formal objectives, of the SBIR program is that it supports the transfer of knowl- edge from universities to the commercial marketplace. This transfer can happen in many different ways. SBIR funding: • Helps university scientists form companies; • Enables small firms to use university faculty and to employ graduate students as specialized consultants; • Permits the use of university laboratory facilities; and • Encourages other less formal types of collaboration.

OCR for page 108
TABLE 4-11 Distribution of Patents  Patent Applications Patent Awards Number of Number of Total Number Percent of Percent of Number of Number of Total Number Percent of Percent of Patents Companies of Patents Applicants Applications Patents Companies of Patents Recipients Patents Received 0 1,060 0 62.3 0.0 0 1,202 0 70.7 0.0 1 378 378 22.2 23.6 1 319 319 18.8 34.0 2 124 248 7.3 15.5 2 99 198 5.8 21.1 3 60 180 3.5 11.3 3 36 108 2.1 11.5 4 25 100 1.5 6.3 4 14 56 0.8 6.0 5 17 85 1.0 5.3 5 8 40 0.5 4.3 6 11 66 0.6 4.1 6 11 66 0.6 7.0 7 4 28 0.2 1.8 7 3 21 0.2 2.2 8 2 16 0.1 1.0 10 2 20 0.1 2.1 9 2 18 0.1 1.1 11 2 22 0.1 2.3` 10 6 60 0.4 3.8 15 2 30 0.1 3.2 11 4 44 0.2 2.8 17 1 17 0.1 1.8 12 1 12 0.1 0.8 20 2 40 0.1 4.3 13 2 26 0.1 1.6 Total 1,701 937 100 100 14 1 14 0.1 0.9 25 1 25 0.1 1.6 100 3 300 0.2 18.8 Total 1,701 1,600 100 100 SOURCE: NRC Phase II Survey, Question 18.

OCR for page 108
5 SBIR PROGRAM OUTPUTS BOX 4-7 Intellectual Property and Company Strategy Among NSF Awardee Case Studies A number of NSF case studies showed extensive use of patenting: • araday Technology has had 23 patents issued in the U.S. and three foreign F issued, since its founding in 1991. Historically, this amounts to 1.4 issued pat- ents per employee. Patents and the fees they generate are the central focus of Faraday’s business strategy, and the company investigates citations by other companies of its patents to obtain knowledge about potential customers. • mmersion Corporation has more than 270 patents issued in the U.S. and an- I other 280 pending in the U.S. and abroad. According to Immersion, its patent portfolio is at the heart of its wealth-generation capacity. • SCA has received a trademark on its most recent insect lure technology, I which is expected to generate substantial growth for the company. • anguage Weaver has more than 50 patents pending worldwide, which under- L pin its commercialization approach. MER has had more than a dozen patents granted. • icroStrain reported patenting to be “very important” to its commercialization M strategy. • VE reported 34 issued U.S. patents and more than 100 patents issued N worldwide. • /J Technologies has had seven patents granted and has a number of others T pending. Over a third (36.5 percent) of all NRC Phase II Survey respondents indicated that there had been involvement by university faculty, graduate students, and/or a university itself in developed technologies. Based on the responses of all survey respondents, this involvement took a number of forms (see Table 4-13). The wide range of roles played by university staff and students underscores the multiple ways in which SBIR projects enhance the knowledge base of the na- tion. Involvement in these projects provides different opportunities for university staff than those available within academia. Many of the companies interviewed for this study noted that their university connections were extremely important. Radiation Monitoring Devices said that they spent more than $1 million annually on contracted research at universities under their SBIR awards.49 Advanced Targeting Systems, in San Diego, has forged an extended and successful research partnership with a senior scientist at 49 Interview with Michael Squillante, vice president, Radiation Monitoring Devices, June 10, 2005.

OCR for page 108
TABLE 4-12 Publications  Submitted Published Number of Number of Total Percent of Percent of Total Number of Number of Total Percent of Percent of Total Publications Respondents Publications Respondents Publications Publications Respondents Publications Respondents Publications 0 902 0 53.0 0.0 0 929 0 54.6 0.0 1 253 253 14.9 8.4 1 252 252 14.8 8.7 2 213 426 12.5 14.1 2 208 416 12.2 14.4 3 116 348 6.8 11.5 3 108 324 6.3 11.2 4 65 260 3.8 8.6 4 65 260 3.8 9.0 5 63 315 3.7 10.5 5 57 285 3.4 9.9 6 15 90 0.9 3.0 6 11 66 0.6 2.3 7 5 35 0.3 1.2 7 4 28 0.2 1.0 8 16 128 0.9 4.2 8 18 144 1.1 5.0 9 4 36 0.2 1.2 9 3 27 0.2 0.9 10 18 180 1.1 6.0 10 16 160 0.9 5.6 11 3 33 0.2 1.1 11 4 44 0.2 1.5 12 7 84 0.4 2.8 12 5 60 0.3 2.1 13 2 26 0.1 0.9 13 2 26 0.1 0.9 15 3 45 0.2 1.5 15 3 45 0.2 1.6 17 1 17 0.1 0.6 17 1 17 0.1 0.6 20 1 20 0.1 0.7 20 1 20 0.1 0.7 23 1 23 0.1 0.8 22 1 22 0.1 0.8 25 1 25 0.1 0.8 25 1 25 0.1 0.9 30 2 60 0.1 2.0 30 2 60 0.1 2.1 33 1 33 0.1 1.1 33 1 33 0.1 1.1 37 1 37 0.1 1.2 37 1 37 0.1 1.3 40 1 40 0.1 1.3 40 1 40 0.1 1.4 45 1 45 0.1 1.5 41 1 41 0.1 1.4 50 1 50 0.1 1.7 50 3 150 0.2 5.2 52 2 104 0.1 3.5 60 1 60 0.1 2.1 60 1 60 0.1 2.0 75 1 75 0.1 2.6 75 1 75 0.1 2.5 165 1 165 0.1 5.7 165 1 165 0.1 5.5 Total 1,701 2,882 100 100 Total 1,701 3,013 100 100 SOURCE: NRC Phase II Survey, Question 18.

OCR for page 108
7 SBIR PROGRAM OUTPUTS TABLE 4-13 University Involvement in SBIR Projects 2% The Principal Investigator (PI) for this Phase II project was a faculty member. 3% The Principal Investigator (PI) for this Phase II project was an adjunct faculty member. 22% Faculty or adjunct faculty member(s) work on this Phase II project in a role other than PI, e.g., consultant. 15% Graduate students worked on this Phase II project. 13% University/College facilities and/or equipment were used on this Phase II project. 3% The technology for this project was licensed from a University or College. 5% The technology for this project was originally developed at a University or College by one of the percipients in this phase II project. 17% A University or College was a subcontractor on this Phase II project. SOURCE: NRC Phase II Survey. the University of Utah, which they described as “critical to the development and testing of their products.”50 These stories and the many others like them that we encountered suggest that the flow of information and funding between small businesses and universities working within the SBIR framework is not simple or unidirectional. For example, the constant flow of feedback, testing, and insights between university research- ers and Advanced Targeting Systems staff helped to move the company forward toward product deployment into new research areas while providing real-life problems for research to address. What is now clear from this research—and could certainly be the subject of further analysis—is the extent to which universities themselves see SBIR as a mechanism for technology transfer, commercialization, and additional funding for university researchers. 4.6.3.1 University Faculty and Company Formation Data from the NRC Firm Survey strongly support the hypothesis that SBIR has encouraged university faculty to form companies to commercialize their inventions and technologies. Two-thirds of all responding companies had at least one academic founder, and more than a quarter had more than one. The same survey found that about 36 percent of founders were most recently employed in an academic environment before founding the new company. These data and evidence from case studies strongly suggests that SBIR has indeed encouraged academic scientists to bring their work to a more commercial environment. 50 See Advanced Targeting System case study in National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit.

OCR for page 108
 AN ASSESSMENT OF THE SBIR PROGRAM 4.7 CONCLUSIONS The extended discussion of metrics and indicators in the Methodology Re- port show quite conclusively that there is no single simple metric that captures “results” from the program. Each of the four congressional mandates is best as- sessed separately, and within each, there are a multiple issues to be addressed. Bearing all these points in mind, it is still possible to summarize the results of our research in straightforward terms. Commercialization. Approximately 30-40 percent of Phase II projects pro- duce innovations that reach the market, with a few generating substantial returns. Other indicators of commercialization, such as licensing activities, marketing partnerships, and access to and utilization of further investments from both private and public sources, all confirm that while returns are highly skewed, the results in general are positive, and are consistent with other public and private sources of early-stage financing. Agency Mission. We found many examples of funded projects that clearly made a major contribution to the mission of the awarding agency. To an increas- ing extent, SBIR is becoming more firmly integrated into the overall R&D strat- egy of each agency. At the procurement agencies—DoD and NASA—we found that data from the agencies and from NRC surveys indicated that many SBIR projects did result in utilization by the agencies. We also determined that the agencies had made important efforts to improve the alignment between agency needs and SBIR implementation, and that procedures for encouraging or ensur- ing alignment were in place at all agencies. Traditionally hostile views of the program among many acquisition and R&D managers have begun to shift in a much more positive direction. Many projects made a very valuable contribution to agency mission, and these contributions were often far out of proportion to the commercial returns involved.51 Support for Small Business, and Woman- and Minority-owned Busi- nesses. SBIR significantly supports small high technology businesses in general, and the NRC research team discovered that SBIR had an important catalytic ef- fect in terms of company foundation—providing the critical seed money to fund a company’s first steps. SBIR also had strongly influenced companies’ decisions to initiate individual projects: more than 65 percent of NRC Phase II Survey respondents at every agency believed that their projects would not have gone forward without SBIR, and, of the remainder, most believed that the projects 51The use of Savi RFID tracking systems reportedly enabled large reductions in shipping require- ments for U.S. force deployments. The potential savings far outweighs the commercial value of the tracking devices. See the Savi Technology case study in Appendix D of National Research Council, An Assessment of the SBIR Program at the Department of Defense, op. cit.

OCR for page 108
9 SBIR PROGRAM OUTPUTS would have been delayed and/or would have had a reduced scope. While some efforts have been made to increase participation by woman- and minority-owned businesses, more needs to be done. At DoD and NIH data on these businesses were incomplete or missing. At all of the agencies except NASA, what data there was suggested that there were areas of concern in the program (different areas at different agencies), and it did not appear that the agencies had focused sufficient resources and management attention on these areas. Support for the Advancement of Scientific and Technical Knowledge. The program funds cutting edge research, as it was designed to do. Most of the agencies use “technical innovation” in some form as a critical selection parame- ter. Outcomes in the form of patents and peer-reviewed scientific publications are encouraging, with about two-thirds of recipients publishing at least one related scientific paper. Surveys of program managers and technical points of contact at DoD, NASA, and DoE indicated that they saw SBIR projects as approximately equivalent in research quality to non-SBIR projects.52 Moreover, case studies indicate that knowledge generated in SBIR-funded research can be picked up indirectly in many ways, and often continues to be productive long after the original project has been concluded. It is therefore appropriate to conclude that the program is meeting all four of the congressional objectives. 52After results were normalized to eliminate the most favorable and most negative responses.