Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
4 SBIR Program Outcomes 4.1â INTRODUCTION As the SBIR program approached its twentieth year of operation, the U.S. Congress asked the National Research Council to conduct a âcomprehensive study of how the SBIR program has stimulated technological innovation and used small businesses to meet federal research and development needsâ and to make recommendations on still further improvements to the program. The Small Business Innovation Act, sets out four goals for the program: â(1) to stimulate technological innovation; (2) to use small business to meet federal research and development needs; (3) to foster and encourage participa- tion by minority and disadvantaged persons in technological innovation; and (4) to increase private-sector commercialization derived from Federal research and development.â The legislation does not set priorities among these four objectives. However, discussions with congressional staff suggest that commercialization has become increasingly important to Congress. Still, it remains important to assess each of the four objectives; each should be taken as equally important in evaluating the achievements and challenges of the SBIR program. These four objectives help to define the structure and content of this chapter. Assessing program outcomes against these four objectives entails numerous â See the SBIR Reauthorization Act of 2000 (H.R. 5667âSection 108). â The Small Business Innovation Development Act (PL 97-219). 55
56 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION methodological challenges. These challenges, discussed in detail in the National Research Councilâs Methodology Report, are briefly reviewed below. 4.1.1â Compared to What? Assessment usually involves comparisonâcomparing programs and activi- ties, in this case. Three kinds of comparison seem possible: with other programs at each agency, among SBIR programs at the various agencies, and with early- stage technology development funding in the private sector, such as venture capital activities. Yet, as we see below, the utility of each of these three types of comparison is limited. Comparison with Other NASA Programs Within NASA, no other program is dedicated to support innovative small businesses. This fundamental difference in objectives makes it difficult to com- pare the NASA SBIR program with other programs at the agency. Comparison with Other SBIR Programs Comparing the NASA SBIR program with those at other agencies is superfi- cially more useful. However, as discussed in Chapter 1 of this volume, the SBIR programs at each of the agencies are shaped by the different agency missions. This, in turn, is reflected in the different mechanisms and approaches taken by the agencies. Agencies whose mission is to develop technologies for internal agency use via procurementânotably DoD and NASAâhave a different orien- tation from agencies that do not procure technology and are instead focused on developing technologies for use outside the agency. There are important differences between the two âprocurementâ agencies. At DoD, once an SBIR technology is proven, there are opportunities for integration of that technology into a very substantial stream of acquisitions dollars. At NASA, such proven technologies may also be taken up for use by the agencyâÂoften only for one or two copies of a technology, for use on NASA space missions. Thus, the character of commercialization is quite different. â National Research Council, An Assessment of the Small Business Innovation Research ÂProgramâ Project Methodology, Washington, DC: The National Academies Press, pp. 20-21, 2004. For a broader discussion of the scope and limitations of surveys by the University of Michigan Survey Research Center, see Robert M. Groves, Floyd J. Fowler, Jr., Mick P. Couper, James M. Lepkowski, Eleanor Singer, and Roger Tourangeau, Survey Methodology, Boston, MA: WileyBlackwell, 2004.
SBIR PROGRAM OUTCOMES 57 Comparison with Early-stage Venture Capital Finally, SBIR might be compared with venture capital (VC) activities, but there are important differences. VC funding is typically supplied later in the development cycle when innovations are in, or close, to market. Indeed, most venture investments are made with the expectation of an exit from the company within three years. VC investments are also typically larger than SBIR awards. In 2007, the median investment made by VC firms in a company was $7.6 mil- lion, compared to less than $1 million for a NASA SBIR over a two to three year cycle. VC investments are also focused on companies, not projects, and often come both with substantial management support and influence (such as through seats on the companyâs board). 4.1.2â Multiple Metrics The lack of direct comparators means that multiple metrics must be de- ployed, using a wide array of information sources. This is what the NRC Com- mittee has done: â¢ The NRC Phase II Survey covers every firm that received a Phase II award between 1992 and 2001 inclusive. â¢ The NRC Phase I Survey covers projects that failed to proceed beyond Phase I. â¢ The NRC Project Manager Survey. â¢ Case Studies commissioned by the NRC Committee provide context and illustration, in addition to user perspectives of the program. â¢ Interviews with agency staff both within and outside the SBIR program office, as well as other experts inform the Committeeâs findings. â¢ NASA Databases, in particular the NASA awards database, as well as a NASA outcomes assessment, have provided basic information about the program. While the surveys broke important ground, and provided a central base of information on which considerable parts of the assessment are based, it is impor- tant to note that surveys of innovation awards can suffer from several forms of survey bias. These issues are discussed in Box 4-1. â 2007 saw some 2,648 deals, with an overall capital investment of $29.9 billion, according to data from National Venture Capital Association. See <https://www.pwcmoneytree.com/MTPublic/ns/index. jsp>. âFor a more detailed discussion of the methodology, see National Research Council, An Assessment of the Small Business Innovation Research ProgramâProject Methodology, op. cit. âAccess the NASA Commercialization Metrics Survey at <http://www.sbir.nasa.gov/SBIR/survey. html>.
58 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION BOX 4-1 Multiple Sources of Bias in Survey Response Large innovation surveys involve multiple sources of bias that can skew the results in both directions. Some common survey biases are noted below. These biases were tested for and responded to in the NRC surveys.a â¢ Successful and more recently funded firms are more likely to respond. Re- search by Link and Scott demonstrates that the probability of obtaining research project information by survey decreases for less recently funded projects and it increased the greater the award amount.b Nearly 40 percent of respondents in the NRC Phase II Survey began Phase I efforts after 1998, partly because the number of Phase I awards increased, starting in the mid 1990s, and partly because winners from more distant years are harder to reach. They are harder to reach as time goes on because small businesses regularly cease operations, are acquired, merge, or lose staff with knowl- edge of SBIR awards. â¢ Success is self-reported. Self-reporting can be a source of bias, although the di- mensions and direction of that bias are not necessarily clear. In any case, policy analy- sis has a long history of relying on self-reported performance measures to represent market-based performance measures. Participants in such retrospectively analyses are believed to be able to consider a broader set of allocation options, thus making the evaluation more realistic than data based on third-party observation.c In short, company founders and/or principal investigators are in many cases simply the best source of information available. â¢ Survey sampled projects at firms with multiple awards. Projects from firms with multiple awards were underrepresented in the sample, because they could not be ex- pected to complete a questionnaire for each of dozens or even hundreds of awards. â¢ Failed firms are difficult to contact. Survey experts point to an âasymmetryâ in their ability to include failed firms for follow-up surveys in cases where the firms no longer exist.d It is worth noting that one cannot necessarily infer that the SBIR project failed; what is known is only that the firm no longer exists. â¢ Not all successful projects are captured. For similar reasons, the NRC Phase II Survey could not include ongoing results from successful projects in firms that merged or were acquired before and/or after commercialization of the projectâs technology. The survey also did not capture projects of firms that did not respond to the NRC invitation to participate in the assessment. â¢ Some firms may not want to acknowledge the full SBIR contribution to a projectâs success. Some firms may be unwilling to acknowledge that they received important benefits from participating in public programs for a variety of reasons. For example, some may understandably attribute success exclusively to their own efforts. â¢ Commercialization lag. While the NRC Phase II Survey broke new ground in data collection, the amount of sales madeâand indeed the number of projects that gener- ate salesâare inevitably undercounted in a snapshot survey taken at a single point in time. Based on successive data sets collected from NIH SBIR award recipients, it is estimated that total sales from all responding projects will likely be on the order of 50
SBIR PROGRAM OUTCOMES 59 percent greater than can be captured in a single survey.e This underscores the impor- tance of follow-on research based on the now-established survey methodology. 14 12 Sales (Millions of Dollars) 10 8 6 4 2 0 1 2 3 4 5 6 7 8 9 10 11 12 Survey taken Years after Phase II Award FIGURE B-4-1â Survey bias due to commercialization lag. These sources of bias provide a context for understanding the response rates to the NASA Figure B-4-1.eps NRC Phase I and Phase II Surveys conducted for this study. For the NRC Phase II Survey, of the 534 NASA firms that could be contacted out of a sample size of 779, 181 responded, representing a 34 percent response rate. The NRC Phase I Survey captured 9 percent of the 3,363 awards made by NASA over the period of 1992 to 2001. See Appendixes B and C for additional information on the surveys. aFor a technical explanation of the sample approaches and issues related to the NRC surveys, see Appendixes B and C. bAlbert N. Link and John T. Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Programâs Intramural Research Initiative, London: Routledge, 2005. cWhile economic theory is formulated on what is called ârevealed preferences,â meaning individu- als and firms reveal how they value scarce resources by how they allocate those resources within a market framework, quite often expressed preferences are a better source of information especially from an evaluation perspective. Strict adherence to a revealed preference paradigm could lead to misguided policy conclusions because the paradigm assumes that all policy choices are known and understood at the time that an individual or firm reveals its preferences and that all relevant markets for such preferences are operational. See Gregory G. Dess and Donald W. Beard, âDimensions of Organizational Task Environments.â Administrative Science Quarterly 29:52-73, 1984; and Albert N. Link and John T. Scott, Public Accountability: Evaluating Technology-Based Institutions, Norwell, MA: Kluwer Academic Publishers, 1998. dAlbert N. Link and John T. Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Programâs Intramural Research Initiative, op. cit. eData from NIH indicates that a subsequent survey taken two years later would reveal very substantial increases in both the percentage of firms reaching the market, and in the amount of sales per project. See National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009.
60 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 4.1.3â NASAâs Changing Program Priorities NASAâs mission objectives are defined in very specific termsâplacing a man on Mars and returning him safely, for example. These mission objectives require the development and acquisition of highly specialized technologies, often with limited commercial applications. There is, for example, little need for Mars- hardened technologies on Earth. To address NASAâs unique needs, subtopic and selection decisions for the NASA SBIR program were managed by those with a primary interest in using SBIR technologies for NASA missions. Seeking, at the same time to improve the commercialization outcomes of its SBIR program, NASA also provided support for technology incubators. In time, NASA recognized that this compromise was not successful. On one hand, the NASAâs mission realities limited the commercial potential of NASA SBIR projects. On the other hand, the focus on commercialization limited a tighter linkage between SBIR projects and agency mission objectives. Recognizing this challenge, NASAâs senior management shifted the program in 2006 away from this uneasy balance between commercialization and mission support objectives, and to a new commitment focused almost exclusively on mis- sion support. As a result of this reorganization, commercialization has become a secondary objective for the agencyâs SBIR program. The reorganization changed in the structure of the program from one where key decisions were primarily made by centers and specific topic managers to one where those decisions are made at Headquarters by Mission Directorate staff, with input from the centers and project managers. With authority for strategic planning, NASAâs Mission Directorate staff were seen as best placed to identify the technology areas with greatest needs, and hence to set the appropriate priori- ties for the SBIR program. By moving SBIR from the periphery of NASA plan- ning to become a potentially valuable solution to current mission challenges, the reorganization is designed to ensure that SBIR is used to the maximum extent possible to help develop technologies needed by NASA. 4.2â COMMERCIALIZATION: A LONG-TIME PROGRAM PRIORITY Commercialization of SBIR-funded technologies has been a key congres- sional objective for the SBIR program since its inception. In fact, the programâs initiation in the early 1980s in part reflected a concern that American investment in research was not adequately deployed to the nationâs competitive advantage. â Unlike DoD, NASA does not itself constitute a major market for commercial sales. And at the same time, the technologies developed specifically for NASA have relatively little relevance to the private marketplaceâfor example, compared to some applications developed at NIH. â These administrative changes, made in 2006, are described in Chapter 5 (Program Management).
SBIR PROGRAM OUTCOMES 61 Directing a portion of federal investment in R&D to small businesses was thus seen as a new means of meeting the mission needs of federal agencies while increasing the participation of small business and thereby the proportion of in- novation that would be commercially relevant. Congressional and Executive branch interest in the commercialization of SBIR research has increased over the life of the program. Drawing from a 1992 GAO study10 that focused on commercialization, the 1992 reauthorization specifi- cally âemphasize[d] the programâs goal of increasing private-sector commercial- ization of technology developed through Federal research and developmentâ 11 and noted the need to âemphasize the programâs goal of increasing private-sector commercialization of technology developed through Federal research and devel- opment.â The 1992 reauthorization also changed the order in which the programâs objectives are described, moving commercialization to the top of the list. 12 The term âcommercializationâ is subject to widely varying interpretations. Several agencies have taken it to mean âfirst saleââthat is, the first sale of a product in the market place, whether to public- or private-sector clients. This definition, however, misses significant components of commercialization that do not result in a discrete sale. It also fails to provide any guidance on how to evaluate the scale of commercialization, an important element in assessing the degree to which SBIR programs successfully encourage commercialization. The NRC methodology has determined that multiple metrics can and should be used to assess the extent of commercialization. The following sections review commercialization outcomes for NASA through 2005. consequently, the recent change in emphasis to mission support is not addressed. â A growing body of evidence, starting in the late 1970s and accelerating in the 1980s indicates that small businesses were assuming an increasingly important role in both innovation and job creation. See, for example, J. O. Flender and R. S. Morse, The Role of New Technical Enterprise in the U.S. Economy, Cambridge, MA: MIT Development Foundation, 1975, and David L. Birch, âWho Creates Jobs?â The Public Interest, 65:3-14, 1981. Evidence about the role of small businesses in the U.S. economy gained new credibility with the empirical analysis by Zoltan Acs and David Audretsch of the U.S. Small Business Innovation Data Base, which confirmed the increased importance of small firms in generating technological innovations and their growing contribution to the U.S. economy. See Zoltan Acs and David Audretsch, âInnovation in Large and Small Firms: An Empirical Analysis,â The American Economic Review, 78(4):678-690, September 1988. See also Zoltan Acs and David Audretsch, Innovation and Small Firms, Cambridge, MA: MIT Press, 1990. 10â U.S. General Accounting Office, Small Business Innovation Research Program Shows Success But Can Be Strengthened, GAO/RCEDâ92â37, Washington, DC: U.S. General Accounting Office, 1992. 11â PL 102-564, October, 28, 1992. 12â These changes are described by R. Archibald and D. Finifter in âEvaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approachâ in National Research Council, SBIR: An Assessment of the Department of Defense Fast Track Initiative, Charles W. Wessner, ed., Washington, DC: National Academy Press, 2000.
62 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 4.2.1â Assessing Commercialization Clear metrics for assessing commercialization are elusive.13 It is not possible to quantify in full all commercialization from a given research project, for the reasons listed below. â¢ Numerous additional development steps are often needed after the re- search has been concluded. Thus, a single, direct line between research inputs and commercial outputs rarely exists in practice; cutting-edge research is only one contribution among many leading to a successful commercial product. â¢ Markets themselves have major imperfections, often caused by informa- tion asymmetries. Hence high qualityâeven path-breakingâresearch does not always result in commensurate commercial returns. â¢ There are often long lags between an early-stage research project and an eventual commercial product. This means that for a significant number of the more recent SBIR projects, commercialization is still in process, and salesâoften substantial salesâwill be made in the future (see the âsnapshot effectâ discussed in Box 4-2). â¢ Research rarely results in stand-alone products. Often, the output from an SBIR project is combined with other technologies. The SBIR technology may provide a critical element in developing a winning solution, but that commercial impact is hard to measure in simple dollars. â¢ In some cases, the full value of an âenabling technologyâ that can be used across industries is difficult to capture. All this is to say that commercialization results must be viewed with caution. Our ability to track them is limited. Indeed, it appears highly likely that quantifi- cation of research awards through surveys substantially understates the true com- mercial impact of SBIR projects. In addition, a specific award cannot lay claim to all subsequent commercial successes, even though the technology developed with the award may have contributed to many significant outcomes.14 4.2.2â Commercialization Indicators and Benchmarks This report uses four sets of indicators to assess commercialization success quantitatively: 13â See National Research Council, An Assessment of the Small Business Innovation Research Â rogramâProject Methodology, op. cit. P 14â Data on infusion/commercialization of SBIR technologies from 1983 to 1996 was included in NASAâs 2002 Commercial Metrics publication. Follow-on data gathering via this Commercial Tech- nology Division initiative has been halted pending further funding. Of NASAâs 1,739 SBIR Phase II awards during the target period, about 15 percent developed technologies used in NASA or other programs via Phase III funding, and 31 percent commercialized in the private sector. See Commercial Metrics; NASA Commercial Technology Division; October, 2002; Fig 4.
SBIR PROGRAM OUTCOMES 63 1. Sales and licensing revenues (âsalesâ hereafter, unless otherwise noted). Revenues flowing to a company from the commercial marketplace and/or through government procurement constitute the most obvious measure of com- mercial success. They are also an important indicator of uptake for the product or service. Sales indicate that the result of a project has been sufficiently positive to convince buyers that the product or service is the best available solution. Yet if there is general agreement that sales are a key benchmark, there is no such agreement on what constitutes âsuccess.â Companies, naturally enough, focus on projects that contribute to the bottom lineâthat are profitable. Agency staff provide a much wider range of views. Some view any sales a substantial success for a program focused on such an early stage of the product and devel- opment cycle, while others seem more ambitious.15 Some senior executives in the private sector view only projects that generated cumulative revenues at $100 million or more as a complete commercial success.16 Rather than seeking to identify a single sales benchmark for âsuccess,â it therefore seems more sensible to assess outcomes against a range of benchmarks reflecting these diverse views, with each marking the transition to a greater level of commercial success. 2. Phase III activities within NASA. As noted above, Phase III activities within NASA are a primary form of commercialization for NASA SBIR projects. These activities are considered in the mission support section (Section 4.3). 3. R&D investments and research contracts. Further R&D investments and contracts are good evidence that the project has been successful in some sig- nificant sense. These investments and contracts may include partnerships, further grants and awards, or government contracts. 4. Sale of equity. This is a clear-cut indicator of commercial success or market expectations of value. Key metrics include: a. Equity investment in the company by independent third party. b. Sale or merger of the entire company. Sales and Licensing Revenues A basic question on commercialization is whether results from a project have reached the marketplace. The NRC Phase II Survey17 indicates that about 15â Interviews with SBIR program coordinators at DoD, NIH, NSF, and DoE. 16â Pete Linsert, CEO, Martek Biosciences, Inc., NRC Committee Meeting, June 5, 2005. 17â Much of the primary data in this section of the report was derived from the NRC Phase II Sur- vey. The NRC Phase II Survey of projects provides recent evidence on the extent by which NASA SBIR award recipients have achieved commercialization and/or progress toward commercialization. The survey provides information on sales, modes of commercialization, and on steps important to achieving commercialization, including marketing activities, interactions with other companies and
64 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 46 percent of NASA respondents had generated some sales or licensed their tech- nology, and that a further 14 percent still expected sales though they had none at the time of the survey. In addition, 3 percent were still in the research stage of the project.18 (See Figure 4-1.) These results are broadly in line with other sources of information on com- mercial outcomes from SBIR program, including those at other agencies such as the DoD commercialization database, the NIH Phase II Survey, and the NSF Phase II survey.19 Distribution of Sales Research on early-stage financing strongly suggests that a pronounced skew to the results is likely, and this turns out to be the case. Most projects that reach the market generate minimal revenues. A few awards generate substantial results, and a small number bring in large revenues.20 Of the 74 NASA SBIR Phase II projects reporting sales greater than $0, aver- age sales per project were $1,154,156. About 40 percent of the total sales dollars were due to the two NASA projects responding to the survey that had received $5,000,000 or more in sales. The highest cumulative sales figure reported was $15,000,000. investors, and attraction of funding from non-SBIR sources. It also provides information on employ- ment effects, including the extent to which woman and minorities are involved in the projects as principal investigators. Finally, it explores the extent to which the reported effects are believed by survey respondents to be attributed to impacts of the SBIR program. See Appendix B for additional information about the NRC Phase II Survey, including response rates. 18â See Finding G on venture capital and SBIR in Chapter 2 of National Research Council, An AssessÂ ment of SBIR at the National Institutes of Health, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009. 19â See the NRC assessments of the SBIR program at DoD and NIH for discussion of these sources. National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit.; National Research Council, An Assessment of the SBIR Program at the National Science Foundation, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2008; ÂNational Research Council, An Assessment of the SBIR Program at the Department of Defense, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009. 20â See John H. Cochrane, âThe Risk and Return of Venture Capital,â Journal of Financial Eco- nomics, 75(1):3-52, 2005. Drawing on the VentureOne database Cochrane plots a histogram of net venture capital returns on investments that âshows an extraordinary skewness of returns. Most returns are modest, but there is a long right tail of extraordinary good returns. 15 percent of the firms that go public or are acquired give a return greater than 1,000 percent! It is also interesting how many modest returns there are. About fifteen percent of returns are less than 0, and 35 percent are less than 100 percent. An IPO or acquisition is not a guarantee of a huge return. In fact, the modal or âmost probableâ outcome is about a 25 percent return.â See also Paul A. Gompers and Josh Lerner, âRisk and Reward in Private Equity Investments: The Challenge of Performance Assessment.â Journal of Private Equity, 1 (Winter 1977):5-12. Steven D. Carden and Olive Darragh, âA Halo for Angel Investorsâ The McKinsey Quarterly, 1, 2004 also show a similar skew in the distribution of returns for venture capital portfolios.
SBIR PROGRAM OUTCOMES 65 No sales yetâSales expected (14%) Discontinued without sales No sales yetâNone (30%) expected (7%) In development (3%) Sales or licensing (46%) FIGURE 4-1â Results from NASA Phase II projects. SOURCE: NRC Phase II Survey. Based on responses to Phase II Survey questions 1a, NASA Figure 4-1.eps 1b, 3a, and 3b. These figures appear lower than those for other agencies, notably DoD and NIH. However, direct comparisons of results from the NRC Phase II Survey are not valid because of survey response issues. And it should be noted that the very high degree of skew combined with smaller number of awards at NASA means that comparisons are likely to be even more inaccurate (NASA may simply not have made enough awards to generate a statistically significant number of big winnersâfirms with more than $10 million in salesâthough it might be a matter of concern if current trends continued indefinitely). This distribution is reflected in Figure 4-2. More than 80 percent of the projects reporting sales greater than zero had $1 million or less in sales, as seen in Figure 4-3. The numerous projects with relatively low sales (below $1 million) are also in line with our understanding of commercialization within NASA itself. Accord- ing to the SBIR liaison office at the Space Operations Mission Directorate, the average Phase II award at NASA is on the order of $500,000-600,000. 21 This is of course sharply lower than those at DoD, and reflects the particular needs and objectives of NASA programs. As a result, however, a an SBIR project that was successful from NASAâs perspectiveâeven one that resulted in technologies be- ing adopted for a space flight missionâmight well generate commercial returns of less than $1 million. 21â Interview with Jason Crusan, Program Integration Office, Space Operations Mission Directorate, NASA, December 7, 2007.
66 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION $5M to <$10M More than $10M (1.5%) (1.5%) $0 to <$100K $1M to <$5M (17.6%) (14.7%) $100K to <$1M (64.7%) FIGURE 4-2â Distribution of projects with sales >$0. SOURCE: NRC Phase II Survey,NASA Figure 4-2.eps Detailed responses to question 4b. The fact that 3 percent of projects generated more than $5 million in com- mercial returns is also approximately in line with results from other sources at other agencies. Future Sales from Existing Projects A complete accounting of all sales from the projects funded during 1992- 2002 (the focus of the NRC Phase II Survey) will be possible only some years in the future. Many projects have only recently reached the market, so the bulk of their sales will be made in the future and are not captured in these survey data, which effectively capture initial sales (see Box 4-2). According to NASA staff, full commercialization of NASA SBIR projects usually occurs only 7 years after completion of the Phase II award.22 22â Interview with Carl Ray, SBIR Program Director, November 12, 2007.
SBIR PROGRAM OUTCOMES 67 $0 to <$100K (0.7%) $10M+ $100K to <$1M (28.2%) (27.2%) $5M to <$10M (11.3%) $1M to <$5M (32.6%) FIGURE 4-3â Distribution of sales, by total sales (percent of total sales dollars). Detailed responses to question 4b. NASA Figure 4-3.eps SOURCE: NRC Phase II Survey. Responses to the NRC Phase II Survey also indicate that respondents expect to commercialize more in the future. About 66 percent of the NRC Phase II Survey respondents at NASA with no sales still expected sales in the future (14 percent of all projects responding).23 Sales by Sector The NRC Phase II Survey asked respondents to identify the customer base for the products. Results for NASA projects are described in Table 4-1. It is perhaps surprising that only 17 percent of sales went to NASA, plus some share of the 11 percent going to prime contractors for DoD and NASA. However, this may simply reflect the relatively small dollar size of sales to NASA, where only a few units of a given technology may be needed for a par- ticular mission. The data also suggest significant spillover between NASA SBIR awards and DoD acquisitions. The 11 percent share of export markets might on first glance appear surprising, but space operations are a highly specialized field, 23â NRC Phase II Survey, Question 3.
68 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION BOX 4-2 Underestimating Commercial Outcomes from the SBIR Program: The Impact of Systematic Characteristics of Survey Analysis Among the SBIR agencies, only DoD requires that firms enter commercialization data into a database when applying for subsequent awards. This detailed database is a pow- erful source of information, primarily about DoD-oriented firms and projects. We would recommend that other agencies consider implementing the same requirement, and that all agencies utilize the existing DoD database for this purpose to minimize costs. In the absence of such data, analysis of commercialization continues to rely on survey data. These data have important strengths and weaknesses. The NRC Phase II Survey was sent to all firms with SBIR Phase II awards from 1992- 2002. This represents the first effort to generate responses from the entire population of winning firms. The data generated are therefore the best available. However, there are three key sets of limitations, all of which have the effect of understatingâperhaps very substantiallyâthe amount of commercialization achieved. These three limitations can be called the âmultiple-awards effect,â the âsnapshot effect,â and the ârecent awards effect.â The Multiple-awards Effect Because some firms have received many awards, it is not feasible or reasonable to expect them to answer a similar questionnaire about each award that they received. As a result, the NRC Phase II Survey limited the number of questionnaires sent to multiple winners, sending one questionnaire per project to firms that had won three Phase II awards or less, and questionnaires to a sampling of awarded projects for firms with more than three awards. The effect has been to bias survey responses away from firms with multiple awards. This matters when there are systemic differences between the results provided by these different groups of firms. And the NRC Phase II Survey indicates that firms with multiple awards are in fact likely to generate higher levels of commercialization than firms with smaller numbers of awards. Using data from the DoD commercialization database to test this hypothesis, we found that firms receiving more than 15 awards generated an average of $1.39 million in sales per project; firms with fewer than 15 awards generated only $0.75 million per project. Firms with more awards generated on average 85 percent more sales per project.a Thus, the selection bias away from firms with multiple awards appears likely to have had a significant downward impact on commercialization estimates. The Snapshot Effect Well-designed surveys provide an important insight into outcomes from SBIR projects. Necessarily, however, they provide a view of outcomes at the moment that the survey was completed. For almost all products and services, sales follow some form of bell-shaped curve: relatively slow sales as products begin to penetrate the market, growth in sales until the market is saturated or competing products emerge, and decline until the product has been superseded. The shape of the curve differs between products, of course, and the
SBIR PROGRAM OUTCOMES 69 entire curve can be completed in a matter of months for some software sales, or in de- cades for niche products in extremely long cycle industries (e.g., weapons platforms). The survey, however, takes a cross-section of the bell curve. It asks about levels of commercialization at a particular point in time. In essence, it asks about past sales, but can generate little reliable data on future sales. Thus, the average sales data generated by surveys reflects average sales to date. Using some simple analytic techniques, it is possible to estimate that on average, the NRC Phase II Survey (and other similar surveys such as the NIH Phase II Survey) ex- cluded approximately 50 percent of the total lifetime sales of the products and services generated from SBIR awards. This hypothesis is supported by recent data from NIH, where the first resurvey of firms was done in 2005, 3 years after the initial 2002 survey. Results from the survey indicate that the number of firms with some sales increased from 29 percent of surveyed firms to 63 percent (this partly reflects the number of firms still in precommercialization at the time of the first survey). The Recent-awards Effect The snapshot effect is further exacerbated by the distribution of responses to the sur- veys. For two reasons, responses are tilted toward awards from more recent years. First, the number of awards has been rising rapidly, especially at NIH and DoD since the late 1990s. As a result, a larger number of awards are concentrated in recent years. Second, firms with awards from many years ago are harder to find, and are less likely to respond to surveys. As one commentator notes, âthere are no SBIR shrinesâ at SBIR recipient companiesâno one may remember receiving an award 10 years ago; the company may be out of business; the PI may have left. As a result, awards that are more recent generate a higher percentage response rate. The results of the factors are clear. At NIH for example, of the original 758 survey re- spondents, 258â34 percent of all respondentsâreached the market after the date of the first survey. The first survey captured less than half of the projects that had reached the market three years later, in 2005. This is unsurprising but very important. Responses come preponderantly from projects where awards were made relatively recentlyâprecisely the projects where the snap- shot effect is particularly important. Conclusions It is at this stage not possible to provide accurate estimates for the impact of these effects on commercialization estimates drawn from surveys. The limited evidence avail- able to date suggests that the effect may be to reduce commercialization estimates by at least 50 percent, and possibly considerably more. This analysis strongly suggests that follow-up surveys will be especially important, as they provide critical data for making precisely the assessments and modifications to the analysis that will be necessary to improve accuracy in the future. aSee National Research Council, An Assessment of the SBIR Program, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2008, Chapter 4.
70 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION TABLE 4-1â Percentage of Sales by Type of Customer Customer Percent Domestic private sector 35 Department of Defense (DoD) 22 Prime contractors for DoD or NASA 11 NASA 17 Other federal agencies 3 State or local governments 0 Export markets 11 Other 0 SOURCE: NRC Phase II Survey, Question 5. and it is possible that technologies adopted for NASA are well placed for adop- tion by foreign space agencies. Licensee Sales and Related Revenues Licensee sales are one indicator of the extended effects of SBIR beyond the immediate awardee company and may capture important indirect commercial successes. However, the dataâwhere survey respondents report sales not made by their own companyâshould be treated with an additional degree of caution as respondents do not necessarily have as accurate information about another company as they have about their own. Licensing activity within the NASA SBIR program is limited. Only 3 projects at NASA report licensee sales greater than $0, with the largest being $300,000. 24 It thus seems reasonable to conclude that licensing is not a substantial mechanism for commercialization among NASA Phase II award recipients. Although the numbers of licenses are limited, their impact can be significant, as the case studies show. For example, TiNi Alloy licensed its pneumatic valve technology for use by Lee Inc. in meeting NASAâs needs for latching valves.25 TiNi Alloyâs Frangibolt(TM) has become a standard component on satellites (a shape memory alloy powered separation device), and was used for the Clem- entine Space Mission. The device has been used by TRW, the European Space Agency, and Lockheed-Martin. In addition, TiNiâs pinpuller was used on the Mars Global Explorer, and is scheduled for use in NASAâs STEREO program. 4.2.3â Additional Investment Funding Further investment in a recipient company related to the SBIR award project is another indication that the project work is of value. On average, NASA SBIR 24â NRC Phase II Survey, Question 4. 25â See TiNi Alloy case study in Appendix E.
SBIR PROGRAM OUTCOMES 71 TABLE 4-2â Sources of Additional Investments in SBIR Projects Source Average Dollars a. Non-SBIR federal funds 133,829 b. Private investment (1) U.S. venture capital 0 (2) Foreign investment 1,381 (3) Other private equity 3,825 (4) Other domestic private company 24,150 c. Other sources (1) State or local governments 13,812 (2) College or universities 966 d. Not previously reported (1) Your own company 100,450 (2) Personal funds 3,121 Total 281,534 NOTE: N=181 for NASA. See Table App-B-1 for a breakdown of Survey Response Rates. SOURCE: NRC Phase II Survey, Question 23. projects received almost $800,000 from non-SBIR sources, with over half of survey respondents (51.6 percent) reporting some additional funds for the project from a non-SBIR source (see Table 4-2). To put this in perspectiveâand again noting that sampling issues limit exact comparisons among agenciesâthe average additional investment reported by NASA firms is less than half that reported by projects at all agencies.26 Once again, this may reflect in part, the small size of the internal market at NASA. So far as the source of funds is concerned, NASA projects generated no ad- ditional investment from venture capital, and largely depended for funding on internal company sources (about 35 percent of the total) and non-SBIR federal funding (just under half) (see Table 4-2). The lack of venture capital funding is unsurprising, as NASA-focused firms are almost by definition not working in markets large enough to interest venture investors. Overall, 72 respondents (40 percent of all 181 NASA responses) reported non-SBIR funding greater than $0, with one firm receiving $15 million, and three others $1 million or more (although many more reported additional funding without committing to a specific dollar amount). Equity Sales Sales of equity by NASA SBIR awardees to others represent transfers of knowledge. Among the NRC Phase II survey respondents, activities to transfer equity centered on sales of technology rights to other domestic companies and 26â Again, direct comparisons across agencies are invalidated by survey response issues. However, such comparisons can indicate areas for possible concern and future research.
72 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION TABLE 4-3â Equity Sales of NASA Phase II Awardees to U.S. and Foreign Companies and Investors Sale of Company Technology Partial Sale of Sale of Merger Rights Company Company Final Ongoing Final Ongoing Final Ongoing Final Ongoing Focus of Interactions (%) (%) (%) (%) (%) (%) (%) (%) Interactions with 0 0 1 7 2 1 1 3 U.S. Companies and Investors Interactions with 0 0 1 3 0 0 0 0 Foreign Companies and Investors SOURCE: NRC Phase II Survey, Question 12. investors rather than sales abroad. Table 4-3 shows that much of this activity was still in process at the time of the survey. At the time of the survey, none of the awardee companies had been sold to foreign companies or investors. Equity sales are sometimes an essential element in commercialization strategy. In some cases, companies with the ability to commercialize are lo- cated outside the U.S., and they may require ownership as a condition for commercializing.27 Additional SBIR Funding Aside from providing non-SBIR funds, the federal government in many cases makes further investments via the SBIR program itself. This provides some indication that the technology and work completed to date are of continuing value to the agency. The NRC Phase II Survey asked respondents how many additional Phase I and Phase II awards followed each initial award, related to the original project. (See Table 4-4.) About 35 percent of respondents reported receiving at least one additional related Phase II award, and slightly under half reported at least one additional Phase I award.28 However, a few projects received many related awards: 7.5 percent of re- 27â For example, according to Brodd there are no volume lithium-ion battery manufacturers in the U.S. and this may influence commercialization strategies of small companies performing R&D in lithium-ion battery. Ralph J. Brodd, Factors Affecting U.S. Production Decisions: Why are there no Volume Lithium-Ion Battery Manufacturers in the United States? ATP Working Paper Series, Work- ing Paper 05-01, June 2005. 28â Note that these subsequent awards may have been made by other agencies.
SBIR PROGRAM OUTCOMES 73 TABLE 4-4â Related SBIR Awards Related Phase I Awards Related Phase II Awards Number of Awards Percent of Responses Number of Awards Percent of Responses 0 52.2 0 65.4 1 18.9 1 15.1 2 11.9 2 12.6 3 6.9 3 2.5 4 2.5 4 1.3 5 2.5 5 1.3 6 0.6 6 0.6 7 0.6 7 0.6 8 1.3 12 0.6 9 0.6 12 1.3 19 0.6 SOURCE: NRC Phase II Survey. Question 20 spondents reported at least 5 related Phase I awards, and 7 percent received at least three related Phase II awards. SBIR Impact on Further Investment The NRC Phase II Survey sought additional information about the impact of the SBIR program on company efforts to attract third party fundingâthe âhalo effectâ mentioned by some interviewees, who suggested that an SBIR award acted as form of validation for external inventors.29 The fact that 60 percent of NASA SBIR respondents reported no outside funding, and that none at all received venture funding, suggests that receiving a NASA Phase II SBIR award may have only a limited effect in improving the likelihood of external funding for these recipients.30 4.2.4â Small Company Participation and Employment Effects Growing employment is another indicator of commercial success. It also provides evidence that the program is supporting small business. The median size of company receiving SBIR awards is relatively smallâfar 29â For a discussion of the âhalo effectâ from awards by the Advanced Technology Program, see Maryann Feldman and Maryellen Kelley âLeveraging Research and Development: The Impact of the Advanced Technology Program,â in National Research Council, The Advanced Technology Program: Assessing Outcomes, Charles W. Wessner, ed. Washington, DC: National Academy Press, 2001. 30â See Chapter 4 in National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit.
74 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 60 Num ber of Responding Firm s 50 40 30 20 10 0 0 1-5 6-20 21-50 51-100 >100 Number of Employees at Time of Award FIGURE 4-4â Distribution of companies, by number of employees at time of award. SOURCE: NRC Phase II Survey, Question 16. NASA Figure 4-4.eps lower than the 500 employee limit imposed by the SBA. Most awards go to very small companies. Among NRC Phase II Survey respondents, about just over a quarter had between one and five employees at the time of award. A majority (64 percent) of respondents had 20 employees or fewer at the time of the Phase II award. The NRC Phase II Survey sought detailed information about the number of employees at the time of the award and at the time of the survey and about the direct impact of the award on employment. Overall, the survey data showed that the average employment gain at each responding firm from the date of the SBIR award to the time of the survey was 16 full-time equivalent employees. Of course, few companies that went out of business have responded to the survey, so this question is particularly skewed toward firms that have been at least somewhat successful. Table 4-5 shows that the percentage of companies with more than 50 em- ployees grew from 24.5 to 31.5 percent of respondents. 26.5 percent of firms reported that they were in the smallest size group (1 to 5 employees) at the time of their first Phase II award. Only 16 percent remained at that size at the time of the survey. All other size groups have increased their share of the reporting firms. One of the reporting firms has outgrown the SBIR program size limitation of 500 employees (it had 520 at the time of the survey), and the second and third largest firms had 370 and 366 current employees, respectively. The NRC Phase II Survey also sought to identify employment gains that were the direct result of the award. Respondents estimated that specifically as a result of the SBIR project, their firm was able to hire an average of 1.3 employ- ees, and to retain 1.4 more.31 31â NRC Phase II Survey, Question 16.
SBIR PROGRAM OUTCOMES 75 TABLE 4-5â Employment at Phase II Respondent Companies, at the Time of Award and at Time of Survey At Time of Award At Time of Survey Number of Employees Number Percent Number Percent 0 11 6.8 3 1.9 1-5 43 26.5 26 16.0 6-20 49 30.2 56 34.6 21-50 21 13.0 26 16.0 51-100 16 9.9 16 9.9 >100 22 13.6 35 21.6 Total 162 100 162 100 SOURCE: NRC Phase II Survey, Question 16. 4.2.5â Sales of Equity and Other Company-level Activities Company-level operations may offer another set of indicators for measuring commercial activity, as these may capture activities that indicate commercial value even absent sales. The NRC Phase II Survey explored whether SBIR awardees had finalized agreements or ongoing negotiations on various company- level activities. This information is summarized in Table 4-6. The impact of these activities is hard to gauge using quantitative assess- ment tools only. Box 4-3 illustrates how research conducted using SBIR funding seeded an entire generation of spin-off companies and joint ventures in a technol- ogy of potential significance for homeland security. TABLE 4-6â Company-level Activities Foreign Companies/ U.S. Companies/Investors Investors Finalized Ongoing Finalized Ongoing Agreements Negotiations Agreements Negotiations Activities (%) (%) (%) (%) a. Licensing agreement(s) 6 11 8 5 b. Sale of company 1 3 0 0 c. Partial sale of company 2 1 0 0 d. Sale of technology rights 1 7 1 3 e. Company merger 0 0 0 0 f. Joint Venture agreement 1 4 0 2 g. Marketing/distribution agreement(s) 7 5 6 2 h. Manufacturing agreement(s) 2 6 2 0 i. R&D agreement(s) 16 10 4 1 j. Customer alliance(s) 11 9 6 0 k. Other Specify____________ 3 4 1 1 SOURCE: NRC Phase II Survey, Question 12.
76 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION BOX 4-3 Detecting Toxins at a Distance: The Case of Intelligent Optical Systemsa Intelligent Optical Systems (IOS) has developed a system for using the entire length of a specially-designed fiber-optic cable as a sensor for the detection of toxins and other agents. This bridges the gap between point detection and stand- off detection, making it ideal for the protection of fixed assets.b Intelligent Optical Systems has leveraged its SBIR-supported research to develop subsidiaries and spin-offs. This activity has generated private investments of $23 million in support of activities oriented toward the rapid transition to commercially viable products. Since January 2000, IOS has formed two joint ventures, spun out five companies to commercialize various IOS proprietary technologies, and finalized licensing/ technology transfer agreements with companies in several major industries. Optimetrics manufactures and markets active and passive integrated optic com- ponents based on IOS-developed technology for the telecommunication industry. Maven Technologies was formed to enhance and market the Biomapper technolo- gies developed by IOS. Optisense manufactures and distributes gas sensors for the automotive, aerospace, and industrial safety markets, and will be providing H2 and O2 optical sensor suites designed to enhance the safety of NASA launch operations. Optical Security Sensing (OSS), which is IOSâs newest spin-off com- pany, was formed to commercialize chemical sensors for security and industrial applications. IOS currently employs 40 scientists, and almost 80 percent of its revenues come from non-SBIR sources. The company currently holds 13 patents, with an ad- ditional 13 applications pending. aSee âSBIR and STTR Success Story for Intelligent Optical System,â accessed at <http:// grants.nih.gov/grants/funding/sbir_successes/160.htm>. bPoint detection means the contaminant comes into physical contact with the sensor and it is analyzed. In standoff detection, the sensor sees the contaminant at a distance and recog- nizes it, but the contaminant never comes in contact with the sensor. Other companies that were at least initially strongly SBIR dependent have also utilized the spin-off approach. Creare, Inc., has spun off more than 12 com- panies, which together generate more than $250 million in annual revenues and employ 1,500 people.32 Luna Innovations has created five new companies since 2000 while opening additional branches in Charlottesville, Danville, Roanoke, Hampton Roads, and Mclean, Virginia and Baltimore, Maryland.33 32â See Creare, Inc., case study in Appendix E. 33â See Luna Innovations, Inc., case study in Appendix E.
SBIR PROGRAM OUTCOMES 77 4.2.6â Commercialization: Conclusions While accepting the view that there is no single, simple metric for determin- ing the commercial success of an early stage R&D program such as SBIR, numer- ous metrics do provide the basis for making a broad determination of commercial outcomes at SBIR. These data, taken together, support the view that while the NASA SBIR pro- gram has traditionally had a strong commercial focus, with considerable efforts to bring projects to market, overall success in this area has been elusive. The number of major commercial successes has been few, and while this is normal for early stage high-risk projects, the fact that no NASA project reports more than $15 million in related sales does indicate the challenges that NASA companies face. Despite the challenges, the overall commercialization rate for the NASA SBIR program has been comparable to those identified using other data at other agencies. The NRC Phase II survey respondents indicated that 46 percent of NASA SBIR projects had reached the market. These conclusions broadly align with views from within the agency: according to the NRC Project Manager Sur- vey, about 35 percent of projects are believed to have commercialized. 34 Still, the small number of big winners means that overall commercialization from the program has been limited. Average commercialization is well below $1 million per project.35 The structural difficulties facing commercialization for NASA-funded SBIR projects have been a major factor in the 2006 restructuring of the SBIR program. The restructuring was also driven by a recognition at NASA that the number of projects reporting sales made to NASAâthe adoption of SBIR technology by NASAâwas also relatively low. Only 17 percent of projects with sales report that these went to NASA. The impact of this switch from a commercialization focus to an agency mission focus has been noted by companies interviewed for this report. For ex- ample, DCT (a small Ohio software maker), finds that NASA is, in its experience, increasingly focusing the SBIR program on specific space-related needs that have with little commercial significance. Narrowly focused topics with specific mission objectives significantly limit opportunities for commercialization, in the view of DCT.36 34â NRC Project Manager Survey, Table App-D-19. 35â NRC Phase II Survey, Question 4. From the 181 projects that responded to the survey, 78 reported a year of first sale and only 74 reported sales greater than zero. Their average sales were $1,154,156. Over half of the total sales dollars were due to 7 projects, each of which had $3,200,000 or more in sales. The highest reporting project had $15 million in sales. Similarly, of the 19 projects that reported a year of first licensee sale, only 3 reported actual licensee sales greater than zero. Their average sales were $127,000. Ninety-nine percent of the total dollars was due to one project, which had $30,000 or more in licensee sales. 36â is worth noting that the shift in emphasis will have negative effects as well as positive ones. It At DCT, earlier awards funded nearly 100 percent of company R&D, which implies that DCT will
78 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION Internal tracking capabilities at NASA are very limited. The last update to the internal outcomes spreadsheet apparently halted before data for the FY1998 Phase II awards could be entered. Better utilization of the NASA SBIR program by the agency must be matched by improved and timelier tracking of outcomes. 4.3â AGENCY MISSION Agency missions vary substantially by agencyâindeed, each agency has a unique mission by design. Thus, each agency must address the extent to which the SBIR program supports its mission. However, some more general observa- tions can be made. An assessment of the extent to which SBIR supports an agencyâs mission can be divided into two areas: â¢ Procedural alignmentâthe extent to which the procedures of the agency SBIR program are aligned with the needs of the agency. â¢ Program outcomesâthe extent to which outcomes from the program have the effect of supporting the agency mission. In addition, the subjective views of program managers can be introduced as an important source of evidence about program effectiveness. It is important to note that the different missions of the agencies mean that some agencies define agency mission support more narrowly, or at least have much tighter metrics for assessing this element of the program. In particular, the procurement agenciesâprimarily DoD and NASAâassess contribution to agency mission primarily against the extent to which the agency itself uses out- puts from the SBIR program. In contrast, the nonprocurement agenciesâNIH, NSF, and, to a great extent, DoEâsee support for mission much more broadly: for NIH, for example, support for mission can be construed as anything that im- proves medical knowledge or public health. 4.3.1â Procedural Alignment of SBIR Programs and Agency Mission at NASA A procedural assessment reviews the steps taken by each agency program to ensure that the design and procedures of their SBIR program are aligned with the needs of the agency. be increasingly unable to continue making software advances. In particular, they see themselves less able to compete with European companies where governments support research and technical implementation.
SBIR PROGRAM OUTCOMES 79 BOX 4-4 NASA Mission NASAâs mission is to pioneer the future in space exploration, scientific discovery, and aeronautics research. SOURCE: NASA Web site. Access at <http://www.nasa.gov/about/highlights/what_does _nasa_do.html>. Topics and Solicitations To align SBIR with its agency mission, NASA must ensure that the topics that guide applicants (and at NASA define the limits of what can be funded) are themselves aligned with agency mission needs. These topics are published in an annual solicitation of proposals. Within each solicitation, specific subject areas of interest are defined by individual topics and subtopics. These can be focused tightly on a specific problem or requirement, or they may broadly outline an area of technical interest to the agency. Aside from NIH, which expressly indicates that its topics are guidelines, and not mandatory limits or boundaries on research that could be funded, all the SBIR funding agencies use topics to specify the technical boundaries of the research they are prepared to fund. In doing so, they are specifically delimiting areas of technical interest to the agency. At NASA, the structure of the SBIR solicitation, and the technical taxono- mies used to define topics and subtopics, parallel the structure and taxonomies of the Mission Directorate technology program roadmaps. This supports alignment between SBIR projects and agency mission. This is prima fascia confirmation that the SBIR programs support agency mission: Unless there is evidence that agencies are generating topics that are not aligned with the agency missionâand our analysis and interviews with staff and awardees found no trace of thisâthe use of topics and solicitations indicates that agencies are working to ensure that awards are aligned with the stated scientific and technical needs of the agency. However, what is less clear is how the alignment of topics meshes with the changing management priorities. For SBIR, the pre-2006 management structure strongly favored the priorities of the centers and individual researchers, over the priorities of the Mission Directorates and Headquarters. Thus while the SBIR program pre-2006 was closely aligned with NASA mission needs, these were not always the priorities of the Mission Directorates or the technology programs that eventually funded the take-up (âinfusionâ as NASA calls it) of SBIR-funded technologies into NASA space missions.
80 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION Award Selection Process The selection of awards can also support an agencyâs mission, to the extent that the process reflects the agenciesâ priorities. A wide range of awards proce- dures are used at the various agencies and these may differ substantially even between components of the same agency. For example, DoD, Army, and Navy use different approaches, staff, and methodologies for selecting awardees. At NASA, evaluation of Phase I proposals is performed by NASA scientists and engineers at the center(s) identified in the Solicitation as responsible for the applicable subtopic. While initial rankings are developed at the Centers, where the relevant tech- nical expertise resides, final decisions are made by the NASA Source Selection official after input form the Mission Directorates. A high center ranking does not guarantee funding. However, this process too has changed since 2006; Mission Directorate input is now much stronger. According to Parminder Ghuman of the Science Mission Directorate, of the 95 Phase I proposals recommended for fund- ing by the centers, 86 eventually received an award after MD review.37 Mr. Ghuman also notes that the commercialization section of proposals is ac- corded relatively little weight in the NASA selection process. This is unsurprising given the limited commercial success of NASA-funded projects in general. 38 Phase II selection is strongly influenced by the report of NASAâs Phase I technical representative (the COTR); these staff a responsible for managing the Phase I and help to determine whether Phase II funding is appropriate. After 2005, increased efforts have been made to ensure that the COTR for a Phase I project is also likely to be the official running the section of the Mission Directorateâs technology program where any post Phase II work will be funded. NASA has now also formalized the previous practice of providing additional SBIR awards to centers that provide matching fundsâa clear sign that the pro- posed project is a high priority item. The new program is called Phase IIe. There are no data either on the number of projects that were funded this way in the past, or on the impact of formalizing this part of the selection program. 39 Overall, then, the selection process is designed to ensure that funded pro- posals are in all cases aligned with both the broad mission of the agency and the specific technical needs of the agency in designated areas. Those needs are written by the project managers who will be responsible for meeting them, and thus have every incentive to ensure that the topics published are those that meet their most important needs. Interviews with NASA staff also determined that SBIR funds can constitute a significant proportion (often more than 50 percent) of the funding available to a program manager for immediate and flexible deployment. Most NASA 37â Interview with Parminder Ghuman, Science Mission Directorate, December 7, 2007. 38â Ibid. 39â Interview with Paul Mexcur, NASAâs SBIR & STTR Program Manager, June 2005.
SBIR PROGRAM OUTCOMES 81 fundsâlike DoDâare committed years in advance, so flexible funding like SBIR can be highly prized. 4.3.2â Program Outcomes and Agency Mission In contrast to the discussion above, program outcomes for agency mission are more difficult to assess. All of the methodological difficulties in assessing outcomes discussed at the beginning of this chapter apply here; moreover, (unlike commercialization) there are few widely accepted benchmarks. Like other agencies, NASA maintains a list of âsuccess stories,â describing SBIR awards that meet congressional goals. Some of these are focused on agency mission. However, the stories themselves, while illustrative of the power of the program to help develop new technologies, are of variable quality. The NRC Phase II Survey data suggest that NASA SBIR has found limited take-up within the agency. Only about 10 percent of NASA Phase II awards have attracted NASA Phase III funding for either further development or purchase of product or service. However, the NRC Project Manager Survey of agency technical managers (COTR) provided an additional source of important information in this area. These agency staff are in charge of the research areas within which SBIR awards are made. The survey sought to measure the quality of SBIR research from the perspective of technical staff who managed both SBIR and non-SBIR programs. The NRC therefore surveyed all COTRs with responsibilities for SBIR projects at NASA. 184.108.40.206â Results from the NRCâs Project Manager Survey COTRs tend to be involved with the SBIR Phase II projects to which they are assigned. This involvement gives them a unique basis for evaluating the quality, usefulness, and value of each project to NASA. We used the survey of project managers to develop measures of these dimensions of program outcomes. Program managers at other agenciesâprimarily DoD but also DoEâwere also included in the survey. Project Quality COTRs were asked to rank their SBIR projects in terms of quality on a scale of 1 to 10. The mean score was 6.98 (with a standard deviation of 1.846) as seen in Table 4-7. This metric however does not adjust for âtoughness of the grader.â To compensate for the fact that different project managers have different standards in mind when evaluating a research project, COTRs were also asked to rate the average quality of other research projects conducted for their research unit/office. They gave this other research a mean score of 7.45 (standard deviation of 1.268). (See Table 4-8.)
82 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION TABLE 4-7â Ratings for SBIR Projects Measure of Quality of SBIR Project Total Sample NASA Mean Score 6.93 6.98 Standard Deviation 2.072 1.846 Median Score 7 8 Sample Size 513 82 SOURCE: NRC Project Manager Survey. TABLE 4-8â Ratings for Non-SBIR Research Projects Measure of Average Quality of Research (non-SBIR) Total Sample NASA Mean Score 7.29 7.45 Standard Deviation 1.594 1.268 Median Score 8 7 Sample Size 513 82 SOURCE: NRC Project Manager Survey. A comparison of the scores for the SBIR and non-SBIR projects allows us to gauge how COTRs view the relative quality of the SBIR projects. The differ- ences in means between the score for the sample of SBIR and non-SBIR projects are given in Table 4-9. For the total sample of NASA SBIR projects, the SBIR projects were on average somewhat lower.40 Linkage to Research Mission The NRC Project Manager Survey also addressed the usefulness of the re- search for the research unit/office, and in particular the extent of linkages between the SBIR research project and other research conducted by the agency. About 44 percent of NASA responses indicated no linkage, compared with about 30 per- cent at other agencies. Approximately a quarter of NASA managers encouraged firms to continue their research through further SBIR awards, or through other agency-funded research projects, compared with nearly 50 percent at other agen- cies. About 10 percent of projects were âblind alleys.â (See Table 4-11.) Overall, these responses suggest that while COTRâs at NASA do see value in SBIR projects, they have seen significantly less value than in other similar re- search projects, and they have made less effort to use the results of SBIR projects in other agency/unit research. 40â As judged by NASA project managers, the difference in the measure of project quality was 0.476. Using a statistical ât-test,â these differences are statistically significant (at the .01 level). In interpret- ing this statistic, it is worth keeping in mind that the magnitude of the difference in perceived research quality is rather smallâon a scale from 1-10, the difference is only .476.
SBIR PROGRAM OUTCOMES 83 TABLE 4-9â Mean Difference in Scores SBIR Quality Minus Average Non- SBIR Project Quality Measure of Difference in Scores of Project Quality Total Sample NASA Mean Difference in ScoreâSBIR Quality Minus Average Non-SBIR â.364* â.476** Project Quality Standard Deviation 2.15 2.10 NOTES: *Statistically significant at the .01 level; **Statistically significant at the .05 level. SOURCE: NRC Project Manager Survey. Comparative Value of SBIR Projects Another way to look at the value of SBIR-funded research from the agencyâs perspective is to consider the utility of money spent on SBIR projects compared with money spent on other R&D. As Table 4-12 shows, about 30 percent of NASA respondents indicated that SBIR projects gave fewer mission benefits than the average dollar spent on other contracts, while 18 percent thought SBIR projects gave more benefits. Fifty per- cent saw the benefits as equivalent. Once again, NASA COTRs had a less positive view of SBIR projects than did those at other agencies surveyed by the NRC. Abundance of Fundable SBIR Proposals A majority of NASA COTRâs (56 percent) believe that, in general, their research office/unit receives more good SBIR proposals than they can fund (see Table 4-13). About one-fifth reported more money on hand than high quality TABLE 4-10â Distribution of Scores SBIR Quality and Average Non-SBIR Project Quality (NASA) Quality of Research Average Quality of Research (SBIR) (Non-SBIR) Score of Number of Number of Project Quality Responses Percent Responses Percent 1 0 0.00 0 0.00 2 2 2.44 0 0.00 3 3 3.66 0 0.00 4 4 4.88 1 1.22 5 7 8.54 7 8.54 6 10 12.20 9 10.98 7 20 24.39 19 23.17 8 20 24.39 31 37.80 9 12 14.63 13 15.85 10 4 4.88 2 2.44 TOTAL 82 100.00 82 100.00 SOURCE: NRC Project Manger Survey.
84 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION TABLE 4-11â Effect of SBIR Projectâs Research on Your Research Unit Number of Responses Percent Other Other Agencies NASA Agencies NASA No, this project was a separate project, and the 130 36 30.2 43.9 knowledge generated by this SBIR contract has had no impact on the other research we conduct or sponsor. Yes, this project produced results that have 127 21 29.5 25.6 been useful to us, and we have tried to follow up on the ideas initiated in this SBIR contract by encouraging the firm to apply for additional SBIR awards. Yes, this project produced results that have 208 26 48.3 31.7 been useful to us, and we have tried to follow up on the ideas initiated in this SBIR contract in other research we conduct or sponsor. Yes, but this project found a blind alley, so we 41 8 9.5 9.8 have not followed up on this line of inquiry. Total Sample 431 82 N/A N/A NOTE: Multiple answers were permitted. SOURCE: NRC Project Manager Survey. proposals. This suggests that the marginal value of increased SBIR funding is likely to be high. This response helps to confirm the view that SBIR in general receives high quality applications, and that these are more applications that could be funded than there is funding available. TABLE 4-12â Comparative Dollar Value of Projects Number of Responses Percent Other Other Dollar Value Agencies NASA Agencies NASA SBIR project had fewer benefits for your 119 26 27.6 31.7 agencyâs mission than the average dollar spent on other contracts you sponsor SBIR project had more benefits for your 128 15 29.7 18.3 agencyâs mission than the average dollar spent on other research contracts you sponsor Same Benefits 184 41 42.7 50.0 TOTAL SAMPLE SIZE 431 82 100.0 100.0 SOURCE: NRC Project Manager Survey.
SBIR PROGRAM OUTCOMES 85 TABLE 4-13â Relative Number of Fundable SBIR Projects Number of Responses Percent Other Other Agencies NASA Agencies NASA More fundable proposals than can fund 180 40 65.2 56.3 About the right number of proposals 66 16 23.9 22.5 Fewer fundable proposals than can fund 30 15 10.9 21.1 Total Sample Size 276 71 100.0 100.0 SOURCE: NRC Project Manager Survey âOwnership Effectâ One important finding from the NRC Project Manager Survey was that early âownershipâ of an SBIR project by agency staff leads to much more positive views of project outcomes. This is in itself not a surprising findingâprojects where managers were involved at the design stage are more likely to align with their larger research agendas. However, the size of this effect was substantial. The COTR sample was broken into two subsamples: project managers with a potentially strong degree of ownership in the project; and those with less potential for ownership. In this context, âownershipâ means that the project manager had a potential stake in the project as demonstrated either by being involved in defining the topic or being involved with the recipient firm before the Phase I proposal. Table 4-14 shows that managers with ownership had a much higher opinion of their SBIR projects than those with more of a connection to their projects. â¢ The ownership managers has a much more favorable view of research quality. â¢ Three quarters of the nonownership group saw the SBIR-funded research as ânot useful,â compared with under 30 percent for the ownership group. TABLE 4-14â Analysis of Ownership EffectsâNASA Area of Interest Ownership Group (n=55) Remaining Project Managers (n=27) Research Quality â.236 â.963 Usefulness of Research a. No, not useful 1 6 29.09% 2 0 74.07% b. Yes, more SBIR 1 6 29.09% 5 18.52% c. Yes, general follow-up 2 3 41.82% 3 11.11% d. Yes, but blind alley 8 14.54% 0 0.00% Mission Benefits (Q6) a. More than average 20.00% 14.81% b. Same as average 52.73% 44.44% c. Less than average 27.27% 40.74% SOURCE: NRC Project Manager Survey.
86 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION â¢ Almost four times the share of ownership respondents indicated that they were looking for non-SBIR support for subsequent research or acquisition. â¢ The ownership group also identified stronger mission benefits, though the differences between the groups were not so pronounced on this area. Cause and effect are not identified here. However, evidence from elsewhereâ e.g., the NRC Phase III Conference Reportâindicates that ownership effects of this kind can be encouraged by agency management and policies, and also that their impact persists into Phase III. Project Managers and Phase III Project managers were asked whether their projects received Phase III fund- ing from NASA. As shown in Table 4-15, slightly fewer than 16 percent of Phase II Projects got additional funding from NASA. More than half did not, and just over a quarter of respondents did not know. This finding quite closely tracks results from the NRC Phase II Survey. This Phase III funding was evenly distributed between direct procurement, incorporation of the technology into a larger system, and additional non-SBIR research funding. These results were broadly comparable with NRC COTR Survey data from other agencies (including of course DoD). They also indicate significant room or improvement in the linkage between the program managers who fund SBIR research and outcomes from that research, both at NASA and at other agencies. Overall, these results suggest that NASAâs SBIR program will require substantial changes to achieve its recent increased emphasis on âspin-inâ outcomes. Project Manager Involvement Overall, NASA projects managers indicated extensive involvement in the projects they funded (see Table 4-16). However, the proceedings from the NRC Phase III Conference and other agency interviews at NASA and DoD indicate that the timing of that involvement is important.41 About 40 percent of NASA respondents reported that they became involved with the SBIR project before Phase Iâi.e., during the topic development stage. However, this was true for almost 70 percent of project mangers reporting from other agencies. This difference may possibly result in part form the extensive efforts made by DoD since the late 1990s to improve the alignment between topic development, SBIR program managers, and the acquisitions community. The 2006 structural changes in the SBIR program are partly designed to address this specific issue. The actual role of the project manager also appears to vary from project-to- 41â See the conference remarks by Michael McGrath of the U.S. Navy summarized in National Re- search Council, SBIR and the Phase III Challenge of Commercialization, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2007, pp. 59-62.
SBIR PROGRAM OUTCOMES 87 TABLE 4-15â SBIR Project Received Phase III Funding from Your Agency Number of Responses Percent Other Other Phase III Funding from your AgencyâForm Agencies NASA Agencies NASA Direct procurement of the product of this SBIR 10 5 2.3 6.1 Procurement through incorporation of the result 15 3 3.5 3.7 of this project into system Further non-SBIR R&D funding 48 5 11.1 6.1 No Phase III from agency 253 46 58.7 56.1 Unknown 105 23 24.4 28.0 Total 431 82 100.0 100.0 SOURCE: NRC Project Manager Survey. TABLE 4-16â When Did Project Manager Become Involved in Project? Number of Responses Percent Other Other When Involved in SBIR Project? Agencies NASA Agencies NASA Before Phase I 300 34 69.6 41.5 After Phase I, Before Phase II 81 37 18.8 45.1 After Phase II started, Before Phase II 42 11 9.7 13.4 completed After Phase II completed 8 0 1.9 0.0 Total 431 82 100.0 100.0 SOURCE: NRC Project Manager Survey. project, and from project manager to project manager. Table 4-17 summarizes the project managersâ role(s) with respect to the particular SBIR project in question. Over 97 percent of NASA respondents claimed a âtechnicalâ role, while only 5 percent claimed a âfinancialâ role and 12 percent claimed a âcommercialâ role. Finally, NASA project management appears more closely focused on techni- cal issues, and much less on financial controls. However, it is unclear whether this distinction has any impact on commercialization or other outcomes. Evidence from the NRC Project Manager Survey: Conclusions The NASA SBIR project managers in our sample appear to be, for the most part, engaged with the SBIR program. Many were involved with their projects early and often. In general, they ranked the quality of SBIR research as similar to that of non-SBIR research. Most believed their projects ere useful to NASAâs
88 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION TABLE 4-17â Role of Project Manager with Respect to this SBIR Project Number of Responses Percent Other Other Role of Project Manager Agencies NASA Agencies NASA Technical 413 80 95.8 97.6 Financial 105 4 24.4 4.9 Commercialization Assistance 74 10 17.2 12.2 Other 50 5 11.6 6.1 Total Sample Size 431 82 N/A N/A NOTE: Multiple responses permitted. SOURCE: NRC Project Manager Survey. mission, and the majority of respondents reported that they had more high qual- ity SBIR proposals than they could fund. NASA SBIR project managers seem uninvolved in Phase III activity. 220.127.116.11â Case Studies and Agency Mission A second avenue for gathering important information about the impact of the SBIR program on NASAâs mission lies in the use of case studies. These are collected in Appendix E, but it is also worth highlighting summaries of some of the more notable cases below. It should be understood that these cases are spe- cifically designed to highlight cases where SBIR did indeed make a significant difference to NASAâs space mission. Restoring the Hubble Space Telescope: The CreareâNCS Cryocooler The NCS Cryocooler was used on the Hubble Space Telescope to restore the operation of the telescopeâs near-infrared imaging device. This was a mission- critical adjustment to the operations of the Hubble, in light of the failure of a key component. The Independent Space Science Board Report on the project concluded that On the technical side, the cryocooler system had been successfully flown and tested in space; and great care had been taken to characterize the thermal, me- chanical, and electro-optical changes that NICMOS had undergone late in 1998 and during the rapid warm-up of January 1999. We now know a great deal more about the technical issues surrounding the feasibility of successfully prolong- ing the life of NICMOS through the installation of a cryocooler. We commend the GSFC and Creare teams for developing and successfully flight testing the cryocooler on such a short time scale.42 42â Report of the Independent Science Review: NICMOS Cryocooler, March 4-5, 1999. <http://www. stsci.edu/observing/nicmos_cryocooler_isr1999.html>.
SBIR PROGRAM OUTCOMES 89 The technologies that were required to build that cryogenic refrigerator started being developed in the early 1980s as one of Creareâs first SBIR projects. Over 20 years, Creare received over a dozen SBIR projects to develop the tech- nologies that ultimately were used in the cryogenic cooler. Additionally, Creare has been awarded âPhase IIIâ development funds from programmatic areas that were ten times the magnitude of the cumulative total of SBIR funds received for fundamental cryogenic refrigerator technology de- velopment. However, until the infrared imaging device on the Hubble telescope failed due to the unexpectedly rapid depletion of the solid nitrogen used to cool it, there had been no near-term application of the technologies that Creare had developed. Monitoring the Space Shuttleâs Surface: The Wireless Data Acquisition Project In light of the Columbia disaster, NASA became aware of the need to de- velop new technologies that could wirelessly transmit data from key points on the surface of space vehicles to instruments inside the vehicle, for download to mission controllers on Earth. Figure 4-5 shows how the Johnson Space Center used SBIR to develop some of the critical technologies used for the acquisition of wireless dataâa key component in monitoring space flight. Data developed using this technology development program was used to monitor the leading edge of space shuttle wingsâa new requirement for the shuttle after the Columbia tragedy. The chart shows that SBIR technologies were infused into the overall wire- less instrumentation system at several points in its 6-year development cycle. 4.3.3â Conclusions: Agency Mission The case studies and data from the NRC Project Manager Survey both show that NASA has in the past successfully used SBIR to develop technologies that were critically important for some NASA missions. However, NASA itself determined that the existing approach, driven largely from the bottom up by the needs and interests of specific centers, was not work- ing well enough. Accordingly, as detailed in Chapter 5 (Program Management) NASA started to change the way SBIR works at the agency specifically to im- prove outcomes related to supporting agency mission. The new approach is designed to ensure a tighter alignment between the needs of the agency, as expressed through requirements and roadmaps developed by the Mission Directorates, and the selection of topics and proposals. In effect, NASA is seeking to switch the primary emphasis of the SBIR pro- gram from commercialization to support for agency mission. One challenge that Mission Directorate staff are now focusing on is that there are currently no met- rics in place to help assess the success or otherwise of this change of emphasis.
90 SCAT 1995 On-Orbit Data Recording Extreme Environment SBIR WSDS 1996 Ultra-low Power GPSCON Low-rate RF Sensors 1997 Spread-Spectrum WLAN Relaying RF Network WDAS MicroWIS SBIR External RTD 1998 High-speed WLAN Radio Medium-rate Real Programmable Rate Modular Architecture Time Sensors High-rate Data Recorder with RF Interface 1999 SWIS WATS MicroTemp High Accuracy Data Real-Time EVA Deployed Acq. Synchronization 10yr Lifetime 2000 MicroSGU Relaying Network FPP IWIS Very High-rate Wireless Sensors Synchronization 2001 Large flash MicroTAU ELMWIS memory 2002 Wideband MicroTAU Wireless Data Acquisition Systems Local Data Processing Local Data Processing 2003 MMA Long Duration Long Life Long Life Operation ISS Power Continuous Data 2004 Acq. Mode Wing Leading Edge UltraWIS SBIR EWIS Extreme 2005 Standards-based Low Power Impact Integration IDAA STTR DIDS SBIR Detection FIGURE 4-5â Technology Development Tree for Wireless Data acquisition. SOURCE: John Saiz, Johnson Space Center, Houston, TX. NASA Figure 4-5.eps Set for landscape, smallest type is 5.72-pt.
SBIR PROGRAM OUTCOMES 91 4.4â SUPPORT FOR SMALL, WOMAN-OWNED, AND DISADVANTAGED BUSINESSES 4.4.1â Support for Woman- and Minority-owned Firms Support for woman and disadvantaged persons is one of the four primary congressional objectives for the SBIR program. In the context of SBIR, NASA has used, as its primary metric, the extent of support for woman- and minority- owned businesses. Award Patterns The trends for both Phase I and Phase II awards at NASA are relatively flat for both woman- and minority-owned businesses, with a recent positive shift for women. The absence of detailed applications data for woman- and minority-owned businesses at NASA means that it is not possible at present to determine whether these trends are the result of an increase in the number of applications from woman- and minority-owned firms, improved success rates, a combination of both, or some other factor. Woman-owned Firms 30 Minority-owned Firms Either Percent of All Phase I Awards 25 20 15 10 5 0 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 Year NASA Figure 4-6.eps FIGURE 4-6â NASA SBIR Phase I awards, by demographic group, 1992-2005. SOURCE: National Aeronautics and Space Administration.
92 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 30 Woman-owned Firms Minority-owned Firms 25 Either Percent of All Phase II Awards 20 15 10 5 0 1997 1998 1999 2000 2001 2002 2003 2004 FIGURE 4-7â NASA SBIR Phase II awards, by demographic group, 1997-2004. SOURCE: National Aeronautics and Space Administration. NASA Figure 4-7.eps 4.4.2â Small Business Support At one level, the SBIR program obviously provides support for small busi- ness, in that it gives funding only to businesses with no more than 500 employ- eesâthe SBA definition of a small R&D business. This positive view is strongly reflected in the case studies undertaken by the NRC for this volume. Companies utilize SBIR for a range of purposes, and founders and senior staff interviewed for the NRC study were in many cases will- ing to credit a government funding program at least partially for their success. However, it has been less clear whether SBIR has provided additional sup- port for small business, or simply aggregates existing small business research dollars under the programâs umbrella. Project-level Impacts One way of measuring SBIRâs impact is to ask awardees whether their projects would have been implemented without SBIR program funding. Data in Figure 4-8 from the NRC Phase II Survey strongly suggest that SBIR provides funding that plays a determinant role to most of the projects that receive it. According to the NASA respondents, about 68 percent of projects would likely not have proceeded at all without SBIR.43 This finding reflects the known 43â NRC Phase II Survey, Question 13.
SBIR PROGRAM OUTCOMES 93 Definitely yes (3%) Probably yes (15%) Definitely not (32%) Uncertain (14%) Probably not (36%) FIGURE 4-8â Would the project proceed without SBIR funding? (Percent of respondents). NASA Figure 4-8.eps SOURCE: NRC Phase II Survey, Question 13. difficulties in funding high-risk early-stage research in all scientific fields. SBIR seems to provide critical funding necessary to fund many early-stage projects. Respondents also indicated that many of the 18 percent of projects that were âdefinitelyâ or âlikelyâ to have continued in the absence of SBIR funding would have had significant delays and other changes.44 Forty-three percent of these re- spondents noted that the projectâs scope would have been narrower. Seventy-six percent of projects that would have continued would have been delayed, and 48 percent expected the delay would have been at least 24 months.45 In short, SBIR has a profound effect on project initiation for the high tech projects that it funds and, in turn, on the commercialization outcomes of these projects. Impacts Different Types of Companies Professor Irwin Feller has suggested a typology to describe five common profiles of companies supported by SBIR funding. This typology captures the critical differences in company capabilities and aspirations. 44â Ibid. 45â NRC Phase II Survey, Question 15.
94 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 1. Start-up firm. This is a new firm, typically without marketable prod- ucts, and usually with minimal funding and limited personnel resources. 2. R&D Contractor. As described by Reid Cramer, these firms make a strategic choice to specialize in the performance of R&D rather than in marketing products or services.46,47 3. Technology Firm. These firms have developed a core technology, which is then deployed into products and services. 4. Scientific firm. These businesses are described by Reid Cramer as âfirms [that] are generally small and were founded by scientists to explore whether a particular research areas can generate ideas or products that might at- tract investorsâ48 5. Transformational firm. These companies start out as highly (or par- tially) dependent on SBIR or other government R&D contracts, which they use to develop a product that turns out to have considerable commercial value. This leads the company to become a production-oriented commercial vendor, with a concomitant decrease in the role of SBIR on the firm progression. Examples of these firm types can be seen in the case studies of SBIR awardees. Further research in this area may help to establish better, how the NASA SBIR program supports these different kinds of businesses, and businesses at different stages of development. Conclusions SBIR supports small high technology businesses at a time when other sources of financial support are especially difficult to find. Businesses use these funds for a variety of purposes, in pursuit of several distinct strategies. Awards data also indicate the role of woman- and minority-owned firms at NASA. It would be helpful if further analysis in this area focused on the role and incidence of minority and female Principal Investigators (PI), as these positions my be an important stepping stone on the path to forming the kinds of companies that can qualify for SBIR awards. 4.5â SBIR AND THE EXPANSION OF KNOWLEDGE Quantitative metrics for assessing knowledge outputs from research pro- grams are well-known, but far from comprehensive. Patents, peer-reviewed pub- 46â See for example Polymer case study in National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit. Polymer was in its early years primarily a con- tractor, but has since developed many cutting-edge products of its own. 47â See Cramer, Reid, âPatterns of Firm Participation in the Small Business Innovation Research Program in Southwestern and Mountain States,â in National Research Council, SBIR: An Assessment of the Department of Defense Fast Track Initiative, op. cit. 48â Ibid.
SBIR PROGRAM OUTCOMES 95 TABLE 4-18â Patents from NASA SBIR Projects Number of Patents Applied Received 0 121 128 1 30 25 2 8 7 3 2 1 Total 161 161 SOURCE: NRC Phase II Survey, Question 18. lications, and, to a lesser extent copyrights and trademarks, are all widely used metrics, and are discussed in detail below. However, these metrics do not capture the entire transfer of knowledge in- volved in programs such as SBIR. It is therefore quite important to understand that the quantitative metrics discussed below are only an indicator of the expansion of knowledge; they reflect that expansion but do not fully capture it. In particular, they say little about the impact of that knowledge. 4.5.1â Patents According to the Small Business Administration, small businesses produce 13 to 14 times more patents per employee than large patenting firms. These pat- ents are twice as likely as large firm patents to be among the one percent most cited.49 The data show that 40 projectsâabout 25 percent of respondentsâreported at least one patent application, and that 33 projects (20 percent) generated at least one patent. No projects generated more than three applications, and only one received three or more patents. 4.5.2 Scientific Publications Publication in peer-reviewed journals and conference proceedings are a standard method for disseminating scientific knowledge. Several case study in- 49â Accessed on May 16, 2007, at <http://app1.sba.gov/faqs/faqindex.cfm?areaID=24>. Drawing on seminal empirical research, Acs and Audretsch found that small businesses have comparatively higher rates of innovationâspecifically, that âthe number of innovation increases with increased industry R&D expenditures but at a decreasing rate. Similarly, while the literature has found a some- what ambiguous relationship between concentration and various measures of technical change, our results are unequivocalâindustry innovation tends to decrease as the level of concentration rises.â See Zoltan J. Acs and David B. Audretsch, âInnovation in Large and Small Firms: An Empirical Analysis,â op. cit.
96 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION terviewees noted that publication in peer-reviewed journals was an essential part of the firmâs work, and provided valuable exposure. The NRC Phase II Survey asked respondents about this aspect of these ac- tivities as well. (See Table 4-19 for a summary of the results.) About 40 percent of respondents reported at least one peer-reviewed publication. Five projects reported at least ten such publications. These data fit well with case studies and interviews, which suggested that some SBIR companies are proud of the quality of their research. Publications are featured prominently on many company Web sites. Publications therefore fill two important roles in the study of SBIR programs: â¢ First, they provide an indication of the quality of the research being conducted with program funds. In this case, more than half of the funded projects were of sufficient value to generate at least one peer-reviewed publication. â¢ Second, publications are themselves the primary mechanism through which knowledge is transmitted within the scientific community. The existence of the articles based on SBIR projects is therefore direct evidence that the results of these projects are being disseminated widely, which in turn means that the congressional mandate to support the creation and dissemination of scientific knowledge is being met. We note that like other SBIR agencies, NASA does not have evaluation programs in place to compare knowledge effects within and outside the SBIR program. Tracking Knowledge Dissemination by Citation Analysis Citation studies have been used extensively to show the transfer of knowl- edge from federally funded projects to others outside the walls of the funded TABLE 4-19â Publications from NASA SBIR Phase II Awards Number of Publications Number of Responses 0 97 1 20 2 19 3 6 4 4 5 8 8 2 10 3 12 1 30 1 Total 161 SOURCE: NRC Phase II Survey, Question 18.
SBIR PROGRAM OUTCOMES 97 projects, thereby demonstrating the wider potential impact of the federal funds. In the case of paper-to-patent citations, this is done by examining references to scientific and engineering papers on the front pages of U.S. patents. References are also made to previously issued patents. Both sets of patent and nonpatent references comprise the âprior artâ of patents. Citation analysis has been used at various times by the U.S. Department of Energy, the National Institute of Standards and Technology, the Agricultural Re- search Service, the National Science Foundation50, and other federal agencies to show movement of knowledge from scientific research programsâwhere impacts are difficult to measureâto industrial technologyâwhere impact measurement is more tractable.51 Patent citation trees are routinely used by ATP, for example, to show the dissemination of technical knowledge via patents from completed projects to other companies and other organizations.52 No evidence was found, however, of publication or patent citation analysis by the NASA SBIR program. Further, no evidence was found of the systematic collection by NASA of the detailed publication, and patent data from SBIR proj- ects needed to support citation studies. Yet, as indicated by the results of the NRC Phase II Survey, patents and scientific publications are being produced by the NASA SBIR program. Hence, opportunities exist to encourage program participants to publish when it will not compromise their ability to commercialize. Both publication and patent citation analysis could be used to demonstrate and track knowledge dissemination from NASA SBIR projects to others. 4.5.3â Licensing Licensing agreements depend on the protection of the intellectual property. They are another indicator of the creation and dissemination of knowledge. Respondents reported licensing as an important activity they engaged in with other companies and investors both in the U.S. and abroad. Table 4-20 shows the frequency with which respondents said they had finalized or were negotiating licensing agreements to commercialize technologies resulting from the referenced award. Respondents formed licensing agreements with both foreign companies and investors and with domestic companies and investors. The use of licensing signals the underlying importance of intellectual prop- 50â The referenced use of citation analysis by NSF lies outside the NSF SBIR program. NSF sup- ported extensive work by CHI Research, Inc. to develop and âcleanâ databases needed to perform publication citation analysis. 51â For an example of a citation study performed for a federal R&D program, see J. S. Perko and Francis Narin, CHI Research, Inc., âThe Transfer of Public Science to Patented Technology: A Case Study in Agricultural Science,â Journal of Technology Transfer 22(3):65-72. 52â Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Reportâ Number 2, NIST Special Publication 950-2, Gaithersburg, MD: National Institute of Standards and Technology, pp. 266-270.
98 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION TABLE 4-20â Licensing Activities of Phase II Surveyed Awardees with U.S. and Foreign Companies and Investors Finalized Ongoing Focus of Interactions Agreements (%) Negotiations (%) Interactions with U.S. Companies and Investors 6 11 Interactions with Foreign Companies and Investors 8 5 SOURCE: NRC Phase II Survey, Question 12. erty protection to high-tech small businesses. Case-study results also highlight the importance of intellectual protection and licensing activities as a major com- mercialization strategy for several small businesses. Licensing activities tend to increase the diffusion of a technologyâs effect, and as noted by Jaffe, licensing tends to increase spillover effects, particularly market spillovers. 53 4.5.4â Partnerships of Small Firms with Other Companies and Investors Partnering with other organizations and people also accomplishes knowl- edge transfer. For small companies, the formation of partnerships with other companies is often an essential strategy for commercializing a technology. The larger companies they partner with often have manufacturing capacity, market- ing know-how, and distribution paths in place. Awardees whose technology is far upstream of consumer goods may need to: partner with other companies for the additional research needed to integrate their technologies into larger systems; partner with Original Equipment Manufacturers who purchase the awardeesâ out- put as intermediate goods; and form alliances with customers to more effectively reach markets. The NRC Phase II Survey provided insight about the kinds of partnerships being formed by SBIR recipients. As shown in Table 4-21, partnerships for R&D, for marketing and distribution, with customers, and for manufacturing were found to be formed by these awardees. Licensing agreements may or may not entail close partnering, whereas the other listed forms generally do require close alliances and partnering. 4.5.5â Interactions Among Small Firms and Universities Many companies with NASA also have relationships with universities through which knowledge is created and disseminated. Many funded projects in- 53â Adam Jafee, Economic Analysis of Research Spillovers: Implications for the Advanced Technol- ogy Program, NIST GCR 97-708, Gaithersburg, MD: National Institute of Standards and Technology, pp. 42-44.
SBIR PROGRAM OUTCOMES 99 TABLE 4-21â Percent of Phase II Surveyed Awardees Forming Partnerships with U.S. and Foreign Companies and Investors With U.S. Companies and With Foreign Companies Investors and Investors Ongoing Ongoing Finalized Negotiations Finalized Negotiations Partnering for: (%) (%) (%) (%) Licensing Agreement(s)a 6 11 8 5 R&D Agreement(s) 16 10 4 1 Marketing/Distribution Agreement(s) 7 5 6 2 Customer Alliance(s) 11 9 6 0 Manufacturing Agreement(s) 2 6 2 0 Joint Venture Agreement(s) 1 4 0 2 SOURCE: NRC Phase II Survey, Question 12. volve university faculty, graduate students, and/or university developed technolo- gies. University faculty and students establish small businesses. Faculty members serve as proposal reviewers. Universities assist firms with proposal preparation and sub-contracts or consult on projects. They also sometimes provide facilities and equipment to assist projects. The NRC Firm Survey showed that over 62 percent of all respondents had at least one founder with an academic background. Around 31 percent of company founders had been most recently employed by a college or university prior to founding the company. The NRC Phase II Survey showed that 29 percent of NASA projects involved some form of university involvement. The survey data show the prime mode of involvement to be faculty members or adjunct faculty members working on the referenced project in a role other than PIâas a consultant, for example. The next most frequent modes of involvement were those of universities/colleges as subcontractors, graduate students working on the project, and university or col- lege facilities or equipment being used on the project. In some instances, project technologies were originally developed in universities or colleges by one of the participants in the referenced projects. On occasion, the technologies for the ref- erenced projects were licensed from a university or college. Table 4-22 indicates the extent to which each type of university involvement occurred in the sample Phase II projects. The NRC Phase II Survey results show that the NASA SBIR plays some role in moving research concepts out of the university. Of the Phase II survey proj- ects, 4 percent involved technology that was originally developed at a university by a project participant. Two percent of the technologies in the Phase II survey projects were licensed from a university. In addition, some of the case-study firms were found to have on-going affiliations with universities. Although only 7 percent of the Phase II awards reported that they had re- ceived assistance in Phase I or Phase II proposal preparation, universities were
100 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION TABLE 4-22â Involvement by Universities and Colleges in NRC Phase II Survey Projects Type of Relationship Between Referenced Project and Respondents Reporting Universities/Colleges the Relationships (%) Faculty members or adjunct faculty member worked on the project in a 17 role other than PI. Graduate students worked on the project. 15 University/College facilities and/or equipment were used on the project. 13 A university or college was a subcontractor on the project. 16 The technology for this project was originally developed at a university or 4 college by one of the participants in the referenced project. The technology for the project was licensed from a university or college. 2 The Principal Investigator (PI) for the project was at the time of the 1 project an adjunct faculty member. The Principal Investigator (PI) for the project was at the time of the 2 project a faculty member. SOURCE: NRC Phase II Survey, Question 31. responsible for providing most of that assistance. When asked to evaluate the usefulness of the proposal assistance, five of the 11 Phase II recipients which reported receiving assistance rated it as âvery usefulâ and the other six rated it as âuseful;â none said it was ânot useful.â As shown in Table 4-23, overall, 29 percent of the respondents reported involvement in their Phase II project by faculty, graduate students, or university- developed technologies. This result is reinforced by the sample of firm case studies. Of the 22 firms, 14 demonstrated important linkages with the university sector. Many SBIR projects therefore do seem to promote the transfer of knowl- edge between the private sector (the awardee) and universities. TABLE 4-23â In Executing the Phase II Award Was There Involvement by University Faculty, Graduate Students, and/or University-developed Technologiesâ(N=161) Response Percent of Respondents Yes 29 No 71 SOURCE: NRC Phase II Survey, Question 31.
SBIR PROGRAM OUTCOMES 101 4.5.6â Assessing Knowledge Expansion Developing and disseminating knowledge derived in some part from SBIR projects depend on both the riskiness of the project and the often indirect ways though which knowledge spreads. Risk Profile One question about the SBIR program is the extent to which it funds projects that are truly innovative. This is a difficult and important area. There are pressures on program manag- ers to ensure that levels of commercialization are highâyet commercialization outcomes are inversely related to the riskiness of the research: Very high risk projects are less likely to reach the market than modest adjustments to a technol- ogy that already has customers. In interviews, program managers at all agencies recognize this potential difficulty. However, in the main they remain focused on the need to enhance commercialization. The risk of insufficient innovation is both lower priority and, perhaps in the long run more important, is not easily assessed. Sales can be counted; innovation lies in the eye of the beholder. It is therefore important that agencies continue to use existing indicators to monitor the riskiness of the projects they fund, and to seek to develop new ones. Much information can be found in a better understanding of why and when projects fail: â¢ The NRC Phase II survey reported that technical difficulties were one important reason for discontinuing Phase II projectsâthey were the fifth most cited reason. â¢ The NRC Phase I survey also suggested that technical risk among NASA projects was high. Of the Phase I projects that did not get a follow-on Phase II award, a leading reason was technical barriers. Indicators, Not Measures of Benefit No economic benefits are generated from knowledge efforts until the knowl- edge flows are actually used by others to develop new and improved products, processes, and services. Hence, collection of data on knowledge generation and dissemination activities does not provide direct measurement of impacts. Such data can, however, serve to construct indicators of potential impacts. Examples of possible indicators are number of patents per research dollar, characteristics of collaborative networks formed, and sales of commercialized goods and services. Trends in these and other indicators may indicate that developments are occurring along an indirect pathâas would be expected for projects that are progressing toward the generation of broad impacts.
102 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION It is apparent from the NRC Phase II Survey results that it would be possible to compile multiple indicators of knowledge generation and dissemination and early commercialization achievements from NASA SBIR projects, and to track them over time. Thus far, however, it appears that such indicators have been developed only partially and on an ad hoc basis. It appears that more could be done to systematically compile and track indicators of knowledge generation and dissemination if desired. 4.5.7â Conclusions on SBIRâs Knowledge Impact Given its descriptions and proposal selection criteria which emphasize the achievement of broad impacts (i.e., not just the commercialization goal), it might be expected that NASAâs SBIR program would demonstrate a strong interest in measures of knowledge outputs. But, in fact, broader impacts appear to be de- fined by the program largely as commercial results and the infusion of the new technologies in the agencyâs mission. Little evidence was found that the program pays much attention to knowledge outputs per se. As a first step, NASA (and other agencies) might consider requiring that recipient firms provide bibliographic citations for papers appearing in peer- reviewed journals, the proceedings of scientific societies, or conference reports, as part of their outcome reporting process. 4.6â CONCLUSIONS There is no single simple metric that adequately captures âresultsâ from the program, as discussed in the NRCâs Methodology Report for the SBIR assess- ment.54 Each of the four congressional mandates is best assessed separately, and within each, there are a multiple issues to be addressed. Bearing all these points in mind, it is still possible to summarize the results of our research in straightforward terms. 4.6.1â Commercialization Approximately 30-40 percent of Phase II projects produce innovations that reach the market, with a small number generating substantial returns. Other indi- cators of commercialization, such as licensing activities, marketing partnerships, and access to and utilization of further investments from both private and public sources, all confirm that while returns are highly skewed, and the results in gen- eral are positive, firms operating within the NASA SBIR program face significant 54â National Research Council, An Assessment of the Small Business Innovation Research ÂProgramâ Project Methodology, op. cit.
SBIR PROGRAM OUTCOMES 103 structural barriers that make it hard to develop substantial markets based on the highly NASA-specific technologies the agency tends to fund. 4.6.2â Agency Mission While it is difficult to find good data on the extent to which SBIR supports NASAâs mission we can conclude the following: â¢ The SBIR program has been aligned with the agency needs, primarily through the topic development and award selection processes. This process has been considerably altered by changes made in FY2006. â¢ Outcomes from the SBIR program appear to be aligned with agency needs, although the small number of projects that are selected for NASA Phase III funding helped to drive the FY2006 reforms. â¢ Agency staff in general indicate that SBIR awards are of comparable quality to other NASA research projects. This analysis also indicates that early involvement with and âownershipâ of SBIR projects by NASA technical staff is an important factor in the successful utiliza- tion of SBIR for agency purposes. 4.6.3â Support for Woman- and Minority-owned Businesses NASAâs SBIR program supports the participation of minority and woman- owned small business in innovation research.55 More widely, SBIR significantly supports small high technology businesses in general, and the NRC research determined that SBIR had an important cata- lytic effect in terms of company foundationâproviding the critical seed money to fund a companyâs first steps. SBIR also had strongly influenced companiesâ decisions to initiate individual projects: 68 percent of NRC Phase II Survey respondents at NASA believed that their projects would not have gone forward without SBIR, and, of the remainder, most believed that the projects would have been delayed and/or would have had a reduced scope.56 4.6.4â Support for the Advancement of Scientific and Technical Knowledge The program funds cutting-edge research, as it was designed to do. One of the key selection criteria at NASA is âtechnical innovation.â NASA SBIR funding also supports the dissemination of knowledge through traditional vectors such as peer-reviewed publications, as well traditional indica- 55â See Chapter 3 of this report for more details. 56â NRC Phase II Survey, Questions 13-14.
104 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION tors that valuable intellectual property has been produced, such as patents. About 40 percent of projects led to at least one peer-reviewed publication, and about 20 percent of projects generated at least one successful patent application. It is therefore appropriate to conclude that the NASA SBIR program is meet- ing all four of the congressional objectives.