4
Outcomes

4.1
INTRODUCTION

Identifying the specific outcomes resulting from an early-stage R&D program such as SBIR is challenging.1 The long lag between input (funding) and output (possible products and services), combined with the frequent need for multiple inputs for successful technology development, make definitive assessments of the link between a single input and a complex output difficult. In addition there are very substantial data collection problems, as awardees and agencies cannot consistently capture outcomes for all supported projects. Many early-stage research projects generate little that is tangible in the form of products and services while a few projects can generate very large returns. The large skew means that anything short of an all-inclusive analysis risks missing important contributions from the program.

While keeping these caveats in mind, this chapter seeks to provide as broad an assessment of outcomes from the DoD SBIR program as possible. It will focus on whether SBIR is meeting its four congressional objectives. These are “(1) to stimulate technological innovation; (2) to use small business to meet federal research and development needs; (3) to foster and encourage participation by minority and disadvantaged persons in technological innovation; and (4)

1

For a summary of the challenges of tracking specific SBIR outcomes, see National Research Council, SBIR: Program Diversity and Assessment Challenges, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2004, pp. 32-35. Data in this chapter are derived from the NRC Phase II Survey unless otherwise specified.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 86
4 Outcomes 4.1 INTRODUCTION Identifying the specific outcomes resulting from an early-stage R&D pro - gram such as SBIR is challenging.1 The long lag between input (funding) and output (possible products and services), combined with the frequent need for mul- tiple inputs for successful technology development, make definitive assessments of the link between a single input and a complex output difficult. In addition there are very substantial data collection problems, as awardees and agencies cannot consistently capture outcomes for all supported projects. Many early-stage re - search projects generate little that is tangible in the form of products and services while a few projects can generate very large returns. The large skew means that anything short of an all-inclusive analysis risks missing important contributions from the program. While keeping these caveats in mind, this chapter seeks to provide as broad an assessment of outcomes from the DoD SBIR program as possible. It will focus on whether SBIR is meeting its four congressional objectives. These are “(1) to stimulate technological innovation; (2) to use small business to meet federal research and development needs; (3) to foster and encourage participa - tion by minority and disadvantaged persons in technological innovation; and (4) 1 For a summary of the challenges of tracking specific SBIR outcomes, see National Research Council, SBIR: Program Diersity and Assessment Challenges, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2004, pp. 32-35. Data in this chapter are derived from the NRC Phase II Survey unless otherwise specified. 

OCR for page 86
7 OUTCOMES to increase private sector commercialization derived from federal research and development.”2 4.2 COMMERCIALIZATION 4.2.1 Background Bringing new technologies developed under the research supported by SBIR awards to the marketplace has been a central objective of the SBIR program since its inception. The program’s initiation in the early 1980s in part reflected a concern that American investment in research was not being transformed ad - equately into products that could generate greater wealth, more employment, and increased competitiveness. Directing a portion of federal investment in R&D to small businesses was thus seen as a new means of meeting the mission needs of federal agencies while increasing the participation of small business and thereby the proportion of innovation that would be commercially relevant.3 Congressional and Executive Branch interest in the commercialization of SBIR research has increased over the life of the program. A 1992 GAO study4 focused on commercialization in the wake of con- gressional expansion of the SBIR program in 1986.5 The 1992 reauthorization specifically “emphasize[d] the program’s goal of increasing private sector com - mercialization of technology developed through federal research and develop - ment6 and noted the need to “emphasize the program’s goal of increasing private sector commercialization of technology developed through federal research and development.” The 1992 reauthorization also changed the order in which the 2The Small Business Innovation Development Act (PL 97-219). 3A growing body of evidence, starting in the late 1970s and accelerating in the 1980s indicates that small businesses were assuming an increasingly important role in both innovation and job creation. See, for example, J. O. Flender and R. S. Morse, The Role of New Technical Enterprise in the U.S. Economy, Cambridge, MA: MIT Development Foundation, 1975, and David L. Birch, “Who Creates Jobs?” The Public Interest, 65:3-14, 1981. Evidence about the role of small businesses in the U.S. economy gained new credibility with the empirical analysis by Zoltan Acs and David Audretsch of the U.S. Small Business Innovation Data Base, which confirmed the increased importance of small firms in generating technological innovations and their growing contribution to the U.S. economy. See Zoltan Acs and David Audretsch, “Innovation in Large and Small Firms: An Empirical Analysis,” The American Economic Reiew 78(4):678-690, Sept 1988. See also Zoltan Acs and David Audretsch, Innoation and Small Firms, Cambridge, MA: The MIT Press, 1990. 4 U.S. Government Accounting Office, Federal Research: Small Business Innoation Research Shows Success but Can Be Strengthened, GAO/RCED-92-37, Washington, DC: U.S. Government Accounting Office, March 1992. 5 PL 99-443, October 6, 1986. 6 PL 102-564 October, 28, 1992.

OCR for page 86
 SBIR AT THE DEPARTMENT OF DEFENSE program’s objectives are described, moving commercialization to the top of the list.7 The term “commercialization” means, “reaching the market,” which some agency managers interpret as “first sale”—that is the first sale of a product in the market place, whether to public or private sector clients. This definition, how- ever misses significant components of commercialization that do not result in a discrete sale. It also fails to provide any guidance on how to evaluate the scale of commercialization, an important element in assessing the degree to which SBIR programs successfully encourage commercialization. The metrics for assessing commercialization can also be elusive. It’s not straightforward, for example, to calculate the full value of an “enabling technology” that can be used across indus - tries. Also elusive is the value of materials that enable a commercial service. 8 In light of the difficulties in measuring commercialization effectively, the Navy SBIR program manager has suggested that a firm’s success in securing Phase III funding from an agency be substituted for the current weight accorded commercialization in the Commercialization Achievement Index (CAI) measure used during the proposal selection competition (the CAI is discussed in more de - tail in Chapter 6). Given the earlier noted variations in the S&T needs, sizes, and institutional arrangements of the services and units, applying this performance measure consistently is not always self evident. In fact, efforts to identify Phase III results may have been unduly limited. The initial 1982 SBIR legislation noted that Phase III is not time-bound and can come long after the end of the Phase II; Phase III can include private sector sales. The law indicates that commercialization “may also involve non-SBIR, government- funded production contracts with a federal agency. . . .”9 Moreover, Phase III funding comes via a wide variety of mechanisms. Firms receive modifications to add Phase III federal R&D funding to Phase II SBIR contracts, they have won production contracts or R&D contracts competitively, sold to prime contractors, received additional private sector funding, and sold products commercially. All of these are Phase III activities in accordance with the legislation and with the SBA policy directive. Consequently, a narrow defi - nition of Phase III, as a noncompetitively awarded further R&D or production 7These changes are described by R. Archibald and D. Finifter in “Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach” in National Research Council, The Small Business Innoation Research Program: An Assessment of the Department of Defense Fast Track Initiatie, Washington, DC: National Academy Press, 2000. 8 For a discussion of this and related methodological challenges, see, National Research Council, Assessment of the Small Business Innoation Research Program—Project Methodology, Washington, DC: The National Academies Press, 2004, accessed at . 9 U.S. Government Accounting Office, Federal Research: Small Business Innoation Research Shows Success but Can Be Strengthened, GAO/RCED-92-37, op. cit., p. 14.

OCR for page 86
9 OUTCOMES contract, and one that is only captured if properly entered in the DD350 report is too limited as an approach. In addition, DD350 documentation of Phase III funding does not occur until at least 1 year following completion of any Phase II enhancement awards, the form itself is often not filled out completely or appropriately, as data tests run by BRTRC indicate.10 DoD staff indicate that Navy makes a considerable effort to ensure that its DD350 forms fully capture SBIR Phase III activities to the maxi- mum extent possible. Other agencies do not.11 In sum, while they do provide one important measure of commercializa- tion and one that could be used more effectively, under current circumstances, the DD350 reports may not provide sufficiently comprehensive or accurate data on which to make definitive determinations about the success of DoD SBIR commercialization. Indeed, the multiple goals of the SBIR program mean that multiple measures are appropriate for evaluation. 4.2.2 Proposed Commercialization Indicators and Benchmarks This report uses three sets of indicators to quantitatively assess commercial - ization success: 1. Sales and licensing revenues (“sales” hereafter, unless otherwise noted). Revenues flowing into a company from the commercial marketplace constitute the most obvious measure of commercial success. They are also an important indicator of uptake for the product or service. Sales indicate that the result of a project has been sufficiently positive to convince buyers that the product or service is the best available solution. Yet if there is general agreement that sales are a key benchmark, there is no such agreement on what constitutes “success.” Companies, naturally enough, focus on projects that contribute to the bottom line—that are profitable. Agency staff provide a much wider range of views. Some view any sales a substantial success for a program focused on such an early stage of the product and develop- ment cycle, while others seem more ambitious.12 Some senior executives in the private sector viewed only projects that generated cumulative revenues at $100 million or more as a complete commercial success.13 Rather than seeking to identify a single sales benchmark for “success,” it therefore seems more sensible to simply assess outcomes against a range of 10 Peter Cahill, BRTRC, private communication, December 1, 2006. 11 Michael Caccuitto, DoD SBIR/STTR Program Administrator, Interview, November 28, 2006. 12 Interviews with SBIR program coordinators at DoD, NIH, NSF, and DoE. 13 Pete Linsert, CEO, Martek, Inc., Meeting of the Committee for Capitalizing on Science, Tech - nology, and Innovation: An Assessment of the Small Business Innovation Research Program, June 5, 2005.

OCR for page 86
90 SBIR AT THE DEPARTMENT OF DEFENSE benchmarks reflecting these diverse views, with each marking the transition to a greater level of commercial success: a. Reaching the market—A finished product or service has made it to the marketplace. b. Reaching $ million in added cumulatie sales (beyond SBIR Phases I and II)—The approximate combined amount of standard DoD Phase I and Phase II awards. c. Reaching $ million in cumulatie sales—A modest commercial success that may imply that a company has broken even on a project. d. Reaching $0 million in cumulatie sales—A full commercial success. 2. Phase III activities within DoD. As noted above, Phase III activities within DoD are a primary form of commercialization for DoD SBIR projects. These activities are considered in Section 4.3 and Chapter 5. 3. R&D investments and research contracts. Further R&D investments and contracts are good evidence that the project has been successful in some sig - nificant sense. These investments and contracts may include partnerships, further grants and awards, or government contracts. The benchmarks for success at each of these levels should be the same as those above, namely: a. Any R&D additional funding. b. Additional funding of $1 million or more. c. Additional funding of $5 million or more. d. Funding of $50 million or more. 4. Sale of equity. This is a less clear-cut indicator of commercial success. but it is unlikely investors or competitors would buy equity in a company that had not shown its ability to produce something of significant value. Key metrics include: a. Equity investment in the company by independent third party. b. Sale or merger of the entire company. 4.2.3 Sales and Licensing Revenues from DoD SBIR Awards The most basic of all questions on commercialization is whether a project produced a good or service that reached the marketplace. Figure 4-1 shows the status of surveyed projects. It shows that only a fairly small percentage of SBIR projects have been discontinued with no hope of ever generating sales (26 percent), though it is likely that a significant portion of the projects currently in development will also fail to achieve significant commercial success.

OCR for page 86
9 OUTCOMES No answer Phase II Project not (1%) completed yet Discontinued/abandoned Projects in development, (5%) - no sales expecting sales (26%) (18%) Projects in development, not expecting sales (4%) Projects with some sales/revenues (46%) FIGURE 4-1 Status of surveyed projects. NOTE: The NRC deployed two surveys to the population of Phase II recipients as part of the research conducted for this project. The NRC Phase II Survey focused on individual projects. In addition, the NRC Firm Survey was sent to every firm receiving a Phase II award between 1992 and 2001, and focused on firm-level questions. Unless otherwise stated, all references here to the NRC Survey4-1to the project survey. are SOURCE: NRC Phase II Survey. These data indicate that 46 percent of surveyed projects reported some revenues from their project; a further 18 percent were still in development and expected sales, and 5 percent had not yet completed Phase II. 4.2.3.1 Sales Ranges Early-stage technology projects are inherently risky. As a result, there is a very skewed distribution of results. Many projects generate no commercial results at all, and relatively few of those that do reach the market have substantial com- mercial successes (see Figure 4-2). The data suggest that at DoD—as at other agencies—the overwhelming majority of sales are concentrated in the $0–$1 million range. Ten percent of reporting projects generated at least $5 million in revenues, while more than 65 percent of respondents with sales reported total sales of less than $1 million rev - enues (as of May 2005, the date of the survey). As a result of this very skewed distribution, the mean amount of sales for all companies that reported sales was $2,894,834, while the median was $500,000. Underreporting of Sales Results. The average total sales for older projects is much higher than for recent ones. In fact, the average sales for the 176 reporting DoD projects awarded Phase II contracts from 1992 to 1994 was $2.78 million, whereas the average sales for the 415 reporting DoD projects awarded Phase II

OCR for page 86
9 SBIR AT THE DEPARTMENT OF DEFENSE 70 65.1 60 50 40 Percent 30 24.9 20 10 6.6 2.6 0.8 0 > 0 and <1M 1M to <5M 5M to 10M to 50M + <10M <50M Sales (Dollars) FIGURE 4-2 Sales by sales range. SOURCE: NRC Phase II Survey. contracts between 1999 to 2001 was only $982,000. This difference in part re - 4-02 flects the number of recent awardees whose products have yet to be commercially introduced or fully exploited commercially. Reported aggregate and average sales data up through the survey date of May 2005 are therefore only a partial estimate of the total commercial impact of the 920 awards covered by the NRC Phase II Survey.14 According to former senior DoD staff, average major DoD weapons system R&D cycle is approximately 12 years (before production)—so SBIR 14 Using the trendline shown in Figure 4-3, we find that the best fit generates average sales of ap - proximately $5.5 million per project for those with awards in 1992, declining to averages sales of $1 million for those in 2001. Note that these data cover only firms reporting some sales. The trendline gives us a means of estimating the eventual sales generated by each project, using simplified as - sumptions (notably, that all sales end by May 2005 (the date of the NRC Phase II Survey), and that commercialization remains constant across time (in fact, it is likely to have increased as agencies have increased their focus on supporting projects with better prospects of commercial success). Other assumptions tend to reduce the size of the estimated revenues. Using these estimates to project forward, we find that by the time commercialization of all projects is completed (i.e., ten years after the last project funded, or 2011), total revenues generated by the projects reporting revenues so far is $2.13 billion. These data suggest that the reported revenues as of May 2005 may understate eventual total rev- enues as of 2011 by as much as 50 percent.

OCR for page 86
9 OUTCOMES 9,000,000 8,000,000 Average Total Sales (Dollars) 7,000,000 6,000,000 5,000,000 4,000,000 3,000,000 2,000,000 1,000,000 0 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 Year of Award FIGURE 4-3 Average sales, by year of award, plus trendline. SOURCE: NRC Phase II Survey. TABLE 4-1 Reported and 4-03 Projected Revenues for Companies that Reported Sales as of May 2005 Amount ($) 1,094.0 Reported Sales (Millions of Dollars) 1,040.5 Total Sales (Millions of Dollars) 2,134.5 Average Sales per Project with Sales ($) 5,646,787 Average Sales per Project—All Projects ($) 2,151,699 SOURCE: NRC Phase II Survey. No Response (0%) Phase II Not Complete (5%) Sales Not Expected Repor ted (31%) Sales (46%) Sales Expected (18%) FIGURE 4-4 Reported sales and expectations. SOURCE: NRC Phase II Survey. 4-4

OCR for page 86
9 SBIR AT THE DEPARTMENT OF DEFENSE TABLE 4-2 For Companies Anticipating Sales, Year of Expected First Sale Year of Expected Sales Number of Projects Percent of Projects 2005 37 22.0 2006 70 41.7 2007 31 18.5 2008 18 10.7 2009 5 3.0 2010 7 4.2 Total 168 NOTE: Survey Date: 2005. SOURCE: NRC Phase II Survey. products for defense sales would tend to have a long time lag. 15 Thus, Figure 4-3 should not be taken to mean that sales are declining; it largely reflects the extent of the lag in DoD-oriented sales. Sales Concentration. Total revenues from sales are highly concentrated in the very few projects that have generated at least $5 million in cumulative rev- enues. Just under 75 percent of all cumulative sales were accounted for by the 38 projects (out of 920 overall) that reported at least $5 million in sales. This very high concentration confirms the view that from the perspective of sales, the SBIR program at DoD generates a few major winners, rather than a more widely dis- persed range of more modest successes. This is similar to commercial outcomes from early-stage R&D programs. 4.2.3.2 Sales Expectations About a quarter of projects reported that they expected sales in the future. Of those companies not yet reporting sales on their projects, about 67 percent still expect them. Most of the respondents that had not yet received sales expected sales to come in the very near future, as shown by Table 4-2. Of those expected sales, 80 percent anticipated that their first sale would occur within 3 years. The data in Figure 4-5 show, for projects that have received sales, the time that elapsed between the Phase II award and the first sales. Survey responses indicate that 87.6 percent of first sales occurred within 4 years of the award date. This relatively short time from award to first sale is supported by the comments of John Williams, Navy SBIR Manager.16 However, it must be stressed that 15 Dr. Jacques Gansler, former Under Secretary of Defense for Acquisition, Technology and Logis - tics, January 29, 2007. 16 John Williams, U.S. Navy SBIR program manager, presentation at SBTC SBIR in Rapid Transi - tion Conference, Washington, DC, September 27, 2006.

OCR for page 86
9 OUTCOMES 25 20 15 Percent 10 5 0 -10 -6 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 Elapsed Years Between Phase II Award and Year of First Sale FIGURE 4-5 Time elapsed between award and first sales—frequency distribution. SOURCE: NRC Phase II Survey. 4-05 interviews and cases strongly support the view that the bulk of sales will occur some years after the date of first sale. The latter is therefore best seen as a lead - ing indicator for sales. These data from the NRC’s May 2005 Phase II Survey help us to evaluate claims of companies that they will generate sales in the future. About 25 percent of all DoD survey respondents made this claim. However, the likelihood of commercialization diminishes substantially with time elapsed since the award. Projects still expecting sales are clustered toward the 2000–2001 timeframe, but even here the likely window of opportunity for success appears to be closing rapidly. The median time to first sale is before the end of the second year after the award. This analysis suggests that though a considerable number of companies anticipate sales in the future, the actual likelihood of this occurring is relatively low. By the start of the 9th year after the Phase II award is made, 99 percent of projects that will eventually report sales have done so; the likelihood a project without sales reporting sales after the 8th year is less than 1 percent. Similarly, projects not reporting sales by the start of year five have a 13.8 percent chance of eventually reporting sales (to put it yet another way, by the end of the fourth year after the award year, 86.2 percent of projects that eventually make sales will have started to do so).17 Table 4-3 helps us to determine likely revenues for companies that expect 17This is not to say that all sales will have been completed by the end of that year, only that it is quite reasonable to apply these percentages to the more recent awards, as a way of estimating eventual sales data.

OCR for page 86
9 SBIR AT THE DEPARTMENT OF DEFENSE TABLE 4-3 Total Projected Sales for Companies Without Sales That Still Expect Them Projects Historical Success Projected Total Revenues Year of Award Expecting Sales Percentage ($) 1992 3 0.0 0 1993 4 0.0 0 1994 4 0.3 58,730 1995 8 0.8 352,381 1996 6 0.3 88,095 1997 13 0.3 190,873 1998 18 2.4 2,378,571 1999 19 3.7 3,905,556 2000 32 4.8 8,457,143 2001 61 10.1 34,034,127 Total 49,465,476 NOTE: Projected sales are calculated by multiplying the number of companies reporting that they still expect sales, by the percentage likelihood that a company in that award year will in fact generate sales, by the average total sales for all companies that did record more than $0 in sales. SOURCE: NRC Phase II Survey. them, based on historical records. The total projected revenues for these compa - nies is relatively low—about $50 million, or less than 2.5 percent of all projected sales. We can therefore conclude that while ongoing revenues from companies with some sales in hand will be relatively large (see above), the revenues ex - pected from companies that had not yet reported some sales as of May 2005 are likely to be of limited significance. These findings suggest that while the product cycle for the entire defense industry may be long—a well-known characteristic of major defense systems— the first sales cycle for most SBIR-related products is actually relatively short. Most successful project start receiving initial sales revenues within 4 years of the award, while large sales tend to come considerably later due to the defense procurement cycle.18 Sales and Projected Sales: Conclusions. The NRC Phase II Survey pro- vided the following summary data regarding sales and projected sales: • 378 out of a total of 920 respondents ( 41.1 percent) report sales greater than $0. • The average reported sales is $1.3 million for all projects (n=920), and $3.2 million for those reporting sales greater than $0 (n=378). 19 18This paragraph concerns only first sales. The bulk of sales occur at some unknown period after the first sale. 19When projected out to 2011, estimated average sales per project are $5.6 million for projects already reporting some sales, and $2.2 million for all projects.

OCR for page 86
7 OUTCOMES TABLE 4-18 Projects Reporting Patent Applications and Awards Projects with Patent Applications Projects with Patents Awards Number Percent Number Percent Yes 281 34.4 205 25.1 No 535 65.6 611 74.9 Total 816 816 SOURCE: NRC Phase II Survey. tion for most of the case study firms. Many SBIR awardees generate products and services that are inputs into the larger weapons systems being managed by large defense contractors. Without strong patent protection, firms are wary about entering into subcontracting relationships with prime contractors. They fear that the larger firm will imitate, reverse engineer, or preemptively patent function - ally equivalent technologies, thus eliminating the SBIR firm’s major or only market.74 However, some case study firms are resource constrained in pursuing an active patent protection strategy. Other firms see patents as a limited, relatively ineffective or inefficient form of intellectual property protection given their tech - nology domain or the size of the prospective market. Thus, some firms do not pursue patents, relying instead on trade secrets and know-how to protect their in - tellectual property. Choice of this strategy is influenced in part by the assessment that patents provide little economic benefit if the dominant customer of the firm’s product is the U.S. government, which under SBIR is entitled to royalty-free use of resulting technology. Also, this strategy avoids the costs of patenting and the associated public disclosure of related proprietary technological knowledge. 4.5.1 Patents As noted above, SBIR awards are generally a significant patent stream. Table 4-18 reports findings on patents and publications, as well as other forms of intellectual property rights protection, received by NRC Phase II Survey re- spondents as a result of Phase II awards. The NRC Phase II Survey data indicate that about one-fourth of respondents received patents related to the relevant 74The Night Vision Corp. . The United States of America patent case filed before the United States Court of Federal Claims represents a variation on these concerns. In this case, Night Vision sued the United States claiming that the Air Force shared with another firm, Insight, prototypes that it had developed, and that Insight had then disassembled the prototypes, violating its SBIR data rights. See U.S. Court of Federal Claims, No. 03-1214C. 25 May 2005. See also U.S. Court of Appeals for the Federal Circuit, 06-5048, November 22, 2006.

OCR for page 86
 SBIR AT THE DEPARTMENT OF DEFENSE SBIR-funded project, and about 34 percent of projects generated at least one patent application. These data are similar, though slightly lower, than those reported at NIH. They indicate that at a minimum, one-third of projects generated knowledge that was judged by the firm to be sufficiently unique and commercially important to be worth the significant expense of patent filing. And a quarter of all projects reported that government examiners agreed and awarded at least one patent. This is a very significant finding. It can be viewed as addressing the tip of the knowledge iceberg. Only a small percentage of the knowledge generated dur- ing a research project meets the relatively stringent tests indicated above. Most research outcomes are not sufficiently unique to qualify for patent protection. And even unique knowledge must pass formidable internal hurdles before patent protection is sought, as the process is expensive and time-consuming. Most companies file only one project-related patent. However, a few file many, as shown by Figure 4-15. Because it is sometimes assumed that smaller firms have more limited access to the funds and expertise necessary to file patents, we examined the relationship between patent filing and firm size (see Figure 4-16). By developing a “patenting ratio”—the ratio of firms with at least one patent awarded to all responses, by size of firm—the data seems to show that projects at firms with more than 75 employees are less likely to generate patents than those with less than 75 employees. The former generate 0.23 patents per project, the latter more than twice as many, 0.56, although this may be in part because larger firms are more likely to engage in manufacturing and other activities in addition to R&D. It may also reflect differences among sectors—for example, between information technology and less patent prone sectors. 3-5 Patents (12%) 6-10 Patents (2%) 2 Patents (21%) 11-20 Patents (2%) 1 Patent (63%) FIGURE 4-15 Distribution of patenting activities—Responding projects. SOURCE: NRC Phase II Survey. 4-15

OCR for page 86
9 OUTCOMES 500+ 301-500 201-300 Number of Employees 101-200 76-100 51-75 41-50 31-40 26-30 21-25 16-20 11-15 6-10 0-5 0 0.2 0.4 0.6 0.8 1 1.2 Patenting Ratio (patents awarded/responses) FIGURE 4-16 Patents and firm size at time of survey. SOURCE: NRC Phase II Survey. This is somewhat counterintuitive, as larger firms tend to have more re - 4-16 sources, and patenting is expensive. Case studies do suggest that for these larger firms with better access to financial resources, SBIR projects tend to be deployed on research that is not in the company’s core strategic plan, and is therefore less likely to be seen as having immediate commercial benefit (necessary to attract the internal resources for patenting). Analysis of the scientific importance of the patents listed has not been pos- sible, as the patents themselves were not disclosed in the course of the survey. 4.5.1.1 Knowledge Generation vs. Commercialization? It has also been suggested that there might be a disconnect between research- oriented firms and commercially oriented firms, and that SBIR programs should be adjusted to focus more effectively on the latter. This is—partly—the purpose of including the CAI in SBIR proposal reviews. The NRC Phase II Survey can help to test the hypothesis that commercially oriented companies were focused on areas different from research-oriented firms. If this hypothesis was correct, we would anticipate a distinction between firms that report IP-related activities (filing patents, copyrights, and trademarks, and seeking publications) and those undertaking marketing activities (preparation of marking plans, hiring marketing staff, etc.). In fact, Figure 4-17 shows that no such distinction is observed. 40 percent of respondents report both IP- and marketing-related activities, and a further 22 percent report neither. About 28 percent appear to fit the “commercially oriented”

OCR for page 86
0 SBIR AT THE DEPARTMENT OF DEFENSE 45 40 35 Percent of Responses 30 25 20 15 10 5 0 No Pending + No Pending + Pending + No Pending + No Marketing Mar keting Mar keting Mar keting FIGURE 4-17 Patenting and marketing activities tend to be complements not substitutes. SOURCE: NRC Phase II Survey. model with no IP activities, and just over 10 percent fit the “research-oriented” model with no marketing activities. 4-17 It thus appears that the SBIR program has been quite successful in encourag- ing firms engaging in early-stage research to focus on the commercial applica - tions that might be drawn from that work. Two-thirds of the respondents in this sample report some specific marketing activities related to the project. 4.5.2 Scientific Publications Publications fill two important roles in the study of SBIR programs. First, they provide an indication of the quality of the research being conducted with program funds. More than half of the DoD-funded projects were of sufficient value to generate at least one peer-reviewed publication. Second, scientific pub - lications are themselves the primary mechanism thorough which knowledge is transmitted within the scientific community. The existence of papers based on SBIR projects is therefore direct evidence that the results of these projects are being disseminated widely, which in turn means that the congressional mandate to support scientific outcomes is being met. Unlike NIH, where scientific publication is at the core of the enterprise for both the agency and the investigators (who are overwhelmingly drawn from

OCR for page 86
 OUTCOMES academic environments), publication is not always viewed as an unmixed bless - ing in the DoD environment. Even where founders had advanced degrees, their professional careers in the military and industry had moved them away from emphasizing peer-reviewed publications as a mode for establishing priority or disseminating research findings. More common in the security industry are pre - sentations at professional meetings, or briefings with sponsors and users. Knowl - edge in general seems to be viewed more pragmatically and commercially, being released publicly in some cases but not in others. Bearing these points in mind, considerable scientific publication still comes from DoD-funded SBIR research. Out of the 816 projects responding to the rel - evant question in the NRC Phase II Survey, 348 (42.6 percent) reported at least one scientific peer-reviewed publication related to the project, and some reported many more (see Figure 4-18). This compares with 53.5 percent at NIH. About 15 percent of DoD projects with publications had published a single paper, but one company had published 114 papers on the basis of its SBIR, two others had published at least 100, and two more had published between 50 and 100 papers. These data fit well with case studies and interviews, which suggested that SBIR companies are proud of the quality of their research, and justifiably so given their success in publication. Publications are featured prominently on many company Web sites, and companies like SAM—among many others—made a point during interviews that their work was of the highest technical quality as measured in the single measure that counts most in the scientific community, peer-reviewed publications. 30+ 11-30 (1%) (1%) 6-10 (6%) 1(32%) 5 (7%) 4 (8%) 3 (16%) 2 (29%) FIGURE 4-18 Project-related scientific peer-reviewed publications. NOTE: This figure provides a breakout of the number of publications among those that reported at least one scientific peer-reviewed publication. SOURCE: NRC Phase II Survey. 4-18

OCR for page 86
 SBIR AT THE DEPARTMENT OF DEFENSE 4.5.3 SBIR and the Universities There is anecdotal evidence from case study interviews and papers presented at the 2004 Technology Transfer Society Conference in Albany that university views about SBIR have begun to change.75 Traditionally, universities have been wary of SBIR, partly because it might “distract” faculty from the pure pursuit of knowledge, and partly because it may lure faculty away from academic work altogether into commercial research. More fundamentally, SBIR has been seen as diverting resources from the other activities of research agencies—notably NSF and NIH—whose funds would likely find their way to universities or university researchers. This view is changing. As universities themselves have become much more concerned about the commercial side of research—witness the dramatic in- crease in university patenting and licensing activities reported by NSF and the widespread introduction of IP-related clauses into faculty contracts—they have begin to see SBIR as a positive source of funding for research. Universities are increasingly looking toward mechanism of technology transfer as they increase their ties to their surrounding locales. As a part of this effort, many universities now make a conscious effort to inform faculty about SBIR, even helping faculty put together SBIR applications. A quarter of projects indicated that there had been involvement by universi - ties’ faculty, graduate students, or the university itself in developed technologies. For the response to a follow-up question on the variety of relationships this en - compasses, see Table 4-19.76 The wide range of roles played by university staff and students indicate once more the multiple ways in which SBIR projects increase the knowledge base of the nation. Involvement in these projects provides different opportunities for university staff than those available within the academy. The results of this change in perspective were certainly indicated in the course of case studies, where a number of SBIR recipient firms indicated the importance of their ties to universities. These stories suggest that the flow of information and funding between small businesses and universities working within the SBIR framework is neither simple nor unidirectional. The constant flow of feedback, testing, and insights between university researchers and staff at SBIR awardees helped to move those compa- nies forward toward product deployment into new research areas. One further impact of SBIR has been to facilitate transitions of both tech- 75 See. 76 Seequestions 30 and 31 in the NRC Phase II Survey, presented in Appendix B. Of the 837 re - spondents to these questions, 25 percent responded in the positive (and 75 percent in the negative) to question 30 as to whether there was any involvement by university faculty, graduate students, and/or university developed technologies in executing this award. Responses to Question 31, shown in Table 4-19 address any relationships between the respondent’s firm on the Phase II project being surveyed the same 837 respondents who answered Question 30, not just those who answered “yes.”

OCR for page 86
 OUTCOMES TABLE 4-19 University Involvement in SBIR Projects 1.3% The Principal Investigator (PI) for this Phase II project was at the time of the project a faculty member. 1.3% The Principal Investigator (PI) for this Phase II project was at the time of the project an adjunct faculty member. 13.6% Faculty member(s) or adjunct faculty member(s) work on this Phase II project in a role other than PI, e.g., consultant. 11.4% Graduate students worked on this Phase II project. 9.2% University/college facilities and/or equipment were used on this Phase II project. 2.2% The technology for this project was licensed from a university or college. 3.9% The technology for this project was originally developed at a university or college by one of the percipients in this Phase II project. 12.5% A university or college was a subcontractor on this Phase II project. NOTE: Survey respondents could check more than one category. SOURCE: NRC Phase II Survey (n = 837). nologies and researchers from university labs to the commercial environment. Data from the NRC Firm Survey (using data for all agencies) strongly support this hypothesis, with 66 percent of SBIR companies including at least one aca - demic as founder, and 28 percent having more than one academic as a founder (see Figure 4-19). The same survey found that about one-third of founders were most recently employed in an academic environment before founding the new company. These data and evidence from case studies strongly indicate that SBIR has indeed encouraged some academic scientists to work in a more commercial environment. What is not clear from this research is the extent to which universities them- selves see SBIR as a mechanism for technology transfer, commercialization, and additional funding for university researchers. These questions are beyond the scope of the current study but merit additional research. 4.5.4 Inventions and Indirect Knowledge This analysis has understandably focused on the data available from the NRC Phase II Survey about the IP-related activities of firms. However, it must be stressed that these are only the formal IP-related activities. Every project gen- erates a very wide range of less formal, less easily captured knowledge effects, which are nonetheless important despite being very difficult or even impossible to quantify. The case of Thermacore, described in Box 4-9, provides some insights into this kind of program effect.

OCR for page 86
 SBIR AT THE DEPARTMENT OF DEFENSE BOX 4-8 Brimrose Corporation Brimrose Corporation was founded by Dr. Ronald G. Rosemeier while still a Ph.D. student in Johns Hopkins University material science program. After gradu- ation, working as a post-doctoral student at the University of Maryland, he started writing SBIR proposals. After three years of submitting unsuccessful proposals, he was awarded four Phase I’s, whose total value approximated $200,000. He started hiring his first em- ployees and began applying for loans at banks. Because banks were not willing to give him loans backed by the SBIR awards, he amassed charges of $100,000 on credit cards. Six months later he wrote the follow-on Phase II proposals, receiving awards on three of them for a total of $1.5 million. With these funds, he was able to hire additional employees, and expand opera- tions. At that time 10 percent of the firm’s revenues were from commercial sales (selling X-ray imaging at tradeshows) and 90 percent from the SBIR awards. As the firm started commercializing new products, this percentage shifted to 80 per- cent commercial revenue and 20 percent SBIR revenue. Most of the R&D team and few of the support staff were hired under SBIR related activities. Brimrose began operations with 6 employees; by 2005 its employment level had reached approximately 60 employees. Overall, throughout its history, Brimrose has re- ceived 65 Phase I and 28 Phase II SBIR awards. The firm’s major lines of business are industrial process control spectroscopy in the pharmaceutical and petrochemical industries, nondestructive testing and evaluation and novel opto-electronics devices. Its business model is to specialize in applied R&D. A few of the SBIR programs have directly resulted in commercial products but most have led to product improvements. The firm’s commercialization strategy emerged from and has been greatly en- hanced through its participation in SBIRs. The Phase I and Phase II SBIR funding allowed them to determine the feasibility of new technology and develop it to the point of prototype development without allocation of significant internal resources. Following prototype development, Brimrose used internal funds from previous commercial sales to bring the technology to the point of commercial availability. Thus, the SBIR funds lowered the company’s financial burden by decreasing the risks associated with new technology development. 4.6 UNDERSTANDING OUTCOMES: EMPIRICAL FINDINGS Overlapping methods were used to address questions relating to commer- cialization of DoD’s SBIR research. First, a Web-based survey was conducted of firms that had received at least one Phase II award between the years 1992– 2001.77 Second, case histories were prepared of a sample of firms that had received Phase II awards. Thirty-one firms representing a cross-section of DoD sponsoring agencies and states were selected for study. Third, an additional Web- 77Appendix B contains a full account of the survey methodology.

OCR for page 86
 OUTCOMES BOX 4-9 Thermacore: Creating Knowledge and Capacity Many of the benefits to participants and their clients are hard to quantify. Thermacore, for example, believes that its experiences under the SBIR program provided it with a “brain pool” of “know-how” related to manufacturing reliability. These tacit skills have contributed to the firm’s ongoing competitive position even as patents on its initial core technologies have expired. Thermacore was founded by Yale Eastman, an RCA employee, in 1970. The firm started as a “garage” start-up, focused on RCA-abandoned heat pipe tech- nologies for solar applications. Throughout most of the 1970s, the company remained small, with no more than ten employees, working on industry and gov- ernment R&D contracts. In the 1980s, it began to grow via non-SBIR and SBIR R&D contracts primarily from NASA, DoD, and DoE. Thermacore and SBIR. Thermacore began active pursuit of SBIR awards in the early 1980s. While it remained eligible, Thermacore received 82 SBIR awards from several government agencies, including DoD, NASA, and DoE. This substantial and repeated support, provided by several agencies over a number of years, highlights the way in which SBIR supports technologies that are complex and require a number of incremental technological advances to transition an R&D concept into a viable commercial product. Thermacore describes its growth as a case of the marketplace catching up to its technology. In the early 1990s, Thermacore was approached by Intel to discuss the possibility of mass producing its heat pipe technology for use in the rapidly growing market for personal computers. With financial support from a venture capital firm, Thermacore took the risk of setting up a production line before receiv- ing orders. Subsequently, it received large orders from several major computer manufacturers, such as HP, Dell, IBM, and Sun. Reflecting its transition from an R&D to a production-oriented firm, contract R&D projects and OEM work now account for only 6 percent of revenues; 94 percent comes from sale of commercial products. After its sale in 2001, following the retirement of its founder, Thermacore be- came ineligible for SBIR awards because it was a wholly owned subsidiary of a larger company. It does however continue to do some SBIR-funded research as a subcontractor to small firms conducting Phase I and II research. based survey addressed Phase I awards. Additional data were collected from the DoD CAI index database. Collectively, these methods strongly suggest that DoD companies funded by SBIR do commercialize their results, and that the rate of commercialization appears to be increasing. Thus, the charge of low commercialization by multiple- award winners no longer appears to be supported by the data.78 If it even was a 78The 1992 GAO study that identified this “trend” is often cited by the GAO investigators, multiple caveats are not. For example, the limited time frame of the study for several of the years studied, e.g., 1987 awards reviewed in 1990. See U.S. Government Accounting Office, Federal Research: Small Business Innoation Research Shows Success but Can Be Strengthened, GAO/RCED-92-37, op. cit.

OCR for page 86
 SBIR AT THE DEPARTMENT OF DEFENSE Unknown (5%) >1 (28%) 0 (29%) 1 (38%) FIGURE 4-19 Number of academics as founders. SOURCE: NRC Firm Survey. problem in the past, it is now to a considerable extent resolved, with some very large winners graduating from the program in various ways (e.g., Foster-Miller’s purchase by a foreign 4-19 new made it ineligible) and also clear evidence that Fgure firm, which some multiple-award winners are now reducing their dependence on SBIR, and to generate more substantial commercial results (e.g., Radiation Monitoring, PSI). The findings from the three approaches are generally consistent with one another and complementary. The case studies provide explanations for findings generated via the two sets of surveys. The findings from the separate approaches at times though tend to emphasize different aspects of SBIR’s program impacts mainly because they frame the question of commercialization differently. The survey findings, in keeping with congressional and Executive Branch usage, de - fine commercialization in terms of economic outcome measures, such as sales, investment, employment, and patents. The case histories contain data on these variables, but also narratives about firm formation, business strategies, intellectual property rights strategies, and processes of technological innovation and diffusion of innovations. They also contain limited data on the gestation processes connect- ing SBIR awards and commercial outcomes. The review of outcomes described in this chapter provides an overview of an effective program, meeting a wide variety of program goals. The DoD awards are generating new knowledge evidenced in publications and successful patent -

OCR for page 86
7 OUTCOMES ing activity, new products and processes to maintain equipment, improve sup - ply management, and develop new products to better support and defend those charged with combat missions. The program also provides valuable linkage be- tween university professors and students and the commercial and defense market place. By growing and nurturing the defense industrial base, the program is also encouraging high-tech entrepreneurship, thereby increasing innovation, encour- aging competition, and offering greater choices. The SBIR program is helping DoD to meet the new and often sudden challenges of a turbulent world.