National Academies Press: OpenBook
« Previous: 3 SBIR Awards at DoD
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

4
Outcomes

4.1
INTRODUCTION

Identifying the specific outcomes resulting from an early-stage R&D program such as SBIR is challenging.1 The long lag between input (funding) and output (possible products and services), combined with the frequent need for multiple inputs for successful technology development, make definitive assessments of the link between a single input and a complex output difficult. In addition there are very substantial data collection problems, as awardees and agencies cannot consistently capture outcomes for all supported projects. Many early-stage research projects generate little that is tangible in the form of products and services while a few projects can generate very large returns. The large skew means that anything short of an all-inclusive analysis risks missing important contributions from the program.

While keeping these caveats in mind, this chapter seeks to provide as broad an assessment of outcomes from the DoD SBIR program as possible. It will focus on whether SBIR is meeting its four congressional objectives. These are “(1) to stimulate technological innovation; (2) to use small business to meet federal research and development needs; (3) to foster and encourage participation by minority and disadvantaged persons in technological innovation; and (4)

1

For a summary of the challenges of tracking specific SBIR outcomes, see National Research Council, SBIR: Program Diversity and Assessment Challenges, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2004, pp. 32-35. Data in this chapter are derived from the NRC Phase II Survey unless otherwise specified.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

to increase private sector commercialization derived from federal research and development.”2

4.2
COMMERCIALIZATION

4.2.1
Background

Bringing new technologies developed under the research supported by SBIR awards to the marketplace has been a central objective of the SBIR program since its inception. The program’s initiation in the early 1980s in part reflected a concern that American investment in research was not being transformed adequately into products that could generate greater wealth, more employment, and increased competitiveness. Directing a portion of federal investment in R&D to small businesses was thus seen as a new means of meeting the mission needs of federal agencies while increasing the participation of small business and thereby the proportion of innovation that would be commercially relevant.3

Congressional and Executive Branch interest in the commercialization of SBIR research has increased over the life of the program.

A 1992 GAO study4 focused on commercialization in the wake of congressional expansion of the SBIR program in 1986.5 The 1992 reauthorization specifically “emphasize[d] the program’s goal of increasing private sector commercialization of technology developed through federal research and development6 and noted the need to “emphasize the program’s goal of increasing private sector commercialization of technology developed through federal research and development.” The 1992 reauthorization also changed the order in which the

2

The Small Business Innovation Development Act (PL 97-219).

3

A growing body of evidence, starting in the late 1970s and accelerating in the 1980s indicates that small businesses were assuming an increasingly important role in both innovation and job creation. See, for example, J. O. Flender and R. S. Morse, The Role of New Technical Enterprise in the U.S. Economy, Cambridge, MA: MIT Development Foundation, 1975, and David L. Birch, “Who Creates Jobs?” The Public Interest, 65:3-14, 1981. Evidence about the role of small businesses in the U.S. economy gained new credibility with the empirical analysis by Zoltan Acs and David Audretsch of the U.S. Small Business Innovation Data Base, which confirmed the increased importance of small firms in generating technological innovations and their growing contribution to the U.S. economy. See Zoltan Acs and David Audretsch, “Innovation in Large and Small Firms: An Empirical Analysis,” The American Economic Review 78(4):678-690, Sept 1988. See also Zoltan Acs and David Audretsch, Innovation and Small Firms, Cambridge, MA: The MIT Press, 1990.

4

U.S. Government Accounting Office, Federal Research: Small Business Innovation Research Shows Success but Can Be Strengthened, GAO/RCED-92-37, Washington, DC: U.S. Government Accounting Office, March 1992.

5

PL 99-443, October 6, 1986.

6

PL 102-564 October, 28, 1992.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

program’s objectives are described, moving commercialization to the top of the list.7

The term “commercialization” means, “reaching the market,” which some agency managers interpret as “first sale”—that is the first sale of a product in the market place, whether to public or private sector clients. This definition, however misses significant components of commercialization that do not result in a discrete sale. It also fails to provide any guidance on how to evaluate the scale of commercialization, an important element in assessing the degree to which SBIR programs successfully encourage commercialization. The metrics for assessing commercialization can also be elusive. It’s not straightforward, for example, to calculate the full value of an “enabling technology” that can be used across industries. Also elusive is the value of materials that enable a commercial service.8

In light of the difficulties in measuring commercialization effectively, the Navy SBIR program manager has suggested that a firm’s success in securing Phase III funding from an agency be substituted for the current weight accorded commercialization in the Commercialization Achievement Index (CAI) measure used during the proposal selection competition (the CAI is discussed in more detail in Chapter 6). Given the earlier noted variations in the S&T needs, sizes, and institutional arrangements of the services and units, applying this performance measure consistently is not always self evident.

In fact, efforts to identify Phase III results may have been unduly limited. The initial 1982 SBIR legislation noted that Phase III is not time-bound and can come long after the end of the Phase II; Phase III can include private sector sales. The law indicates that commercialization “may also involve non-SBIR, government-funded production contracts with a federal agency….”9

Moreover, Phase III funding comes via a wide variety of mechanisms. Firms receive modifications to add Phase III federal R&D funding to Phase II SBIR contracts, they have won production contracts or R&D contracts competitively, sold to prime contractors, received additional private sector funding, and sold products commercially. All of these are Phase III activities in accordance with the legislation and with the SBA policy directive. Consequently, a narrow definition of Phase III, as a noncompetitively awarded further R&D or production

7

These changes are described by R. Archibald and D. Finifter in “Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach” in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, Washington, DC: National Academy Press, 2000.

8

For a discussion of this and related methodological challenges, see, National Research Council, Assessment of the Small Business Innovation Research Program—Project Methodology, Washington, DC: The National Academies Press, 2004, accessed at <http://books.nap.edu/catalog.php?record_id=11097#toc>.

9

U.S. Government Accounting Office, Federal Research: Small Business Innovation Research Shows Success but Can Be Strengthened, GAO/RCED-92-37, op. cit., p. 14.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

contract, and one that is only captured if properly entered in the DD350 report is too limited as an approach.

In addition, DD350 documentation of Phase III funding does not occur until at least 1 year following completion of any Phase II enhancement awards, the form itself is often not filled out completely or appropriately, as data tests run by BRTRC indicate.10 DoD staff indicate that Navy makes a considerable effort to ensure that its DD350 forms fully capture SBIR Phase III activities to the maximum extent possible. Other agencies do not.11

In sum, while they do provide one important measure of commercialization and one that could be used more effectively, under current circumstances, the DD350 reports may not provide sufficiently comprehensive or accurate data on which to make definitive determinations about the success of DoD SBIR commercialization. Indeed, the multiple goals of the SBIR program mean that multiple measures are appropriate for evaluation.

4.2.2
Proposed Commercialization Indicators and Benchmarks

This report uses three sets of indicators to quantitatively assess commercialization success:

  1. Sales and licensing revenues (“sales” hereafter, unless otherwise noted). Revenues flowing into a company from the commercial marketplace constitute the most obvious measure of commercial success. They are also an important indicator of uptake for the product or service. Sales indicate that the result of a project has been sufficiently positive to convince buyers that the product or service is the best available solution.

    Yet if there is general agreement that sales are a key benchmark, there is no such agreement on what constitutes “success.” Companies, naturally enough, focus on projects that contribute to the bottom line—that are profitable. Agency staff provide a much wider range of views. Some view any sales a substantial success for a program focused on such an early stage of the product and development cycle, while others seem more ambitious.12 Some senior executives in the private sector viewed only projects that generated cumulative revenues at $100 million or more as a complete commercial success.13

    Rather than seeking to identify a single sales benchmark for “success,” it therefore seems more sensible to simply assess outcomes against a range of

10

Peter Cahill, BRTRC, private communication, December 1, 2006.

11

Michael Caccuitto, DoD SBIR/STTR Program Administrator, Interview, November 28, 2006.

12

Interviews with SBIR program coordinators at DoD, NIH, NSF, and DoE.

13

Pete Linsert, CEO, Martek, Inc., Meeting of the Committee for Capitalizing on Science, Technology, and Innovation: An Assessment of the Small Business Innovation Research Program, June 5, 2005.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

benchmarks reflecting these diverse views, with each marking the transition to a greater level of commercial success:

  1. Reaching the market—A finished product or service has made it to the marketplace.

  2. Reaching $1 million in added cumulative sales (beyond SBIR Phases I and II)—The approximate combined amount of standard DoD Phase I and Phase II awards.

  3. Reaching $5 million in cumulative sales—A modest commercial success that may imply that a company has broken even on a project.

  4. Reaching $50 million in cumulative sales—A full commercial success.

  1. Phase III activities within DoD. As noted above, Phase III activities within DoD are a primary form of commercialization for DoD SBIR projects. These activities are considered in Section 4.3 and Chapter 5.

  2. R&D investments and research contracts. Further R&D investments and contracts are good evidence that the project has been successful in some significant sense. These investments and contracts may include partnerships, further grants and awards, or government contracts. The benchmarks for success at each of these levels should be the same as those above, namely:

    1. Any R&D additional funding.

    2. Additional funding of $1 million or more.

    3. Additional funding of $5 million or more.

    4. Funding of $50 million or more.

  1. Sale of equity. This is a less clear-cut indicator of commercial success. but it is unlikely investors or competitors would buy equity in a company that had not shown its ability to produce something of significant value. Key metrics include:

    1. Equity investment in the company by independent third party.

    2. Sale or merger of the entire company.

4.2.3
Sales and Licensing Revenues from DoD SBIR Awards

The most basic of all questions on commercialization is whether a project produced a good or service that reached the marketplace. Figure 4-1 shows the status of surveyed projects. It shows that only a fairly small percentage of SBIR projects have been discontinued with no hope of ever generating sales (26 percent), though it is likely that a significant portion of the projects currently in development will also fail to achieve significant commercial success.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
FIGURE 4-1 Status of surveyed projects.

FIGURE 4-1 Status of surveyed projects.

NOTE: The NRC deployed two surveys to the population of Phase II recipients as part of the research conducted for this project. The NRC Phase II Survey focused on individual projects. In addition, the NRC Firm Survey was sent to every firm receiving a Phase II award between 1992 and 2001, and focused on firm-level questions. Unless otherwise stated, all references here to the NRC Survey are to the project survey.

SOURCE: NRC Phase II Survey.

These data indicate that 46 percent of surveyed projects reported some revenues from their project; a further 18 percent were still in development and expected sales, and 5 percent had not yet completed Phase II.

4.2.3.1
Sales Ranges

Early-stage technology projects are inherently risky. As a result, there is a very skewed distribution of results. Many projects generate no commercial results at all, and relatively few of those that do reach the market have substantial commercial successes (see Figure 4-2).

The data suggest that at DoD—as at other agencies—the overwhelming majority of sales are concentrated in the $0–$1 million range. Ten percent of reporting projects generated at least $5 million in revenues, while more than 65 percent of respondents with sales reported total sales of less than $1 million revenues (as of May 2005, the date of the survey). As a result of this very skewed distribution, the mean amount of sales for all companies that reported sales was $2,894,834, while the median was $500,000.

Underreporting of Sales Results. The average total sales for older projects is much higher than for recent ones. In fact, the average sales for the 176 reporting DoD projects awarded Phase II contracts from 1992 to 1994 was $2.78 million, whereas the average sales for the 415 reporting DoD projects awarded Phase II

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
FIGURE 4-2 Sales by sales range.

FIGURE 4-2 Sales by sales range.

SOURCE: NRC Phase II Survey.

contracts between 1999 to 2001 was only $982,000. This difference in part reflects the number of recent awardees whose products have yet to be commercially introduced or fully exploited commercially. Reported aggregate and average sales data up through the survey date of May 2005 are therefore only a partial estimate of the total commercial impact of the 920 awards covered by the NRC Phase II Survey.14 According to former senior DoD staff, average major DoD weapons system R&D cycle is approximately 12 years (before production)—so SBIR

14

Using the trendline shown in Figure 4-3, we find that the best fit generates average sales of approximately $5.5 million per project for those with awards in 1992, declining to averages sales of $1 million for those in 2001. Note that these data cover only firms reporting some sales. The trendline gives us a means of estimating the eventual sales generated by each project, using simplified assumptions (notably, that all sales end by May 2005 (the date of the NRC Phase II Survey), and that commercialization remains constant across time (in fact, it is likely to have increased as agencies have increased their focus on supporting projects with better prospects of commercial success). Other assumptions tend to reduce the size of the estimated revenues.

Using these estimates to project forward, we find that by the time commercialization of all projects is completed (i.e., ten years after the last project funded, or 2011), total revenues generated by the projects reporting revenues so far is $2.13 billion.

These data suggest that the reported revenues as of May 2005 may understate eventual total revenues as of 2011 by as much as 50 percent.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
FIGURE 4-3 Average sales, by year of award, plus trendline.

FIGURE 4-3 Average sales, by year of award, plus trendline.

SOURCE: NRC Phase II Survey.

TABLE 4-1 Reported and Projected Revenues for Companies that Reported Sales as of May 2005

 

Amount ($)

 

1,094.0

Reported Sales (Millions of Dollars)

1,040.5

Total Sales (Millions of Dollars)

2,134.5

Average Sales per Project with Sales ($)

5,646,787

Average Sales per Project—All Projects ($)

2,151,699

SOURCE: NRC Phase II Survey.

FIGURE 4-4 Reported sales and expectations.

FIGURE 4-4 Reported sales and expectations.

SOURCE: NRC Phase II Survey.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

TABLE 4-2 For Companies Anticipating Sales, Year of Expected First Sale

Year of Expected Sales

Number of Projects

Percent of Projects

2005

37

22.0

2006

70

41.7

2007

31

18.5

2008

18

10.7

2009

5

3.0

2010

7

4.2

Total

168

 

NOTE: Survey Date: 2005.

SOURCE: NRC Phase II Survey.

products for defense sales would tend to have a long time lag.15 Thus, Figure 4-3 should not be taken to mean that sales are declining; it largely reflects the extent of the lag in DoD-oriented sales.

Sales Concentration. Total revenues from sales are highly concentrated in the very few projects that have generated at least $5 million in cumulative revenues. Just under 75 percent of all cumulative sales were accounted for by the 38 projects (out of 920 overall) that reported at least $5 million in sales. This very high concentration confirms the view that from the perspective of sales, the SBIR program at DoD generates a few major winners, rather than a more widely dispersed range of more modest successes. This is similar to commercial outcomes from early-stage R&D programs.

4.2.3.2
Sales Expectations

About a quarter of projects reported that they expected sales in the future. Of those companies not yet reporting sales on their projects, about 67 percent still expect them.

Most of the respondents that had not yet received sales expected sales to come in the very near future, as shown by Table 4-2. Of those expected sales, 80 percent anticipated that their first sale would occur within 3 years.

The data in Figure 4-5 show, for projects that have received sales, the time that elapsed between the Phase II award and the first sales. Survey responses indicate that 87.6 percent of first sales occurred within 4 years of the award date. This relatively short time from award to first sale is supported by the comments of John Williams, Navy SBIR Manager.16 However, it must be stressed that

15

Dr. Jacques Gansler, former Under Secretary of Defense for Acquisition, Technology and Logistics, January 29, 2007.

16

John Williams, U.S. Navy SBIR program manager, presentation at SBTC SBIR in Rapid Transition Conference, Washington, DC, September 27, 2006.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
FIGURE 4-5 Time elapsed between award and first sales—frequency distribution.

FIGURE 4-5 Time elapsed between award and first sales—frequency distribution.

SOURCE: NRC Phase II Survey.

interviews and cases strongly support the view that the bulk of sales will occur some years after the date of first sale. The latter is therefore best seen as a leading indicator for sales.

These data from the NRC’s May 2005 Phase II Survey help us to evaluate claims of companies that they will generate sales in the future. About 25 percent of all DoD survey respondents made this claim. However, the likelihood of commercialization diminishes substantially with time elapsed since the award. Projects still expecting sales are clustered toward the 2000–2001 timeframe, but even here the likely window of opportunity for success appears to be closing rapidly. The median time to first sale is before the end of the second year after the award.

This analysis suggests that though a considerable number of companies anticipate sales in the future, the actual likelihood of this occurring is relatively low. By the start of the 9th year after the Phase II award is made, 99 percent of projects that will eventually report sales have done so; the likelihood a project without sales reporting sales after the 8th year is less than 1 percent. Similarly, projects not reporting sales by the start of year five have a 13.8 percent chance of eventually reporting sales (to put it yet another way, by the end of the fourth year after the award year, 86.2 percent of projects that eventually make sales will have started to do so).17

Table 4-3 helps us to determine likely revenues for companies that expect

17

This is not to say that all sales will have been completed by the end of that year, only that it is quite reasonable to apply these percentages to the more recent awards, as a way of estimating eventual sales data.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

TABLE 4-3 Total Projected Sales for Companies Without Sales That Still Expect Them

Year of Award

Projects Expecting Sales

Historical Success Percentage

Projected Total Revenues ($)

1992

3

0.0

0

1993

4

0.0

0

1994

4

0.3

58,730

1995

8

0.8

352,381

1996

6

0.3

88,095

1997

13

0.3

190,873

1998

18

2.4

2,378,571

1999

19

3.7

3,905,556

2000

32

4.8

8,457,143

2001

61

10.1

34,034,127

Total

 

 

49,465,476

NOTE: Projected sales are calculated by multiplying the number of companies reporting that they still expect sales, by the percentage likelihood that a company in that award year will in fact generate sales, by the average total sales for all companies that did record more than $0 in sales.

SOURCE: NRC Phase II Survey.

them, based on historical records. The total projected revenues for these companies is relatively low—about $50 million, or less than 2.5 percent of all projected sales. We can therefore conclude that while ongoing revenues from companies with some sales in hand will be relatively large (see above), the revenues expected from companies that had not yet reported some sales as of May 2005 are likely to be of limited significance.

These findings suggest that while the product cycle for the entire defense industry may be long—a well-known characteristic of major defense systems—the first sales cycle for most SBIR-related products is actually relatively short. Most successful project start receiving initial sales revenues within 4 years of the award, while large sales tend to come considerably later due to the defense procurement cycle.18

Sales and Projected Sales: Conclusions. The NRC Phase II Survey provided the following summary data regarding sales and projected sales:

  • 378 out of a total of 920 respondents ( 41.1 percent) report sales greater than $0.

  • The average reported sales is $1.3 million for all projects (n=920), and $3.2 million for those reporting sales greater than $0 (n=378).19

18

This paragraph concerns only first sales. The bulk of sales occur at some unknown period after the first sale.

19

When projected out to 2011, estimated average sales per project are $5.6 million for projects already reporting some sales, and $2.2 million for all projects.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

TABLE 4-4 Customer Base

Customer

Percent of Total Sales

Domestic private sector

21

Department of Defense

38

Prime contractors for DoD or NASA

12

NASA

1

Other federal agencies

1

State or local governments

1

Export markets

11

Other

16

Total

100

SOURCE: NRC Phase II Survey.

  • Finally, additional sales from projects with no reported sales as of May 2005 are likely to be of limited importance—less than $50 million in total.20

4.2.3.3
Sales by Sector

The NRC Phase II Survey asked respondents to identify the customer base for their products. The responses are summarized in Table 4-4.

While the fact that half of sales went either to DoD or DoD/NASA prime contractors is not surprising, the balance between the two is somewhat at odds with comments made by many interviewees and speakers at the NRC Phase III conference. Those comments indicated that it was very hard for SBIR firms to sell directly to DoD and that sales had to be mediated through the primes. The data above suggest that this is much less the case than conventional wisdom would suggest, as more than one-third of sales went directly to DoD, and these sales constitute the largest single sector market for DoD SBIR recipients. However, it is also possible that the question was asked with insufficient precision, or that some of these were limited sales—e.g., prototypes to DARPA.

Phase II projects lead to several forms of new products and processes, with some new technologies having multiple characteristics. Allowing for more than one response, the most prevalent form reported was “hardware” (60 percent), which may occur as a final product, component, or intermediate product. “Hardware” was followed by “software” (32 percent) and “process technology” (23 percent). Of note is that some reported outputs occur in the forms of “new or improved service capability” (18 percent) and “research tool” (15 percent).21

20

However, given the highly skewed nature of sales outcomes, it is entirely possible that one of the companies that does reach the market after May 2005 will turn out to be a major success, but there is no way of predicting whether that will be the case.

21

Other federal agencies report a significantly higher percent of research tools and educational materials (26 percent and 13 percent respectively for the other four agencies in aggregate).

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

BOX 4-1

Multiple Sources of Bias in Survey Response

Large innovation surveys involve multiple sources of bias that can skew the results in both directions. Some common survey biases are noted below. These biases were tested for and responded to in the NRC surveys.a

  • Successful and more recently funded firms are more likely to respond. Research by Link and Scott demonstrates that the probability of obtaining research project information by survey decreases for less recently funded projects and it increased the greater the award amount.b Nearly 40 percent of respondents in the NRC Phase II Survey began Phase I efforts after 1998, partly because the number of Phase I awards increased, starting in the mid-1990s, and partly because winners from more distant years are harder to reach. They are harder to reach as time goes on because small businesses regularly cease operations, are acquired, merge, or lose staff with knowledge of SBIR awards.

  • Success is self-reported. Self-reporting can be a source of bias, although the dimensions and direction of that bias are not necessarily clear. In any case, policy analysis has a long history of relying on self-reported performance measures to represent market-based performance measures. Participants in such retrospectively analyses are believed to be able to consider a broader set of allocation options, thus making the evaluation more realistic than data based on third-party observation.c In short, company founders and/or principal investigators are in many cases simply the best source of information available.

  • Survey sampled projects at firms with multiple awards. Projects from firms with multiple awards were underrepresented in the sample, because they could not be expected to complete a questionnaire for each of dozens or even hundreds of awards.

  • Failed firms are difficult to contact. Survey experts point to an “asymmetry” in their ability to include failed firms for follow-up surveys in cases where the firms no longer exist.d It is worth noting that one cannot necessarily infer that the SBIR project failed; what is known is only that the firm no longer exists.

  • Not all successful projects are captured. For similar reasons, the NRC Phase II Survey could not include ongoing results from successful projects in firms that merged or were acquired before and/or after commercialization of the project’s technology. The survey also did not capture projects of firms that did not respond to the NRC invitation to participate in the assessment.

  • Some firms may not want to fully acknowledge SBIR contribution to project success. Some firms may be unwilling to acknowledge that they received important benefits from participating in public programs for a variety of reasons. For example, some may understandably attribute success exclusively to their own efforts.

  • Commercialization lag. While the NRC Phase II Survey broke new ground in data collection, the amount of sales made—and indeed the number of projects that generate sales—are inevitably undercounted in a snapshot survey taken at a single point in time. Based on successive data sets collected from NIH SBIR award recipients, it is estimated that total sales from all responding projects will likely be on the order of 50 percent greater than can be captured in a single survey.e This underscores the importance of follow-on research based on the now-established survey methodology.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
FIGURE B-4-1 Survey bias due to commercialization lag.

FIGURE B-4-1 Survey bias due to commercialization lag.

These sources of bias provide a context for understanding the response rates to the NRC Phase I and Phase II Surveys conducted for this study. For the NRC Phase II Survey for DoD, of the 2,191 firms that could be contacted out of a sample size of 3,055, 920 responded, representing a 42 percent response rate. The NRC Phase I Survey captured 9 percent of the 13,103 awards made by DoD between 1992 to 2001. See appendixes B and C for additional information on the surveys.

  

aFor a technical explanation of the sample approaches and issues related to the NRC surveys, see Appendix B.

  

bAlbert N. Link and John T. Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Program’s Intramural Research Initiative, London: Routledge, 2005.

  

cWhile economic theory is formulated on what is called “revealed preferences,” meaning individuals and firms reveal how they value scarce resources by how they allocate those resources within a market framework, quite often expressed preferences are a better source of information especially from an evaluation perspective. Strict adherence to a revealed preference paradigm could lead to misguided policy conclusions because the paradigm assumes that all policy choices are known and understood at the time that an individual or firm reveals its preferences and that all relevant markets for such preferences are operational. See (1) Gregory G. Dess and Donald W. Beard, “Dimensions of Organizational Task Environments,” Administrative Science Quarterly, 29: 52-73, 1984. (2) Albert N. Link and John T. Scott, Public Accountability: Evaluating Technology-Based Institutions, Norwell, MA: Kluwer Academic Publishers, 1998.

  

dAlbert N. Link and John T. Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Program’s Intramural Research Initiative, op. cit.

  

eData from NIH indicates that a subsequent survey taken two years later would reveal very substantial increases in both the percentage of firms reaching the market, and in the amount of sales per project. See National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
4.2.3.4
Sales by Size of Company (Employees)

Another important question is whether the size of a SBIR contract recipient seems to significantly affect commercialization. It would seem, for instance, that companies that are extremely small would have more difficulty managing both the research and marketing functions needed for commercial success.

The data shown in Table 4-5 indicates that very small companies do tend to have less commercialization success (as measured by cumulative sales).

Further analysis suggests that companies larger than 25 employees report significantly better sales outcomes, as shown by Table 4-6.

Companies with more than 25 employees seem to consistently outperform companies with less than 25 employees, in terms of projects that generate at least $5 million in sales. The former account for 31.5 percent of all responding projects, but 74.5 percent of all projects reporting sales of at least $5 million.

One possible hypothesis for explaining this difference, based on discussions at the NRC Phase III Conference and with case study companies, might be that acquisitions officers are more comfortable engaging with larger and presumably more stable companies that will likely have a longer track record (all other things being equal).

4.2.3.5
Sales by Licensees

Licensing revenues are an important source of commercialization activity for SBIR companies. Indeed, interviews with staff and awardees suggest that in some

TABLE 4-5 Cumulative Project Sales by Company Size at Time of Survey

Number of Employees

Cumulative Project Sales (Number of Projects)

Total

Percent

<$100K

$100K to < $1M

$1M to <$5M

$5M to <$50M

>$50M

0–5

19

33

6

2

 

60

15.8

6–10

12

24

9

6

 

51

13.5

11–15

12

21

10

3

 

46

12.1

16–25

8

19

24

2

 

53

14.0

26–50

7

21

17

17

 

62

16.4

51–100

6

12

20

8

 

46

12.1

101–250

6

20

8

5

 

39

10.3

251–500

1

5

6

5

2

17

5.0

500+

0

2

0

1

 

3

0.8

Missing

 

 

 

 

 

 

 

Total

71

157

100

49

2

377

 

Percent

18.7%

41.4%

26.4%

12.9%

0.5%

 

100.0%

SOURCE: NRC Phase II Survey.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

TABLE 4-6 Sales Outcomes by Size of Company at Time of Survey

Number of Employees

Sales Outcomes (Percent Distribution)

Percent of responses

Percent

<$100K

$100K to < $1M

$1M to <$5M

$5M to <$50M

>$50M

0–5

31.7

55.0

10.0

3.3

0.0

15.8

100.0

6–10

23.5

47.1

17.6

11.8

0.0

13.5

100.0

11–15

26.1

45.7

21.7

6.5

0.0

12.1

100.0

16–25

15.1

35.8

45.3

3.8

0.0

14.0

100.0

26–50

11.3

33.9

27.4

27.4

0.0

16.4

100.0

51–100

13.0

26.1

43.5

17.4

0.0

12.1

100.0

101–250

15.4

51.3

20.5

12.8

0.0

10.3

100.0

251–500

5.3

26.3

31.6

26.3

10.5

5.0

100.0

500+

0.0

66.7

0.0

33.3

0.0

0.8

100.0

All Responses

18.7

41.4

26.4

12.9

0.5

100.0

100.0

SOURCE: NRC Phase II Survey.

important cases, SBIR has been commercialized primarily not by the awardee company, but by a licensee.

The NRC Phase II Survey asked respondents to estimate sales by licensees. Table 4-7 shows what they reported.

As with direct sales data, these responses suggest both that a large majority of licensee sales are less than $1 million, and that there are a few very substantial licensing streams. However, this is an area that seems to be underreported in the survey. This may be due to a lack of information, as indicated by the considerable number of firms reporting sales by licensees, but no first date for those sales. Total sales reported for licensees were $124.9 million. Of this, $88 million (70.5 percent) came from the 6 projects reporting more than $10 million in licensee sales.

Case analysis also suggests that for some technologies, licensing may be the only realistic method of commercializing a product. For many research compa-

TABLE 4-7 Sales by Licensees

 

Count

Percent

>0 and <$1M

17

51.5

$1M to <$5M

10

30.3

$5M to <$10M

2

6.1

$10M to <$50M

4

12.1

$50M+

0

0.0

Total

33

100.0

SOURCE: NRC Phase II Survey.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

nies, the decision to begin manufacturing is seen as a high-risk venture that is often rejected in favor of a licensing-based strategy.

4.2.3.6
Employment Effects
4.2.3.6.1
Employment Effects

Employment analysis provides another method for identifying commercial success; it also offers another indication of support for small business.

The median size of company receiving SBIR awards at DoD is relatively small—far lower than the 500 employee limit imposed by the SBA (as shown in Table 4-8).

Just over 41 percent of respondents had 15 or fewer employees, while about 15 percent had more than 100. Three firms exceeded the 500-employee SBA limit for participation in the SBIR program (having grown since receiving the award). The median size of SBIR awardees at the time of the award was 10 employees; 62 percent of respondents had 15 employees or fewer.

Although employment is not a direct indicator for commercialization, it is clear that the two are related. Commercialization tends to require additional staff, and that additional staff requires additional revenues to support it. Those revenues, in turn, normally have to come from successfully commercializing a product.

4.2.3.6.2
Employment Gains

The NRC Phase II Survey sought detailed information about the number of employees SBIR awardees had at the time of the relevant award, the number of employees the awardees had at the time of the survey, and the estimated direct impact of the award on employment. Overall, it showed that the average employment gain at each responding firm since the date of the SBIR was 29 full-time

TABLE 4-8 Distribution of Companies, by Employees, at Time of the Survey

Number of Employees

Number of Firms

Percent of Firms

0–5

60

15.9

6–10

51

13.5

11–15

46

12.2

16–25

53

14.1

26–50

62

16.4

51–100

46

12.2

101–250

39

10.3

251–500

17

4.5

500+

3

0.8

Total

377

100.0

SOURCE: NRC Phase II Survey.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
FIGURE 4-6 Total firms employment change at firms between time of Phase II award and time of survey.

FIGURE 4-6 Total firms employment change at firms between time of Phase II award and time of survey.

SOURCE: NRC Phase II Survey.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

equivalent (FTE) employees. In addition, on average, respondents estimated that, specifically as a result of the SBIR project, their firms were able to hire 2.5 FTE employees, and to retain 2.3 more.22

The NRC Phase II Survey results do track with information from case studies, where several interviewees noted that a Phase II award typically funded the addition of slightly more than one full-time researcher for two years plus overhead23 but firms then grew on future sales and third-party funding.

Respondent firms report modest employment growth as a result of the specific SBIR award addressed in the survey. Forty-eight percent of respondents reported that they had added no employees as a result of receiving the Phase II award; 42 percent reported adding between one to five additional employees to work on the project. Five percent reported adding between 6–20 employees, and only two percent reported adding more than 20.24

Eight percent of firms were true “start-ups” at the time of the surveyed award, having no employees.25 At the time of their responses to the survey, only one percent of the firms remained in this category. At the time they submitted the Phase II proposal, 67 percent of SBIR awardees had 20 employees or fewer. In 2005, when they completed the survey, the firms had grown, and this percentage had dropped to 48 percent. At the upper end of the distribution, whereas 13 percent of respondents had between 21–50 employees at the time of their proposal submission and only eight percent had more than 100 employees, these firms had also grown, and these percentages had increased to 22 percent and 18 percent respectively by 2005.

4.2.4
Additional Investment, Funding, and Other Partnerships

Post-SBIR investment in a company is a powerful validation that its work is of value. About 53 percent of DoD respondents said that they had received some additional funding related to the surveyed project, other than further SBIR awards. As with sales, the distribution of funding is highly skewed, with a few companies receiving most of the additional investment (see Table 4-9).

The average investment was about $850,000 ($1.6 million per project if only those that received additional funding are counted).

22

NRC Phase II Survey, Questions 16b and 16c.

23

Interview with Josephina Card, Sociometrics, Inc.

24

See the remarks by Joshua Lerner of the Harvard Business School in National Research Council, The Small Business Innovation Research Program: Challenges and Opportunities, Charles W. Wessner, ed., Washington, DC: National Academy Press, 1999, p. 23.

25

Other than the founder, who may not have been drawing full-time equivalent income from the award.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

TABLE 4-9 Further Investments in SBIR Projects

 

Number of Responses

Investment ($)

$50M+

2

106,700,000

$5M to <$50M

32

324,151,193

$1M to <$5M

96

202,819,919

$100K to <$1M

241

90,112,919

<$100K

80

3,291,603

Total Investments

451

727,075,634

Average

 

1,612,141

No additional investment

402

0

Average (all responses)

 

852,375

SOURCE: NRC Phase II Survey, DoD awards database.

4.2.4.1
Sources of Investment Funding

Conventional wisdom is that the bulk of funding for near-market development comes from venture capitalists and angel investors. However, data from the NRC Phase II Survey—shown in Table 4-10—do not validate the conventional view.

About one-quarter of all further investments come from non-SBIR federal funds, while only 3.8 percent are from U.S. venture capital companies. However, the picture does change somewhat when we consider the amount of funding, rather than the number of investments (see Figure 4-7 and Table 4-11).

TABLE 4-10 Sources of Investment Funding (NRC)

Source of Investment

Number of Investments

Percent

Non-SBIR Federal Funds

205

25.8

Private Investment from U.S. Venture Capital

30

3.8

Your Own Company

274

34.4

Private Investment from Other Private Equity

55

6.9

Private Investment from Other Domestic Private Company

105

13.2

Private Investment from Foreign Investment

19

2.4

Personal Funds

65

8.2

State or Local Government

31

3.9

College or Universities

12

1.5

Total

796

100.0

SOURCE: NRC Phase II Survey.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
FIGURE 4-7 Sources of additional DoD SBIR company funding.

FIGURE 4-7 Sources of additional DoD SBIR company funding.

SOURCE: NRC Phase II Survey.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

TABLE 4-11 Average Additional Funding by Category, for Firms That Received Additional Investment in Each Category.

Source of Investment

Number of Investments Reported

Average Investment ($)

Non-SBIR Federal Funds

205

1,621,339

Private Investment from U.S. Venture Capital

30

5,192,267

Your Own Company

274

305,257

Private Investment from Other Private Equity

55

1,292,124

Private Investment from Other Domestic Private Company

105

500,981

Private Investment from Foreign Investment

19

810,788

Personal Funds

65

215,597

State or Local Government

31

178,608

Colleges or Universities

12

138,939

SOURCE: NRC Phase II Survey. See also Table App-A-37.

While venture funding did not provide the largest total amount of additional support, it did offer the largest average support per project funded, at $5.2 million per project. This is in line with the growing size of VC-funded deals, which in 2007 averaged at $8 million per project.26

However, venture funding supported only 30 projects—less than 4 percent of the 920 responses. Of course, the relatively small number of VC investments overall means that SBIR-funded companies may still account for a significant share of all VC investments.27

Self-finance continues to be the source of additional funding for many companies. More than one-third of respondents with additional funding indicated that additional funds came from their own company, although this accounted for only 1 percent of total funding received. Just under 10 percent reported funding from other private equity sources, which seems in most cases likely to mean angel funding. The widespread use of private investment funding (probably angel

26

According to the 2007 First Quarter Capital Report by Dow Jones VentureOne and Ernst & Young, LLP, “The median deal size reached $8 million, up from $7 million in the first quarter of 2006 and making it the highest quarterly median round size since the fourth quarter of 2000.” Ernst and Young, “U.S. Venture Capital Investment Increases to 8 percent to $6.96 Billion in First Quarter of 2007,” April 23, 2007. Accessed at <http://www.ey.com/global/content.nsf/US/Media_-_Release_-_04-23-07DC> on May 24, 2007.

27

In fact, venture capital firms often use SBIR as a signal of quality in making investment decisions. According to John Cottrell of L3 Communications, 12 percent of U.S. VC investment has involved firms with SBIR funding. John Cottrell, L3 Communications. Presentation at SBTC SBIR in Rapid Transition Conference, Washington, DC, September 27, 2006.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

funding) is not surprising, and the average amount invested is $1.2 million per funded project.

Investments from state and local government and academic sources played a limited role in terms of numbers and amount of funding provided (less than 5.5 percent of all investments in projects with additional funding). These sources also provided relatively low amounts of funding per project.

4.2.4.2
SBIR Impact on Further Investment

The NRC Phase II Survey also sought additional information about the impact of the SBIR program on company efforts to attract third-party funding. Some case study companies mentioned that SBIR awards can have a “halo effect,” acting as a form of validation for external inventors.28

Case study interviews provided mixed views on this perception. Some interviewees strongly supported the view that SBIR helps to attract investment, while others claimed that the halo effect was weaker than commonly thought—views that may reflect the individual firm or university experience.

Forty-six percent of DoD SBIR respondents did not attract any outside funding, and venture funding only accounted for 3.8 percent of investments. This suggests that an SBIR award is in itself no guarantee of further external funding, or (possibly) that outside funding was not needed.

4.2.5
Additional SBIR Funding

Aside from third-party investment, the federal government in many cases makes further investments via the SBIR programs itself. The NRC Phase II Survey attempted to determine how many additional SBIR awards followed each initial award.

The data shown in Table 4-12 indicate that 43.5 percent of respondent projects had at least one related SBIR award. These data suggest that while SBIR awards are to some extent concentrated, this effect is not overwhelming. 56.5 percent of respondents report no additional related SBIR awards at all, although small businesses reported that it takes multiple awards (often complementary) to build a product for effective sale.

28

E.g., Neurocrine, Illumina. See National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009, Appendix D. See also Maryann Feldman “Assessing the ATP: Halo Effects and Added Value” in National Research Council, The Advanced Technology Program: Assessing Outcomes, Washington, DC: National Academy Press, 2001.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

TABLE 4-12 Related SBIR Awards

Number of Phase II Awards

Number of Companies

Percent

28

1

0.1

9

1

0.1

7

4

0.5

6

6

0.7

5

12

1.5

4

23

2.8

3

29

3.6

2

106

13.0

1

172

21.2

0

459

56.5

SOURCE: NRC Phase II Survey.

4.2.6
Sales of Equity and Other Corporate-level Activities

The NRC assessment explored the different types of activities ongoing or completed among surveyed companies.

The data in Table 4-13 show that marketing-related activities were most widespread, with licensing agreements related to 33.4 percent of projects, and marketing/distribution agreements to 22.5 percent. Agreements likely to involve the direct transfer of equity were much less common. Only 3.6 percent of respondents reported finalized or ongoing mergers, while only 6.3 percent reported a sale of the company. Note, however, that the question asked specifically for outcomes that were the “result of the technology developed during this project”29—a very tight description.

Activities with foreign partners were, unsurprisingly, substantially lower than similar activities with U.S. partners. Again, marketing-related activities were most widespread.

In addition, the NRC Firm Survey30 determined that three firms (with SBIR awards at DoD) had had initial public offerings, and that a further three planned such offerings for 2005/2006. Seventy-five out of 445 companies at all agencies had established one or more spin-off companies (16.9 percent).31

The diversity of these outcomes underscores the challenge of early-stage

29

NRC Phase II Survey, Question 12.

30

The NRC Firm Survey did not assign companies (as opposed to projects) to specific agencies, as many had received awards from more than one agency.

31

The NRC Firm Survey was sent to senior executives at all firms receiving Phase II awards between 1992 and 2001.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

TABLE 4-13 Equity-related Activities

Activities

U.S. Companies/Investors

Foreign Companies/Investors

All Companies/Investors

Finalized (%)

Ongoing (%)

Total (%)

Finalized (%)

Ongoing (%)

Total (%)

Surveys

Total

%t

Licensing Agreement(s)

16

16

32

3

5

8

619

207

33.4

Sale of Company

1

5

6

0

1

1

619

39

6.3

Partial Sale of Company

1

4

5

0

1

1

619

33

5.3

Sale of Technology Rights

4

10

14

1

3

4

619

95

15.3

Company Merger

0

3

3

0

1

1

619

22

3.6

Joint Venture Agreement

4

8

12

1

2

3

619

83

13.4

Marketing/Distribution Agreement(s)

11

9

20

5

4

9

619

139

22.5

Manufacturing Agreement(s)

3

9

12

3

2

5

619

94

15.2

R&D Agreement(s)

14

14

28

3

3

6

619

175

28.3

Customer Alliance(s)

14

14

28

5

3

7

619

173

27.9

Other

2

2

4

0

1

1

619

24

3.9

SOURCE: NRC Phase II Survey.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

technology development and the multiple paths followed by small companies as they partner, license, are acquired, and/or attract additional capital.

Some companies have made a practice of acquiring SBIR firms. Titan Corp. purchased 12 SBIR-funded companies, and was then acquired itself by L3 Communications. L3 purchased a further 14 SBIR-funded companies, meaning that it has bought a total of 27 SBIR-funded companies. GE and Invitrogen are other examples of companies that have made a number of acquisitions among SBIR-funded companies.32

Arguably this is a powerful validation of the value expected by some SBIR awards and the opportunity represented by the technologies developed through the program. It also underscores the difficulties small companies face in developing a product set independently.

4.2.7
Initiatives to Improve Commercialization Outcomes

Prior to the 1992 reauthorization of SBIR, DoD agencies focused on using SBIR to meet DoD research and development goals. Growth in SBIR funding in late 1980s coincided with an overall reduction in funding for 6.2 (applied) research.33 Consequently, many DoD laboratories began using SBIR as a substitute for what would have been Broad Agency Announcements (BAA) to get their research program accomplished. Topics which in prior years would have been BAA topics were made into SBIR topics. Reportedly, for many topics prior to 1992, topic authors and selection panels paid little attention to the commercial potential of either a topic or an award. Thus, SBIR was a direct and subordinate aspect of DoD’s R&D program: it was used when DoD needed to investigate a technology or wanted a prototype built, but lacked either the funds or the budgetary flexibility to take these steps.34

This was the background against which the increased push for commercialization should be viewed. Consequently, a number of efforts have been made to improve commercialization outcomes from the SBIR program.

4.2.7.1
DoD Initiatives

During the period after 1992, DoD’s topic generation focus gradually changed, with increasingly greater emphasis on topics that could result in sales to DoD or within the commercial marketplace. Several influences contributed to this changed emphasis. In 1994, the Principal Deputy Under Secretary of Defense for Acquisitions chartered a Process Action Team (PAT) to develop a comprehensive

32

John Cottrell, L3 Communications, Presentation at SBTC SBIR in Rapid Transition Conference, Washington, DC, September 27, 2006.

33

DoD uses a coding system to describe kinds of R&D, ranging from 6.1 (basic research) to 6.7 (testing and deployment). 6.2 is the DoD code for applied research.

34

Peter Cahill, BRTRC, private communication, December 1, 2006.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

set of recommendations to improve SBIR contracting and funding processes; increase the commercialization of SBIR research in both military and private sector markets; and expand program outreach, particularly to socially and economically disadvantaged small businesses and woman-owned businesses.

These recommendations resulted in a number of changes for the program. Over the next few years, DoD undertook a wide range of initiatives, including:

  • Creation of the Fast Track program, which provided benefits for companies that could show third-party investment in their projects.35

  • Efforts to reduce award processing time.

  • Establishment of pre-release procedures for each solicitation, allowing potential applicants to talk with topic authors.

  • An increase in outreach activities toward woman and minority business owners and PIs.

  • New efforts to train small business owners in commercialization (see discussion of the Navy outreach and training program below).

  • New requirements that firms identify commercialization strategy in proposals (partly reflecting new congressional mandates in this area).

  • Implementation of the Commercialization Achievement Index (CAI), which created a commercialization metric for multiple Phase II award winners.

For a period in the mid- to late 1990s, the Director of Defense Research and Engineering (DDR&E) strongly pushed dual-use technology which could find markets in both the commercial and military sectors. The DoD Office of Small and Disadvantaged Business Utilization (SADBU) SBIR Program Manager also emphasized the importance of the commercial sector, on the assumption that commercially attractive SBIR topics and awards would widen the downstream market for the technology as well as attract private sector investment, thus reducing the cost of the technology and making it more affordable for DoD. However, at the service and component level, the primary emphasis continued to be on what was good for agency mission needs as defined at the technical monitor level. If commercial sales resulted, that was a just a bonus.36

4.2.7.2
The Fast Track Initiative

In 1996, DoD SBIR Solicitation 96.1 established, on a two-year pilot basis, a “fast track” SBIR process for companies which during their Phase I projects identified independent third-party investors that would match Phase II SBIR funding. Fast Track projects received (1) interim SBIR funding between Phases

35

See below for a description of this program.

36

Peter Cahill, BRTRC, private communication, December 1, 2006.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

I and II, (2) priority for Phase II funding, and (3) an expedited Phase II selection decision and award.

As early as 1992, DoD’s Ballistic Missile Defense Organization (BMDO) had begun to reward applications whose technologies demonstrated commercial private sector interest in the form of investment potential. This BMDO initiative, called “co-investment,” was effectively an informal “fast track” program. Under this approach, the evaluation process for Phase II proposals gave preference to applicants who could demonstrate that they would commit internal funding to the research or that they had financial or in-kind commitments from third parties to bring the technology to market in Phase III. With that commitment, applicants received essentially continuous funding from Phase I to Phase II.

In October 1995, DoD launched a broader Fast Track initiative to attract new firms and encourage commercialization of SBIR-funded technologies throughout the Department. With this initiative, DoD sought to improve commercialization through preferential evaluation and efforts to close the funding gap that could develop between Phase I and Phase II grants. The Fast Track program addressed this gap by providing expedited review and essentially continuous funding from Phase I to Phase II as long as applying firms could demonstrate that they had obtained third-party financing for their technology. In this context, third-party financing could mean that another company or government agency had agreed to invest in or purchase the SBIR firm’s technology; it could also mean a venture capital commitment to invest in the firm or that other private capital is available.

The expedited decision-making process for the Phase II award is justified from the agency’s perspective because outside funding validates the commercial promise of the technology. More broadly, the Fast Track program sought to address the need to shorten government decision cycles in order to interact more effectively with small firms focused on rapidly evolving technologies.

Based on commissioned case studies, surveys, and empirical research, the Moore Committee’s 2000 report37 suggested that the Fast Track initiative was meeting its goals of encouraging commercialization and attracting new firms to the program.38 Consequently, the Committee recommended that Fast Track be continued and expanded where appropriate.

37

National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, op. cit.

38

As always, there are caveats and limitations to this research. The first limitation concerns the relatively short time that the Fast Track program had been in place. This necessarily limited the Committee’s ability to assess the impact of the program. The case studies and surveys constituted what was clearly the largest independent assessment of the SBIR program at the Department of Defense, the study was nonetheless constrained by the limitations of the case-study approach and the size of the survey sample. The study nevertheless represents the largest external review of the Fast Track program and SBIR undertaken until the current review.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
4.2.7.3
The Commercialization Achievement Index (CAI)39

The efforts to increase the focus on and the success of commercialization led to the introduction of the Commercialization Achievement Index (CAI) in 1999. Companies with five or more Phase II awards were required to submit data about commercial outcomes as part of the application process for further awards. Following review of its SBIR and STTR programs as part of OMB’s Program Assessment Rating Tool (PART) process, DoD modified the CAI reporting requirements in its FY2006 SBIR Program Solicitation. The new requirements (Section 3.5d) specify that:

  1. Firms with four or more completed Phase II projects will receive a CAI score. Formerly, a CAI score was assigned only to firms with five or more completed Phase II projects.

  2. Firms with a CAI at the 10th percentile or below may receive no more than half of the evaluation points available for commercial potential criteria. Formerly this provision applied to firms with a CAI at the 5th percentile or below.

  3. DoD will now comprehensively examine the company commercialization review (CCR) data supplied by all firms participating in the program. Formerly, the review consisted mainly of periodic cross-checks.

The projected effect of these changes is to (a) slightly increase the number of firms whose new proposals are subject to criteria relating to past commercialization results, and (b) reduce the evaluation scores of more firms whose prior Phase II projects have not produced desirable levels of commercialization relative to other awardees.

CAI—A positive development? Efforts to find metrics for commercialization should be applauded. Eventual insertion of SBIR-developed products and services into DoD acquisition programs is an important objective of the program. Finding ways to ensure that money is targeted to firms with a good commercialization records ensures a positive return to the program (although it also implies multiple awards to a single firm).

However, the case histories suggest that developing such metrics is challenging. The length of time needed for the development of a commercially viable technology or for market demand to match an emerging technology may be longer than allowed for in the CAI. Thus the CAI may not be a sufficiently accurate measure of the commercial impact of SBIR-funded technologies at DoD for its use to be expanded further with the current metrics.

39

The Company Commercialization Report (CCR) was first required in DoD SBIR Solicitation 93.2 as a result of the 1992 SBIR reauthorization. The CCR was redefined and formatted as Appendix E to any proposal in DoD SBIR Solicitation 96.1. The CCR became an electronic submission with an embedded evaluation of a firm’s history of commercializing prior Phase II awards, the Commercialization Achievement Index (CAI), in DoD SBIR Solicitation 99.2.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

It is also worth pointing out that CAI is a limited tool focused exclusively on commercialization. SBIR’s other objectives are not addressed. Moreover, use of the CAI omits several important economic benefits of the SBIR program that contribute to the viability, growth, and profitability of small firms. These components of value are discussed in other sections of this chapter.

Thus, while CAI is undoubtedly a useful tool, it is important that DoD—and the selection process in particular—remain aware of its limitations.

4.2.8
Commercialization: Conclusions

As noted in the introduction to this section, there is no single metric for identifying commercial success. Instead, multiple indicators, and multiple metrics within those indicators, are needed to develop a broad assessment of commercialization within the DoD SBIR program.

This assessment supports the view that there has been considerable effort to bring SBIR projects at DoD to the market, with some substantial success. Even though the number of spectacular commercial successes (as is typical in most early-stage research efforts) has been few, the overall commercialization effort is substantial.40

Products are coming to market and significant licensing and marketing efforts are underway for many projects. Approximately 40 percent of projects generate products or services that eventually reach the marketplace (an unusually high percentage). These data all paint a picture of a program where the commercialization objective is well understood by award recipients and by the agency.

Overall, integrating survey findings and case study narratives suggests that firm experiences under the DoD SBIR program do not appear to differ markedly from private sector firms that invest in their own R&D. Christensen and Raynor, for example, note that recent surveys of private sector R&D report: “Over 60 percent of new product efforts are scuttled before they ever reach the market, and of the 40 percent that do see the light of day, 40 percent fail to become profitable and are withdrawn from the market.”41

This approximate benchmark puts an even more positive light on DoD SBIR commercialization outcomes, since private sector R&D is typically weighted far more to product development than to the science and technology activities (6.1–6.3) within the DoD SBIR portfolio; thus private sector R&D should be expected to produce comparatively higher success rates.

However, as we shall see in Chapters 5 and 6, there are still considerable barriers to SBIR awardees achieving success within the DoD acquisition system itself—which represents an area of significant potential improvement.

40

The distortion is echoed in venture and angel investments, both of which are often downstream of SBIR, which normally makes earlier-stage investments.

41

C. Christensen and M. Raynor, Innovator’s Solution, Boston, MA: Harvard Business School, 2003, p. 73.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

4.3
AGENCY MISSION

One of the core legislative objectives of the SBIR program is that it contributes to each agency’s mission. DoD’s SBIR program has nurtured many technological innovations that have made significant contributions to DoD’s mission capabilities.42 SBIR-spawned innovations have contributed to enhanced combat capabilities, and provided technological solutions to meet sudden, unexpected military threats.

Detailed analysis of Phase III at DoD is provided in Chapter 5.

4.3.1
Unique Benefits of SBIR at DoD

4.3.1.1
Enhanced Flexibility and Innovation

SBIR-funded projects have proven to be of especial value in generating technological approaches to new, unexpected problems that have arisen in ongoing military engagements in Afghanistan and Iraq. The high degree of flexibility characteristic of small firms means that SBIR has provided DoD with an increased number of suppliers capable of quickly responding on short notice43 to unanticipated battlefield situations (such as the use of improvised explosive devices [IEDs] in Iraq).

DoD has used the SBIR program to move quickly from identification of a DoD need to issuance of Phase I and then Phase II awards, and then to rapid deployment of operational equipment to meet pressing needs. Among the success stories contained in the case studies are the use of unmanned aerial vehicles for collecting over the horizon intelligence (Advanced Ceramics), the development of hand-held language translators (Marine Acoustics/Voxtex), the invention of radio detection and explosive devices to combat improvised explosive devices (First RF), and the production of an automated ammunitions sorter (Cybernet).

According to DoD SBIR program officials, SBIR has also proven an important and successful means of attracting the interest of small, high-technology firms to address specific R&D and operational needs where the potential market is too small to attract the interest of large defense contractors or venture-backed firms.

DoD officials point to a number of other benefits from the SBIR program:

  • It has increased the number of potential suppliers for new technologies, and also created new opportunities for these firms to partner together in new undertakings.

42

Dr. Charles Holland, Deputy Under Secretary for Science and Technology, Department of Defense, “Meeting Mission Needs,” in National Research Council, SBIR and the Phase III Challenge of Commercialization, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2007.

43

Dr. Mike McGrath, Deputy Assistant Secretary for RDT&E, U.S. Navy.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
  • It has helped DoD personnel learn about a set of high-tech firms with whom they would not otherwise ordinarily have contact.44

  • It allows DoD to draw on the creativity of the small- and medium-sized firms that comprise the SBIR community, and of the commitment of these firms to serve the needs of Services and agencies.

  • The SBIR program also has served as a filter to determine if firms have the technical, management, and financial capabilities to become reliable suppliers to DoD, whether on additional Phase III awards or for consideration in subsequent procurement competitions.

Survey responses from 347 SBIR technical monitors or Technical Points of Contact (TPOC) indicate that they perceive of DoD SBIR projects’ research to be of high quality—on average close to the quality estimated for non-SBIR awards.45 On a ten-point scale, where 10 represented the best research ever produced in the research unit/office in which the TPOC was located, SBIR awards received a mean score of 6.95. This average score was slightly below the mean score of 7.27 for non-SBIR research projects.46 This difference may be explained by outliers among the surveyed group.

4.3.1.2
Usefulness

SBIR projects were also found to have affected the way in which the DoD unit/office conducted research or supported research in other contracts. Fifty-three percent of TPOC respondents indicated that the specific SBIR project referred to in the survey produced results that were useful to them and which they had followed up on in other research.

Another indicator of the relative cost-effectiveness of the knowledge or informational contribution of SBIR projects to the design of DoD’s research program was that one-third of TPOC respondents noted that the SBIR project had more benefits for the agency’s mission than the average dollar spent on other re-

44

Firms express this outcome as follows: Had it not been for SBIR, their business with DoD services or agencies would not have developed. Services likely would have stayed with their pre-existing sources of supply. Program managers are too busy with multiple contracts to search out or respond attentively to new sources of technology. Their orientation is to hire a contractor to solve problems, not necessarily to seek out the most technologically innovative performer. The SBIR program requires that they become involved with small firms, to look at technical options, and to allow for increased competition in the selection of R&D performers.

45

NRC Program Manager Survey in National Research Council, An Assessment of the SBIR Program at the National Aeronautics and Space Administration, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009—contains a full description of the survey methodology and findings.

46

The difference in mean scores was statistically significant (at the .01 level), but was attributable to the considerably higher percentage of SBIR than non-SBIR projects that received low scores (e.g., 3 or below).

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

search contracts that the TPOC sponsored. Forty percent of respondents reported the same level of benefits per dollar spent; 27 percent reported fewer benefits per dollar spent. This at a minimum indicates that TPOC respondents saw SBIR dollars as generating research of quality equal to that of funding spent through other mechanisms—an important result in an environment where SBIR has often been seen as an unwanted congressional imposition on research managers’ discretion with research funding.

4.3.1.3
Low-cost Technological Probes

SBIR projects also provide DoD project managers with relatively low-cost explorations of novel scientific and technological approaches, the outcomes of which serve to increase the effectiveness of DoD’s laboratories and R&D facilities. While SBIR awardees overheads are normally not lower than universities (though this varies with company size) SBIR firms normally do have a much lower overhead than laboratories or large corporations.

4.3.1.4
Cost Savings

A number of DoD SBIR projects, for example RLW’s technique for real-time monitoring of ship maintenance and SAVI’s radio frequency detection system for monitoring shipments, have also been directed specifically at cost reduction (see case studies).

4.3.2
Assessment of SBIR’s Contributions to DoD Missions

4.3.2.1
Previous Assessment Efforts

There is currently no comprehensive formal measurement of DoD SBIR performance. The Navy considers the Phase III reported in the DD350 its primary tool for such purposes, emphasizing once again the stress laid on active insertion into weapons programs by the services as the core metric of success from the agency perspective. However, the other services have not emphasized the DD350 to the same degree, and there are substantial interservice differences in the extent to which the DD350 reporting system actually captures program outcomes.47

Of special concern to DoD SBIR program officials is that the CCR does not fully capture the program’s mission benefits and that undue use of commercialization measures will undercut core processes of technological development. They note that the closer a technology effort is to basic research, the less likely it is that its impacts will be captured in the PART system. Latent in these concerns is the

47

These limitations are emphasized by the DoD’s primary data subcontractor on SBIR. Pete Cahill, BRTRC, private communication, December 1, 2006.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

implication that program managers and laboratory personnel will rebalance their R&D portfolios to perform well according to PART and GPRA measures. They will select short-term product development topics that will improve their individual or unit annual performance reports but will diminish, over time, the rate at which agency R&D contributes to significantly enhanced agency performance.

DoD did make one prior effort to more fully analyze the performance of its SBIR program. In 1996–1997, the Director of Defense Research and Engineering (DDR&E) in conjunction with the DoD Small and Disadvantaged Business Utilization Office (SADBU) SBIR Program Manager, contracted BRTRC, a technology research firm, to study the commercialization of DoD SBIR awards. The DDR&E chose not to publish the results of this internal study. The survey’s responses, however, were shared with GAO. GAO used and referenced these results in two reports to Congress.48

DoD’s SBIR monitoring and reporting activities currently emphasize reporting on compliance. These data are suitable for documenting attainment of selected program objectives, such as support for woman- and minority-owned firms, but not to those related to mission support.

Conceptual and empirical difficulties clearly exist in documenting these contributions. Mission support (or “benefits”) and “sales revenue” are not identical. Thus, for example, it is hard to exactly measure the “benefit” of the SBIR-supported technology developed by Ophir Corp., which made stealth bombers more “stealthy.” Is the benefit (a) the $27.5 million in the firm’s sales, (b) an estimate for the increase in aircraft survivability (and, ultimately, lives saved), (c) the enhanced force capability, or (d) the sum of all of the above?

These complexities are not unique to the Ophir project. They are relevant to many SBIR projects that successfully promoted an agency mission. In most cases, direct measurement of agency mission impacts is simply not quantifiable—as NIH found when seeking to identify the public health impacts of SBIR awards. To give another example, how do you determine the contribution (or value) to mission support of Vista Controls Corporation’s electronic computing card in the ballistic computer of the M-1 tank? How do we answer the question, “how many lives and how much equipment are saved by getting off the first shot and hitting a target before it can hit you?”

Sometimes educated guesses can be made. Pentagon logisticians estimated that the SaviTag (an SBIR-developed bar-coding device for managing the logistics of military and nonmilitary material) could have saved $2 billion dollars had it been used in Desert Shield. The technology was used when U.S. forces were deployed in Bosnia and it is now in widespread use in Iraq.49 While the survey

48

U.S. General Accounting Office, Observations on the Small Business Innovation Research Program, GAO/RCED-98-132, Washington, DC: U.S. General Accounting Office, April 1998, and U.S. General Accounting Office, Evaluation of Small Business Innovation Research Can Be Strengthened, GAO/RCED-99-114, Washington, DC: U.S. General Accounting Office, June 1999.

49

Available at <http://www.savi.com/products/SaviTag_654.pdf>.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

and other data make it relatively easy to compute Savi’s sales, this amount has little relationship to the cost savings the technology provided to the department, the alternative uses of resources it enabled and the efficiency gains and savings permitted, especially during crises when resources are constrained.

4.3.2.2
Data Issues

The quality of the data available to DoD to assess the impacts of its SBIR program is also uneven. The DoD commercialization database contains information on Phase II awards for all agencies, not just DoD. Projects awarded prior to 1999 are reported/updated only if the firm is applying for further SBIR. For awards since 1999, updates on the project are required one year after the start of Phase II, at the completion of Phase II, and subsequently when the contractor submits a new SBIR or STTR proposal to DoD. Firms that do not submit a new proposal to DoD are asked (but not required) to provide updates on an annual basis after the completion of Phase II.50

If a company failed and ceased to exist, was very successful and outgrew SBIR, was sold to a larger firm, or became disenchanted with SBIR or DoD prior to 1999, no information on that firm will exist in the commercialization database. Over 30 percent of the Phase II awards made by DoD prior to 2000 have no information in the commercialization database. For more recent years, the data is more complete; for the period 1992–2002, over 82 percent of DoD Phase II awardees have entries in the commercialization database. The voluntary nature of reporting after the Phase II contract is finished clearly limits the completeness of the data.

“Success stories,” another frequently employed indicator of SBIR’s contribution to DoD’s mission objectives-success stories, also have limitations. DoD’s compilation of “success stories” is somewhat uneven. In practice, DoD accepts all submissions offered by firms, topic authors, and program managers. This approach cannot be completely comprehensive or representative, being shaped more by the initiatives and motivations of individuals rather than from any agencywide attempt at systematic coverage. There also are no visible quality control checks to insure the accuracy of specific statements. Moreover, since the content of success stories tends to focus on technological performance and sales,

50

The consequence of collecting data on pre-1999 awards only if a firm submits a new proposal can be seen in the following example. In terms of reported sales, the seventh most commercially successful award reported in CCR at $92 million was the Air Force’s award to SPEAKR Engineering. This award Phase II award however was made in 1983. SPEAKR did not submit an SBIR proposal in 1999, 2000, 2001, or 2002. Thus, there is no record of this $92 million in the commercialization database until the second solicitation of 2003, when SPEAKR submitted a new Phase I proposal. At the time of this submission, their proposal also reported three other DoD Phase II awards (one of which had $14 million in sales).

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

it is unlikely that many of the other benefits of the DoD SBIR program are fully accounted for, e.g., firm knowledge effects.

These challenges frequently arise in assessments of the economic and noneconomic impacts of other public sector investments. But answers to these questions, couched variously in terms of benefit-cost, cost-effectiveness, or rate-of-return analysis, exist for related R&D programs at several federal agencies. In addition, an extensive, long-standing theoretical and empirical literature exists on assessment of the impacts of technologies comparable to those developed under DoD’s SBIR awards.51

The distinctive aspect of the current state of affairs in assessing the contribution of the DoD’s SBIR program to missions is therefore not the inherent complexity of the challenges faced in producing acceptable estimates (although, admittedly, it is difficult), but the absence of such studies.

Interviews with DoD officials indicate that the primary reason for this absence is the lack of internal budget, staff, and expertise to systemically monitor, document, and assess the contribution of the SBIR program to its mission objectives. DoD program managers note that they are legislatively required to use all appropriated SBIR funds for awards to firms, and are explicitly prohibited (by Title 15, Chapter 14A) from using these funds for in-house activities such as program assessment. In the absence of nonprogram resources, DoD finds itself without the resources to maintain an adequate database or analytical studies to respond to questions about program management and program outcomes. When the department does initiate changes, it is not well positioned to systematically assess their impact.

Notwithstanding these constraints, the Department of Defense has made a significant effort to evaluate its SBIR program. For many years, the department has relied on an outside consulting firm (BRTRC) to maintain a database and, in the past, to carry out some analysis of the program. The department has also led the way among SBIR agencies in external assessment, commissioning exploratory workshops and then a major review of the Fast Track program by the National Academies in 1999–2000. More recently, Defense was an early contributor to the current study and has commissioned additional work with the Academies and with the RAND Corporation. Despite the real resource constraints described above, the department leads all other agencies in its efforts to better

51

Notable examples include several studies of the economic benefits of health related research: S. C. Silverstein, H. H. Garrison, and S. J. Heinig, “A Few Basic Economic Facts about Research in the Medical and Related Life Sciences,” FASEB 9:833-840, 1995; D. Cutler, Your Money or Your Life, New York: Oxford University Press, 2005; agricultural research: R. Evenson, P. Waggoner, and P. Ruttan “Economic Benefits from Research: An Example from Agriculture,” Science, 205(14 September):1101-1107, 1979, and R. Ruegg and I. Feller, A Toolkit for Evaluating Public R&D Investment—Models, Methods, and Findings from ATP’s First Decade, NIST GCR 03-857, Gaithersburg, MD: National Institute of Standards and Technology, 2003.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

understand the program’s operations and accomplishments as well as the impact of new policy initiatives.

That said, program managers recognize that current resources do not allow them to collect and maintain the data needed to better understand the program, nor to provide an effective tracking system and internal analysis that would facilitate external analysis of the program.52

4.4
SUPPORT FOR SMALL BUSINESS AND FOR MINORITY- AND WOMAN-OWNED BUSINESSES

This section provides an analysis of the ways in which this support helps different kinds of companies, and the overall impact of SBIR on small business activities at DoD. At one level, SBIR obviously provides support for small business, in that it provides funding only to businesses with no more than 500 employees—the SBA definition of a small business.

4.4.1
Small Business Shares of DoD Funding

Data from DoD show that for all contracts (not just SBIR), small business direct awards have for the past 15 years been approximately equal to the subcontracts to small business reported by the prime contractors (see Figure 4-8).

4.4.2
Project-level Impacts

A basic question concerning any “intervention” is whether the observed outcome would have occurred in the absence of the intervention. In terms of the SBIR program, this question generally relates to whether or not an R&D project would have been undertaken by a firm in the absence of an award, and if so, what affect the award had on the size, scope, or final characteristics of the project. After all, if most SBIR projects would have gone forward even without the SBIR contract, then the program’s impact would appear to be limited.

4.4.2.1
Project Initiation and Beyond

Figure 4-10 reports firm responses to the question of whether they would have undertaken the specific project in question in the absence of the SBIR award.53 Over one-third (37 percent) of responses stated that the project would “definitely not” have been undertaken, and another third (33 percent) said “prob-

52

As noted in the introduction, a major challenge for the Academies’ assessment of the program was the absence of the necessary data with regard to outcomes and program impact.

53

This question was asked of the 618 respondents who had completed their Phase II project and then continued to develop the project.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
FIGURE 4-8 DoD direct and subcontract awards to small businesses.

FIGURE 4-8 DoD direct and subcontract awards to small businesses.

NOTE: Subcontract Awards category includes only reportable subcontracts. A significant portion of small business awards are for a variety of necessary services, e.g., facilities maintenance, but are not high-tech research and product development.

SOURCE: Presentation by Dr. James Finley, Deputy Under Secretary for Acquisition and Technology, DoD. SBTC Conference on SBIR in Transition, September 27, 2006. Washington, DC.

FIGURE 4-9 Outcomes for firms not successful in receiving Phase II.

FIGURE 4-9 Outcomes for firms not successful in receiving Phase II.

SOURCE: NRC Phase I Survey.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
FIGURE 4-10 Project-level impacts—Phase II recipients.

FIGURE 4-10 Project-level impacts—Phase II recipients.

SOURCE: NRC Phase II Survey.

ably not,” Only 13 percent of responses indicated that the project “definitely” or “probably” would have gone forward in the absence of SBIR.

These responses strongly imply that the SBIR contract had a substantial impact in terms of the firm’s decision to carry out the proposed project. The NRC also surveyed companies that did not receive a Phase II award. Almost half did not pursue the technology.

Case studies suggest that the “definitely not” respondents can be divided into two groups. Some SBIR recipients claim that without the SBIR award the firm itself would either not have been formed or would not have survived for long. SBIR provided incentives and opportunities for the founders to leave their existing occupation and start a firm, and the initial working capital essential for pursuing the proposed research.

For existing firms with ongoing R&D or production activities, the primary value of the SBIR program was that SBIR topics indicate a well-funded potential market for the firm’s technology.54 In effect, technology-intensive firms may be seen as constantly searching for potential applications for their core technologies. At any point in time, there are more technology paths to pursue than resources allow. SBIR topics focus a firm’s attention on a specific technological and market

54

N. Rosenberg, “The Direction of Technological Change: Inducement Mechanisms and Focusing Devices,” Economic Development and Cultural Change 18:1-24, 1969.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
FIGURE 4-11 Further project impacts.

FIGURE 4-11 Further project impacts.

SOURCE: NRC Phase II Survey.

objective. And when the firm’s application is successful, an SBIR award provides the capital, contacts, and versatility needed to pursue product development.

The absence of SBIR funding would have had a significant impact even on the 13 percent of respondents who said that they probably or definitely would have undertaken the project even without the SBIR award. Half of these respondents reported that the scope of the project would have been narrower,55 and 62 percent expected the project would have been delayed. More than a fifth of these companies expected a delay of at least 24 months; an additional 49 percent thought the delay would have been at least 12 months.

In addition, 78 percent of all respondents noted that the project would have taken longer to complete in the absence of a Phase II award. Thus, in the absence of the award, there would have been a twofold impact of the firm’s efforts to transform an R&D concept into a marketable product: The start of the project would have been delayed and project would have taken longer to complete.

There is therefore little doubt that the SBIR awards had, in the vast majority of cases a significant, often decisive, impact on the company’s ability to undertake the research supported by the work.

55

This last response presumably reflects the narrowing of an R&D concept to mesh with the scope of the topic solicitation announcement.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

BOX 4-2

Cybernet—Start-Up Case Study

SBIR has many different impacts at different firms, but one clear impact has been on the decision to found the firm. Only a relatively small percentage of firms were founded directly “because” of SBIR, but Cybernet is one of them.

Heidi Jacobus, the founder, was working on a doctoral thesis on human-computer interaction at the University of Michigan in 1988. She also worked in the university library on a project developing an indexed reference book on the SBIR program. Soon after, she saw her thesis topic listed as a DARPA topic of interest. She distilled her thesis proposal into an SBIR proposal. Subsequent to the submission of the proposal, she received a telephone call from a DARPA official stating that her proposal was the “best” he had ever read.

The feedback from DARPA was the motivating event that gave Jacobus the courage to found Cybernet. She submitted SBIR proposals to other agencies, receiving awards from NASA and the Army, followed by the award from DARPA that had catalyzed the firm’s founding. Cybernet was a bootstrap operation: The firm’s offices were housed in her daughter’s bedroom. Jacobus had to learn the basics of government contracting and accounting procedures, such as overhead rates, allowable expenditures, and related provisions. She accomplished this by reading manuals obtained at the regional SBA office, purchasing technical assistance from local consultants, and receiving assistance from the regional Small Business Development Center.

Cybernet’s core mission is the development and application of robotics technology solutions to human-machine interaction. It applies its work to a diverse set of defense and nondefense industries. The firm has drawn on the rich tradition and ready availability of robotics and related manufacturing expertise in the Ann Arbor region to steadily increase its manufacturing activities, and also sells to the private sector. Cybernet currently derives about 70 percent of revenues from federal contracts and 30 percent from the private sector.

4.4.3
Multiple-award Winners and New Firms in the Program

Some critics argue that the DoD SBIR program tends to favor “established” small firms with well-developed ties to decision makers in the funding stream.

In a certain sense, this is entirely natural. Firms that are successful in winning contracts and then in delivering results will tend to seek to duplicate these successful efforts, and awarding officers will become familiar with these firms and confident that they can deliver a quality product in a timely fashion.

In interviews, agency staffers have also noted that SBIR is a critically important mechanism for “proving out” potential vendors: The risk is relatively low, contracts are tightly time bound and relatively small, and positive experiences can be built upon quickly as the timeline between solicitations is short. Thus, some staffers see SBIR very much as a means of validating the capabilities of potential

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

vendors, who when successful can be added to the technical resource pool available for meeting agency needs.56

These program characteristics imply that the distribution of awards will be skewed, and that some companies will, over time, win a considerable number of awards—particularly firms with a wide range of technical capabilities who are able to address the requirements for many topics in a given solicitation.

At the same time, however, it is important that the program not be captured by a group of regular winners to the extent that potential new vendors are frozen out, and either fail to win or are so discouraged that they no longer apply. Equally, it is important to ensure that firms do not simply become specialists in winning awards, without ever producing commercial results or in the larger sense addressing agency needs for quality research and product prototypes.

The question of new winners is addressed in Chapters 3 and 6, based on DoD awards data. However, additional information has been developed using the NRC’s Phase II Survey.

4.4.3.1
New Firms

Most NRC Phase II Survey respondents had received only a few SBIR awards. Most significantly, just under one-third (29 percent) had no prior Phase I award and were entirely new to the program. This underscores the competitive nature of the program, namely that a significant number of awards have regularly gone to new firms.57

Another 42 percent of respondents had had five or fewer prior Phase I awards. Five firms reported 96 or more prior Phase I awards. Excluding these five firms, the remaining 561 firms averaged less than four prior Phase I awards.

The average number of previous Phase II awards for firms in this sample was 2.45. Forty-six percent of respondents had no prior Phase II awards and another 36 percent had five or less. The same five firms accounted for all respondents with more than 35 previous awards. The remainder of the firms averaged 1.7 prior Phase II awards.

Respondents were queried about the number of prior Phase I awards in related technologies to the technology embedded in the Phase II project being surveyed. Firms reported an average of 1.5 related Phase I awards with 46 percent of respondents having no prior related Phase I awards, and an additional 48 percent having five or fewer prior related awards. In terms of related Phase II awards, the average number reported by firms was 0.8: Fifty-six percent has no

56

Comments by John Williams, Navy SBIR Program Manager, at Navy Opportunity Forum, September 2005.

57

The substantial percentage of new entrants each year was revealed in the 2000 NRC survey. See National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, op. cit.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
FIGURE 4-12 Percentage of projects reporting prior SBIR awards.

FIGURE 4-12 Percentage of projects reporting prior SBIR awards.

SOURCE: NRC Phase II Survey.

prior related Phase II award and another 42 percent had 5 or less prior related Phase II awards.

These data make clear that the DoD SBIR program is open to new entrants, with a significant percentage of new entrants for both Phase I and Phase II.

4.4.3.2
Understanding Multiple-award Winners

Commercialization and Multiple-award Winners. Increased attention to commercialization also has implications for a group of companies that win multiple SBIR awards. As noted above, there are understandable reasons why some firms have become proficient at winning SBIR awards just as some large contractors and some universities become proficient in their domains. One critical question is whether these firms do more than just win awards: Do they provide good value to DoD and do they help the DoD SBIR program meet the congressionally mandated objectives?

This is a harder question to answer than it appears. Firms may be performing work that is important to DoD, but for which there is a limited market both at DoD and commercially. So commercial results in and of themselves are not a sufficient metric, and can indeed be misleading if they become the central focus of analysis.58

On the other hand, companies that receive large numbers of awards should be providing work of value to the agency, and one important way to tell whether such work is being done is whether the agency continues to fund the project after Phase II is completed. Again, it is important to consider commercialization metrics only as one indicator of value—an important indicator, but only one of

58

One important output of research is new ideas that are often picked up and applied by others. This is a major objective of government sponsorship. For more, see Section 4.5 of this chapter on SBIR and the expansion of knowledge.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

several. Moreover, case studies clearly show that firms can perform useful work that is not well reflected in commercial results, for example by “answering” a research question of providing a product that has no follow-on sales.

Keeping this complexity of goals and outcomes in mind, commercial results do matter. How well do multiple-winner firms commercialize? Part of the answer lies in Table 4-14 and Figure 4-13. Table 4-14 shows commercialization by number of previous awards. Figure 4-13 focuses on lack of commercialization—the percentage of projects reporting no sales at all, by number of previous awards.

By its three-phase structure, the SBIR program suffers from an implicit linear myth, namely that a single grant for a single project is sufficient to fully develop a technology and drive long-term growth of the company. In practice, a single grant is often not sufficient to commercialize a product. Often multiple related projects, complementary technologies, and varied funding sources are needed, in addition to effective management, to bring a product to market. With regard to additional awards, NRC Phase II Survey data indicate that returns are maximized where firms have received 10–25 previous Phase II awards.

4.4.4
Differing Uses of SBIR by Firms

The expansive scope of the NRC study and its use of multiple methods provide an opportunity to place conventional measures of commercialization, such as sales, within the larger context of firm formation, business strategy, and long-term growth.

Earlier assessments have contained typologies of the several different types

TABLE 4-14 Sales by Number of Previous Awards

Additional Prior SBIRs

Sales Reported (Percent of Responses)

Total Responses

0

<$100K

$100K to < $1M

$1M to <$5M

$5M to <$50M

$50M+

0

46.1

13.3

22.3

13.7

4.3

0.4

256

1

63.2

9.2

13.2

10.5

3.9

0.0

76

2

50.6

10.4

20.8

15.6

2.6

0.0

77

3–5

51.1

7.5

22.6

12.8

4.5

1.5

133

6–10

56.7

7.8

15.6

14.4

5.6

0.0

90

11–15

54.5

3.0

12.1

15.2

15.2

0.0

33

16–25

56.1

2.4

17.1

12.2

12.2

0.0

41

26–50

51.1

12.8

14.9

14.9

6.4

0.0

47

51–100

78.6

3.6

14.3

3.6

0.0

0.0

28

101+

70.6

5.9

19.6

3.9

0.0

0.0

51

Total

53.7

9.4

19.1

12.6

4.8

0.4

832

SOURCE: NRC Phase II Survey.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
FIGURE 4-13 Projects with zero dollars in sales, by number of prior awards.

FIGURE 4-13 Projects with zero dollars in sales, by number of prior awards.

SOURCE: NRC Phase II Survey.

of firms participating in the SBIR program.59 These typologies are especially useful for extending the analysis beyond the focus on “young, high-technology firms” to the more diverse array of small, high-technology firms that actually participate in the SBIR program. They also help to explain certain outcomes.

Fieldwork conducted during the course of the NRC SBIR study, coupled with review of earlier typologies, produces the following classification of firms that participate in the DoD SBIR program:

  • Start-up firms.

  • R&D contractors.

  • Product-oriented firms.

  • Defense-oriented technology-based firms.

59

See Reid Cramer, “Patterns of Firm Participation in the Small Business Innovation Research Program in Southwestern and Mountain States,” in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, op. cit.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
4.4.4.1
Start-up Firms

These are newly formed firms whose early viability is directly connected to receipt of an SBIR award. Some firms were formed specifically to be eligible for SBIR funding; others were existing early-stage firms that would probably have failed without SBIR.

The “start-up firm” category captures the firm in its embryonic form, a transitional stage before the firm evolves into other categories. Firms in this group provide material for exploring SBIR’s contribution to new firm formation. In many cases, they also show how SBIR helps to grow the manufacturing base and thus to expand the technological options available to meet DoD mission goals.

For these start-up firms, SBIR Phase I and Phase II awards may provide the initial or critical operating and start-up capital needed to explore leading edge technologies. And, as many interviews—as well as previous NRC studies60 and academic analyses61—indicate, that capital is not readily available from any private sector source. Start-up firms are rarely good candidates for venture capital funding. This is especially true for the relatively narrow, regulatory constrained defense market.

Finally, firms also credit the entrepreneurial behavior of selected SBIR program managers as having catalyzed the firm’s formation. Among DoD Services and units, DARPA program managers are most often cited as having played this role. In the case of Cybernet, for example, the interest of a DARPA program officer in the proposal submitted by the firm’s founder led the founder to forego an opportunity to complete a Ph.D. and instead venture out to form a firm. Personal encouragements by DARPA program managers to ex-military and ex-industry employees who were employed as consultants to DoD or defense-related firms also are recounted as having led these individuals to form firms based on the prospects of successfully competing for SBIR awards.

One striking aspect of DoD SBIR start-up firms is that many of the founders are former employees of large defense and aerospace prime contractors, who often left because they wanted to run their own business. They describe themselves as refugees from former firms that had grown too large and bureaucratic,

60

R. Archibald and D. Finifter, “Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach” in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, op. cit.; D. Audretsch, J. Weigand, and C. Weigand, “Does the Small Business Innovation Research Program Foster Entrepreneurial Behavior” in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, op. cit., pp. 160-193.

61

See Lewis M. Branscomb, Kenneth P. Morse, and Michael J. Roberts, Managing Technical Risk: Understanding Private Sector Decision Making on Early Stage Technology-based Projects, NIST GCR 00-787, Gaithersburg, MD: National Institute of Standards and Technology, 2000.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

TABLE 4-15 Was Company Founded Because of SBIR Program?

 

Number of Responses

Percent of Responses

No

342

74.8

Yes

49

10.7

Yes, in part

66

14.4

 

457

100.0

NOTE: Data reported in Table 4-15 are for firms with at least one DoD award. NRC Firm Survey results reported in Appendix B are for all agencies (DoD, NIH, NSF, DoE, and NASA).

SOURCE: NRC Firm Survey.

or as exiles from small firms that had shelved development of a technology they strongly believed in.62

Responses to the NRC Firm Survey indicate that 25 percent of firms were founded entirely or in part as a result of SBIR awards (See Table 4-15). These data are important because they suggest that SBIR funding should be considered one of the more important sources of seed capital for new high-technology companies and entrepreneurs.

This finding was supported by previous analysis. As one scholar reported in the first Academy workshop on the SBIR program, “the picture that emerges is a program that is working effectively and appears to be playing a positive role in stimulating small firm creation.” This represents a significant contribution in two respects. As noted in the introduction, from a public perspective this is desirable because there are significant knowledge spillovers associated with R&D, that is, the benefits of R&D do not accrue only to those making the investment.

Second, information problems hamper investors’ efforts to identify promising technologies. The large number of companies seeking financing and the uncertainty involved with innovative business proposals pose significant risk assessment challenges for potential investors—challenges that may result in underinvestment in new technologies.63

4.4.4.2
R&D Contractors

A second generic type of participant in the DoD SBIR program are R&D contractors that specialize in the performance of R&D, strategically positioning

62

See for example the remarks of David O’Hara of Parallax Research who left his previous firm, one that he believed was insufficiently focused on the commercial potential of their SBIR awards, in order to found Parallax and pursue commercialization. National Research Council, The Small Business Innovation Research Program: Challenges and Opportunities, op. cit., p. 25.

63

This problem may be especially acute in small firms. See the remarks by Joshua Lerner of the Harvard Business School in National Research Council, The Small Business Innovation Research Program: Challenges and Opportunities, op. cit., p. 23.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

BOX 4-3

First RF: Addressing New Defense Needs

FIRST RF Corporation was founded by Farzin Lalezari and Theresa Boone in 2003. Lalezari was born in Iran, and emigrated to the United States in 1971, while a high school student, following the imposition of a death sentence on his father, who was serving as Iran’s Minister of Education, by the Khomeini regime. Upon graduation, he joined Ball Aerospace, where he advanced to position of chief scientist and director of research.

At Ball, Lalezari’s research led to 25 patents, all assigned to the firm. Lalezari left to form FIRST RF because of disenchantment with the bureaucratization and technological stagnation of large firms, and their overemphasis on short-term profit measures designed to meet the requirements of stock market analysts.

FIRST RF’s core technology focus is advanced antennas and RF systems. Lalezari used the SBIR solicitation of topics to focus on a specific problem. In his view, one of the primary benefits of the SBIR program is that it is seen as forcing firms to “think out of the box,” while simultaneously providing innovators with access to users.

Lalezari reports writing about 12 SBIR proposals during the firm’s first year of operation. The firm received awards on seven of these proposals, a number described as a national record for a start-up company. In late 2003, it submitted a Phase I proposal for an Army-generated topic related to the detection of improvised explosive devices.

By the time its Phase I project was finished, the firm had delivered production prototypes for use by U.S. military forces in Iraq. In 2004, the firm entered a structured competition against 27 other firms, including major defense contractors such as Raytheon and BAE for volume production of IED countermeasure devices. It won the competition, receiving an initial $21.5 million contract from the Army, with delivery scheduled for December 2005.

This Army contract has been followed by several additional contracts with DoD prime contractors.

themselves as a DoD equivalent to private sector industrial R&D labs. R&D contracts constitute a substantial portion of revenues for these firms, and a relatively high percentage of those revenues (especially in the firm’s early years) may come from SBIR awards. These awards may in part come from multiple federal agencies. Typically, firm dependence on SBIR declines over time as the firm becomes more of an established supplier of its technical services, gaining revenues from standard procurement contracts, integrating into downstream products, and entering private sector markets.

SBIR—Main source of federal funding for early-stage technology development. SBIR provides over 85 percent of federal financial support for early-stage development. SBIR provided over 20 percent of funding for early-stage development from all sources in 1998.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
FIGURE 4-14 Estimate of federal government funding flows to early-stage technology development.

FIGURE 4-14 Estimate of federal government funding flows to early-stage technology development.

NOTE: Based on total funding for ATP, SBIR, and STTR programs.

SOURCE: Lewis M. Branscomb and Philip E. Auerswald, Between Invention and Innovation: An Analysis of Funding for Early Stage Technology Development, NIST GCR 02-841, Gaithersburg, MD: National Institute of Standards and Technology, 2002.

However, even if their work leads to production prototypes, R&D contracting firms often choose not to become extensively involved in downstream manufacturing lest it detract from an emphasis on R&D. They deliberately limit their scope to R&D activities in which they have competitive expertise, plus some selective, (usually) customized manufacturing. Additional revenues may come from licensing proprietary technology or income from spin-off manufacturing subsidiaries.

Such firms operate within self-imposed ceilings on growth. They tend not to seek the external capital usually needed to expand operations because to do so would dilute founder control and equity.64 This strategy results in a high percentage of revenues from R&D contracts, as the result of a specific commercial strategy.

Even where the venture capital community has expressed interest, not all

64

E. Roberts, Entrepreneurs in High Technology: Lessons from MIT and Beyond, Oxford, UK: Oxford University Press, 1991, p. 328.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

BOX 4-4

TRIDENT SYSTEMS: The Challenge of Moving Upstream

The data strongly suggest that there are different levels and degrees of commercial success, and the Trident Systems case study indicates that stepping up from one level to the next can be challenging.

Trident Systems was established by Nicholas Karangelen in 1985. Initially, the firm operated primarily as a “services” company, consulting to major DoD contractors, such as General Electric and Westinghouse, but lacking its own prime contracts. This work helped build Trident’s reputation as a knowledgeable, reliable performer and gave it new insights into DoD requirements.

Trident began submitting SBIR proposals around 1986, submitting four unsuccessful proposals before winning an award. It views the SBIR program as one of the few available contract mechanisms that provide “size appropriate competition”—opportunities for small firms to compete for DoD’s R&D and procurement contracts.

In 1988, Trident became a prime contractor on a Navy system development project related to antisubmarine warfare. The stability of the contract, the quality of the work Trident performed under it, and the business relationships developed during performance of the contract launched Trident on the growth trajectory it has experienced since the late 1980s.

Trident has grown primarily by expanding its business around its core competencies in requirements analysis for weapons systems, systems engineering, and, more recently, systems design. The company now serves as prime contractor on many programs and has recently moved further into downstream integration.

Difficulties in Moving Beyond Phase II. Trident’s success in developing DoD related technologies, especially under the SBIR program, has not led to proportionate successes in landing procurement contracts. In part this may be due to the attitudes of acquisition managers. Trident believes that DoD acquisition offices are reluctant to recognize the value of small firms and their technologies, and are more concerned with maintaining the status quo and avoiding risk. Prime contractors too are seen by Trident as generally unwilling to bring in a promising externally developed (and potentially disruptive) technology when they either have or believe they can develop an internally developed alternative.

SBIR awardees are eager to enter into an agreement lest they be required to relinquish ownership and control. The basic motivation for starting a firm for several of the founders included in this study and the stylized workings of the start-up/venture capital markets diverge at this point. As noted, several of the firm founders in this study in effect are “new world” immigrants from what they see as an “old world” of large, bureaucratic defense and aerospace contractors. As such, they are wary of entering into business relationships that dilute their selection of research topics or require that they meet the profit/sales targets of outside investors, even if the consequence is a smaller firm.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

Two hedging, or mid-ground mechanisms, also at times permit founders to maintain autonomy and increase personal incomes. First, they can retain ownership of their initial firm, which retains an emphasis on R&D, while entering into equity agreements with outside investors in the launching of subsidiaries that focus on manufacturing. Second, as a form of life-cycle choice, some founders eschew venture capital and other outside investment during their “work years,” and then cash out their founder’s profits in the form of equity arrangements as they near retirement.

Besides concern about dilution of ownership prerogatives, another contributing factor in the decision of several firms to refrain from growing via addition of manufacturing and marketing capabilities is that their management perceives the cultures and worldviews of R&D personnel as being distinctively different from (and perhaps superior to) those of personnel engaged in the more downstream activities. The difficulties of integrating the cultures of R&D, manufacturing, and marketing within a single firm were noted by several respondents. One founder observed that in his previous employment settings, he had seen three efforts to reorient a firm from being a service provider to providing both services and equipment; each effort failed. In his view, a firm must decide between one or the other. This philosophy does not, however, rule out the above-noted strategy of spinning off subsidiaries if and when R&D leads to marketable products requiring volume production. Ultimately, firm orientation depends on management preferences—and opportunity—but this does not detract from the versatile and timely contributions successful firms can make to meeting Defense R&D needs.

4.4.4.3
Product-oriented Firm

Product-oriented firms are firms with an existing core technical competency and commercially viable products or services that seek to grow by expanding into, or within, the DoD market.

SBIR topics and funds can assist product-oriented firms in starting new R&D initiatives. They can also help the companies adjust to adverse changes in market conditions. SBIR funds often enable the research small businesses need to make a major course correction when such funding would otherwise be difficult to obtain within the company’s existing resources.

For some more established firms, SBIR awards help augment existing product lines and broaden R&D portfolios. SBIR topics both focus a firm’s attention on potential new uses for its technologies and expertise, and provide the incentive needed for the firm’s management to devote resources to the agency’s R&D priorities rather than to other uses.

Product-oriented firms tend to have a mixed portfolio of products and customers, often switching their R&D and commercial activities between defense and nondefense markets as opportunities arise. In some cases, such as ACR (see Box 4-6), SBIR has been critical in helping companies adapt when the high risks

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

BOX 4-5

PSI Physical Sciences, Inc.

Physical Sciences, Inc., was established in 1973. The founders left Avco-Everett to start their own firm partly because they sought a smaller research and working environment; Avco-Everett at the time had about 900 employees.

PSI’s growth was initially modest, based on contracts with the Air Force and DoE. By the early 1980s, it had approximately $10m in revenues and a staff of 35–50. After a decline in the early 1980s, PSI diversified its federal customer base as well as its range of technological expertise. As it has grown, SBIR awards have contributed a diminishing portion of firm revenues, falling from a peak of about 60 percent in the late 1990s to a projected 35 percent in FY2006.

Since the uses of optical technology have dramatically expanded, the firm’s technological and market bases have widened to encompass applied R&D, production operations, and bundling of “hands-on” service delivery with the application of newly developed products, especially in the areas of instrument development, diagnostics, and monitoring. PSI has strategically positioned itself in an R&D market niche defined by multidisciplinary expertise and research infrastructure in specialized high-tech areas too small to attract major investments by large DoD prime contractors, while at the same time too mission-driven to elicit competition from universities.

The firm’s successes led to opportunities in new directions, but the founding vision was to maintain owner/employee control of the firm. Hence the firm remains focused on R&D and prototype development rather than manufacturing, which would require additional external capital.

Some of the firm’s contracts with DoD involve development of specialized, one of a kind technologies. These can meet critical DoD needs, but may constitute a market with a small sales volume. Other DoD contracts led to the development of technologies, mainly in the area of instruments, that the firm does seek to market to the private sector. For example, PSI’s SBIR-funded development of sensor technology to detect methane gas leaks has been sold to gas utilities. In general, sales to the private sector are largely based on technologies developed for DoD under SBIR awards—a classic case of spin-out development.

PSI will engage in limited production of specific instruments for DoD and other federal agencies. When its technological developments lead to commercially viable products, PSI follows a mixed strategy. One approach is to form new firms, with new, independent management, that operate as partially owned spin-outs. Shaping this business decision is the firm’s view that the “cultures” and operational needs of contract R&D and manufacturing firms differ sufficiently that it is more efficient to operate them as separate entities rather than attempt to combine them into one larger firm. Conversely, PSI also creates wholly owned subsidiaries, focused on R&D activities, which have become eligible for SBIR competition on their own (as long as PSI remains a small business).

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

involved in a production-oriented strategy (with attendant high overheads and substantial dependence on specific markets) leads to a crisis for the firm.

For several of the larger (yet still “small business”), private sector firms with established product lines within the case study sample, SBIR funding provides a source of incremental revenue that can be used to determine the feasibility of new technologies without having either to cut back on manufacturing or marketing or seek access to external capital. A pattern evident in the case histories, consistent with the description of the strategic, commercially oriented approach to SBIR topics employed by firms, is for a firm to seek out topics that will permit it to test or demonstrate the applicability of its generic technology to a wider set of applications (and thereby markets). Thus, the SBIR project becomes the test bed to address specific technical problems (e.g., weight, durability, processing speed, luminescence, bandwidth, etc.) of interest to DoD, which if solved could also open up or enhance the competitiveness of the firm’s products in different markets (e.g., medical equipment, computer games, first responder equipment, etc.)

4.4.4.4
Defense-oriented Technology-based Firm

These are existing firms whose core competency is the integration of leading edge R&D and product development for DoD. The firms use R&D—including SBIR awards—to advance the performance capabilities of a technology, and then engage in an ongoing search to adapt the core technology to a widened set of applications and users, including initially DoD but also potential customers in the public and private sectors.

SBIR awards can provide an opportunity to explore the broader applicability of core technologies most immediately to DoD’s needs, as stated in solicitation topics, as well as to new uses and markets.

The typology described above is useful for understanding some of the different strategies of small high-tech firms participating in SBIR. These can include: (a) seeking to transition from an R&D provider into a supplier of products and processes, subsequently followed by movements into product development, manufacturing, and marketing; (b) failing to make this transition successfully, possibly because of the difficulties of integrating R&D and manufacturing functions (and cultures) within the same organization, and then reverting to its prior R&D specialization; (c) deliberately rejecting the downstream transition strategy by choosing to remain a specialized supplier of R&D services. These points are reflected in the discussion below.

4.4.5
How Firms Use SBIR: Commercialization Case Study Results

As noted earlier, different types of firms use SBIR awards in different ways to achieve different objectives. A series of case studies were conducted to flesh out these varied approaches to the program. They explore the workings of the

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

BOX 4-6

Advanced Ceramics Research (ACR)

From its inception in 1989, ACR sought to become a product development company, capable of manufacturing products for varied industries. The company knew about, but initially rejected, the SBIR program on the grounds that the 5–7 percent allowable profit was too low.

ACR shifted perspectives on the program, however, and has since participated in SBIR programs at DoD, NASA, DoE, and NSF. SBIR awards accounted for nearly all of ACR’s revenues by 1992, but by 2005 only 15–20 percent of its sales were projected to come from STTR/SBIR.

Drawing in part on the advanced research being done at the University of Michigan and its own expertise in both advanced ceramics and manufacturing, ACR developed a general purpose technology for converting AutoCAD drawings first into machine-readable code, and then to direct generation of ceramic, composite, and metal parts.

One market with considerable potential was “flexible carriers for hard-disk drives” for the electronics industry. ACR’s aggressive marketing soon helped the company become a major supplier to firms such as SpeedFam Corporation, Komag, Seagate, and IBM. Demand for this product line grew rapidly, enabling the firm to go to a three-shift, seven-day-a-week operation. Demand for ACR’s electronic products grew fast during the 1990s, from 5,000 to 60,000 units monthly, and ACR built a new 30,000 square foot plant. The electronics market for ACR’s products, however, declined abruptly in 1997, when two of its major customers shifted production to Asia. The loss of its carrier business was a major reversal for the firm. Heavy layoffs resulted, and employment declined to low of about 28 employees in 1998.

The next 2 years are described as a period of reinvention for survival. The firm’s R&D division, formerly a money-loser, became its primary source of revenue. The explicit policy was to undertake only that R&D which had discernible profit margins and the opportunity for near-term commercialization.

Since 2000, ACR has received funding from the Office of Naval Research (ONR) to develop a new low-cost, small unmanned aerial vehicle (UAV), initially for whale watching around Hawaii, in support of the Navy’s underwater sonic activities. Once developed, the UAV’s value as a low-cost, highly flexible, more general purpose battlefield surveillance tool soon became apparent, and ONR provided additional funding to further refine the UAV for use in Iraq.

ACR is now actively engaged in the continued development and marketing of Silver Fox, a small, low-cost UAV, supported by awards under DoD’s SBIR (and STTR) program, which have funded collaboration between ACR and researchers at the University of Arizona, University of California-Berkeley, the University of California-Los Angeles, and MIT.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

SBIR program, how SBIR awards enter into the business history and strategy of a firm, and the process of technological innovation at the firm level.

4.4.5.1
SBIR and Entrepreneurship

Several themes emerge. The first is the connection between SBIR funding and entrepreneurship in defense-related industries. The NRC Firm Survey data indicate that, for 78 percent of the respondent firms, at least one founder’s most recent employment was at a private company. For 28 percent of the firms, at least one founder came directly from a university and, for 7 percent of the firms, at least one founder was most recently employed by the government.65 The case studies provide additional detail on prior employment. In a majority of the 31 cases, the firm’s founder (or founders) was an individual who had worked for a large defense or aerospace firm, and who had left a relatively senior or secure position to develop and commercialize a specific technology.66

The founders had several reasons for making this move. In some cases, it reflected a desire to start out on one’s own; reflecting what one founder called his “entrepreneurial heritage.” In other cases, it reflected a desire to pursue development of a technology that had been sidetracked in the larger R&D portfolio of a prime contractor, as in the case of Trident Systems, above. Some founders revealed that their move was caused by frustration and disappointment that decisions and values within a formerly technologically innovative firm had given way to preoccupation with short-term financial targets.

The high percentage of former defense-related employees among the set of firm founders helps to explain some of the data noted above. It is consistent with the limited involvement of third parties in providing technical assistance, as many SBIR firms were already familiar with DoD’s needs and with federal contracting and cost accounting practices. It also reinforces the view that SBIR firms are not necessarily oriented toward rapid financial returns along the lines of the traditional VC model.

Several of the firms explored in the case studies were true “garage” start-ups. Founders located their first activities within garages, basements, or children’s bedrooms. Several reported relying on their own savings or funds from relatives to start their businesses. A number recounted a period of serving as consultants to their former employers, customers of those employers, or DoD organizations with whom they had formerly worked, as a transition stage in which they crystallized their technological visions and embryonic business plans while still earning an income.

65

These data are for DoD only. NRC Firm Survey results reported in Appendix B are for all agencies (DoD, NIH, NSF, DoE, and NASA).

66

Examples here include Applied Signal Technology, Bihrle Applied Research, Custom Manufacturing & Engineering, First RF Corporation, JX Crystals, Physical Sciences, Inc., Scientific Research Corporation and Systems, and Process Engineering Corporation.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

These activities served mainly to launch their businesses. The challenge for many was “working capital” to meet payroll for even a few employees, purchase the components necessary to develop and build even rough prototypes (which were seen as a competitive edge toward winning a Phase II award), and to pay for unavoidable overheads. Founders reported mortgaging their homes, drawing down their children’s college savings accounts, and foregoing salary for extended periods of time.

At this point in the development process, prospects for venture capital funding were nil. More importantly, their access to bank loans was also limited. Founders reported that a Phase I or even Phase II award usually did not provide sufficient collateral for the firm to qualify for a bank loan. State government programs likewise were seen as better suited to later-stage development, or as dependent on securing funds from a third party, which the state would then seek to leverage. Some firms noted that the payback requirements of state loans were more demanding than those offered by banks.

As a result of all these factors, start-up firms clearly found SBIR Phase I and Phase II awards to be an indispensable source of initial seed capital.

4.4.5.2
Firm Complexity

The case histories also offer a more complex picture of firm behaviors than suggested by the stylized dichotomy of “commercializers” and “mills” that pervades many discussions about the SBIR program, at least with respect to DoD. DoD awards data indicate the presence of a small number of firms with more than 50 Phase I and Phase II awards. Case studies indicate that some of these firms have purposefully positioned themselves to be primarily, but not exclusively, performers of contract R&D. Their business strategy is not necessarily to pursue SBIR awards, but to address the specific, sometimes “one-off,” research, testing, and evaluation needs of one or more services and agencies.

As noted above, earlier reductions in DoD 6.2 funding had led services and agencies to turn to the SBIR program to undertake R&D projects that were deleted or delayed as a result of budget cutbacks. Interviews with DoD SBIR managers suggest that although this practice has been reduced by subsequent thrusts to promote dual-use technology and more commercialization, it nevertheless remains part of the portfolio of approaches used by topic authors to achieve mission-oriented R&D objectives.

A group of small, high-tech, defense-oriented firms have positioned themselves to serve these niche R&D markets. Implicitly, the SBIR awards received by such niche firms attest to the needs of DoD Services and agencies for specialized extramural R&D services. The awards also underscore the flexible (but still competition-based) use the R&D managers are making of the SBIR topic generation and solicitation processes to get needed research accomplished in a difficult budgetary environment.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

The diversity of these awards highlights the way in which small firms can meet important needs at DoD, not least by addressing problems that provide too small a market to justify the interest of larger defense contractors. These firms may have advantages over academic institutions as well, including specialized equipment, willingness to engage in classified research, and, especially important, the capacity to address demanding time schedules, generated by newly emerging national security needs.

4.4.5.3
SBIR and Competition: A Useful Characteristic for Small Firms

The highly competitive character of DoD’s SBIR selection process, described in Chapter 6, is well recognized, and, in many respects, valued by firms. Especially at the Phase I stage, firm executives view the selection process as generating a level playing field. Because they perceive the SBIR award process to be so highly competitive, firms report using a fine mesh filter in deciding whether or not to invest internal resources in preparing an SBIR proposal. During interviews, several firms emphasized that they submitted Phase I proposals only if they could foresee a definable product and market.67 Unsuccessful firms also noted that they often received valuable feedback from DoD personnel about why their proposal was not funded. The technical commentary contained in this feedback often provided useful guidance for reworking the technology outlined in the initial proposal, akin to “revise and resubmit” commentary found in refereed journal or proposal review panels.

Another recurrent theme to emerge from the case histories is that firms do not see SBIR awards as “free” goods.68 Preparing a Phase I proposal is not costless to the firm. It involves the direct expenditure of funds for staff time, as well as the allocation of the firm’s limited pool of researchers to pursuing SBIR awards rather than alternative contracts.

Firms with DoD awards often have received awards from at least one other agency (NASA being the most frequently cited couplet). Thus, submission of proposals to DoD, or any other agency, not only entail costs in the direct sense of expenditures for staff time and ancillary contract services, but also represents an opportunity cost in submitting proposals to one agency instead of another. Interviewed firms describe careful calculations about how many proposals to write about which topics and to which agencies. Several firms report using in-

67

This is not to say that these forecasts were accurate. As later described, firm histories highlight how technologies have often evolved into quite different applications than markets and customers had originally conceived.

68

Sample selection issues affect the generalizability of this account. The case studies recount the experiences of Phase II winners only. Thus, it is possible that the motivations of firms who were unsuccessful in securing Phase I awards or those who had Phase I awards but not Phase II awards differs from this sample. Indeed, one of implicit determining factors between winners and losers of Phase II awards, given a Phase I award, may be the very characteristic of having an a priori commercial target in mind when the Phase I project was initially submitted.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

ternal policy and review committees to apportion resources to support staff time to write proposals.

4.4.5.4
Indirect/Nonlinear Commercialization

Another theme to emerge from the case histories is that the initial topic and the resulting commercial use are not always the same. A related subtheme is that this process of matching needs and capabilities often takes several years.

Two examples illustrate the times lags and unexpected outcome that frequently characterize new technologies and small firm growth. One example is Savi Technology’s well-known success with radio identification devices, now widely employed in both commercial and defense markets. It was originally intended as a product that would be installed in children’s shoes as a way for parents to monitor their location. A second example is Starsys Research Corporation’s success as both a performer of R&D and manufacturer of components for launch release systems and satellite capture systems, with diversified funding from the Air Force, MDA, NASA, and the commercial sector. These accomplishments are based on a core technology for nonelectronic thermal control systems originally aimed at the commercial water heater market. Thermacore International’s core technology involves the conversion of heat to electricity. Started in 1970 at the time of the shift in national energy and environmental policy from nuclear to solar power, the firm worked on developing heat pipe technologies for solar applications under a series of SBIR and non-SBIR contracts from DoD, NASA, and DoE. As the market for personal computers grew, so did the importance of finding solutions to dissipating the computers’ internal heat. The marketplace thus created new uses for the firm’s technology, allowing it to quickly expand into a major component supplier to HP, Dell, IBM, and Sun.

4.4.5.5
The Importance of the DoD SBIR Program for the Participating Firms

The single overriding theme that emerges from the case histories is the importance assigned by these firms to the DoD SBIR program. Beyond its contribution to the firm’s history, SBIR is seen by the companies as a beneficial national investment strategy in technological innovation, in the birth and growth of small, high-technology firms, and in enhanced national defense capabilities. These assessments are independent of whether or not the firm is currently eligible to submit an SBIR proposal, whether or not it currently has an SBIR award, the absolute and relative dollar amount of SBIR revenues to total firm revenues, and whether or not the firm operates within defense or nondefense markets. Indeed, some of the strongest statements on behalf of the program emanate from firms that are no longer eligible for the program or whose growth and current prosperity is based on DoD awards made a decade ago or more. Looking back, these now-

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

TABLE 4-16 SBIR Impacts on Company Growth

 

Number of Responses

Percent of Responses

Less than 25%

132

29.5

25% to 50%

100

22.4

51% to 75%

78

17.4

More than 75%

137

30.6

Total

447

100.0

NOTE: Data reported in Table 4-16 are for firms with at least one DoD award. NRC Firm Survey results reported in Appendix B are for all agencies (DoD, NIH, NSF, DoE, and NASA).

SOURCE: NRC Firm Survey.

successful firms credit the DoD SBIR program with providing critical infusions of funds in their formative period.

As reported in the case studies, limited access to capital to scale up production is often a significant hurdle to commercialization. Not unexpectedly, the primary sources of expansion capital reported are retained earnings, bank loans, angel capital, and venture capital, with a small amount of state government augments. Some firms report ongoing discussions with venture capital representatives about new infusions of funds to permit expansion. Other firms report less positive interactions with the venture capital community. Some are too new, too small, or too early in the product development process to have elicited any interest from investors in the venture capital community, hence the importance of SBIR awards as a validation of the technology and, indirectly, of the firm’s potential.

4.4.6
SBIR and Firm Growth

Estimates about the effect of the SBIR program as a whole (and individual projects in particular) on company growth must involve an element of judgment and one best provided by the firm itself, survey respondents did provide their own estimates of the SBIR program’s impact on their development (see Table 4-16).

Almost half of respondents (48 percent) indicated that more than half of the growth experienced by their firm was directly attributable to SBIR. This too is evidence of the powerful influence of SBIR on the development trajectories of firms winning SBIR awards.

4.5
SBIR AND THE EXPANSION OF KNOWLEDGE

Quantitative metrics for assessing knowledge outputs from research programs are well known, though far from comprehensive. Patents, peer-reviewed publications, and, to a lesser extent, copyrights and trademarks are all widely used

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

metrics. They are each discussed in detail below. However, it is also important to understand that these metrics do not capture the entire transfer of knowledge involved in programs such as SBIR. Michael Squillante, Vice-President for Research at Radiation Monitoring Devices, Inc., points out that there may be very substantial benefits from the development and diffusion of knowledge that is simply not reflected in any quantitative metric:

For example, our research led to a reduction in the incidence of stroke following open-heart surgery. Under an NIH SBIR grant we developed a tool for medical researchers who were examining the causes of minor and major post-operative stroke occurring after open-heart surgery.69

It is therefore critically important to understand that the quantitative metrics discussed below are an indicator of the expansion of knowledge. They reflect that expansion but do not entirely capture it. In particular, they say little about the impact of knowledge which generated no patent, no commercial sales, and no impact on the company’s bottom line. As can be seen from Squillante’s example above, some of these unquantifiable technological developments can remarkably improve outcomes for other actors (like open-heart surgeons).

BOX 4-7

“Results! Why, man, I have gotten a lot of results. I know several thousand things that won’t work.”

Thomas A. Edison

In addition, there is also strong evidence within the literature for the existence of what have been called “indirect effects”—spillover effects that are not captured within the context of a single project or even a single company, but may nonetheless make an important contribution to the field. Even if a project fails to reach the market, and is eventually shuttered, knowledge gained can be important in several ways: it can help other companies (and DoD itself) avoid technological dead ends (as exemplified by Edison’s quote above); it can create knowledge that is then used in subsequent projects, inside or outside the company; it can expand human capital, by helping the PI to learn more; and it can support the transition from a solely scientific orientation to one with more commercial understanding. None of these important effects can be easily captured in a quantitative analysis using the currently available tools.

Many commercial uses of new knowledge are discussed above in the sections on commercialization. But new knowledge may also be made a public good via open publications and presentations, even when such dissemination could

69

Michael Squillante, testimony presented to the NRC research team, June 11, 2004.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

TABLE 4-17 Patents, Copyrights, Trademarks and/or Scientific Publications

Number Applied For/Submitted

Number Received/Published

836

Patents

398

71

Copyrights

62

211

Trademarks

176

1,028

Scientific Publications

990

SOURCE: NRC Phase II Survey.

limit the firm’s ability to later claim intellectual property. Limits on the scope of patents and multiple forms of “leakage’ of proprietary knowledge may also lead to spillover benefits, as others employ the new knowledge without paying the inventing firm for its use.70

As shown in Table 4-17, the 816 firms responding to this question on the NRC Phase II survey applied for 836 patents, and had, at the time of the survey, received 398. These firms reported having published 990 related scientific publications, with an additional 38 under review. In addition, 593 Phase I awards that did not directly result in a Phase II generated 108 patents, with eight applications in review, and 157 scientific publications, with an additional four under review.71

Of the 564 DoD respondents that answered this question, 42 percent indicated zero patents, and 28 percent reported receiving one or two patents. The average number of patents per firm was 3.5. A few firms invested heavily in patenting SBIR innovations, however. Slightly less than ten percent of the firms reported ten or more patents, and four firms reported receiving 50 patents or more.72 The activities of the 25 firms included in the case study portion of the study suggest the probability of sizeable increases in coming years in the number of patents received by SBIR firms.

The absence of comparison groups as well as case study findings point to the need to exercise great care in interpreting these survey data. Lacking a comparative yardstick that provides patent or publication data on other, comparable firms, it is not possible to say whether the reported numbers for SBIR awardees are high or low.73

Patents are seen as indispensable forms of intellectual property rights protec-

70

For an informed discussion of this phenomenon, see A. Jaffe, Economic Analysis of Research Spillovers: Implications for the Advanced Technology Program, NIST GCR 97-708, Gaithersburg, MD: National Institute of Standards and Technology, 1996.

71

NRC Phase I Survey.

72

These data indicate the tendency of firms which have received DoD SBIR awards, to patent, not the number of patents exclusively tied to DoD only awards. Some number of these reported patents are likely attributable to SBIR/STTR awards from other agencies.

73

Examples include number of patents per million dollars of sales, or number of patents per number of SBIR awards, to gauge the effectiveness of the SBIR program across agencies.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

TABLE 4-18 Projects Reporting Patent Applications and Awards

 

Projects with Patent Applications

Projects with Patents Awards

Number

Percent

Number

Percent

Yes

281

34.4

205

25.1

No

535

65.6

611

74.9

Total

816

 

816

 

SOURCE: NRC Phase II Survey.

tion for most of the case study firms. Many SBIR awardees generate products and services that are inputs into the larger weapons systems being managed by large defense contractors. Without strong patent protection, firms are wary about entering into subcontracting relationships with prime contractors. They fear that the larger firm will imitate, reverse engineer, or preemptively patent functionally equivalent technologies, thus eliminating the SBIR firm’s major or only market.74

However, some case study firms are resource constrained in pursuing an active patent protection strategy. Other firms see patents as a limited, relatively ineffective or inefficient form of intellectual property protection given their technology domain or the size of the prospective market. Thus, some firms do not pursue patents, relying instead on trade secrets and know-how to protect their intellectual property. Choice of this strategy is influenced in part by the assessment that patents provide little economic benefit if the dominant customer of the firm’s product is the U.S. government, which under SBIR is entitled to royalty-free use of resulting technology. Also, this strategy avoids the costs of patenting and the associated public disclosure of related proprietary technological knowledge.

4.5.1
Patents

As noted above, SBIR awards are generally a significant patent stream. Table 4-18 reports findings on patents and publications, as well as other forms of intellectual property rights protection, received by NRC Phase II Survey respondents as a result of Phase II awards. The NRC Phase II Survey data indicate that about one-fourth of respondents received patents related to the relevant

74

The Night Vision Corp. v. The United States of America patent case filed before the United States Court of Federal Claims represents a variation on these concerns. In this case, Night Vision sued the United States claiming that the Air Force shared with another firm, Insight, prototypes that it had developed, and that Insight had then disassembled the prototypes, violating its SBIR data rights. See U.S. Court of Federal Claims, No. 03-1214C. 25 May 2005. See also U.S. Court of Appeals for the Federal Circuit, 06-5048, November 22, 2006.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

SBIR-funded project, and about 34 percent of projects generated at least one patent application.

These data are similar, though slightly lower, than those reported at NIH. They indicate that at a minimum, one-third of projects generated knowledge that was judged by the firm to be sufficiently unique and commercially important to be worth the significant expense of patent filing. And a quarter of all projects reported that government examiners agreed and awarded at least one patent.

This is a very significant finding. It can be viewed as addressing the tip of the knowledge iceberg. Only a small percentage of the knowledge generated during a research project meets the relatively stringent tests indicated above. Most research outcomes are not sufficiently unique to qualify for patent protection. And even unique knowledge must pass formidable internal hurdles before patent protection is sought, as the process is expensive and time-consuming.

Most companies file only one project-related patent. However, a few file many, as shown by Figure 4-15.

Because it is sometimes assumed that smaller firms have more limited access to the funds and expertise necessary to file patents, we examined the relationship between patent filing and firm size (see Figure 4-16).

By developing a “patenting ratio”—the ratio of firms with at least one patent awarded to all responses, by size of firm—the data seems to show that projects at firms with more than 75 employees are less likely to generate patents than those with less than 75 employees. The former generate 0.23 patents per project, the latter more than twice as many, 0.56, although this may be in part because larger firms are more likely to engage in manufacturing and other activities in addition to R&D. It may also reflect differences among sectors—for example, between information technology and less patent prone sectors.

FIGURE 4-15 Distribution of patenting activities—Responding projects.

FIGURE 4-15 Distribution of patenting activities—Responding projects.

SOURCE: NRC Phase II Survey.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
FIGURE 4-16 Patents and firm size at time of survey.

FIGURE 4-16 Patents and firm size at time of survey.

SOURCE: NRC Phase II Survey.

This is somewhat counterintuitive, as larger firms tend to have more resources, and patenting is expensive. Case studies do suggest that for these larger firms with better access to financial resources, SBIR projects tend to be deployed on research that is not in the company’s core strategic plan, and is therefore less likely to be seen as having immediate commercial benefit (necessary to attract the internal resources for patenting).

Analysis of the scientific importance of the patents listed has not been possible, as the patents themselves were not disclosed in the course of the survey.

4.5.1.1
Knowledge Generation vs. Commercialization?

It has also been suggested that there might be a disconnect between research-oriented firms and commercially oriented firms, and that SBIR programs should be adjusted to focus more effectively on the latter. This is—partly—the purpose of including the CAI in SBIR proposal reviews.

The NRC Phase II Survey can help to test the hypothesis that commercially oriented companies were focused on areas different from research-oriented firms. If this hypothesis was correct, we would anticipate a distinction between firms that report IP-related activities (filing patents, copyrights, and trademarks, and seeking publications) and those undertaking marketing activities (preparation of marking plans, hiring marketing staff, etc.).

In fact, Figure 4-17 shows that no such distinction is observed. 40 percent of respondents report both IP- and marketing-related activities, and a further 22 percent report neither. About 28 percent appear to fit the “commercially oriented”

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
FIGURE 4-17 Patenting and marketing activities tend to be complements not substitutes.

FIGURE 4-17 Patenting and marketing activities tend to be complements not substitutes.

SOURCE: NRC Phase II Survey.

model with no IP activities, and just over 10 percent fit the “research-oriented” model with no marketing activities.

It thus appears that the SBIR program has been quite successful in encouraging firms engaging in early-stage research to focus on the commercial applications that might be drawn from that work. Two-thirds of the respondents in this sample report some specific marketing activities related to the project.

4.5.2
Scientific Publications

Publications fill two important roles in the study of SBIR programs. First, they provide an indication of the quality of the research being conducted with program funds. More than half of the DoD-funded projects were of sufficient value to generate at least one peer-reviewed publication. Second, scientific publications are themselves the primary mechanism thorough which knowledge is transmitted within the scientific community.

The existence of papers based on SBIR projects is therefore direct evidence that the results of these projects are being disseminated widely, which in turn means that the congressional mandate to support scientific outcomes is being met.

Unlike NIH, where scientific publication is at the core of the enterprise for both the agency and the investigators (who are overwhelmingly drawn from

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

academic environments), publication is not always viewed as an unmixed blessing in the DoD environment. Even where founders had advanced degrees, their professional careers in the military and industry had moved them away from emphasizing peer-reviewed publications as a mode for establishing priority or disseminating research findings. More common in the security industry are presentations at professional meetings, or briefings with sponsors and users. Knowledge in general seems to be viewed more pragmatically and commercially, being released publicly in some cases but not in others.

Bearing these points in mind, considerable scientific publication still comes from DoD-funded SBIR research. Out of the 816 projects responding to the relevant question in the NRC Phase II Survey, 348 (42.6 percent) reported at least one scientific peer-reviewed publication related to the project, and some reported many more (see Figure 4-18).

This compares with 53.5 percent at NIH. About 15 percent of DoD projects with publications had published a single paper, but one company had published 114 papers on the basis of its SBIR, two others had published at least 100, and two more had published between 50 and 100 papers.

These data fit well with case studies and interviews, which suggested that SBIR companies are proud of the quality of their research, and justifiably so given their success in publication. Publications are featured prominently on many company Web sites, and companies like SAM—among many others—made a point during interviews that their work was of the highest technical quality as measured in the single measure that counts most in the scientific community, peer-reviewed publications.

FIGURE 4-18 Project-related scientific peer-reviewed publications.

FIGURE 4-18 Project-related scientific peer-reviewed publications.

NOTE: This figure provides a breakout of the number of publications among those that reported at least one scientific peer-reviewed publication.

SOURCE: NRC Phase II Survey.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

4.5.3
SBIR and the Universities

There is anecdotal evidence from case study interviews and papers presented at the 2004 Technology Transfer Society Conference in Albany that university views about SBIR have begun to change.75 Traditionally, universities have been wary of SBIR, partly because it might “distract” faculty from the pure pursuit of knowledge, and partly because it may lure faculty away from academic work altogether into commercial research. More fundamentally, SBIR has been seen as diverting resources from the other activities of research agencies—notably NSF and NIH—whose funds would likely find their way to universities or university researchers.

This view is changing. As universities themselves have become much more concerned about the commercial side of research—witness the dramatic increase in university patenting and licensing activities reported by NSF and the widespread introduction of IP-related clauses into faculty contracts—they have begin to see SBIR as a positive source of funding for research. Universities are increasingly looking toward mechanism of technology transfer as they increase their ties to their surrounding locales. As a part of this effort, many universities now make a conscious effort to inform faculty about SBIR, even helping faculty put together SBIR applications.

A quarter of projects indicated that there had been involvement by universities’ faculty, graduate students, or the university itself in developed technologies. For the response to a follow-up question on the variety of relationships this encompasses, see Table 4-19.76

The wide range of roles played by university staff and students indicate once more the multiple ways in which SBIR projects increase the knowledge base of the nation. Involvement in these projects provides different opportunities for university staff than those available within the academy.

The results of this change in perspective were certainly indicated in the course of case studies, where a number of SBIR recipient firms indicated the importance of their ties to universities.

These stories suggest that the flow of information and funding between small businesses and universities working within the SBIR framework is neither simple nor unidirectional. The constant flow of feedback, testing, and insights between university researchers and staff at SBIR awardees helped to move those companies forward toward product deployment into new research areas.

One further impact of SBIR has been to facilitate transitions of both tech-

75

See <http:///www.t2society.org>.

76

See questions 30 and 31 in the NRC Phase II Survey, presented in Appendix B. Of the 837 respondents to these questions, 25 percent responded in the positive (and 75 percent in the negative) to question 30 as to whether there was any involvement by university faculty, graduate students, and/or university developed technologies in executing this award. Responses to Question 31, shown in Table 4-19 address any relationships between the respondent’s firm on the Phase II project being surveyed the same 837 respondents who answered Question 30, not just those who answered “yes.”

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

TABLE 4-19 University Involvement in SBIR Projects

1.3%

The Principal Investigator (PI) for this Phase II project was at the time of the project a faculty member.

1.3%

The Principal Investigator (PI) for this Phase II project was at the time of the project an adjunct faculty member.

13.6%

Faculty member(s) or adjunct faculty member(s) work on this Phase II project in a role other than PI, e.g., consultant.

11.4%

Graduate students worked on this Phase II project.

9.2%

University/college facilities and/or equipment were used on this Phase II project.

2.2%

The technology for this project was licensed from a university or college.

3.9%

The technology for this project was originally developed at a university or college by one of the percipients in this Phase II project.

12.5%

A university or college was a subcontractor on this Phase II project.

NOTE: Survey respondents could check more than one category.

SOURCE: NRC Phase II Survey (n = 837).

nologies and researchers from university labs to the commercial environment. Data from the NRC Firm Survey (using data for all agencies) strongly support this hypothesis, with 66 percent of SBIR companies including at least one academic as founder, and 28 percent having more than one academic as a founder (see Figure 4-19). The same survey found that about one-third of founders were most recently employed in an academic environment before founding the new company.

These data and evidence from case studies strongly indicate that SBIR has indeed encouraged some academic scientists to work in a more commercial environment.

What is not clear from this research is the extent to which universities themselves see SBIR as a mechanism for technology transfer, commercialization, and additional funding for university researchers. These questions are beyond the scope of the current study but merit additional research.

4.5.4
Inventions and Indirect Knowledge

This analysis has understandably focused on the data available from the NRC Phase II Survey about the IP-related activities of firms. However, it must be stressed that these are only the formal IP-related activities. Every project generates a very wide range of less formal, less easily captured knowledge effects, which are nonetheless important despite being very difficult or even impossible to quantify. The case of Thermacore, described in Box 4-9, provides some insights into this kind of program effect.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

BOX 4-8

Brimrose Corporation

Brimrose Corporation was founded by Dr. Ronald G. Rosemeier while still a Ph.D. student in Johns Hopkins University material science program. After graduation, working as a post-doctoral student at the University of Maryland, he started writing SBIR proposals.

After three years of submitting unsuccessful proposals, he was awarded four Phase I’s, whose total value approximated $200,000. He started hiring his first employees and began applying for loans at banks. Because banks were not willing to give him loans backed by the SBIR awards, he amassed charges of $100,000 on credit cards. Six months later he wrote the follow-on Phase II proposals, receiving awards on three of them for a total of $1.5 million.

With these funds, he was able to hire additional employees, and expand operations. At that time 10 percent of the firm’s revenues were from commercial sales (selling X-ray imaging at tradeshows) and 90 percent from the SBIR awards. As the firm started commercializing new products, this percentage shifted to 80 percent commercial revenue and 20 percent SBIR revenue. Most of the R&D team and few of the support staff were hired under SBIR related activities. Brimrose began operations with 6 employees; by 2005 its employment level had reached approximately 60 employees. Overall, throughout its history, Brimrose has received 65 Phase I and 28 Phase II SBIR awards.

The firm’s major lines of business are industrial process control spectroscopy in the pharmaceutical and petrochemical industries, nondestructive testing and evaluation and novel opto-electronics devices. Its business model is to specialize in applied R&D. A few of the SBIR programs have directly resulted in commercial products but most have led to product improvements.

The firm’s commercialization strategy emerged from and has been greatly enhanced through its participation in SBIRs. The Phase I and Phase II SBIR funding allowed them to determine the feasibility of new technology and develop it to the point of prototype development without allocation of significant internal resources. Following prototype development, Brimrose used internal funds from previous commercial sales to bring the technology to the point of commercial availability. Thus, the SBIR funds lowered the company’s financial burden by decreasing the risks associated with new technology development.

4.6
UNDERSTANDING OUTCOMES: EMPIRICAL FINDINGS

Overlapping methods were used to address questions relating to commercialization of DoD’s SBIR research. First, a Web-based survey was conducted of firms that had received at least one Phase II award between the years 1992–2001.77 Second, case histories were prepared of a sample of firms that had received Phase II awards. Thirty-one firms representing a cross-section of DoD sponsoring agencies and states were selected for study. Third, an additional Web-

77

Appendix B contains a full account of the survey methodology.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

BOX 4-9

Thermacore: Creating Knowledge and Capacity

Many of the benefits to participants and their clients are hard to quantify. Thermacore, for example, believes that its experiences under the SBIR program provided it with a “brain pool” of “know-how” related to manufacturing reliability. These tacit skills have contributed to the firm’s ongoing competitive position even as patents on its initial core technologies have expired.

Thermacore was founded by Yale Eastman, an RCA employee, in 1970. The firm started as a “garage” start-up, focused on RCA-abandoned heat pipe technologies for solar applications. Throughout most of the 1970s, the company remained small, with no more than ten employees, working on industry and government R&D contracts. In the 1980s, it began to grow via non-SBIR and SBIR R&D contracts primarily from NASA, DoD, and DoE.

Thermacore and SBIR. Thermacore began active pursuit of SBIR awards in the early 1980s. While it remained eligible, Thermacore received 82 SBIR awards from several government agencies, including DoD, NASA, and DoE. This substantial and repeated support, provided by several agencies over a number of years, highlights the way in which SBIR supports technologies that are complex and require a number of incremental technological advances to transition an R&D concept into a viable commercial product.

Thermacore describes its growth as a case of the marketplace catching up to its technology. In the early 1990s, Thermacore was approached by Intel to discuss the possibility of mass producing its heat pipe technology for use in the rapidly growing market for personal computers. With financial support from a venture capital firm, Thermacore took the risk of setting up a production line before receiving orders. Subsequently, it received large orders from several major computer manufacturers, such as HP, Dell, IBM, and Sun.

Reflecting its transition from an R&D to a production-oriented firm, contract R&D projects and OEM work now account for only 6 percent of revenues; 94 percent comes from sale of commercial products.

After its sale in 2001, following the retirement of its founder, Thermacore became ineligible for SBIR awards because it was a wholly owned subsidiary of a larger company. It does however continue to do some SBIR-funded research as a subcontractor to small firms conducting Phase I and II research.

based survey addressed Phase I awards. Additional data were collected from the DoD CAI index database.

Collectively, these methods strongly suggest that DoD companies funded by SBIR do commercialize their results, and that the rate of commercialization appears to be increasing. Thus, the charge of low commercialization by multiple-award winners no longer appears to be supported by the data.78 If it even was a

78

The 1992 GAO study that identified this “trend” is often cited by the GAO investigators, multiple caveats are not. For example, the limited time frame of the study for several of the years studied, e.g., 1987 awards reviewed in 1990. See U.S. Government Accounting Office, Federal Research: Small Business Innovation Research Shows Success but Can Be Strengthened, GAO/RCED-92-37, op. cit.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
FIGURE 4-19 Number of academics as founders.

FIGURE 4-19 Number of academics as founders.

SOURCE: NRC Firm Survey.

problem in the past, it is now to a considerable extent resolved, with some very large winners graduating from the program in various ways (e.g., Foster-Miller’s purchase by a foreign firm, which made it ineligible) and also clear evidence that some multiple-award winners are now reducing their dependence on SBIR, and to generate more substantial commercial results (e.g., Radiation Monitoring, PSI).

The findings from the three approaches are generally consistent with one another and complementary. The case studies provide explanations for findings generated via the two sets of surveys. The findings from the separate approaches at times though tend to emphasize different aspects of SBIR’s program impacts mainly because they frame the question of commercialization differently. The survey findings, in keeping with congressional and Executive Branch usage, define commercialization in terms of economic outcome measures, such as sales, investment, employment, and patents. The case histories contain data on these variables, but also narratives about firm formation, business strategies, intellectual property rights strategies, and processes of technological innovation and diffusion of innovations. They also contain limited data on the gestation processes connecting SBIR awards and commercial outcomes.

The review of outcomes described in this chapter provides an overview of an effective program, meeting a wide variety of program goals. The DoD awards are generating new knowledge evidenced in publications and successful patent-

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×

ing activity, new products and processes to maintain equipment, improve supply management, and develop new products to better support and defend those charged with combat missions. The program also provides valuable linkage between university professors and students and the commercial and defense market place. By growing and nurturing the defense industrial base, the program is also encouraging high-tech entrepreneurship, thereby increasing innovation, encouraging competition, and offering greater choices. The SBIR program is helping DoD to meet the new and often sudden challenges of a turbulent world.

Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 86
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 87
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 88
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 89
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 90
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 91
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 92
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 93
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 94
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 95
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 96
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 97
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 98
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 99
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 100
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 101
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 102
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 103
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 104
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 105
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 106
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 107
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 108
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 109
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 110
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 111
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 112
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 113
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 114
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 115
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 116
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 117
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 118
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 119
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 120
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 121
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 122
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 123
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 124
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 125
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 126
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 127
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 128
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 129
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 130
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 131
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 132
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 133
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 134
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 135
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 136
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 137
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 138
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 139
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 140
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 141
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 142
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 143
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 144
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 145
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 146
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 147
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 148
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 149
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 150
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 151
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 152
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 153
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 154
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 155
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 156
Suggested Citation:"4 Outcomes." National Research Council. 2009. An Assessment of the SBIR Program at the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/11963.
×
Page 157
Next: 5 Phase III Challenges and Opportunities »
An Assessment of the SBIR Program at the Department of Defense Get This Book
×
Buy Hardback | $122.00 Buy Ebook | $99.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The SBIR program allocates 2.5 percent of 11 federal agencies' extramural R&D budgets to fund R&D projects by small businesses, providing approximately $2 billion annually in competitive awards. At the request of Congress, the National Academies conducted a comprehensive study of how the SBIR program has stimulated technological innovation and used small businesses to meet federal research and development needs. Drawing substantially on new data collection, this book examines the SBIR program at the Department of Defense and makes recommendations for improvements. Separate reports will assess the SBIR program at NSF, NIH, DOE, and NASA, respectively, along with a comprehensive report on the entire program.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!