5
Quantitative Outcomes
As was noted in Chapter 1 (Introduction), Congress mandated four goals for the Small Business Innovation Research (SBIR) program: (1) to stimulate technological innovation; (2) to use small business to meet Federal research and development needs; (3) to foster and encourage participation by minority and disadvantaged persons in technological innovation; and (4) to increase private-sector commercialization derived from federal research and development.1 This chapter provides an analysis of program outcomes related to the first, second and fourth goals: stimulating technological innovation, using small business to meet Federal research and development needs, and increasing private-sector commercialization of federally funded research.2 The approach analyzes outcomes as revealed primarily by the performance of National Aeronautics and Space Administration (NASA) Phase II awards from fiscal year (FY) 1998 to FY2007 (the period covered by the 2011 Survey). The focus is on Phase II awards rather than Phase I awards because Phase II-funded projects are expected to have business plans and to have progressed toward commercialization.
Although NASA was among the earliest agencies to adopt a fully electronic submission and project management system, the agency has not led the way on tracking outcomes. NASA has recently begun to track outcomes using firm-completed surveys as part of its Electronic Handbook, but these records currently cover less than one-half of the awards made from FY20032012 (596 of 1,362 Phase II awards). Although some highly visible anecdotes have emerged about SBIR technologies being used to great effect by NASA—for example, several SBIR technologies were used with the Mars Rover—NASA does not have systematic data to describe the transition of SBIR
__________________
1 Small Business Innovation Development Act of 1982, P.L. 97-219, July 22, 1982.
2 The second goal of using small businesses to meet federal research and development needs was also discussed to some extent in Chapter 2 (Program Management). The third goal of fostering the participation of women and minorities is the focus of Chapter 6.
technologies within NASA into the Mission Directorates and then into space or other agency applications, nor does it have systematic data on the take-up of SBIR-funded technologies outside NASA.
Therefore, the analysis of outcomes in this report is based primarily on the 2011 Survey, for the period FY1998-2007. The survey methodology is described in detail in Appendix A and a description of the survey response rate and non-respondent bias3 is provided in Box 5-1. The overall target population for the survey reported in this chapter is NASA Phase II awards made FY19982007,4 and most response data is reported at the project-level. Some survey questions, however, collect company-level information (such as number of employees). In cases where company information, as opposed to individual project information, was collected, multiple responses from the same company were averaged. Tables and figures with company-level data are marked as reporting the number and percentage of responding companies. Not all survey recipients completed every survey question; as a result, the number of respondents and the number of responding companies varies.5
The survey data have limitations that signal the need for caution. The response rate for NASA was lower than for some other agencies,6 and the number of responses—although sufficient to provide useful data—was also lower than for other agencies. In addition, the 2011 Survey inevitably captured outcomes at a specific point in time: many projects had not yet generated maximum commercial returns—some were just entering their commercialization phase, while other more mature projects may not generate revenues for many years to come. These caveats are important to bear in mind while reviewing the data in this chapter. As a result, the study findings reported in Chapter 8 are based on not only the 2011 Survey data, but also the case studies and interviews presented in Chapter 7 (Insights).7
COUNTERFACTUALS
It is always difficult to tightly determine the impact of a given SBIR award. Many factors affect the success and failure of companies and projects, and it is often difficult to determine whether a specific factor was a
__________________
3 Multiple sources of bias in survey response are discussed in Box A-1 of Appendix A.
4 See Box 5-1 and Appendix A for a description of filters applied to the starting population.
5 Not all questions were applicable to all respondents, depending on their answers to particular questions. For example, questions 36 and 37, which address sales outcomes, were directed only to respondents reporting sales in response to question 35. In other cases, respondents did not answer particular questions. The reasons for these non-responses are unknown.
6 The reason for the lower response rate is not known. See Box 5-1 for a discussion of potential non-respondent bias in the survey.
7 See Appendix A for a detailed description of the survey methodology. The 2011 Survey questionnaire is reproduced in Appendix C.
necessary condition for success. Worse still, the large number of factors and the multiple paths to success and failure mean that it is unusual to be able to state with confidence that a particular intervention—in this case an SBIR award—constitutes a sufficient condition for a project’s success.
Still, it is worth considering what would have occurred absent SBIR funding from the perspective of those most likely to have detailed knowledge and understanding of their particular projects: the principal investigators (PIs). Accordingly, the 2011 Survey asked a series of questions focused on the likely effect of the absence of SBIR funding. Of course, asking recipients about the impact of funding raises possible conflicts of interest, so results should be interpreted with some caution. However, these are awards some years in the past now, and many respondents no longer apply for SBIR funding for a variety of reasons.
PROJECT GO-AHEAD ABSENT SBIR FUNDING
One approach has been to ask recipients for their own views on the program’s impact on their project or company. In particular, the survey asked respondents whether the project would have been undertaken absent receipt of the SBIR award and whether the scope and timing would have been affected. Responses are summarized in Table 5-1. Seven percent of the respondents indicated that the project probably or definitely would have proceeded without program funding. In contrast, 75 percent thought the project probably or definitely would not have proceeded absent SBIR funding. Nineteen percent of respondents were uncertain.
These data have interesting wider implications for debates about early-stage funding: they suggest a weakness in the “crowding out” hypothesis, as it would appear that awardees at least—presumably those with the closest knowledge of funding prospects for the project—overwhelmingly believed it to be unlikely that alternative funding would be found.
TABLE 5-1 Project Undertaken in the Absence of this SBIR Award
|
Question: In your opinion, in the absence of this SBIR award, would the company have undertaken this project? |
Percentage of Respondents |
Number of Respondents |
|
Definitely yes |
2 |
3 |
|
Probably yes |
5 |
9 |
|
Uncertain |
19 |
34 |
|
Probably not |
41 |
73 |
|
Definitely not |
34 |
60 |
| Total | 100 |
179 |
SOURCE: 2011 Survey, Question 24.
The small number of respondents (12) who believed the project might have proceeded without SBIR funding were asked additional questions about the impact on project scope, duration, and timelines. They indicated the following:
- Project scope would have been narrower (67 percent)
- Project would have been substantially delayed (75 percent)
- Project would have taken longer (75 percent)
- Project would not have hit necessary milestones (75 percent)
Overall, these views indicate that SBIR funding was important not only for the go/no-go decision but also for the eventual shape and indeed likely impact of the project. Delay in bringing projects to conclusion—and hence to the point of potential market entry—can have a disastrous effect, as the window for market entry can be a narrow one.
CHAPTER OUTLINE
The remainder of this chapter is broken into two sections: (1) Quantitative Survey Evidence that NASA Increased Commercialization and (2) Quantitative Survey Evidence that NASA Stimulated Technological Innovation. Commercialization is discussed first, where it is treated as inclusive of both technology infusion into NASA (i.e., meeting federal R&D needs, which was also treated to some extent in Chapter 2) and private-sector commercialization. These sections are preceded by a chapter overview.
Two annexes to this chapter, contained in appendixes F and G of this report, offer supplemental data. Appendix F contains a range of 2011 Survey data on SBIR effects on companies that is not reported in this chapter. Appendix G provides data from the Department of Defense on NASA SBIR awards.
OVERVIEW
Increasing Commercialization
Each agency has its own priorities for the SBIR program, and SBIR-supported technologies can be infused into agency programs or commercialized in the private sector or both. At NASA, the overwhelming emphasis has been on the adoption of SBIR-funded technologies for use within the agency in support of the agency mission, yet not all SBIR funding goes to areas in which NASA has acquisition activity, such as aeronautics, and evidence of commercialization outside NASA was also found as part of the committee’s research as noted below.
NASA contracts for SBIR-funded technologies tend to be relatively small. It appears that small companies whose mission is to work on space
technologies for NASA are likely to remain small, and their projects are unlikely to generate huge commercial outcomes. NASA SBIR companies responding to the 2011 Survey had a median of 10 employees at the time the surveyed award was made and a median of 15 employees at the time of the survey. In some cases, spinoffs from the mission technologies led to contracts and successes outside NASA. Forty-six percent of survey respondents reported some sales, and an additional 26 percent reported that they anticipate future sales. At the same time, the scale of commercialization was limited: one percent of respondents reported project-related sales of over one million dollars.
Just over one-third of projects with sales reported sales to domestic private-sector customers. Only 17 percent of respondents reported that their products were in use at NASA. Almost as many reported use at the Department of Defense (DoD). About 9 percent reported sales to export customers, and 2 percent report sales to NASA prime contractors.
Survey results suggest that subsequent investment in an SBIR project generates potential commercial value even if the project has not yet reached the market. Sixty-five percent of respondents reported additional investment funding. Non-SBIR federal funding was overwhelmingly the most likely source of additional funding (71 percent). No other source was used by more than 10 percent of respondents. About 2 percent reported U.S. venture capital (VC) funding.
Stimulating Technological Innovation
Regarding the first goal, there is ample quantitative evidence that NASA is using its SBIR program to stimulate technology development, enhance science and technology, and add to the scientific knowledge base. Among the evidence are data showing linkage of survey respondents to a university (nearly one-third). Among respondents: 21 percent reported that a research university or college was a subcontractor; 15 percent reported that university faculty worked on the project (not as Principal Investigator [PI]); and 14 percent reported that graduate students worked on the project. Seventy-seven different universities were identified as project partners. Sixty-three percent of respondents reported at least one academic founder, and 29 percent of responding companies reported that the most recent prior employment of the founder was at a university.
Survey results show that SBIR projects generate substantial knowledge-based outputs such as patents and peer-reviewed publications—widely used metrics of the creation and dissemination of technical knowledge. Forty-five percent of responses reported receipt of at least one patent related to the surveyed project. Fourteen percent of responding companies reported receipt of 10 or more patents for all SBIR-funded technologies.8 Eighty-two percent of respondents reported at least one peer-reviewed publication resulting from the
__________________
8 N=64 companies.
surveyed project. Thirty-one percent reported more than three peer-reviewed publications.
Technological innovation can be stimulated by fostering innovative companies, thereby expanding the nation’s capacity for innovation. Forty percent of respondents reported that the company was founded entirely or in part because of the SBIR program (not necessarily just the NASA SBIR program). The receipt of the award targeted by the survey had a “transformative” effect according to 25 percent of respondents and a “substantial positive long term effect” according to 56 percent of respondents.
At the same time, survey results show that NASA program participants in general are not overly dependent on SBIR awards. More than one-quarter of respondents reported that they received no SBIR funding during the most recent fiscal year. Less than one-quarter reported that SBIR accounted for more than 50 percent of company revenues during the most recent fiscal year.
QUANTITATIVE SURVEY EVIDENCE THAT NASA INCREASED COMMERCIALIZATION
This section presents the quantitative survey results related to the success of the NASA SBIR program in meeting its commercialization-related goals, both through the take-up of technology by NASA and commercialization outside NASA. Commercialization is among the more measurable outcomes of the SBIR program and has become a primary benchmark for program performance.9
The priority of the NASA SBIR program is to serve the agency’s mission by providing technology not otherwise available for use by the agency. Sales to commercial non-agency buyers are considered to be an important but secondary goal from the perspective of NASA.10 Although similar to DoD in this respect, the program’s impact on quantitative outcomes is different because NASA represents a much smaller market than DoD for SBIR outputs, making the opportunities to generate sales to NASA fewer and on a much smaller scale.
Therefore, the structure of the program itself limits the commercial opportunities for program participants. On the one hand, the program is explicitly designed to provide tools and capabilities for use by NASA; on the other hand, NASA’s needs for some technologies are of insufficient scale to provide a viable and sustainable commercial market for those same tools and capabilities.11Moreover, the time from technology development in SBIR to
__________________
9 The focus on commercialization, however, should not be allowed to obscure the requirement that the program meet all four congressionally mandated objectives.
10 This perspective is drawn from numerous discussions with agency program managers, Center SBIR directors, and other agency staff.
11Yet it is possible to overstate the uniqueness of NASA relative to other generators of research. While many NASA SBIR projects are in technical domains – e.g., hypersonic vehicles, scramjets, in-situ use of lunar or Martian materials – for which related commercial markets may have not emerged (and may never emerge), many others – e.g., project management software, sensors, heat
eventual acquisition for a program may span years, a decade, or longer, presenting further challenges to small businesses and few market opportunities. That said, SBIR program participants at NASA are small for-profit companies (as for other agencies), and they need to proceed in ways that provide a sustainable path forward. The extent to which they achieve this is the subject of this section.
Defining “Commercialization”
Several important conceptual challenges emerge when seeking to define “commercialization” for the purposes of the SBIR program. Like many apparently simple concepts, commercialization becomes progressively more difficult and complex as it is subjected to further scrutiny.12 For example:
- Should commercialization include just sales or other kinds of revenue, such as licensing fees and funding for further development?
- What is the appropriate benchmark for sales? Is it any sales whatsoever, sufficient sales to cover the costs of awards, sales that lead to breaking even on a project, or sales that reflect a commercial level of success and viability? The latter at least would likely be different for each project in each company.
- Should commercialization include license fees and sales by licensees, which may be many multiples of the sales by the licensors?
- Should commercialization metrics focus only on formally recognized Phase III contracts,13 or should they more widely cover follow-on sales and development activities even when not formally recognized as Phase III?
NASA-supported technologies and program have long development cycles (see Box 5-2). For the purposes of this study, the committee deployed a broad net to capture a range of data. Once acquired, these data were analyzed in a variety of ways to provide insights into this complex topic.14
__________________
and energy management systems, aeronautics, unmanned systems, processing and inspection of composite materials, etc. – have markets outside of NASA programs.
12 Measurement of commercialization also raises questions about time needed to commercialize new technologies. For a discussion of this “commercialization lag,” see Box A-1 in Appendix A. As noted separately in Appendix A, limiting the 2011 Survey to Phase II awards from no later than FY2007 allowed two years for completion of the Phase II awards and an additional two years for commercialization, and this timeframe was consistent with the 2005 Survey.
13 “Phase III” is in the context of DoD a technical term for contracts that are officially recognized as following from an SBIR or Small Business Technology Transfer (STTR) Phase II award. Not all follow-on contracts are so recorded.
14 For an overview of the commercialization metrics and survey used in this study, see Appendix A (Methodology).
Sales and Other Revenues
Perhaps the most used metrics for assessing commercial outcomes for SBIR-type programs are revenues and licensing fees. Although we are cautious about the overuse of these metrics—which the committee has acknowledged by using a wide range of metrics in the current assessment—sales and licensing revenues remain important considerations.
Reaching the Market
The first survey question in this area concerns reaching the market: Did the project generate sales, and if not, are sales expected? The second part of this question is necessary given the long life cycle of some projects. Responses are summarized in Table 5-2. Forty-six percent of respondents reported some sales, and 26 percent expected sales in the future. Although the percentage reporting sales remained flat from the 2005 Survey to the 2011 Survey, which itself is notable given the economic downturn that occurred, the percentage expecting sales in the future increased from 14 percent in 2005 to 26 percent in 2011.
TABLE 5-2 NASA SBIR Sales and Licensing Revenues Reported by 2011 Survey Respondents
| Sales Status for the Surveyed Project | Percentage of Respondents | Number of Respondents | ||
| No sales to date | 54 | 97 | ||
|
No sales to date, nor are sales expected. |
28 | 50 | ||
|
No sales to date, but sales are expected. |
26 | 47 | ||
| Any sales to date | 46 | 82 | ||
|
Sales of product(s) |
35 | 63 | ||
|
Sales of process(es) |
3 | 6 | ||
|
Sales of services(s) |
17 | 30 | ||
|
Other sales (e.g., rights to technology, licensing, etc.) |
3 | 6 | ||
| Total | 100 | 179 |
NOTE: Respondents could report multiple different types of sales for a single project, so the types of sales do not sum to “Any sales to date.”
SOURCE: 2011 Survey, Question 35.
Amount of Sales and Licensing Revenues
The percentage of projects reaching the market is an important metric, but it is not a sufficient determinant of success. It is also important to understand the distribution of sales. Of the 179 responses to the survey, 80 reported sales. The 80 respondents who reported sales to also reported the amount of sales, grouped into tiers. These data are summarized in Table 5-3. Most sales are at the lower end of the scale: 54 percent or respondents reported less than $500,000. Only one percent reported sales of at least $10 million, while about four percent reported sales of $5 million to <$10 million. Overall, 33 percent reported sales of at least $1 million. This compares with 18 percent from the 2005 Survey, which suggests that the scale of commercialization is growing at NASA, although due to the low response rate, the data must be considered with caution.
Markets by Sector
For those projects with sales, the survey asked respondents about the market sectors in which sales were made. As shown in Table 5-4, respondents reported an average of 19 percent of project sales to NASA or NASA prime contractors. But a closer look at the data, reveals that only an average of 2 percent of sales were to a NASA prime contractor. Given that most core NASA programs have a significant share of work performed by prime contractors, this low percentage is deserving of attention if SBIR programs are going to play a larger role in meeting NASA core program needs. As noted in Chapter 2, while nearly three-quarters of survey respondents indicated that the assigned COTR
TABLE 5-3 Distribution of Total Sales Dollars, by Range, Reported by 2011 Survey Respondents Reporting Sales Greater than Zero Dollars
|
Total Sales |
Percentage of Respondents that Reported Sales |
Number of Respondents |
|
Some sales under $100,000 |
28 |
22 |
|
$100,000-$499,999 |
26 |
21 |
|
$500,000-$999,999 |
14 |
11 |
|
$1,000,000-$4,999,999 |
28 |
22 |
|
$5,000,000-$9,999,999 |
4 |
3 |
|
$10,000,000-$19,999,999 |
1 |
1 |
|
$20,000,000-$49,999,999 |
0 |
0 |
|
$50,000,000 or more |
0 |
0 |
| Total | 100 | 80 |
NOTE: This question was directed only to those respondents reporting sales in response to 2011 Survey question 35.
SOURCE: 2011 Survey, Question 36 B1.
was either quite or extremely knowledgeable about the SBIR program, over one-half of respondents gave a low rating of COTR assistance in finding market opportunities (1 or 2 on a 5-point scale), and 58 percent indicated that their COTR was either not very helpful or not at all helpful in connecting the firm with sources of Phase III funding.15
The leading market overall, as shown by Table 5-4, was the domestic private sector (35 percent), followed by DoD and its contractors (24 percent). Nine percent were exports. These results suggest strong crossover between the NASA and DoD market sectors and provide further evidence of the limited market available within NASA, even though take-up within NASA is a primary objective of the NASA SBIR program.
Employment
As with prior surveys, 2011 Survey respondents were asked both about the size of the company at the time of the award and the current size, in terms of number of employees.
The data in Table 5-5 show that, at the time of the award, NASA SBIR Phase II awardees were in general far below the 500 employee limit defining a small company for Small Business Administration (SBA) purposes. Nearly one-third (30 percent) of companies reported fewer than 5 employees; overall, about two-thirds reported fewer than 20 employees at the time of award, up slightly from the 2005 Survey. The number of companies with 100 or more employees
__________________
15 See section on “Working with the Contracting Officer Technical Representative,” especially tables 2-8 and 2-9, in Chapter 2: Program Management and 2011 Survey questions 49, 50, and 52 in Appendix C.
decreased from 13.6 percent in 2005 to 7 percent in 2011. The mean and median numbers of employees were 26 and 10, respectively.
TABLE 5-4 Average Percentage of Project Sales by Market Sector, Reported by 2011 Survey Respondents
| Market Sector | Mean Value (Percent) Reported by Respondents that Reported Sales |
|
Domestic private sector |
35 |
|
NASA |
17 |
|
Department of Defense (DoD) |
14 |
|
Export markets |
9 |
|
Prime contractors for DoD |
10 |
|
Other federal agencies |
4 |
|
Prime contractor for NASA |
2 |
|
State or local governments |
1 |
|
Other (Specify) |
8 |
| Total | 100 |
NOTE: Number of respondents = 82. This question was directed only to those respondents reporting sales in response to 2011 Survey question 35. For this question, each respondent reported a percentage distribution. The values above are calculated by deriving the mean value for all the responses received for each category.
SOURCE: 2011 Survey, Question 37.
TABLE 5-5 Number of Company Employees at Time of Award and at Time of Survey, Reported by 2011 Survey Responding Companies
| Number of Employees | At Time of Award | At Time of Survey | ||
| Percentage of Responding Companies | Number of Responding Companies | Percentage of Responding Companies | Number of Responding Companies | |
| Under 5 | 30 | 23 | 24 | 18 |
| 5 to 9 | 18 | 14 | 12 | 9 |
| 10 to 19 | 19 | 15 | 19 | 14 |
| 20 to 49 | 20 | 15 | 23 | 17 |
| 50 to 99 | 6 | 5 | 11 | 9 |
| 100 or more | 7 | 5 | 11 | 8 |
| Total | 100 | 77 | 100 | 75 |
|
Mean |
26 | 46 | ||
|
Median |
10 | 15 | ||
NOTE: In cases where company information, as opposed to individual projection information, was collected, multiple responses from the same company were averaged.
SOURCE: 2011 Survey, Questions 18A and 18B.
Respondents also provided the current number of employees (at the time of the survey). Although the results report information from surviving companies, the comparisons are useful. Table 5-5 shows that the smallest firms accounted for the largest percentage of awards, with 24 percent of companies having fewer than five employees, a decrease of 6 percentage points from the time of the award. These results indicate that one-fifth of the smallest Phase II companies grew out of this category (or ceased operations). Median employment grew from 10 to 15. The percentage of companies with 100 or more employees increased from 7 percent at the time of award to 11 percent currently, while the percentage of companies with fewer than 20 employees decreased from 67 percent to 55 percent. This distribution is broadly similar to that drawn from the 2005 Survey, except for the percentage of larger firms, defined as 100 or more employees (22 percent in 2005 compared to 11 percent in 2011) and suggests that Phase II awards are associated with increasing firm size.
Further Investment
The ability of SBIR projects and companies to attract further investment has traditionally been a defining metric for SBIR outcomes.16 Sixty-five percent of the respondents had received additional investment,17 an increase from 52 percent in the 2005 Survey.
As with prior surveys, the amount of additional investment is substantially skewed. Table 5-6 summarizes responses from the 50 respondents who reported receipt of additional investment from non-SBIR federal sources. In most cases, the amount of extra investment was quite modest: 64 percent of respondents reported additional funds of less than $500,000, while only two percent of respondents reported additional funds of $50 million or more.
Respondents also reported the sources of additional investments. The funding distribution was dominated by federal non-SBIR sources. Consistent with earlier evidence of strong commercial markets for both NASA and DoD, 71 percent of the average (mean) additional investment of $1.5 million came from federal non-SBIR sources. The dependence on federal non-SBIR funding increased from 2005, when the percentage was 51 percent (see Table 5-7). Other sources individually accounted for less than 10 percent of additional investment, and U.S. venture funding accounted for about 2 percent.
Company-level Commercialization through Mergers and Acquisitions
SBIR companies sometimes commercialize their technology through mergers or other company-level activities. As shown in Table 5-8, 16 percent of responding companies reported that their company had spun off one or more
__________________
TABLE 5-6 Additional Investment by Amount to Surveyed Projects for which Additional Funds were Received, Reported by 2011 Survey Respondents
| Amount of Additional Investment | Percentage of Respondents that Reported Additional Project Investment | Number of Respondents |
|
Some investment under $100,000 |
24 |
12 |
|
$100,000-$499,999 |
40 |
20 |
|
$500,000-$999,999 |
12 |
6 |
|
$1,000,000-$4,999,999 |
20 |
10 |
|
$5,000,000-$9,999,999 |
2 |
1 |
|
$10,000,000-$19,999,999 |
0 |
0 |
|
$20,000,000-$49,999,999 |
0 |
0 |
|
$50,000,000 or more |
2 |
1 |
| Total | 100 | 50 |
NOTE: This question was directed only to those respondents reporting additional project investment in response to 2011 Survey question 33.
SOURCE: 2011 Survey, Question 34.1.
TABLE 5-7 Distribution of All Reported Additional Project Investments by Source of Funds, Reported by 2011 Survey Respondents
| Source of Additional Investment | Percentage of Total Funding Reported by Respondents that Received Additional Project Investments |
|
Non-[SBIR/STTR] federal funding |
71 |
|
U.S. venture capital |
2 |
|
Foreign investment |
5 |
|
Other private equity (including angel funding) |
2 |
|
Other domestic private company |
7 |
|
State or local governments |
0 |
|
Colleges or universities |
0 |
|
Your own company (including money you borrowed) |
9 |
|
Personal funds |
3 |
| Total | 100 |
|
Average additional funding (mean) ($000s) |
1,488 |
|
N = Number of Respondents |
116 |
NOTE: This question was directed only to those respondents reporting additional project investment in response to 2011 Survey question 33. Values reported are percentages of total funding reported.
SOURCE: 2011 Survey, Question 34.
TABLE 5-8 Company-level Changes, Resulting from the SBIR Program, Reported by 2011 Survey Responding Companies
| Company-level Activity | Percentage of Responding Companies | Number of Responding Companies |
|
Established one or more spin-off companies |
16 |
11 |
|
Been acquired by/merged with another firm |
9 |
6 |
|
Made an initial public offering |
3 |
2 |
|
Planning to make an initial public offering in 2011-2012 |
2 |
1 |
|
None of the above |
75 |
53 |
NOTE: Number of Responding Companies = 71. In cases where company information, as opposed to individual projection information, was collected, multiple responses from the same company were averaged. Percentages do not sum to 100 percent because respondents could select more than one answer.
SOURCE: 2011 Survey, Question 10.
new companies. Greater than 9 percent of responding companies indicated that the awardee company had been acquired or merged with another firm. Five percent indicated that the company had made or planned to make an initial public offering (IPO). Three-quarters of responding companies indicated that their companies had not been acquired, had not implemented or planned an IPO, and had not established a spin-off company.
Respondents reported on a range of market-related activities involving agreements between their company and other organizations, which can again be taken as an indication of commercial activity. About 25 percent of respondents identified at least one market-related activity undertaken by their company as a result of the technology developed during the surveyed project.18 Of those companies, 61 percent completed at least one research development (R&D) agreement; 30 percent entered into licensing agreements; and 26 percent entered into customer alliances (see Table 5-9).
Commercialization Training and Marketing
Federal agencies have in recent years provided more commercialization training for SBIR awardees. In some cases this training has been mandatory. NASA does not provide formal commercialization training. However, 22 percent of respondents nonetheless claimed to have received agency-sponsored training related to the surveyed award.19
__________________
TABLE 5-9 Market-oriented Activities Resulting from Project Technology, Reported by 2011 Survey Respondents—Finalized agreements with U.S. companies and investors
| Market-oriented Activity | Percentage of Respondents | Number of Respondents |
|
R&D agreement(s) |
61 |
26 |
|
Licensing agreement(s) |
30 |
13 |
|
Customer alliance(s) |
26 |
11 |
|
Marketing/distribution agreement(s) |
19 |
8 |
|
Manufacturing agreement(s) |
21 |
9 |
|
Sale of technology rights |
12 |
5 |
|
Joint venture agreement |
9 |
4 |
|
Sale of company |
5 |
2 |
|
Partial sale of company |
5 |
2 |
|
Company merger |
0 |
0 |
|
Other |
5 |
2 |
NOTE: Number of Respondents = 43. Percentages do not sum to 100 percent because respondents could select more than one answer.
SOURCE: 2011 Survey, Question 38.1.
The 2011 Survey asked whether the company has at least one full-time staff person for marketing. This new question provides another metric to gauge the extent to which the company has focused on marketing. Only 40 percent of respondents reported that their company had at least one full-time marketing staff, a figure likely explained by the high percentage of very small firms among the respondents.20
Conclusions: Commercialization at the Company Level
Evidence from the 2011 Survey provides useful insight into the commercialization record of SBIR companies at NASA, on a number of dimensions. The data confirm that a substantial percentage of projects do indeed commercialize through sales of products or services and/or advance toward commercialization through the receipt of additional development funding.
Forty-six percent of respondents indicated that their company had already recorded sales of products or services derived from the awarded project. A further 26 percent of respondents were expecting sales in the future (a substantial increase from data collected in the 2005 Survey). (NASA does not have independent data against which the validity of the survey responses can be cross-checked.)
In light of the highly specialized fields in which NASA companies
__________________
20 2011 Survey, Question 12.
operate, which in many cases means that the available market is small, it is not surprising that the scale of commercialization is limited. About two-thirds of respondents reporting project-related sales indicated total sales of $1 million or less. Only 1 percent of respondents reported sales of more than $10 million, and zero percent reported sales of $20 million or more.
SBIR commercialization is also in part associated with take-up by NASA. An average of 19 percent of project sales reported by 2011 Survey respondents were to NASA or NASA primes. This percentage is slightly lower than the average percentage of sales to DoD or DoD primes (24 percent), suggesting strong crossover between the NASA and DoD markets sectors. The 19 percent figure is also considerably lower than the average percentage of sales to the domestic private sector (35 percent). These data reflect a tension within the NASA SBIR program, whose design and operations primarily aim to service the needs of NASA Mission Directorates, which compels NASA small business contractors to sell elsewhere to achieve sufficient scale for financial viability.21 (The report did not ascertain which NASA SBIR topics/subtopics/technologies were linked to higher and lower rates of commercialization within and without NASA, though such analysis would be useful in making program adjustments.)
Additional investment is another important metric for commercialization. Many Phase II projects are not yet ready for the marketplace at the end of the award period, especially at NASA (like DoD), where careful technology readiness assessment must occur and specific levels of readiness must be achieved before interest emerges from acquisitions groups. Sixty-five percent of survey respondents reported that the project received additional investment, mostly for amounts of less than $1 million.
Of the subset of respondents that received additional investment, 71 percent of the average (mean) additional investment of $1.5 million came from federal non-SBIR sources. Other sources individually accounted for less than 10 percent of additional investment, and U.S. venture funding accounted for about 2 percent. The percentages for funding from own company and personal resources were substantially lower than for the 2005 Survey.
QUANTITATIVE SURVEY EVIDENCE THAT NASA STIMULATED TECHNOLOGICAL INNOVATION
Did the NASA SBIR program stimulate technological innovation? Although the committee was unable to develop an appropriate control group to analyze against SBIR awardees, it is at least possible to examine—as have previous Academies and GAO surveys—what the company believed might have happened absent SBIR funding. This counterfactual question presented in the
__________________
21 As with previous surveys, respondents reported whether the funded project was currently in use by a Federal System or Acquisition Program. Twenty-one of respondents reported this to be the case.2011 Survey, Question 57.
opening to this chapter is of course subjective and hypothetical, but the company is best placed to provide some answers.
Given the small commercial market for NASA-specific projects, it is not surprising that only 7 percent of respondents believed that the project would definitely or possibly have proceeded without the SBIR award. Conversely, the majority—three-quarters—of respondents said that the project would likely or definitely not have proceeded without the SBIR award.22 The remainder were uncertain.
This section of the chapter examines a number of measures to examine how the NASA SBIR program has stimulated technological innovation—first examining knowledge outcomes such as patents and then returning the broader topic of fostering innovative companies.
Knowledge Outcomes
Although patents and peer-reviewed papers are not the only metrics of knowledge development and dissemination by small high-tech companies, they offer a useful starting point. Table 5-10 shows the overall number of patents that responding companies reported as being related to any SBIR awards they have received (not just NASA SBIR awards).23 Over three-quarters of respondent companies received at least one such patent, and 14 percent received 10 or more.
Patenting activity is often used to measure innovation, but it does not capture the entire story: patenting is important, but it is also expensive, and SBIR funds cannot legally be used for the purpose of patent filing or patent defense. Many companies interviewed for this report and others in this series indicated that they preferred to keep their technology secret or to rely on first-mover advantages and other market-based leverage to defend their technologies. Finally, patenting matters most when potential competitors may seek to utilize a company’s intellectual property (IP); for many NASA companies, their work and their market is so small and so specialized that this threat is limited.
Standard Intellectual Property (IP) metrics and bibliometrics provide at least a starting point for quantitative analysis of knowledge outcomes. The survey addressed patents, trademarks, copyrights, and peer-reviewed papers.24
Patents are to some degree the life blood of high-tech firms. Because patents at small companies often result from multiple contracts in multiple
__________________
22 2011 Survey, Question 24.
23 Non-responding companies may have patents that are unrelated to any SBIR award(s) that they have received.
24 The values of these knowledge repositories vary. Any unique item, painting, photo, music score, can be copy-written for a modest fee. Trademarks include more processing, as registered trademarks need to be unique in their field so as not to impinge on another prior trademark’s domain. A patent can be valuable IP, and patents have been correlated with prosperity. Refereed journal articles as a metric are generally not as valued outside of academia, where there is no tenure track requirement to publish such articles. The university professors who participate in SBIR may be responsible for production of many of the articles, although case studies have shown that company researchers also publish.
projects, it is important to capture patents related to the surveyed project as well as patents more generally attributable to SBIR-funded research. Seventy-six percent of responding companies reported the award of at least one patent by the time of the 2011 Survey related to any SBIR-funded technology; and 14 percent reported at least 10 related patents.
With regard to IP related to the specific award being surveyed. Table 5-11 shows that about 45 percent of respondents that responded to the relevant survey question reported receipt of at least one patent specifically related to the surveyed award, compared to 20 percent in the 2005 Survey. This result reflects a high degree of technical success, given the cost of patent applications (which provides disincentives to filing) as well as the burden of novelty that a successful application must meet. Four percent of respondents reported receipt of more than three patents specifically related to the surveyed award, compared to zero in the 2005 Survey.
TABLE 5-10 Number of Patents per Company Related to All Company SBIR Awards, Reported by 2011 Survey Responding Companies
| Number of Patents Resulting, at Least in Part, from the Company’s SBIR Awards | Percentage of Responding Companies | Number of Responding Companies |
|
0 |
24 |
15 |
|
1 or 2 |
39 |
25 |
|
3 or 4 |
13 |
8 |
|
5 to 9 |
11 |
7 |
|
10 or more |
14 |
9 |
| Total | 100 | 64 |
| At least 1 | 76 | 49 |
NOTE: In cases where company information, as opposed to individual projection information, was collected, multiple responses from the same company were averaged.
SOURCE: 2011 Survey, Question 11.
TABLE 5-11 Patents Awarded Related to Surveyed Project, Reported by 2011 Survey Respondents
| Patents Submitted for Technology Developed as a Result of the Project | Percentage of Respondents | Number of Respondents |
|
0 |
55 |
58 |
|
1 |
25 |
26 |
|
2 |
11 |
12 |
|
3 |
5 |
5 |
|
More than 3 |
4 |
4 |
| Total | 100 | 105 |
| At least 1 | 45 | 47 |
SOURCE: 2011 Survey, Question 39.1.2.
Copyrights and Trademarks
Eleven of the 13 respondent companies who applied for copyrights related to the surveyed project received at least one copyright.25 The survey revealed a similarly limited interest in project-related trademarks, with only 12 respondent companies (less than 10 percent) having applied for at least one. Similar percentages were reported for receipt of trademarks.26
Peer-reviewed Publications
Publication in peer-reviewed journals and conference proceedings are a standard method for disseminating scientific knowledge. As with the first-round Academies assessment, several case study interviewees noted that publication in peer-reviewed journals was an essential part of the firm’s work.
For the purposes of this assessment, peer-reviewed publications are important for two reasons:
- They validate the quality of the research being conducted with program funds.
- They are the primary mechanism through which knowledge is transmitted within the scientific community.
Eighty-two percent of respondents that responded to the relevant survey question indicated that an employee of the surveyed company had published at least one related scientific paper. This was a significantly larger percentage than in the 2005 Survey results (40 percent). Almost one-third reported publication of more than three related papers (see Table 5-12). The existence of articles based
TABLE 5-12 Peer-Reviewed Scientific Publications Related to the Surveyed Project, Reported by 2011 Survey Respondents
| Number of Scientific Publications Published for the Technology Developed as a Result of the Project | Percentage of Respondents | Number of Respondents |
|
0 |
18 |
23 |
|
1 |
20 |
26 |
|
2 |
19 |
24 |
|
3 |
12 |
16 |
|
More than 3 |
31 |
40 |
| Total | 100 | 129 |
| At least 1 | 82 | 106 |
SOURCE: 2011 Survey, Question 39.4.1.
__________________
on SBIR projects is direct evidence that the results of these projects are being disseminated widely, which in turn means that the Congressional mandate to support the creation and dissemination of scientific knowledge is being met.
Links to Universities
The survey asked a number of questions about the use of university staff and facilities on the surveyed project. Overall, nearly one-third of respondents reported a university connection of some kind.27 The most reported types of linkages were a university or college as a subcontractor; a faculty member working on the project but not as a principal investigator (PI); and graduate students employed on the project (see Table 5-13). Box 5-2 describes a workshop that the committee convened to address a range of issues related to university-SBIR linkages. Linkages to university is an important component in examining evidence that NASA “stimulated technological innovation,” Goal 1 of the SBIR Program. University connections can also benefit SBCs by giving access to technical expertise.
__________________
27 2011 Survey, Question 60.
TABLE 5-13 Project Links to Universities, Reported by 2011 Survey Respondents
| Percentage of Respondents | Number of Respondents | |
|
A university or college was a subcontractor on this project |
21 |
37 |
|
Faculty member(s) or adjunct faculty member(s) worked on this project in a role other than PI |
15 |
26 |
|
Graduate students worked on this project |
14 |
25 |
|
The technology for this project was originally developed at a university or college by one of the participants in this project |
5 |
8 |
|
The PI for this project was at the time of the project an adjunct faculty member |
2 |
4 |
|
The technology for this project was licensed from a university or college |
2 |
3 |
|
The PI for this project was at the time of the project a faculty member |
1 |
1 |
|
Any of the above |
31 |
55 |
| None of the above | 69 | 122 |
NOTE: Number of Respondents = 177. Respondents could select more than one answer.
SOURCE: 2011 Survey, Question 60.
Respondents were also asked to identify the universities with which they worked in various capacities on this project. Although the type of work varied widely, some universities were mentioned by a number of respondents. Overall, 77 different universities and colleges were identified. Those mentioned by two or more respondents are listed in Table 5-14 (see Appendix D for the complete list of university mentions). Some of the names on this list are large state universities, a number of which have in recent years focused on technology transition as well as basic research. We believe these data provide a preliminary indication of the connections between specific universities, university systems, and the NASA SBIR program.
Finally it is worth observing that 63 percent of companies that responded to 2011 Survey question 4 reported that at least one founder had an academic background (see Table 5-15), and 29 percent of companies that responded to 2011 Survey question 5 reported that at least one founder was most recently employed by a college or university (see Table 5-16).
Fostering Innovative Companies
Technological innovation can be stimulated by fostering innovative companies. SBIR programs have a range of effects on companies that affect their ability to work within the innovation ecology of the agency or indeed more
TABLE 5-14 University Participants Mentioned by Two or More 2011 Survey Respondents
| College or University Name | Number of Mentions |
|
Pennsylvania State University |
4 |
|
Purdue University |
4 |
|
Stanford University |
4 |
|
University of California, Berkeley |
4 |
|
Georgia Institute of Technology |
3 |
|
Massachusetts Institute of Technology |
3 |
|
The Ohio State University |
3 |
|
Rutgers University |
3 |
|
University of Alabama |
3 |
|
University of Central Florida |
3 |
|
University of Houston |
3 |
|
University of Illinois at Urbana-Champaign |
3 |
|
Dartmouth College |
2 |
|
Rensselaer Polytechnic Institute |
2 |
|
Texas A&M University |
2 |
|
University of Arizona |
2 |
|
University of Colorado |
2 |
|
University of Florida |
2 |
|
University of Maryland |
2 |
|
University of Minnesota |
2 |
|
University of Notre Dame |
2 |
|
Virginia Polytechnic Institute |
2 |
SOURCE: 2011 Survey, Question 60a.
TABLE 5-15 Number of Academic Founders, Reported by 2011 Survey Responding Companies
| Number of Academic Founders | Percentage of Responding Companies | Number of Responding Companies |
|
None |
37 |
27 |
|
1 |
42 |
30 |
|
2 |
11 |
8 |
|
3 |
6 |
4 |
|
4 |
3 |
2 |
|
5 or more |
1 |
1 |
| Total | 100 | 72 |
| At least 1 | 63 | 45 |
NOTE: In cases where company information, as opposed to individual projection information, was collected, multiple responses from the same company were averaged.
SOURCE: 2011 Survey, Question 4.
generally. In addition, data about companies can help to define the technological space in which the SBIR programs operate. Finally, a review of the SBIR share of overall company activities can provide insights into the degree of dependence on the SBIR program for individual companies.
SBIR Share of R&D Effort and Company Revenues
Respondents estimated how much of their company’s total R&D effort (defined as man-hours of work for scientists and engineers) was devoted to SBIR-funded projects. Overall, 40 percent of companies that responded to the relevant survey question indicated that 10 percent or less of total effort was devoted to SBIR activities during the most recent fiscal year (at the time of the survey), and 28 percent indicated more than one-half (see Table 5-17).
TABLE 5-16 Most Recent Founder Employment, Reported by 2011 Survey Responding Companies
| Most Recent Founder Employment | Percentage of Responding Companies | Number of Responding Companies |
|
Other private company |
69 |
50 |
|
College or University |
29 |
21 |
|
Government |
9 |
6 |
|
Other |
11 |
8 |
NOTE: Number of Responding Companies = 72. In cases where company information, as opposed to individual projection information, was collected, multiple responses from the same company were averaged. Percentages do not sum to 100 percent because respondents could select more than one answer.
SOURCE: 2011 Survey, Question 5.
TABLE 5-17 Percentage of R&D Effort Funded by SBIR for Most Recent Fiscal Year (Reported at Time of 2011 Survey by Responding Companies)
| Percentage of S&E Hours on SBIR | Percentage of Responding Companies | Number of Responding Companies |
|
0 |
25 |
18 |
|
1-10 |
15 |
11 |
|
11-25 |
13 |
9 |
|
26-50 |
21 |
14 |
|
51-75 |
13 |
9 |
|
76-100 |
15 |
11 |
| Total | 100 | 72 |
NOTE: In cases where company information, as opposed to individual projection information, was collected, multiple responses from the same company were averaged.
SOURCE: 2011 Survey, Question 7.
These data correspond fairly closely to responses from another survey question, which (again from the perspective of the time of survey) asked what percentage of company revenues during the most recent fiscal year was accounted for by SBIR awards. As Table 5-18 indicates, 28 percent of companies that responded to the relevant survey question reported having zero SBIR revenues for the most recent fiscal year, while nearly one-quarter reported receiving more than 50 percent of revenues from SBIR. The data are consistent with some companies using SBIR awards to launch and then switching to other sources of funding, and with some companies either continuing to use SBIR awards or returning to seek SBIR awards as a means of engaging with NASA and its technology needs; however, they do not give any indication of how these data may have changed over the life of the responding company.28
Prior SBIR Awards
Although the idealized view of the SBIR program is that ideas are tested in Phase I, prototyped in Phase II, and commercialized in Phase III, real-world efforts typically require multiple iterations. Often projects must restart with an earlier phase, or multiple efforts are needed to meet specific problems.
The 2011 Survey asked respondents to indicate how many prior SBIR Phase I awards they had received that were related to the project and technology
TABLE 5-18 Percentage of Company Revenues from SBIR for Most Recent Fiscal Year (Reported at Time of 2011 Survey by Responding Companies
| Percent of Company Revenues from SBIR at the Time of the Survey | Percentage of Responding Companies | Number of Responding Companies |
|
0 |
28 |
20 |
|
1-10 |
15 |
11 |
|
11-25 |
14 |
10 |
|
26-50 |
21 |
15 |
|
51-75 |
10 |
7 |
|
76-100 |
13 |
9 |
| Total | 100 | 72 |
NOTE: In cases where company information, as opposed to individual projection information, was collected, multiple responses from the same company were averaged. Because survey sample includes inactive awards, some respondents reported zero SBIR revenues for the most recent fiscal year.
SOURCE: 2011 Survey, Question 9.
__________________
28 Future research of the program could measure outcomes against firm type (those formed in response to an SBIR award solicitation and established firms) and explore more deeply the strategic uses of SBR by firms.
TABLE 5-19 Prior SBIR or STTR Phase I Awards Related to the Surveyed Project, Reported by 2011 Survey Respondents
| Number of Awards | Percentage of Respondents | Number of Respondents |
|
0 |
21 |
33 |
|
1 |
41 |
63 |
|
2 |
19 |
29 |
|
3 |
10 |
15 |
|
4 |
5 |
7 |
|
5 or more |
5 |
7 |
| Total | 100 | 154 |
|
1 or more |
80 |
SOURCE: 2011 Survey, Question 40.1.1 and Question 40.1.2.
being surveyed. Table 5-19 shows that more than three-quarters of respondents reported prior Phase I awards related to the project surveyed, and more than a third reported more than 1 prior Phase I awards. These data strongly supports the view that innovative products emerge from clusters of activity, rather than from simple straight line development from Phase I to Phase II to commercialization.
Long-term Impacts on Companies Receiving SBIR Awards
Although SBIR awards have direct effects on specific projects, they can have a longer-term effect on the trajectory of company development, creating capacity and in some cases providing a critical input that transforms long-term outcomes. The survey asked respondents about this directly. The results for those companies that responded to the relevant survey question are summarized in Table 5-20.
Respondents to the survey report an overwhelmingly positive impact. Twenty-five percent of respondents indicated that SBIR had had a transformative effect on their company, and a further 56 percent reported a strongly positive effect. Only one percent of respondents reported negative long-term effects.
Conclusions: Stimulating Technological Innovation
What emerges from these data is a picture of companies that are dynamic centers of technological innovation, a considerable amount of which is protected through the patent system. Seventy-six percent of responding companies reported that their company had received at least one patent based on its work under SBIR contracts, while 45 percent reported at least one patent related to the surveyed project only.
TABLE 5-20 Long-term Impacts of the SBIR Funding on Recipient Companies, Reported by 2011 Survey Responding Companies
| Percentage of Responding Companies | Number of Responding Companies | |
|
Had a transformative effect |
25 |
20 |
|
Had a substantial positive long-term effect |
56 |
46 |
|
Had a small positive effect |
15 |
12 |
|
Had no long-term effect |
2 |
2 |
|
Had a negative long-term effect |
1 |
1 |
| Total | 100 |
81 |
NOTE: In cases where company information, as opposed to individual projection information, was collected, multiple responses from the same company were averaged.
SOURCE: 2011 Survey, Question 45.
SBIR companies participate at a high level in the standard form of technical knowledge dissemination: publishing in peer-reviewed journals. Eighty-two percent of respondents reported publication of at least one article based on the SBIR-funded work, and almost one-third reported publication of more than three such papers.
Finally, some SBIR companies are closely connected to the universities. Nearly one-third of respondents reported a university connection on the surveyed project, across a number of different modalities, and 12 universities were specifically mentioned as playing a role in at least three reported projects. This suggests that SBIR plays an important role in supporting the practical implementation of university research.
Data from companies responding to the 2011 Survey provide evidence regarding the program’s effects on fostering innovative companies. Forty percent of respondents indicated that 10 percent or less of total company R&D effort was devoted to SBIR activities in the most recent fiscal year, and 28 percent of responding companies indicated that zero percent of company revenues came from SBIR for the most recent fiscal year. Still, more than three-quarters of survey respondents reported prior Phase I awards related to the project surveyed, and more than a third reported more than one prior Phase I awards. Together these data suggest that innovative products emerge from clusters of activity, and that some companies receiving SBIR awards are using them to launch their firms before moving on to other sources of funding.
