|
||||||||||||||||
|
||||||||||||||||
The following HTML text is provided to enhance online readability. Many aspects of typography translate only awkwardly to HTML. Please use the page image as the authoritative form to ensure accuracy. 4
|
||||||||||||||||
 |
 |
Page 55 |
 |
 |
 |
|
| ||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 55
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
4
SBIR Program Outcomes
4.1
INTRODUCTION
As the SBIR program approached its twentieth year of operation, the U.S. Congress asked the National Research Council to conduct a “comprehensive study of how the SBIR program has stimulated technological innovation and used small businesses to meet federal research and development needs” and to make recommendations on still further improvements to the program.1
The Small Business Innovation Act, sets out four goals for the program: “(1) to stimulate technological innovation; (2) to use small business to meet federal research and development needs; (3) to foster and encourage participation by minority and disadvantaged persons in technological innovation; and (4) to increase private-sector commercialization derived from Federal research and development.”2
The legislation does not set priorities among these four objectives. However, discussions with congressional staff suggest that commercialization has become increasingly important to Congress. Still, it remains important to assess each of the four objectives; each should be taken as equally important in evaluating the achievements and challenges of the SBIR program. These four objectives help to define the structure and content of this chapter.
Assessing program outcomes against these four objectives entails numerous
1
See the SBIR Reauthorization Act of 2000 (H.R. 5667—Section 108).
2
The Small Business Innovation Development Act (PL 97-219).
OCR for page 56
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
methodological challenges. These challenges, discussed in detail in the National Research Council’s Methodology Report, are briefly reviewed below.3
4.1.1
Compared to What?
Assessment usually involves comparison—comparing programs and activities, in this case. Three kinds of comparison seem possible: with other programs at each agency, among SBIR programs at the various agencies, and with early-stage technology development funding in the private sector, such as venture capital activities. Yet, as we see below, the utility of each of these three types of comparison is limited.
Comparison with Other NASA Programs
Within NASA, no other program is dedicated to support innovative small businesses. This fundamental difference in objectives makes it difficult to compare the NASA SBIR program with other programs at the agency.
Comparison with Other SBIR Programs
Comparing the NASA SBIR program with those at other agencies is superficially more useful. However, as discussed in Chapter 1 of this volume, the SBIR programs at each of the agencies are shaped by the different agency missions. This, in turn, is reflected in the different mechanisms and approaches taken by the agencies. Agencies whose mission is to develop technologies for internal agency use via procurement—notably DoD and NASA—have a different orientation from agencies that do not procure technology and are instead focused on developing technologies for use outside the agency.
There are important differences between the two “procurement” agencies. At DoD, once an SBIR technology is proven, there are opportunities for integration of that technology into a very substantial stream of acquisitions dollars. At NASA, such proven technologies may also be taken up for use by the agency—often only for one or two copies of a technology, for use on NASA space missions. Thus, the character of commercialization is quite different.
3
National Research Council, An Assessment of the Small Business Innovation Research Program—Project Methodology, Washington, DC: The National Academies Press, pp. 20-21, 2004. For a broader discussion of the scope and limitations of surveys by the University of Michigan Survey Research Center, see Robert M. Groves, Floyd J. Fowler, Jr., Mick P. Couper, James M. Lepkowski, Eleanor Singer, and Roger Tourangeau, Survey Methodology, Boston, MA: WileyBlackwell, 2004.
OCR for page 57
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
Comparison with Early-stage Venture Capital
Finally, SBIR might be compared with venture capital (VC) activities, but there are important differences. VC funding is typically supplied later in the development cycle when innovations are in, or close, to market. Indeed, most venture investments are made with the expectation of an exit from the company within three years. VC investments are also typically larger than SBIR awards. In 2007, the median investment made by VC firms in a company was $7.6 million, compared to less than $1 million for a NASA SBIR over a two to three year cycle.4 VC investments are also focused on companies, not projects, and often come both with substantial management support and influence (such as through seats on the company’s board).
4.1.2
Multiple Metrics
The lack of direct comparators means that multiple metrics must be deployed, using a wide array of information sources.5 This is what the NRC Committee has done:
The NRC Phase II Survey covers every firm that received a Phase II award between 1992 and 2001 inclusive.
The NRC Phase I Survey covers projects that failed to proceed beyond Phase I.
The NRC Project Manager Survey.
Case Studies commissioned by the NRC Committee provide context and illustration, in addition to user perspectives of the program.
Interviews with agency staff both within and outside the SBIR program office, as well as other experts inform the Committee’s findings.
NASA Databases, in particular the NASA awards database, as well as a NASA outcomes assessment, have provided basic information about the program.6
While the surveys broke important ground, and provided a central base of information on which considerable parts of the assessment are based, it is important to note that surveys of innovation awards can suffer from several forms of survey bias. These issues are discussed in Box 4-1.
4
2007 saw some 2,648 deals, with an overall capital investment of $29.9 billion, according to data from National Venture Capital Association. See <https://www.pwcmoneytree.com/MTPublic/ns/index.jsp>.
5
For a more detailed discussion of the methodology, see National Research Council, An Assessment of the Small Business Innovation Research Program—Project Methodology, op. cit.
6
Access the NASA Commercialization Metrics Survey at <http://www.sbir.nasa.gov/SBIR/survey.html>.
OCR for page 58
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
BOX 4-1
Multiple Sources of Bias in Survey Response
Large innovation surveys involve multiple sources of bias that can skew the results in both directions. Some common survey biases are noted below. These biases were tested for and responded to in the NRC surveys.a
Successful and more recently funded firms are more likely to respond. Research by Link and Scott demonstrates that the probability of obtaining research project information by survey decreases for less recently funded projects and it increased the greater the award amount.b Nearly 40 percent of respondents in the NRC Phase II Survey began Phase I efforts after 1998, partly because the number of Phase I awards increased, starting in the mid 1990s, and partly because winners from more distant years are harder to reach. They are harder to reach as time goes on because small businesses regularly cease operations, are acquired, merge, or lose staff with knowledge of SBIR awards.
Success is self-reported. Self-reporting can be a source of bias, although the dimensions and direction of that bias are not necessarily clear. In any case, policy analysis has a long history of relying on self-reported performance measures to represent market-based performance measures. Participants in such retrospectively analyses are believed to be able to consider a broader set of allocation options, thus making the evaluation more realistic than data based on third-party observation.c In short, company founders and/or principal investigators are in many cases simply the best source of information available.
Survey sampled projects at firms with multiple awards. Projects from firms with multiple awards were underrepresented in the sample, because they could not be expected to complete a questionnaire for each of dozens or even hundreds of awards.
Failed firms are difficult to contact. Survey experts point to an “asymmetry” in their ability to include failed firms for follow-up surveys in cases where the firms no longer exist.d It is worth noting that one cannot necessarily infer that the SBIR project failed; what is known is only that the firm no longer exists.
Not all successful projects are captured. For similar reasons, the NRC Phase II Survey could not include ongoing results from successful projects in firms that merged or were acquired before and/or after commercialization of the project’s technology. The survey also did not capture projects of firms that did not respond to the NRC invitation to participate in the assessment.
Some firms may not want to acknowledge the full SBIR contribution to a project’s success. Some firms may be unwilling to acknowledge that they received important benefits from participating in public programs for a variety of reasons. For example, some may understandably attribute success exclusively to their own efforts.
Commercialization lag. While the NRC Phase II Survey broke new ground in data collection, the amount of sales made—and indeed the number of projects that generate sales—are inevitably undercounted in a snapshot survey taken at a single point in time. Based on successive data sets collected from NIH SBIR award recipients, it is estimated that total sales from all responding projects will likely be on the order of 50
OCR for page 59
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
percent greater than can be captured in a single survey.e This underscores the importance of follow-on research based on the now-established survey methodology.
FIGURE B-4-1 Survey bias due to commercialization lag.
These sources of bias provide a context for understanding the response rates to the NRC Phase I and Phase II Surveys conducted for this study. For the NRC Phase II Survey, of the 534 NASA firms that could be contacted out of a sample size of 779, 181 responded, representing a 34 percent response rate. The NRC Phase I Survey captured 9 percent of the 3,363 awards made by NASA over the period of 1992 to 2001. See Appendixes B and C for additional information on the surveys.
aFor a technical explanation of the sample approaches and issues related to the NRC surveys, see Appendixes B and C.
bAlbert N. Link and John T. Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Program’s Intramural Research Initiative, London: Routledge, 2005.
cWhile economic theory is formulated on what is called “revealed preferences,” meaning individuals and firms reveal how they value scarce resources by how they allocate those resources within a market framework, quite often expressed preferences are a better source of information especially from an evaluation perspective. Strict adherence to a revealed preference paradigm could lead to misguided policy conclusions because the paradigm assumes that all policy choices are known and understood at the time that an individual or firm reveals its preferences and that all relevant markets for such preferences are operational. See Gregory G. Dess and Donald W. Beard, “Dimensions of Organizational Task Environments.” Administrative Science Quarterly 29:52-73, 1984; and Albert N. Link and John T. Scott, Public Accountability: Evaluating Technology-Based Institutions, Norwell, MA: Kluwer Academic Publishers, 1998.
dAlbert N. Link and John T. Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Program’s Intramural Research Initiative, op. cit.
eData from NIH indicates that a subsequent survey taken two years later would reveal very substantial increases in both the percentage of firms reaching the market, and in the amount of sales per project. See National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009.
OCR for page 60
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
4.1.3
NASA’s Changing Program Priorities
NASA’s mission objectives are defined in very specific terms—placing a man on Mars and returning him safely, for example. These mission objectives require the development and acquisition of highly specialized technologies, often with limited commercial applications.7 There is, for example, little need for Marshardened technologies on Earth.
To address NASA’s unique needs, subtopic and selection decisions for the NASA SBIR program were managed by those with a primary interest in using SBIR technologies for NASA missions. Seeking, at the same time to improve the commercialization outcomes of its SBIR program, NASA also provided support for technology incubators.
In time, NASA recognized that this compromise was not successful. On one hand, the NASA’s mission realities limited the commercial potential of NASA SBIR projects. On the other hand, the focus on commercialization limited a tighter linkage between SBIR projects and agency mission objectives.
Recognizing this challenge, NASA’s senior management shifted the program in 2006 away from this uneasy balance between commercialization and mission support objectives, and to a new commitment focused almost exclusively on mission support.8 As a result of this reorganization, commercialization has become a secondary objective for the agency’s SBIR program.
The reorganization changed in the structure of the program from one where key decisions were primarily made by centers and specific topic managers to one where those decisions are made at Headquarters by Mission Directorate staff, with input from the centers and project managers. With authority for strategic planning, NASA’s Mission Directorate staff were seen as best placed to identify the technology areas with greatest needs, and hence to set the appropriate priorities for the SBIR program. By moving SBIR from the periphery of NASA planning to become a potentially valuable solution to current mission challenges, the reorganization is designed to ensure that SBIR is used to the maximum extent possible to help develop technologies needed by NASA.
4.2
COMMERCIALIZATION: A LONG-TIME PROGRAM PRIORITY
Commercialization of SBIR-funded technologies has been a key congressional objective for the SBIR program since its inception. In fact, the program’s initiation in the early 1980s in part reflected a concern that American investment in research was not adequately deployed to the nation’s competitive advantage.
7
Unlike DoD, NASA does not itself constitute a major market for commercial sales. And at the same time, the technologies developed specifically for NASA have relatively little relevance to the private marketplace—for example, compared to some applications developed at NIH.
8
These administrative changes, made in 2006, are described in Chapter 5 (Program Management).
OCR for page 61
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
Directing a portion of federal investment in R&D to small businesses was thus seen as a new means of meeting the mission needs of federal agencies while increasing the participation of small business and thereby the proportion of innovation that would be commercially relevant.9
Congressional and Executive branch interest in the commercialization of SBIR research has increased over the life of the program. Drawing from a 1992 GAO study10 that focused on commercialization, the 1992 reauthorization specifically “emphasize[d] the program’s goal of increasing private-sector commercialization of technology developed through Federal research and development”11 and noted the need to “emphasize the program’s goal of increasing private-sector commercialization of technology developed through Federal research and development.” The 1992 reauthorization also changed the order in which the program’s objectives are described, moving commercialization to the top of the list.12
The term “commercialization” is subject to widely varying interpretations. Several agencies have taken it to mean “first sale”—that is, the first sale of a product in the market place, whether to public- or private-sector clients. This definition, however, misses significant components of commercialization that do not result in a discrete sale. It also fails to provide any guidance on how to evaluate the scale of commercialization, an important element in assessing the degree to which SBIR programs successfully encourage commercialization. The NRC methodology has determined that multiple metrics can and should be used to assess the extent of commercialization.
The following sections review commercialization outcomes for NASA through 2005. consequently, the recent change in emphasis to mission support is not addressed.
9
A growing body of evidence, starting in the late 1970s and accelerating in the 1980s indicates that small businesses were assuming an increasingly important role in both innovation and job creation. See, for example, J. O. Flender and R. S. Morse, The Role of New Technical Enterprise in the U.S. Economy, Cambridge, MA: MIT Development Foundation, 1975, and David L. Birch, “Who Creates Jobs?” The Public Interest, 65:3-14, 1981. Evidence about the role of small businesses in the U.S. economy gained new credibility with the empirical analysis by Zoltan Acs and David Audretsch of the U.S. Small Business Innovation Data Base, which confirmed the increased importance of small firms in generating technological innovations and their growing contribution to the U.S. economy. See Zoltan Acs and David Audretsch, “Innovation in Large and Small Firms: An Empirical Analysis,” The American Economic Review, 78(4):678-690, September 1988. See also Zoltan Acs and David Audretsch, Innovation and Small Firms, Cambridge, MA: MIT Press, 1990.
10
U.S. General Accounting Office, Small Business Innovation Research Program Shows Success But Can Be Strengthened, GAO/RCED–92–37, Washington, DC: U.S. General Accounting Office, 1992.
11
PL 102-564, October, 28, 1992.
12
These changes are described by R. Archibald and D. Finifter in “Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach” in National Research Council, SBIR: An Assessment of the Department of Defense Fast Track Initiative, Charles W. Wessner, ed., Washington, DC: National Academy Press, 2000.
OCR for page 62
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
4.2.1
Assessing Commercialization
Clear metrics for assessing commercialization are elusive.13 It is not possible to quantify in full all commercialization from a given research project, for the reasons listed below.
Numerous additional development steps are often needed after the research has been concluded. Thus, a single, direct line between research inputs and commercial outputs rarely exists in practice; cutting-edge research is only one contribution among many leading to a successful commercial product.
Markets themselves have major imperfections, often caused by information asymmetries. Hence high quality—even path-breaking—research does not always result in commensurate commercial returns.
There are often long lags between an early-stage research project and an eventual commercial product. This means that for a significant number of the more recent SBIR projects, commercialization is still in process, and sales—often substantial sales—will be made in the future (see the “snapshot effect” discussed in Box 4-2).
Research rarely results in stand-alone products. Often, the output from an SBIR project is combined with other technologies. The SBIR technology may provide a critical element in developing a winning solution, but that commercial impact is hard to measure in simple dollars.
In some cases, the full value of an “enabling technology” that can be used across industries is difficult to capture.
All this is to say that commercialization results must be viewed with caution. Our ability to track them is limited. Indeed, it appears highly likely that quantification of research awards through surveys substantially understates the true commercial impact of SBIR projects. In addition, a specific award cannot lay claim to all subsequent commercial successes, even though the technology developed with the award may have contributed to many significant outcomes.14
4.2.2
Commercialization Indicators and Benchmarks
This report uses four sets of indicators to assess commercialization success quantitatively:
13
See National Research Council, An Assessment of the Small Business Innovation Research Program—Project Methodology, op. cit.
14
Data on infusion/commercialization of SBIR technologies from 1983 to 1996 was included in NASA’s 2002 Commercial Metrics publication. Follow-on data gathering via this Commercial Technology Division initiative has been halted pending further funding. Of NASA’s 1,739 SBIR Phase II awards during the target period, about 15 percent developed technologies used in NASA or other programs via Phase III funding, and 31 percent commercialized in the private sector. See Commercial Metrics; NASA Commercial Technology Division; October, 2002; Fig 4.
OCR for page 63
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
Sales and licensing revenues (“sales” hereafter, unless otherwise noted). Revenues flowing to a company from the commercial marketplace and/or through government procurement constitute the most obvious measure of commercial success. They are also an important indicator of uptake for the product or service. Sales indicate that the result of a project has been sufficiently positive to convince buyers that the product or service is the best available solution.
Yet if there is general agreement that sales are a key benchmark, there is no such agreement on what constitutes “success.” Companies, naturally enough, focus on projects that contribute to the bottom line—that are profitable. Agency staff provide a much wider range of views. Some view any sales a substantial success for a program focused on such an early stage of the product and development cycle, while others seem more ambitious.15 Some senior executives in the private sector view only projects that generated cumulative revenues at $100 million or more as a complete commercial success.16
Rather than seeking to identify a single sales benchmark for “success,” it therefore seems more sensible to assess outcomes against a range of benchmarks reflecting these diverse views, with each marking the transition to a greater level of commercial success.
Phase III activities within NASA. As noted above, Phase III activities within NASA are a primary form of commercialization for NASA SBIR projects. These activities are considered in the mission support section (Section 4.3).
R&D investments and research contracts. Further R&D investments and contracts are good evidence that the project has been successful in some significant sense. These investments and contracts may include partnerships, further grants and awards, or government contracts.
Sale of equity. This is a clear-cut indicator of commercial success or market expectations of value. Key metrics include:
Equity investment in the company by independent third party.
Sale or merger of the entire company.
Sales and Licensing Revenues
A basic question on commercialization is whether results from a project have reached the marketplace. The NRC Phase II Survey17 indicates that about
15
Interviews with SBIR program coordinators at DoD, NIH, NSF, and DoE.
16
Pete Linsert, CEO, Martek Biosciences, Inc., NRC Committee Meeting, June 5, 2005.
17
Much of the primary data in this section of the report was derived from the NRC Phase II Survey. The NRC Phase II Survey of projects provides recent evidence on the extent by which NASA SBIR award recipients have achieved commercialization and/or progress toward commercialization. The survey provides information on sales, modes of commercialization, and on steps important to achieving commercialization, including marketing activities, interactions with other companies and
OCR for page 64
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
46 percent of NASA respondents had generated some sales or licensed their technology, and that a further 14 percent still expected sales though they had none at the time of the survey. In addition, 3 percent were still in the research stage of the project.18 (See Figure 4-1.)
These results are broadly in line with other sources of information on commercial outcomes from SBIR program, including those at other agencies such as the DoD commercialization database, the NIH Phase II Survey, and the NSF Phase II survey.19
Distribution of Sales
Research on early-stage financing strongly suggests that a pronounced skew to the results is likely, and this turns out to be the case. Most projects that reach the market generate minimal revenues. A few awards generate substantial results, and a small number bring in large revenues.20
Of the 74 NASA SBIR Phase II projects reporting sales greater than $0, average sales per project were $1,154,156. About 40 percent of the total sales dollars were due to the two NASA projects responding to the survey that had received $5,000,000 or more in sales. The highest cumulative sales figure reported was $15,000,000.
investors, and attraction of funding from non-SBIR sources. It also provides information on employment effects, including the extent to which woman and minorities are involved in the projects as principal investigators. Finally, it explores the extent to which the reported effects are believed by survey respondents to be attributed to impacts of the SBIR program. See Appendix B for additional information about the NRC Phase II Survey, including response rates.
18
See Finding G on venture capital and SBIR in Chapter 2 of National Research Council, An Assessment of SBIR at the National Institutes of Health, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009.
19
See the NRC assessments of the SBIR program at DoD and NIH for discussion of these sources. National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit.; National Research Council, An Assessment of the SBIR Program at the National Science Foundation, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2008; National Research Council, An Assessment of the SBIR Program at the Department of Defense, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009.
20
See John H. Cochrane, “The Risk and Return of Venture Capital,” Journal of Financial Economics, 75(1):3-52, 2005. Drawing on the VentureOne database Cochrane plots a histogram of net venture capital returns on investments that “shows an extraordinary skewness of returns. Most returns are modest, but there is a long right tail of extraordinary good returns. 15 percent of the firms that go public or are acquired give a return greater than 1,000 percent! It is also interesting how many modest returns there are. About fifteen percent of returns are less than 0, and 35 percent are less than 100 percent. An IPO or acquisition is not a guarantee of a huge return. In fact, the modal or “most probable” outcome is about a 25 percent return.” See also Paul A. Gompers and Josh Lerner, “Risk and Reward in Private Equity Investments: The Challenge of Performance Assessment.” Journal of Private Equity, 1 (Winter 1977):5-12. Steven D. Carden and Olive Darragh, “A Halo for Angel Investors” The McKinsey Quarterly, 1, 2004 also show a similar skew in the distribution of returns for venture capital portfolios.
OCR for page 65
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
FIGURE 4-1 Results from NASA Phase II projects.
SOURCE: NRC Phase II Survey. Based on responses to Phase II Survey questions 1a, 1b, 3a, and 3b.
These figures appear lower than those for other agencies, notably DoD and NIH. However, direct comparisons of results from the NRC Phase II Survey are not valid because of survey response issues. And it should be noted that the very high degree of skew combined with smaller number of awards at NASA means that comparisons are likely to be even more inaccurate (NASA may simply not have made enough awards to generate a statistically significant number of big winners—firms with more than $10 million in sales—though it might be a matter of concern if current trends continued indefinitely). This distribution is reflected in Figure 4-2.
More than 80 percent of the projects reporting sales greater than zero had $1 million or less in sales, as seen in Figure 4-3.
The numerous projects with relatively low sales (below $1 million) are also in line with our understanding of commercialization within NASA itself. According to the SBIR liaison office at the Space Operations Mission Directorate, the average Phase II award at NASA is on the order of $500,000-600,000.21 This is of course sharply lower than those at DoD, and reflects the particular needs and objectives of NASA programs. As a result, however, a an SBIR project that was successful from NASA’s perspective—even one that resulted in technologies being adopted for a space flight mission—might well generate commercial returns of less than $1 million.
21
Interview with Jason Crusan, Program Integration Office, Space Operations Mission Directorate, NASA, December 7, 2007.
OCR for page 94
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
Start-up firm. This is a new firm, typically without marketable products, and usually with minimal funding and limited personnel resources.
R&D Contractor. As described by Reid Cramer, these firms make a strategic choice to specialize in the performance of R&D rather than in marketing products or services.46,47
Technology Firm. These firms have developed a core technology, which is then deployed into products and services.
Scientific firm. These businesses are described by Reid Cramer as “firms [that] are generally small and were founded by scientists to explore whether a particular research areas can generate ideas or products that might attract investors”48
Transformational firm. These companies start out as highly (or partially) dependent on SBIR or other government R&D contracts, which they use to develop a product that turns out to have considerable commercial value. This leads the company to become a production-oriented commercial vendor, with a concomitant decrease in the role of SBIR on the firm progression.
Examples of these firm types can be seen in the case studies of SBIR awardees. Further research in this area may help to establish better, how the NASA SBIR program supports these different kinds of businesses, and businesses at different stages of development.
Conclusions
SBIR supports small high technology businesses at a time when other sources of financial support are especially difficult to find. Businesses use these funds for a variety of purposes, in pursuit of several distinct strategies.
Awards data also indicate the role of woman- and minority-owned firms at NASA. It would be helpful if further analysis in this area focused on the role and incidence of minority and female Principal Investigators (PI), as these positions my be an important stepping stone on the path to forming the kinds of companies that can qualify for SBIR awards.
4.5
SBIR AND THE EXPANSION OF KNOWLEDGE
Quantitative metrics for assessing knowledge outputs from research programs are well-known, but far from comprehensive. Patents, peer-reviewed pub-
46
See for example Polymer case study in National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit. Polymer was in its early years primarily a contractor, but has since developed many cutting-edge products of its own.
47
See Cramer, Reid, “Patterns of Firm Participation in the Small Business Innovation Research Program in Southwestern and Mountain States,” in National Research Council, SBIR: An Assessment of the Department of Defense Fast Track Initiative, op. cit.
48
Ibid.
OCR for page 95
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
TABLE 4-18 Patents from NASA SBIR Projects
Number of Patents
Applied
Received
0
121
128
1
30
25
2
8
7
3
2
1
Total
161
161
SOURCE: NRC Phase II Survey, Question 18.
lications, and, to a lesser extent copyrights and trademarks, are all widely used metrics, and are discussed in detail below.
However, these metrics do not capture the entire transfer of knowledge involved in programs such as SBIR.
It is therefore quite important to understand that the quantitative metrics discussed below are only an indicator of the expansion of knowledge; they reflect that expansion but do not fully capture it. In particular, they say little about the impact of that knowledge.
4.5.1
Patents
According to the Small Business Administration, small businesses produce 13 to 14 times more patents per employee than large patenting firms. These patents are twice as likely as large firm patents to be among the one percent most cited.49
The data show that 40 projects—about 25 percent of respondents—reported at least one patent application, and that 33 projects (20 percent) generated at least one patent.
No projects generated more than three applications, and only one received three or more patents.
4.5.2
Scientific Publications
Publication in peer-reviewed journals and conference proceedings are a standard method for disseminating scientific knowledge. Several case study in-
49
Accessed on May 16, 2007, at <http://app1.sba.gov/faqs/faqindex.cfm?areaID=24>. Drawing on seminal empirical research, Acs and Audretsch found that small businesses have comparatively higher rates of innovation—specifically, that “the number of innovation increases with increased industry R&D expenditures but at a decreasing rate. Similarly, while the literature has found a somewhat ambiguous relationship between concentration and various measures of technical change, our results are unequivocal—industry innovation tends to decrease as the level of concentration rises.” See Zoltan J. Acs and David B. Audretsch, “Innovation in Large and Small Firms: An Empirical Analysis,” op. cit.
OCR for page 96
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
terviewees noted that publication in peer-reviewed journals was an essential part of the firm’s work, and provided valuable exposure.
The NRC Phase II Survey asked respondents about this aspect of these activities as well. (See Table 4-19 for a summary of the results.) About 40 percent of respondents reported at least one peer-reviewed publication. Five projects reported at least ten such publications.
These data fit well with case studies and interviews, which suggested that some SBIR companies are proud of the quality of their research. Publications are featured prominently on many company Web sites.
Publications therefore fill two important roles in the study of SBIR programs:
First, they provide an indication of the quality of the research being conducted with program funds. In this case, more than half of the funded projects were of sufficient value to generate at least one peer-reviewed publication.
Second, publications are themselves the primary mechanism through which knowledge is transmitted within the scientific community. The existence of the articles based on SBIR projects is therefore direct evidence that the results of these projects are being disseminated widely, which in turn means that the congressional mandate to support the creation and dissemination of scientific knowledge is being met.
We note that like other SBIR agencies, NASA does not have evaluation programs in place to compare knowledge effects within and outside the SBIR program.
Tracking Knowledge Dissemination by Citation Analysis
Citation studies have been used extensively to show the transfer of knowledge from federally funded projects to others outside the walls of the funded
TABLE 4-19 Publications from NASA SBIR Phase II Awards
Number of Publications
Number of Responses
0
97
1
20
2
19
3
6
4
4
5
8
8
2
10
3
12
1
30
1
Total
161
SOURCE: NRC Phase II Survey, Question 18.
OCR for page 97
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
projects, thereby demonstrating the wider potential impact of the federal funds. In the case of paper-to-patent citations, this is done by examining references to scientific and engineering papers on the front pages of U.S. patents. References are also made to previously issued patents. Both sets of patent and nonpatent references comprise the “prior art” of patents.
Citation analysis has been used at various times by the U.S. Department of Energy, the National Institute of Standards and Technology, the Agricultural Research Service, the National Science Foundation50, and other federal agencies to show movement of knowledge from scientific research programs—where impacts are difficult to measure—to industrial technology—where impact measurement is more tractable.51 Patent citation trees are routinely used by ATP, for example, to show the dissemination of technical knowledge via patents from completed projects to other companies and other organizations.52
No evidence was found, however, of publication or patent citation analysis by the NASA SBIR program. Further, no evidence was found of the systematic collection by NASA of the detailed publication, and patent data from SBIR projects needed to support citation studies.
Yet, as indicated by the results of the NRC Phase II Survey, patents and scientific publications are being produced by the NASA SBIR program. Hence, opportunities exist to encourage program participants to publish when it will not compromise their ability to commercialize. Both publication and patent citation analysis could be used to demonstrate and track knowledge dissemination from NASA SBIR projects to others.
4.5.3
Licensing
Licensing agreements depend on the protection of the intellectual property. They are another indicator of the creation and dissemination of knowledge.
Respondents reported licensing as an important activity they engaged in with other companies and investors both in the U.S. and abroad. Table 4-20 shows the frequency with which respondents said they had finalized or were negotiating licensing agreements to commercialize technologies resulting from the referenced award. Respondents formed licensing agreements with both foreign companies and investors and with domestic companies and investors.
The use of licensing signals the underlying importance of intellectual prop-
50
The referenced use of citation analysis by NSF lies outside the NSF SBIR program. NSF supported extensive work by CHI Research, Inc. to develop and “clean” databases needed to perform publication citation analysis.
51
For an example of a citation study performed for a federal R&D program, see J. S. Perko and Francis Narin, CHI Research, Inc., “The Transfer of Public Science to Patented Technology: A Case Study in Agricultural Science,” Journal of Technology Transfer 22(3):65-72.
52
Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report—Number 2, NIST Special Publication 950-2, Gaithersburg, MD: National Institute of Standards and Technology, pp. 266-270.
OCR for page 98
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
TABLE 4-20 Licensing Activities of Phase II Surveyed Awardees with U.S. and Foreign Companies and Investors
Focus of Interactions
Finalized Agreements (%)
Ongoing Negotiations (%)
Interactions with U.S. Companies and Investors
6
11
Interactions with Foreign Companies and Investors
8
5
SOURCE: NRC Phase II Survey, Question 12.
erty protection to high-tech small businesses. Case-study results also highlight the importance of intellectual protection and licensing activities as a major commercialization strategy for several small businesses. Licensing activities tend to increase the diffusion of a technology’s effect, and as noted by Jaffe, licensing tends to increase spillover effects, particularly market spillovers.53
4.5.4
Partnerships of Small Firms with Other Companies and Investors
Partnering with other organizations and people also accomplishes knowledge transfer. For small companies, the formation of partnerships with other companies is often an essential strategy for commercializing a technology. The larger companies they partner with often have manufacturing capacity, marketing know-how, and distribution paths in place. Awardees whose technology is far upstream of consumer goods may need to: partner with other companies for the additional research needed to integrate their technologies into larger systems; partner with Original Equipment Manufacturers who purchase the awardees’ output as intermediate goods; and form alliances with customers to more effectively reach markets.
The NRC Phase II Survey provided insight about the kinds of partnerships being formed by SBIR recipients. As shown in Table 4-21, partnerships for R&D, for marketing and distribution, with customers, and for manufacturing were found to be formed by these awardees.
Licensing agreements may or may not entail close partnering, whereas the other listed forms generally do require close alliances and partnering.
4.5.5
Interactions Among Small Firms and Universities
Many companies with NASA also have relationships with universities through which knowledge is created and disseminated. Many funded projects in-
53
Adam Jafee, Economic Analysis of Research Spillovers: Implications for the Advanced Technology Program, NIST GCR 97-708, Gaithersburg, MD: National Institute of Standards and Technology, pp. 42-44.
OCR for page 99
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
TABLE 4-21 Percent of Phase II Surveyed Awardees Forming Partnerships with U.S. and Foreign Companies and Investors
Partnering for:
With U.S. Companies and Investors
With Foreign Companies and Investors
Finalized (%)
Ongoing Negotiations (%)
Finalized (%)
Ongoing Negotiations (%)
Licensing Agreement(s)a
6
11
8
5
R&D Agreement(s)
16
10
4
1
Marketing/Distribution Agreement(s)
7
5
6
2
Customer Alliance(s)
11
9
6
0
Manufacturing Agreement(s)
2
6
2
0
Joint Venture Agreement(s)
1
4
0
2
SOURCE: NRC Phase II Survey, Question 12.
volve university faculty, graduate students, and/or university developed technologies. University faculty and students establish small businesses. Faculty members serve as proposal reviewers. Universities assist firms with proposal preparation and sub-contracts or consult on projects. They also sometimes provide facilities and equipment to assist projects. The NRC Firm Survey showed that over 62 percent of all respondents had at least one founder with an academic background. Around 31 percent of company founders had been most recently employed by a college or university prior to founding the company.
The NRC Phase II Survey showed that 29 percent of NASA projects involved some form of university involvement. The survey data show the prime mode of involvement to be faculty members or adjunct faculty members working on the referenced project in a role other than PI—as a consultant, for example. The next most frequent modes of involvement were those of universities/colleges as subcontractors, graduate students working on the project, and university or college facilities or equipment being used on the project. In some instances, project technologies were originally developed in universities or colleges by one of the participants in the referenced projects. On occasion, the technologies for the referenced projects were licensed from a university or college. Table 4-22 indicates the extent to which each type of university involvement occurred in the sample Phase II projects.
The NRC Phase II Survey results show that the NASA SBIR plays some role in moving research concepts out of the university. Of the Phase II survey projects, 4 percent involved technology that was originally developed at a university by a project participant. Two percent of the technologies in the Phase II survey projects were licensed from a university. In addition, some of the case-study firms were found to have on-going affiliations with universities.
Although only 7 percent of the Phase II awards reported that they had received assistance in Phase I or Phase II proposal preparation, universities were
OCR for page 100
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
TABLE 4-22 Involvement by Universities and Colleges in NRC Phase II Survey Projects
Type of Relationship Between Referenced Project and Universities/Colleges
Respondents Reporting the Relationships (%)
Faculty members or adjunct faculty member worked on the project in a role other than PI.
17
Graduate students worked on the project.
15
University/College facilities and/or equipment were used on the project.
13
A university or college was a subcontractor on the project.
16
The technology for this project was originally developed at a university or college by one of the participants in the referenced project.
4
The technology for the project was licensed from a university or college.
2
The Principal Investigator (PI) for the project was at the time of the project an adjunct faculty member.
1
The Principal Investigator (PI) for the project was at the time of the project a faculty member.
2
SOURCE: NRC Phase II Survey, Question 31.
responsible for providing most of that assistance. When asked to evaluate the usefulness of the proposal assistance, five of the 11 Phase II recipients which reported receiving assistance rated it as “very useful” and the other six rated it as “useful;” none said it was “not useful.”
As shown in Table 4-23, overall, 29 percent of the respondents reported involvement in their Phase II project by faculty, graduate students, or university-developed technologies. This result is reinforced by the sample of firm case studies. Of the 22 firms, 14 demonstrated important linkages with the university sector. Many SBIR projects therefore do seem to promote the transfer of knowledge between the private sector (the awardee) and universities.
TABLE 4-23 In Executing the Phase II Award Was There Involvement by University Faculty, Graduate Students, and/or University-developed Technologies—(N=161)
Response
Percent of Respondents
Yes
29
No
71
SOURCE: NRC Phase II Survey, Question 31.
OCR for page 101
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
4.5.6
Assessing Knowledge Expansion
Developing and disseminating knowledge derived in some part from SBIR projects depend on both the riskiness of the project and the often indirect ways though which knowledge spreads.
Risk Profile
One question about the SBIR program is the extent to which it funds projects that are truly innovative.
This is a difficult and important area. There are pressures on program managers to ensure that levels of commercialization are high—yet commercialization outcomes are inversely related to the riskiness of the research: Very high risk projects are less likely to reach the market than modest adjustments to a technology that already has customers.
In interviews, program managers at all agencies recognize this potential difficulty. However, in the main they remain focused on the need to enhance commercialization. The risk of insufficient innovation is both lower priority and, perhaps in the long run more important, is not easily assessed. Sales can be counted; innovation lies in the eye of the beholder.
It is therefore important that agencies continue to use existing indicators to monitor the riskiness of the projects they fund, and to seek to develop new ones. Much information can be found in a better understanding of why and when projects fail:
The NRC Phase II survey reported that technical difficulties were one important reason for discontinuing Phase II projects—they were the fifth most cited reason.
The NRC Phase I survey also suggested that technical risk among NASA projects was high. Of the Phase I projects that did not get a follow-on Phase II award, a leading reason was technical barriers.
Indicators, Not Measures of Benefit
No economic benefits are generated from knowledge efforts until the knowledge flows are actually used by others to develop new and improved products, processes, and services. Hence, collection of data on knowledge generation and dissemination activities does not provide direct measurement of impacts. Such data can, however, serve to construct indicators of potential impacts. Examples of possible indicators are number of patents per research dollar, characteristics of collaborative networks formed, and sales of commercialized goods and services. Trends in these and other indicators may indicate that developments are occurring along an indirect path—as would be expected for projects that are progressing toward the generation of broad impacts.
OCR for page 102
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
It is apparent from the NRC Phase II Survey results that it would be possible to compile multiple indicators of knowledge generation and dissemination and early commercialization achievements from NASA SBIR projects, and to track them over time. Thus far, however, it appears that such indicators have been developed only partially and on an ad hoc basis. It appears that more could be done to systematically compile and track indicators of knowledge generation and dissemination if desired.
4.5.7
Conclusions on SBIR’s Knowledge Impact
Given its descriptions and proposal selection criteria which emphasize the achievement of broad impacts (i.e., not just the commercialization goal), it might be expected that NASA’s SBIR program would demonstrate a strong interest in measures of knowledge outputs. But, in fact, broader impacts appear to be defined by the program largely as commercial results and the infusion of the new technologies in the agency’s mission. Little evidence was found that the program pays much attention to knowledge outputs per se.
As a first step, NASA (and other agencies) might consider requiring that recipient firms provide bibliographic citations for papers appearing in peer-reviewed journals, the proceedings of scientific societies, or conference reports, as part of their outcome reporting process.
4.6
CONCLUSIONS
There is no single simple metric that adequately captures “results” from the program, as discussed in the NRC’s Methodology Report for the SBIR assessment.54 Each of the four congressional mandates is best assessed separately, and within each, there are a multiple issues to be addressed.
Bearing all these points in mind, it is still possible to summarize the results of our research in straightforward terms.
4.6.1
Commercialization
Approximately 30-40 percent of Phase II projects produce innovations that reach the market, with a small number generating substantial returns. Other indicators of commercialization, such as licensing activities, marketing partnerships, and access to and utilization of further investments from both private and public sources, all confirm that while returns are highly skewed, and the results in general are positive, firms operating within the NASA SBIR program face significant
54
National Research Council, An Assessment of the Small Business Innovation Research Program—Project Methodology, op. cit.
OCR for page 103
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
structural barriers that make it hard to develop substantial markets based on the highly NASA-specific technologies the agency tends to fund.
4.6.2
Agency Mission
While it is difficult to find good data on the extent to which SBIR supports NASA’s mission we can conclude the following:
The SBIR program has been aligned with the agency needs, primarily through the topic development and award selection processes. This process has been considerably altered by changes made in FY2006.
Outcomes from the SBIR program appear to be aligned with agency needs, although the small number of projects that are selected for NASA Phase III funding helped to drive the FY2006 reforms.
Agency staff in general indicate that SBIR awards are of comparable quality to other NASA research projects.
This analysis also indicates that early involvement with and “ownership” of SBIR projects by NASA technical staff is an important factor in the successful utilization of SBIR for agency purposes.
4.6.3
Support for Woman- and Minority-owned Businesses
NASA’s SBIR program supports the participation of minority and woman-owned small business in innovation research.55
More widely, SBIR significantly supports small high technology businesses in general, and the NRC research determined that SBIR had an important catalytic effect in terms of company foundation—providing the critical seed money to fund a company’s first steps. SBIR also had strongly influenced companies’ decisions to initiate individual projects: 68 percent of NRC Phase II Survey respondents at NASA believed that their projects would not have gone forward without SBIR, and, of the remainder, most believed that the projects would have been delayed and/or would have had a reduced scope.56
4.6.4
Support for the Advancement of Scientific and Technical Knowledge
The program funds cutting-edge research, as it was designed to do. One of the key selection criteria at NASA is “technical innovation.”
NASA SBIR funding also supports the dissemination of knowledge through traditional vectors such as peer-reviewed publications, as well traditional indica-
55
See Chapter 3 of this report for more details.
56
NRC Phase II Survey, Questions 13-14.
OCR for page 104
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
tors that valuable intellectual property has been produced, such as patents. About 40 percent of projects led to at least one peer-reviewed publication, and about 20 percent of projects generated at least one successful patent application.
It is therefore appropriate to conclude that the NASA SBIR program is meeting all four of the congressional objectives.