Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
An Assessment of the SBIR Program at the Department of Defense Summary I. INTRODUCTION The Small Business Innovation Research (SBIR) program was created in 1982 through the Small Business Innovation Development Act. The program was designated as having four distinct purposes: “(1) to stimulate technological innovation; (2) to use small business to meet federal research and development needs; (3) to foster and encourage participation by minority and disadvantaged persons in technological innovation; and (4) to increase private sector commercialization innovations derived from federal research and development.”1 As the SBIR program approached its 20th year of operation, the U.S. Congress requested the National Research Council (NRC) of the National Academies to conduct a “comprehensive study of how the SBIR program has stimulated technological innovation and used small businesses to meet federal research and development needs,” and to make recommendations on improvements to the pro- 1 Small Business Innovation Development Act (PL 97-219). In reauthorizing the program in 1992 (PL 102-564) Congress expanded the purposes “to emphasize the program’s goal of increasing private sector commercialization developed through federal research and development and to improve the federal government’s dissemination of information concerning the small business innovation research program, particularly with regard to program participation by woman-owned small business concerns and by socially and economically disadvantaged small business concerns.” The evolution of the SBIR legislation was influenced by an accumulation of evidence beginning with David Birch in the late 1970s suggesting that small businesses were assuming an increasingly important role in both innovation and job creation. This trend gained greater credibility in the 1980s and was confirmed by empirical analysis, notably by Zoltan Acs and David Audretsch of the U.S. Small Business Innovation Data Base, which confirmed the increased importance of small firms in generating technological innovations and their growing contribution to the U.S. economy. See Zoltan Acs and David Audretsch, Innovation and Small Firms, Cambridge, MA: MIT Press, 1990.
OCR for page 2
An Assessment of the SBIR Program at the Department of Defense gram.2 Mandated as a part of SBIR’s reauthorization in late 2000, the NRC study has assessed the SBIR program as administered at the five federal agencies that together make up some 96 percent of SBIR program expenditures. The agencies, in order of program size, are the Department of Defense (DoD), the National Institutes of Health (NIH)3, the National Aeronautics and Space Administration (NASA), the Department of Energy (DoE), and the National Science Foundation (NSF). Based on that legislation, and after extensive consultations with both Congress and agency officials, the NRC focused its study on two overarching questions.4 First, how well do the agency SBIR programs meet four societal objectives of interest to Congress: (1) to stimulate technological innovation; (2) to increase private sector commercialization of innovations (3) to use small business to meet federal research and development needs; and (4) to foster and encourage participation by minority and disadvantaged persons in technological innovation.5 Second, can the management of agency SBIR programs be made more effective? Are there best practices in agency SBIR programs that may be extended to other agencies’ SBIR programs? To satisfy the congressional request for an external assessment of the program, the NRC analysis of the operations of SBIR program involved multiple sources and methodologies. Extensive NRC commissioned surveys and case studies were carried out by a large team of expert researchers. In addition, agency-compiled program data, program documents, and the existing literature were reviewed. These were complemented by extensive interviews and discussions 2 See Public Law 106-554, Appendix I—H.R. 5667, Section 108. 3 The legislation designates the Department of Health and Human Services (DHHS) as the agency responsible for the SBIR program, and some components of DHHS, other than NIH, have SBIR programs. The DHHS program is dominated by NIH awards and the study’s focus remains the NIH, in this case taken to represent the entire department. 4 Three primary documents condition and define the objectives for this study: These are the Legislation—H.R. 5667, the NRC-Agencies Memorandum of Understanding, and the NRC contracts accepted by the five agencies. These are reflected in the Statement of Task addressed to the Committee by the Academies leadership. Based on these three documents, the NRC Committee developed a comprehensive and agreed set of practical objectives to be reviewed. These are outlined in the Committee’s formal Methodology Report, particularly Chapter 3, “Clarifying Study Objectives.” National Research Council, An Assessment of the Small Business Innovation Research Program—Project Methodology, Washington, DC: The National Academies Press, 2004, accessed at <http://books.nap.edu/catalog.php?record_id=11097#toc>. 5 These congressional objectives are found in the Small Business Innovation Development Act (PL 97-219). In reauthorizing the program in 1992, (PL 102-564) Congress expanded the purposes to “emphasize the program’s goal of increasing private sector commercialization developed through federal research and development and to improve the federal government’s dissemination of information concerning small business innovation, particularly with regard to woman-owned business concerns and by socially and economically disadvantaged small business concerns.”
OCR for page 3
An Assessment of the SBIR Program at the Department of Defense BOX S-1 Special Features of the Department of Defense SBIR Program Scale. The SBIR program at DoD is the largest of all the SBIR programs. At $943 million in 2005, DoD accounts for over half the program’s funding. Diversity of Operation. The program is spread across the three services and seven agencies involving widely different missions, ranging from missile defense to Navy submarines to Army support for special forces to the special needs of DARPA. The Acquisition Objective. Unlike some major agency participants in the program (e.g., NIH & NSF), DoD seeks to acquire and use many of the technologies and products developed through the program. For many DoD officials, this is the primary objective of the program. Testing and Certification. Given the demands inherent in the Defense mission, the services have stringent requirements for testing and certification that typically require substantial additional investments before commercialization is realized through DoD acquisition. Similarly, the long lead times involved in the procurement process means that careful attention must be paid to identify and integrate relevant SBIR projects if they are eventually to find a place in a weapons system. Innovation and Experimentation. The department has launched a series of management initiatives and experiments in an effort to enhance the program’s return, especially through greater commercialization, over the last decade. The department has also led the way in commissioning external evaluations of its program. with program managers, program participants, agency ‘users’ of the program, as well as program stakeholders.6 The study as a whole sought to understand operational challenges and measure program effectiveness, including the quality of the research projects being conducted under the SBIR program, the challenges and achievements in commercialization of the research, and the program’s contribution to accomplishing agency missions. To the extent possible, the evaluation included estimates of the benefits (both economic and noneconomic) achieved by the SBIR program, as well as broader policy issues associated with public-private collaborations for technology development and government support for high-technology innovation. Taken together, this study is the most comprehensive assessment of SBIR to 6 The Committee’s methodological approach is described in National Research Council, An Assessment of the Small Business Innovation Research Program—Project Methodology, op. cit. For a summary of potential biases in innovation survey responses, see Box 4-1.
OCR for page 4
An Assessment of the SBIR Program at the Department of Defense date. Its empirical, multifaceted approach to evaluation sheds new light on the operation of the SBIR program in the challenging area of early-stage finance. As with any assessment, particularly one across five quite different agencies and departments, there are methodological challenges. These are identified and discussed at several points in the text. This important caveat notwithstanding, the scope and diversity of the report’s research should contribute significantly to the understanding of the SBIR program’s multiple objectives, measurement issues, operational challenges, and achievements. II. SUMMARY OF KEY PROGRAM FINDINGS The SBIR program at the Department of Defense is meeting the legislative and mission-related objectives of the program. The program is contributing directly to enhanced capabilities for the Department of Defense and the needs of those charged with defending the country. With regard to the specific legislated objectives of the program, the DoD SBIR program is: Achieving significant levels of commercialization.7 Within DoD, commercialization can take multiple forms, sometimes involving insertion in the acquisition process, and/or direct sales to the government of through private commercial markets, licensing of technologies, and the acquisition of SBIR firms by larger Defense suppliers in the private sector. It is also true that the potential for private commercialization is sometimes inhibited by the very nature of the defense mission. Commercialization and the potential for commercialization can be measured in a variety of fashions, ranging from sales and licensing to additional DoD or private investment to acquisition of the technology or the firm by other companies. A significant proportion of the SBIR awards achieve commercialization, although other factors naturally also contribute to this process. For example, 46 percent of projects responding to the NRC Phase II Survey reported some sales or licensing revenues; a further 18 percent anticipated such revenues in the future.8 Commercial success tends to be concentrated. As is true with private sector early-stage projects (e.g., those funded by venture capital), a small number 7 See Finding A in Chapter 2. A more detailed discussion of tools and metrics for assessing commercialization can be found in Chapter 4. It should be noted that the complex character of commercialization means that it cannot be captured in any individual metric. 8 See Figure 4-1.
OCR for page 5
An Assessment of the SBIR Program at the Department of Defense FIGURE S-1 Distribution of sales, by size (percent of projects with >$0 in sales). SOURCE: NRC Phase II Survey. of firms account for a large proportion of commercial success.9 Other SBIR firms often provide valuable services and products but do not reach the highest levels of commercial returns.10 9 SBIR awards often occur earlier in the technology development cycle than where venture funds normally invest. Nonetheless, returns on venture funding tend to show the same high skew that characterizes commercial returns on the SBIR awards. See John H. Cochrane, “The Risk and Return of Venture Capital,” Journal of Financial Economics 75(1):3-52, 2005. Drawing on the VentureOne database Cochrane plots a histogram of net venture capital returns on investments that “shows an extraordinary skewness of returns. Most returns are modest, but there is a long right tail of extraordinary good returns. 15 percent of the firms that go public or are acquired give a return greater than 1,000 percent! It is also interesting how many modest returns there are. About 15 percent of returns are less than 0, and 35 percent are less than 100 percent. An IPO or acquisition is not a guarantee of a huge return. In fact, the modal or ‘most probable’ outcome is about a 25 percent return.” See also Paul A. Gompers and Josh Lerner, “Risk and Reward in Private Equity Investments: The Challenge of Performance Assessment,” Journal of Private Equity 1(Winter 1977):5-12. Steven D. Carden and Olive Darragh, “A Halo for Angel Investors,” The McKinsey Quarterly 1, 2004, also show a similar skew in the distribution of returns for venture capital portfolios. 10 Unlike venture or angel investors, DoD SBIR funds research projects, not R&D companies as a whole. Angel investors or venture capitalists are an appropriate referent group, though not an appropriate group for direct comparison. Venture capital investors normally provide significant equity in exchange for ownership of a significant portion of the firm. Venture investors participate in firm governance and firm strategy and advise and recruit the firm’s management team. SBIR awards essentially support projects.
OCR for page 6
An Assessment of the SBIR Program at the Department of Defense Meeting the agency mission. The DoD SBIR program is contributing significant enhancements to the department’s mission capabilities.11 The innovations developed by SBIR companies are contributing to U.S. technological dominance, reducing the cost of operation of support systems, providing new capabilities, and providing increased responsiveness to new challenges (e.g., improvised explosive devices12). Case studies indicate that SBIR projects have provided valuable mission technologies, some within a very short space of time, and others with very large impacts.13 The highly structured topic development process is designed to ensure that SBIR projects are aligned with specific Defense needs, and the department has made increased efforts to ensure that this is the case. SBIR offers significant advantages for DoD with respect to its mission by providing: Shorter planning horizons, which provide what agency staff see as unique flexibility within the execution year; A low risk “technological probe” or research tool for finding new solutions, new technologies, and new suppliers, contributing to enhanced quality and capacity for systems and operations; Access to technologies and providers otherwise largely excluded from the prime-dominated R&D process at DoD;14 Sole source contracts for successful technologies permitting more rapid acquisition with fewer constraints than normal procurements conducted under the Federal Acquisition Regulation (FAR). In some cases, projects meet agency mission objectives without generating substantial commercial outcomes. Supporting small business and competition.15 The DoD program provides substantial benefits for small business participants in terms of market access, funding, and recognition. The program supports a diverse array of small businesses contributing to the vitality of the defense industrial base while providing greater competition and new options and opportunities for DoD managers. New entrants. The program attracts a substantial number (37 percent) of small business participants who are new to the program each year.16 Formation of new, innovative companies. A significant portion of re- 11 See Finding B in Chapter 2. 12 See Box 4-3. 13 See Section 4.3.2 for references to SBIR companies. 14 See evidence in Chapter 4, especially Section 4.3. 15 See Finding C in Chapter 2. 16 See Figure 3-3.
OCR for page 7
An Assessment of the SBIR Program at the Department of Defense spondents to the NRC Firm Survey (25 percent) reported that they were founded entirely or partly because of an SBIR award.17 Support for academic researchers to transition ideas to the defense market. About 25 percent of projects had some significant relationship to a university.18 Encouragement for university-industry and other partnerships. DoD SBIR funding provides the resources for small firms to engage academic consultants and other private sector partners, as reflected in case studies. Substantial impact on project initiation. A large percentage of the surveyed firms (about 70 percent) reported that their project would definitely or probably not have gone ahead without the SBIR funding; many of the remainder indicated that they would have anticipated substantial delays without the SBIR award.19 Market recognition. An SBIR award provides markets with additional information concerning the technical and commercial potential, generating a certification effect with regard to potential investors and customers.20 Supporting woman-owned small business concerns and by socially and economically disadvantaged small business concerns.21 The SBIR provides important support, notably in light of the contributions noted just above, to minority- and woman-owned firms. Trends in awards to woman-owned firms are positive. Awards to woman-owned firms have continued to increase both in absolute numbers (303 Phase I awards in 2005) and as a percentage of the overall awards (12.9 percent of Phase I awards in 2005).22 Trends in awards to minority-owned firms are more problematic. The share of Phase I awards to minority-owned firms has declined substantially since the mid-1990s, falling below 10 percent for the first time in 2004, where it has remained. (See Figure S-2.) This trend in awards for minority 17 See Table 4-15 (NRC Firm Survey, Question 1). Data reported in Table 4-15 are for firms with at least one DoD award. NRC Firm Survey results reported in Appendix B are for all agencies (DoD, NIH, NSF, DoE, and NASA). 18 NRC Phase II Survey, Question 31. See also Table 4-19. 19 See Figure 4-10 (NRC Phase II Survey, Question 13). 20 Innovation awards “may be a signal to non-government sources of funding, such as banks, venture capital firms, and other potential investors, that the firm has a potential future stream of revenue from a reliable customer (the U.S. government.)” See Joshua Lerner, “’Public Venture Capital’: Rationales and Evaluation,” in National Research Council, The Small Business Innovation Research Program: Challenges and Opportunities, Charles W. Wessner, ed., Washington, DC: National Academy Press, 1999. See also Maryann P. Feldman and Maryellen R. Kelly, “Leveraging Research and Development: The Impact of the Advanced Technology Program,” in National Research Council, The Advanced Technology Program: Assessing Outcomes, Charles W. Wessner, ed., Washington, DC: National Academy Press, 2001, p. 204. 21 See Finding E in Chapter 2. 22 See Figure 3-12 based on the DoD awards database.
OCR for page 8
An Assessment of the SBIR Program at the Department of Defense FIGURE S-2 Number of Phase I DoD SBIR awards to minority-owned firms and percentage of all DoD Phase I awards, 1992–2005. SOURCE: DoD awards database. firms suggests the need for further empirical examination. There may be a need for enhanced outreach by the department.23 Knowledge generation. The DoD SBIR program is generating significant intellectual capital. The program contributes to new scientific and technological knowledge, as SBIR companies have generated numerous publications and patents. This knowledge takes many forms, not all of which are easily measured.24 Publications. Thirty-two percent of DoD survey respondents published at least one peer-reviewed article based on the SBIR project surveyed by the NRC, and about 15 percent of them generated more than five articles.25 Patents. Nearly 35 percent of the DoD projects surveyed generated at least one patent application and just over 25 percent had received a patent related to the project.26 Indirect effects. Interviews by the research team offer strong anecdotal evidence with regard to indirect effects of projects that provide investigators, research staff, and DoD management with knowledge that can become useful in a different context. In some cases, knowledge is transferred from 23 Based on DoD awards database. 24 See Finding F in Chapter 2. 25 See Figure 4-18 (NRC Phase II Survey, Question 18). 26 See Table 4-18.
OCR for page 9
An Assessment of the SBIR Program at the Department of Defense one company to another; in others, the knowledge becomes relevant in a different technical context.27 Nonlinear innovation paths. By its three-phase structure, the SBIR program suffers from an implicit linear myth, namely that a single grant for a single project is sufficient to fully develop a technology and drive long-term growth of the company. In practice, a single grant is often not sufficient to commercialize a product. Often multiple related projects, complementary technologies, and varied funding sources are needed, in addition to effective management, to bring a product to market. With regard to additional awards, NRC survey data indicate that returns are maximized where firms have received 10-25 previous Phase II awards.28 III. SUMMARY OF KEY PROGRAM RECOMMENDATIONS As noted in the Findings section above, the Department of Defense has an effective SBIR program. The recommended improvements listed below should enable the DoD SBIR managers to address the four mandated congressional goals in a more efficient and effective manner. Improve the Phase III transition. DoD should continue and expand its work on improving the Phase III transition (the transition from SBIR-funded Phase I and Phase II research to further research along the road toward commercialization—especially testing and evaluation funded by other DoD sources).29 Areas for possible action include: Provide incentives. Expansion of positive incentives for program officers to utilize the SBIR program for their own research needs, beyond the current requirements for their involvement in topic development.30 Adopt roadmaps. Inclusion of SBIR in the technology and program development roadmaps that guide programs through the TRL stages, and eventually through acquisitions.31 Identify and apply best practices across services and between services. Some components have been particularly successful at Phase III transition. Techniques used by these components may be transferable.32 27 See Section 126.96.36.199. See also Box 4-6 in Chapter 4 for a case study of Advanced Ceramics Research (ACR). The value of ACR’s low-cost, small unmanned aerial vehicle (UAV), initially developed for whale watching around Hawaii, became apparent in Iraq, where it is used as a highly flexible, general-purpose battlefield surveillance tool. 28 See Section 188.8.131.52. 29 See Recommendation A in Chapter 2. 30 See Recommendation A-1 in Chapter 2. 31 See Recommendation A-5 in Chapter 2. 32 See Recommendation D-3 and Box 2-3.
OCR for page 10
An Assessment of the SBIR Program at the Department of Defense Expand linkages. Improved linkages with other programs (e.g., ManTech) might be enhanced to facilitate the Phase III transition.33 Interest primes. Encouragement for the growing interest of the prime contractors on the SBIR Program’s outputs and opportunities for partnering with SBIR companies. Consideration should be given to performance incentives for further encouraging development of SBIR supported technologies.34 Expand commercialization assistance. Expanded commercialization programs that provide training, counseling, and networking opportunities.35 Develop program profile. DoD leadership should take steps to improve the perception of the SBIR program’s potential and accomplishments within the department, address obstacles to fulfilling the program’s potential, and seek resources to enhance program operation and outcomes as a means of generating the attention and partnering needed for the program to operate most effectively.36 Encourage pilot programs. Making changes initially through pilot programs allows DoD to alter selected areas on a provisional basis; a uniform approach is unlikely to work well for all components of a program that funds highly diverse projects with very different capital requirements and very different product development cycles.37 DoD should also identify best practices across the SBIR program, and implement them as appropriate, across the Department.38 Improve management and assessment. Additional funding should be provided for program management and assessment in order to encourage and support the development of a results-oriented SBIR program. Effective management requires additional staff and funding. To manage the program effectively requires better monitoring of awardees, enhanced efforts to facilitate commercialization, the regular collection of higher-quality data and its more systematic assessment. Currently, sufficient resources are not available for these functions. Expand evaluation. DoD should substantially strengthen and expand its evaluation efforts in order to further develop a program culture that is driven by outcomes and backed by internal and external evaluations.39 DoD—like the other SBIR agencies—should be encouraged to develop and provide to Congress a comprehensive annual report on SBIR. This 33 See Recommendation A-6 in Chapter 2. The U.S. Army Manufacturing Technology (ManTech) Program supports the development and implementation of advanced manufacturing technologies for the production of Army Material. 34 See Recommendation A-8 in Chapter 2. 35 See Recommendation A-9 in Chapter 2. 36 See Recommendation B in Chapter 2. 37 See Recommendation E-1 in Chapter 2. 38 See Box 2-2. 39 See Recommendation C in Chapter 2.
OCR for page 11
An Assessment of the SBIR Program at the Department of Defense will enhance program accountability. Such a publication will require additional management funding (noted above). DoD should develop methodologies and capabilities that allow for a sharing of best practices across its services and defense agencies. DoD should enhance existing efforts to develop the collection of data needed to evaluate program outcomes. Provide management funds. To enhance program utilization, management, and evaluation, as called for above, consideration should be given to the provision of additional program management and evaluation funds. There are three ways that this might be achieved:40 Additional funds might be allocated internally, within the existing budgets of the services and agencies, as the Navy has done. Funds might be drawn from the existing set-aside for the program to carry out these activities. The set-aside for the program, currently at 2.5 percent of external research budgets, might be modestly increased, with the goal of providing additional resources for management and evaluation to maximize the program’s return to the nation.41 Increase the participation and success rates of woman- and minority-owned firms:42 Improve data collection and analysis. The Committee strongly encourages the agencies to gather and report the data that would track woman and minority firms as well as principal investigators (PIs), and to ensure that SBIR is an effective road to opportunity. Encourage participation. Develop targeted outreach to improve the participation rates of woman- and minority-owned firms, and strategies to improve their success rates. Encourage emerging talent. Encourage woman and minority scientists and engineers with the advanced degrees to serve as principal investigators (PIs) and/or senior co-investigators (Co-Is) on SBIR projects. 40 See Recommendation F in Chapter 2. 41 Each of these options has its advantages and disadvantages. For the most part, the departments, institutes, and agencies responsible for the SBIR program have not proved willing or able to make additional management funds available. Without direction from Congress, they are unlikely to do so. With regard to drawing funds from the program for evaluation and management, current legislation does not permit this and would have to be modified; therefore the Congress has clearly intended program funds to be for awards only. The third option, involving a modest increase to the program, would also require legislative action and would perhaps be more easily achievable in the event of an overall increase in the program. In any case, the Committee envisages an increase of the “set-aside” of perhaps 0.03 percent to 0.05 percent on the order of $35 million to $40 million per year, or roughly double what the Navy currently makes available to manage and augment its program. In the latter case (0.05 percent), this would bring the program “set-aside” to 2.55 percent, providing modest resources to assess and manage a program that is approaching an annual spend of some $2 billion. Whatever modality adopted by the Congress, without additional resources the Committee’s call for improved management, data collection, experimentation, and evaluation may prove moot. 42 See Recommendation G in Chapter 2.