An Assessment of the SBIR Program at the National Aeronautics and Space Administration (2009)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Summary I.  INTRODUCTION The Small Business Innovation Research (SBIR) program was created in 1982 through the Small Business Innovation Development Act. As the SBIR pro- gram approached its twentieth year of operation, the U.S. Congress requested the National Research Council (NRC) of the National Academies to “conduct a com- prehensive study of how the SBIR program has stimulated technological innova- tion and used small businesses to meet Federal research and development needs” and to make recommendations with respect to the SBIR program. Mandated as a part of SBIR’s reauthorization in late 2000, the NRC study has assessed the SBIR program as administered at the five federal agencies that together make up some 96 percent of SBIR program expenditures. The agencies, in order of program size are the Department of Defense (DoD), the National Institutes of Health (NIH), the Department of Energy (DoE), the National Aeronautics and Space Administration (NASA), and the National Science Foundation (NSF). Based on that legislation, and after extensive consultations with both Con- gress and agency officials, the NRC focused its study on two overarching ques- tions. First, how well do the agency SBIR programs meet four societal objectives of interest to Congress: • To stimulate technological innovation; • To increase private-sector commercialization of innovations; • To use small business to meet federal research and development needs; and • To foster and encourage participation by minority and disadvantaged persons in technological innovation.  

 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION Second, can the management of agency SBIR programs be made more ef- fective? Are there best practices in agency SBIR programs that may be extended to other agencies’ SBIR programs? To satisfy the congressional request for an external assessment of the pro- gram, the NRC conducted empirical analyses of the operations of SBIR based on commissioned surveys and case studies. Agency-compiled program data, program documents, and the existing literature were reviewed. In addition, extensive inter- views and discussions were conducted with project managers, program partici- pants, agency ‘users’ of the program, as well as program stakeholders. The study as a whole sought to answer questions of program operation and effectiveness, including the quality of the research projects being conducted under the SBIR program, the commercialization of the research, and the program’s con- tribution to accomplishing agency missions. To the extent possible, the evaluation included estimates of the benefits (both economic and noneconomic) achieved by the SBIR program, as well as broader policy issues associated with public- private collaborations for technology development and government support for high technology innovation. Taken together, this study is the most comprehensive assessment of SBIR to date. Its empirical, multifaceted approach to evaluation sheds new light on the operation of the SBIR program in the challenging area of early stage finance. As with any assessment, particularly one across five quite different agencies and departments, there are methodological challenges. These are identified and discussed at several points in the text. This important caveat notwithstanding, the scope and diversity of the report’s research should contribute significantly to the understanding of the SBIR program’s multiple objectives, measurement issues, operational challenges, and achievements. This volume presents the com- mittee’s assessment of the SBIR program at the National Aeronautics and Space Administration. This study analyzes program data from the period before the 2006 NASA reorganization, which also altered the management of the SBIR program. None- theless, the results of this study are valuable, not least as a point of reference to see whether the organizational changes make recently have enhanced the effec- tiveness of the NASA SBIR program. II.  SBIR AT NASA Program Size With $103 million in annual awards in 2005, NASA operates the fourth larg- est SBIR program in the federal government. A NASA SBIR Phase I award is currently set at a maximum of $100,000 and lasts for six months. A Phase II award is set at a maximum of $600,000 and 

SUMMARY  Other (3.3%) * NSF (4.3%) $79 million Total = $1.85 billion * NASA (5.6%) $103 million * DoE (5.6%) $104 million * DoD (50.9%) $943 million * HHS (30.4%) $562 million FIGURE S-1  Dimensions of the SBIR program in 2005. SOURCE: U.S. Small Business Administration. Accessed at <http://tech-net.sba.gov/>. NASA Figure S-1.eps lasts for a period of up to two years. NASA has not yet adopted the Phase IIB or Phase II Plus or Fast Track option that exists at some other agencies. Program Management The NASA SBIR program has varied over the years in terms of how central- ized it is. Until recently, program operations were run at each of the 10 NASA field centers with NASA Headquarters, supported by a national office located at Goddard, focusing on the overall administration of the program. Following NASA’s recent reorganization, the program will be less decentralized. It will run through only four field centers (Ames, JPL, Glenn, and Langley) with Ames replacing Goddard as the national office. Each NASA center has an SBIR Field Center Program Manager who admin- isters the program at the respective center. Contracts are managed by NASA’s Contracting Officer at each center with support from the Contract Officer Tech- nical Representative (COTR). The COTR serves as the primary contact within NASA on a contract’s technology focus and objectives. Overall program policy, effectiveness, and assessment are the responsibility of the Headquarters Program Executive. It should be noted that the NRC assessment of SBIR program management at NASA has dealt with a moving target. Because management structures at NASA have changed so extensively, data from past projects are of limited relevance in guiding current management (because structures have already changed since the relevant projects were funded.) 

 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION BOX S-1 The Challenge of Assessing SBIR in a Restructuring NASA As with other parts of NASA, the NASA SBIR program has, experienced sequential waves of reorientation and restructuring. Mission objectives have changed very substantially, far more than at other SBIR agencies. During NASA’s reorganization of 2003-2004, the agency’s SBIR program became a component of the Advanced Space Technology Program within the Ex- ploration Systems Mission Directorate (ESMD), which is charged with implement- ing NASA’s planned exploration of Mars and other space exploration projects. In 2006, further reorganization led a change in the balance of management power between the Mission Directorates and the Centers, with the former assuming much more direct authority over SBIR topic and award selection. Because of this churn, any assessment of program management at NASA must deal with a moving target. Extensive changes in management structures mean that data regarding past activities is of limited relevance in directly guiding current management. However, it provides a basis for NASA to judge if its orga- nizational changes have improved the effectiveness of its SBIR program. Acquisition and Commercialization With the appointment of NASA Administrator Michael D. Griffin in 2005, NASA made “spin-in”—the use of SBIR technology by NASA for mission needs—the SBIR program’s main priority. The shift to mission-purpose uses is accompanied by other fundamental changes underway at NASA: • A new Moon-Mars mission, which is altering the relative position of NASA’s 10 centers; • A redesign of NASA’s Innovation Partnership Program to reduce frag- mentation and to emphasize spin-in; • A new budget model that employs full-cost accounting and incorporates competition for funding among NASA centers; and • Movement of the Innovation Partnership Program to NASA Headquar- ters, giving it higher priority status within the agency. Like DoD (and unlike NSF and NIH) NASA is a procurement agency. How- ever, NASA’s SBIR program has not relied solely on procuring the technologies it funds as the only means of fostering commercialization. It has emphasized mar- ketplace commercialization as well as infusion for mission use. Recent changes in NASA have led to a greater emphasis on infusion for mission use or spin-in outcomes. Furthermore, NASA funds companies from diverse industries across   As noted above, the SBIR program is now part of the Innovative Partnership Program (IPP). 

SUMMARY  BOX S-2 Some Special Features of the NASA SBIR Program Research Topics.  NASA’s Mission Directorates conceive and describe the top- ics each year. Subtopic conception, composition, and development are done by project managers and researchers at the various NASA installations. Acquisition.  Unlike some major participants in the program (e.g. NIH & NSF), NASA seeks to acquire and use many of the technologies and products developed through the program. NASA intends to use SBIR funding increasingly for spin-in and less for spin-out. Electronic Handbook.  Small businesses submit their proposals to NASA through a sophisticated multipurpose online system called Electronic Handbooks and E- Submission (EHB). NASA’s Electronic Handbook is a “set of Internet-based tools that support the paperless documentation and management of complex distrib- uted processes,” including the SBIR program. EHB helps guide users through the program and provides real-time, online, paperless documentation and process management. a fairly broad spectrum of technologies presenting a challenge to NASA’s SBIR program in achieving its commercialization goals. Evaluation Culture NASA has initiated program analysis, experimentation, and evaluation, but a successful effort requires funding and management support over the long term. In 2002, NASA published the results of its Commercial Metrics project, which surveyed all Phase II firms and gathered the commercialization results of NASA SBIR projects. However, this project has been halted pending further funding. NASA posts numerous “success stories” on its Web site. NASA also de- velops “quad charts” that describe the technology of an SBIR project and its potential uses. I.  KEY PROGRAM FINDINGS A. The NASA SBIR program is making significant progress in achieving the congressional goals for the program. Keeping in mind NASA’s unique mission and the recent significant changes to the program, the SBIR program   Access at <http://sbir.nasa.gov/SBIR/success.htm>.   See the Committee’s Finding A in Chapter 2. 

 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION is sound in concept and effective in practice at NASA. With the program- matic changes recommended here, the SBIR program should be even more effective in achieving its legislative goals. B. The NASA SBIR program helps its award recipients achieve significant levels of commercialization. • According to the NRC Phase II Survey, nearly half (some 46 percent) of NASA Phase II projects reach the marketplace and generate revenue.  17.7 percent of those projects with revenues generate revenues greater than $1 million. • From 1983 to 1996, NASA SBIR projects created goods and services that generated over $2.3 billion in revenues in the private economy. •  notable feature of NASA commercialization is that 46 percent of all A sales resulting from Phase II awards went to markets other than the fed- eral government.10 C. SBIR Phase II projects result in substantially useful results for NASA, comparable to other NASA R&D.11 • According to a survey of NASA Contracting Officer’s Technical Repre- sentatives (COTRs):12 Nearly two-thirds (63 percent) deemed SBIR projects to have signifi- cant research value.13 Over a third (34.6 percent) of surveyed projects resulted were deemed by NASA COTRs to have resulted in a product or service of commer- cial value. More than two-thirds (68 percent) of COTRs reported that SBIR spending gave the same or more benefits to the agency mission as other NASA R&D projects.   These changes create discontinuities in program goals that complicate assessment. These important changes are described in the Chapter 5 on Program Management.   These objectives are set out in the Small Business Innovation Development Act (PL 97-219). In reauthorizing the program in 1992, (PL 102-564) Congress expanded the purposes to “emphasize the program’s goal of increasing private-sector commercialization developed through federal research and development and to improve the federal government’s dissemination of information concerning small business innovation, particularly with regard to woman-owned business concerns and by socially and economically disadvantaged small business concerns.”   See the Committee’s Finding B in Chapter 2.   See Figure 4-1, which is based on the NRC Phase II Survey.   See Figure 4-2.   See NASA Commercial Metrics Survey, October 2002, page 1. Access at <http://www.sbir.nasa. gov/SBIR/survey.html>. 10 See Table 4-1. 11 See the Committee’s Finding C in Chapter 2. 12 See the NRC Project Manager Survey in Appendix D of this volume. 13 NRC Project Manager Survey, Table App-D-20. 

SUMMARY  D. The NASA SBIR program stimulates collaboration, technological in- novation and generates new knowledge.14 • About a quarter of projects responding to the NRC Phase II Survey re- ported filing at least one related patent; and a fifth received at least one patent.15 • The NASA SBIR program has stimulated links among NASA, small businesses, and universities. Nearly a third (29 percent) of the NRC Phase II Survey respondents reported having university participation in their projects. E. NASA SBIR provides substantial, frequently decisive, support for small businesses.16 • Firm Initiation.  From the NRC Firm Survey, 20 percent of the NASA respondents stated that they were founded at least in part due to SBIR. 17 • Project Initiation.  Over two-thirds (68 percent) of SBIR Phase II award recipients say that they definitely or probably would not have undertaken the funded research project without the SBIR funding.18 • Developmental funding.  Just under half of the respondents (44 percent) received additional funding for their project subsequent to the receipt of the Phase II award.19 F. NASA’s SBIR program supports the participation of minority- and woman-owned small businesses in innovation research.20 • During the 1997-2004 period, minority-owned firms received 12.18 per- cent of Phase II awards and woman-owned firms received 9.94 percent of Phase II awards (see Figure S-2). • Participation by minority- and woman-owned firms in the SBIR program did not appear to greatly increase or diminish with time (see Figure S-2). G. NASA’s technology transfer program has shifted recently from a focus on commercialization (“spin-out”) to a focus on supplying mission needs (“spin-in” or “infusion.”)21 • This shift has created significant challenges for the SBIR program. • The new emphasis on spin-in requires the creation of a new regional in- 14  See the Committee’s Finding D in Chapter 2. 15  See Table 4-18. 16  See the Committee’s Finding E in Chapter 2. 17  See NRC Phase II Firm Survey, Question 1. 18  See Figure 4-8. 19  See NRC Phase II Survey, Question 22. 20  See the Committee’s Finding F in Chapter 2. 21  See the Committee’s Finding G in Chapter 2. 

 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 30 Woman-owned Firms Minority-owned Firms 25 Either Percent of All Phase II Awards 20 15 10 5 0 1997 1998 1999 2000 2001 2002 2003 2004 Year FIGURE S-2  NASA SBIR Phase II awards, by demographic group, 1997-2004. SOURCE: National Aeronautics and Space Administration. NASA Figure S-2.eps frastructure focused on technology acquisition, not technology generation and diffusion. H. NASA does not provide an appropriate level of resources for assessing the program’s performance. Consequently, NASA SBIR program man- agement is not sufficiently data-driven.22 • Given the size and scope of its SBIR program, NASA does not provide an appropriate level of resources for monitoring and assessing the program’s performance. • Partly because of lack of sufficient funding, the program is not suffi- ciently evidence-based. It lacks clear benchmarks and metrics for success. Program evaluation—while recently improved—needs to be enhanced further. II.  KEY PROGRAM RECOMMENDATIONS A. Additional management resources are needed.23 • Effective management and evaluation requires adequate funding. An 22  See the Committee’s Finding I in Chapter 2. 23  See the Committee’s Recommendation D in Chapter 2. 

SUMMARY  evidence-based program requires high quality data and systematic assessment. •  enhance program utilization, management, and evaluation, the NASA To SBIR program should be provided with additional funding for manage- ment and evaluation. B. NASA should evaluate the impact of NASA’s reorganization on SBIR.24 • Following the recent agency restructuring, NASA seeks to make “spin-in” the main priority for the SBIR program. • NASA should study how the new agency orientation towards spin-in will impact SBIR program outcomes. • The new NASA structure and the Innovative Partnership Program (IPP) should be evaluated in terms of its technology transfer management goals. C. NASA should develop data for evaluation, conduct regular assessments, and report to Congress.25 • The NASA SBIR program should develop a series of specific data objec- tives—identifying both the data needed to run the program well and the means of acquiring those data. • Each year, NASA should provide Congress with a summary report on the SBIR program. This annual report should include descriptive statistics for applications, awards, and outcomes along the dimensions identified in this report, including knowledge creation, technology innovation, and impact on agency mission, as well as commercialization. • NASA should also commission regular external arms-length evaluations to assess the program progress and the impact of new initiative. D. NASA should consider the creation of an independent Advisory Board.26 • This Advisory Board would draw together senior agency management, SBIR managers, and other stakeholders as well as outside experts to review current operations and achievements and recommend changes to the SBIR program. • The Advisory Board could be assembled on the model of the Defense Science Board (DSB) or perhaps the National Science Foundation’s SBIR Advisory Board.27 24  See the Committee’s Recommendation G in Chapter 2. 25  See the Committee’s Recommendation E in Chapter 2. 26  See the Committee’s Recommendation F in Chapter 2. 27  The intent here is to use the DSB or the NSF SBIR Board as a model, not something necessarily to be copied exactly. 

10 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION E. NASA should continue to encourage program flexibility and experimen- tation, followed by evaluation of outcomes.28 • NASA should develop an effective program for developing, deploying, and evaluating pilot initiatives. • NASA should explore how to increase the flexibility of projects, given changes in technology and information. • Guided by regular assessments of outcomes, NASA should expand the NASA Alliance for Small Business Opportunities (NASBO). • NASA should evaluate other agencies’ approaches to commercialization assistance and adopt the best approaches where applicable. 28  For the Committee’s formal recommendation, see Recommendation C in Chapter 2.