The National Aeronautics and Space Administration (NASA) has in recent years—and especially since the 2011 reauthorization—experimented with new initiatives within the Small Business Innovation Research (SBIR) program. These include the following:
- Electronic Handbook (EHB)
- Technology infusion managers (TIMs)
- Phase II Enhancement (PII-E) awards
- Phase II-Expanded (PII-X) awards
- Select Topics
- Commercialization Readiness Pilot (CRP) program
- Enhanced data collection
Three of these new initiatives (the EHB, TIMs, and Phase II-E awards) have been in operation for some time; others are just being implemented (the CRP program and enhanced data collection). Each of these initiatives is discussed in turn in the following sections—with the exception of enhanced data collection, which is discussed in a general way within the EHB section. That section relates that recent additions to the EHB have focused on incorporating data collection tools for outcomes to enhance data collection capabilities.
ELECTRONIC HANDBOOK (EHB)1
Introduction and EHB Overview
Originally developed with funding from a NASA SBIR Phase II contract in 1989, the EHB has evolved into a highly effective grants/contracts
1 See background information at SBIR/STTR Awardee Firm Electronic Handbooks, https:/ehb8.gsfc.nasa.gov/contracts/public/firmHome.do. This is also the registration and login site.
and program management tool. First deployed in 1996, the EHB now provides NASA with a complete end-to-end paperless system for managing the SBIR program. Approximately 6,000 users are currently on the NASA system, and overall the system owner and manager, REI Systems, serves more than 250,000 users annually. An example of NASA best practices, the EHB is now in use at a number of federal agencies and other grant-giving organizations.
The EHB contains seven modules corresponding to the different phases of the SBIR program:
- Solicitation. The Solicitation Development module facilitates the collaborative development of the research topics and subtopics for the annual NASA SBIR solicitation. The final solicitation is published through the EHB.
- Submission. Small Business Concerns (SBCs) electronically submit their Phase I and Phase II SBIR proposals via the Proposal Submissions module.
- Administrative screening. Proposals are administratively screened using the Proposal In-processing module, which tracks proposal status, problems identified, and eventual resolution.
- Review and selection. The Review and Selection module is used for evaluation, ranking, recommendation, selection, and debriefing of Phase I and Phase II SBIR proposals.
- Contract negotiation. The Contract Negotiation and Award module supports the negotiation and award of NASA SBIR Phase I and II proposals selected for award and maintains current and archived contracts.
- Contract administration. The Contract Administration and Closeout module facilitates the contract administration and eventual closeout of NASA SBIR Phase I and Phase II contracts, including the submission, review, and acceptance of contract deliverables. The EHB is currently used to manage invoices and approve payments.
- Tracking. The Tracking, or Post Award Successes Module, provides collection and reporting capabilities of post-Phase II successes including Phase III, infusion, and commercialization. This module is a relatively new addition.
Figure 3-1 highlights the range and complexity of the processes managed within the EHB by illustrating the more than 50 different functional roles played by NASA staff, consultants, and company executives within the NASA SBIR process.
Program and Process Dashboards
The EHB provides state-of-the-art tools for program management. It offers users a series of individually customizable dashboards through which to view data in the system, with drill-down capabilities. This dashboard allows program managers to easily review selected data covering applications and awards for a range of variables. These capabilities are more comprehensive than those available at any of the other SBIR programs, including the Department of Defense (DoD), where considerable efforts have been expended in this direction. Similar dashboards allow managers to closely track progress against defined milestones.
Because the system was designed from the start to be a tracking system for contracts, it provides both applicants/contractors and program managers with appropriately differentiated views into the process.
In addition to the detailed and customizable dashboards described above, the EHB has evolved an extensive set of reporting tools. These include both standard reports using a simple trigger and easily customized searches using a range of fields across a number of data sets. Table 3-1 describes the available search tools.
These search tools allow management to track program activities. They also enable companies and NASA officials to explore opportunities by finding technologies and their owners within the system. The TechSource search tool is publicly available on the NASA website; anyone can search using a range of keywords, categories, and filters.
TABLE 3-1 Search Tools Available Within the EHB
Allows customized “ad hoc” searches across solicitations for topics/subtopics, proposals, awards, firms; Reports on all historical data from 1983 to present; Detailed or graphical report options available
Performs keyword searches on the following database fields and uploaded documents; Abstract, Project Title, PI and COTR, Firm Name, Taxonomy Mappings, Submitted docs: Proposal and Deliverable documents
Post Award Success Reports
Allows customized “ad hoc” searches across all post awards and success opportunities
Canned reports with drill down capabilities: Proposal Statistics; Award Statistics; State-based Statistics; Women/Minority/HubZone; Firm-based Search Tool; Subtopic-based Search Tool; Proposal Info Tool Commercial Metrics Survey Tool
Searchable site, solicitations, and awards
EHB Quick Search
Searchable proposals, awards, firms
SOURCE: REI Systems.
Utilization of the EHB
The EHB is a state-of-the-art system for managing and capitalizing on the NASA SBIR program.2 Discussions with NASA staff within the Research Centers suggest that Field Center and Headquarter (HQ) SBIR staff use the EHB extensively as a management tool and as a resource. The EHB is also used to manage the SBIR program on an operational level. The EHB provides tools through which standard tasks can be accomplished with minimal effort, while facilitating a constant flow of information across the program.
It is concluded that the EHB is an example of best practice that SBIR programs at other agencies should consider for adoption (and indeed that other programs that offer grants or contracts beyond SBIR also should consider). The EHB is currently in use at the Department of Homeland Security SBIR program and the Small Business Administration (SBA). It is in itself an important SBIR success story.
That said, NASA SBIR staff are not taking full advantage of the EHB capabilities to develop a data-driven approach to program management. The EHB tools have become much more useful for this purpose since the agency started collecting outcomes data in 2012, while reaching back to awards in previous years. (The data are collected in ways that also capture the current commercialization status of projects funded in earlier years.) NASA currently has on file outcomes data for about 2,000 SBIR and Small Business Technology Transfer (STTR) projects dating back to the 1990s. Additional records are being added.
Data Collection Related to Outcomes
In 2012, NASA adopted a version of the process used at DoD to collect outcomes data for SBIR and STTR awards. Companies are required to enter data on all known SBIR awards (at NASA and at other agencies); however, entering these data is not a prerequisite for applying for or receiving additional awards from NASA, as it is at DoD. The EHB is similar to DoD’s Company Commercialization Record (CCR): data are similar in kind, and the collection procedures are similar in process,3 but the EHB incorporates a number of changes and possible improvements over CCR. These changes and improvements include more detailed data collection within the EHB.
Several more detailed fields in the EHB specifically address infusion into NASA programs. They describe Technology Readiness Levels (TRLs), the specific technology involved, agency components, and other aspects of the
2 The Navy Program Manager’s database is perhaps the closest match as an electronic toolkit for program management.
3 See National Research Council, SBIR Program at the Department of Defense, Washington, DC: The National Academies Press, 2014, p. 68, for a description of the CCR and related procedures.
infusion process. This is a thorough and well-designed effort to help measure the extent to which SBIR technologies are indeed being used by NASA.
Additional fields also relate to matching funds for some programs such as Phase II-E (discussed below). Once again, these adaptations could help provide managers with a more detailed understanding of matching fund sources and commitments.
The development of this module for the EHB represents an important step toward the creation of a data-driven management culture within the NASA SBIR program. It provides a set of tools with which management can work to identify patterns and thereby opportunities for improvement.
EHB Challenges and Opportunities
As with all information technologies, the need for standardization generates tension with the need for flexibility. The EHB has evolved over time. Table 3-2 shows how new capabilities have gradually been added in response to user and agency needs. Recent additions have focused on the addition of data collection tools for outcomes.
Despite this evolution, there is evidence that several of the new initiatives (such as technology infusion managers, discussed in detail in the next subsection) have not fully adopted the EHB for tracking their activities. Not only has one TIM developed an alternative information system based on Salesforce (a popular cloud Customer Relationship Management (CRM) software), but also another TIM uses a 300-page Word document for the same purpose. Both TIMs find these preferable to the EHB, which suggests that the needs of the TIMs are not being fully met by the EHB.
TABLE 3-2 Addition of Capabilities to EHB—Timeline
|Type of Data||Program Year Data Collection Started|
|Phase I||Phase II|
|Contracts||1997 (1998 for STTR)||1997 (1998 for STTR)|
|Contract Deliverables||2001 (more complete over time)||2000 (more complete over time)|
|Technology Taxonomy Mapping||2005||2005|
|STR Technology Area Mapping||2011||2011|
|TRL||2007 (required in 2008)||2007 (required in 2008)|
|Recommendation Quad Charts||2006||2007|
|Briefing Charts||2001 (Required in 2007)||2000 (Required in 2007)|
|Post Award Successes (Phase II-E, Phase III, Infusion, Commercialization)||APG metrics from FY12 plus Commercialization Metrics data from Firms||APG metrics from FY12 plus Commercialization Metrics data from Firms|
SOURCE: REI Systems.
The Academies4 discussions with NASA staff indicated that program management were not currently using these capabilities to guide the program. These discussions indicated that NASA had not developed (or planned) analytics to link outcomes data with possible explanatory variables or applications with outcomes, and NASA has not provided analytics that compare outcomes, for example by Mission Directorates, Center, technology, type of topic, and companies and projects, based on a range of other possible causal factors such as firm size, demographics, technology, or location. Such analyses can help NASA to develop better topics, identify problems within the process, and find firms and projects that are more likely to be successful.
In sum, the NASA EHB is an example of best practice: The EHB has been adopted by other agencies and programs outside of NASA SBIR, including the SBIR program at the Department of Homeland Security and the SBA. It is in itself an important SBIR success story that can be more widely adopted by other SBIR programs, as well as by other grant and contract programs across the federal government.
However, NASA SBIR staff are not yet taking full advantage of the EHB capabilities to develop a data-driven approach to program management. The EHB tools have the potential to become much more useful for this purpose, particularly since NASA started collecting outcomes data in 2012, and have reached back to collect the outcomes (i.e., current commercialization status) of awards funded in previous years).
NASA SBIR program managers can do more with the EHB to guide the program. They can develop analytics to link outcomes data with possible explanatory variables or applications with outcomes. They can compare outcomes, for example by Mission Directorate, Center, and companies and projects based on a range of possible causal factors such as firm size, demographics, technology, or location.5 These and other analyses can help NASA to develop better topics, identify problems within the process, and find firms and projects that are more likely to be successful.
TECHNOLOGY INFUSION MANAGERS (TIMS)
One potentially important long-running initiative at NASA is the provision of Technology Infusion Managers (TIMs) at each Center. A recent document from Glenn Research Center described the core activities of TIMs:
- Help for NASA program/project managers: TIMs conduct searches of Phase I and Phase II awards, identifying technologies to support a
particular technology interest or the specific needs/requirements of a program/project.
- Help for SBIR small businesses: TIMs facilitate dissemination of information about an SBIR technology of potential interest to program managers.
- Help putting these parties together: TIMs facilitate meetings and web-based teleconferences between NASA program/project managers and SBIR companies. They also provide guidance on seeking post-Phase II funding.6
TIMs are expected to connect with Center technologists, help plan and implement infusion strategies, identify applications and funding sources, report successes, inform prospective users about available and developing technologies, and motivate SBIR advocates. Discussions with TIMs and their supervisors indicate that the implementation of these roles differs among Centers because the Centers have slightly different operations and the Mission Directorates have slightly different needs. Successful TIMs rely on a broad network of contacts that are active in SBIR technology infusions.
Discussions with TIMs and their supervisors also reveal that TIMs generally tend to come from within NASA and to have worked on the operational side, for example as an engineer. They have different levels of knowledge related to commercialization depending on the kind of activity involved. Figure 3-2 provides a schematic illustration: TIM knowledge and understanding declines as the relevant area shifts away from the home center to other centers and then attenuates even further for private-sector entities.
A limited scope of Phase III commercialization expertise is understandable, because TIMs primarily focus on connecting SBIR projects to NASA programs. However, it underscores some of the limitations of the TIM model. Discussions with TIMs and other agency staff suggested that the effectiveness of TIMs varies substantially, but this variation is not substantiated with metrics. NASA did not describe or share metrics for assessing the individual or collective success of TIMs, because it does not have tools in place for tracking TIM activities effectively across Centers or to identify or transfer best practices. This is unfortunate because TIMs develop different strategies and tools, and these natural experiments have the potential to improve operations for the program as a whole if they are appropriately analyzed and transferred effectively.
6 SBIR/STTR Program Office, Glenn Research Center, “Opportunities to Infuse SBIR Technology into NASA Programs: Funding and Strategic Alignment Guidance for the Science Mission Directorate,” 2015.
PHASE II ENHANCEMENT (PII-E) AWARDS
Established in FY2007, the objective of the PII-E program is to “further encourage the advancement of innovations developed under Phase II via an option of R/R&D efforts underway on current Phase II contracts.”7 Firms that can attract an external investor can apply to NASA for matching funds up to a pre-set limit. By using the matching funds approach, NASA can ensure that
7 NASA Phase II-E description, http://sbir.gsfc.nasa.gov/content/post-phase-ii-initiatives, accessed March 7, 2015.
there is either commercialization or investment interest from a third party. The PII-E program requires that the matching funds come from a third party (external to the NASA SBIR program, though potentially from other sources within NASA).
The PII-E program has evolved somewhat since its inception. The maximum program contribution was $250,000 until 2011, and has been $150,000 since then. The maximum amount of agency funding counting both Phase II and Phase II-E is now $900,000, down from $1 million in 2007-2011 (see Table 3-3). Reasons for these changes were not provided. Select Topics can be funded at a higher level (see Select Topics section, below).
Phase II-E Award Patterns
From 2007 to 2011, NASA made 94 PII-E awards, using about $12.5 million in NASA SBIR funding and attracting about $16.0 million in matching funds (see Table 3-4). Data for 2013 and 2014 are not yet available from NASA (as of May 2015).
TABLE 3-3 Evolution of the NASA Phase II-E Program
|Applicable Period/Solicitation||Minimum Non-SBIR/STTR Funding Required||Corresponding SBIR/STTR Program Contribution||Maximum Cumulative Award (Phase II + Phase II-E Match)|
|April 2016 – onwards||$25,000||1:1 match to a maximum of $150,000||$900,000 (SBIR and STTR) $1,650,000 (SBIR Select)|
1:1 match to a maximum of $125,000
$875,000 (SBIR and STTR) $1,625,000 (SBIR Select)
1:1 match to a maximum of $250,000
SOURCE: NASA Phase II-E description, http://sbir.gsfc.nasa.gov/content/post-phase-ii-initiatives, accessed March 7, 2015.
TABLE 3-4 SBIR PII-E Awards at NASA, 2007-2011
|Year||Number of PII-E SBIR Awards||SBIR Phase II-E Funding (Dollars)||Matching Contribution (Dollars)|
SOURCE: Awards data provided by NASA.
In the context of the NASA SBIR program, these are relatively modest numbers. During the same time period, NASA made a total of 1,605 awards, so Phase II-E awards accounted for only 5.8 percent of NASA awards. The SBIR program provided $53.7 million in funding during this period, and PII-E awards accounted for 2.8 percent of SBIR program expenditures. The 94 PII-E awards were distributed to 76 different companies, with the two most prolific companies each receiving four awards and $600,000 in SBIR funds.
Phase II-E—Sources of Matching Funds
The PII-E program is designed in part to help Phase II awardees connect to other funding sources, especially within NASA. Figure 3-3 shows the distribution of matching funds and confirms that NASA is the largest single source of funding for the 94 PII-E projects at 55 percent, followed by the private sector at 27 percent and DoD at 10 percent.
Phase II-E Outcomes
Although Phase II-E projects account for less than 6 percent of all NASA SBIR Phase II contracts from 2007 to 2011, they account for 155 out of 735 projects, or 21.1 percent, in the commercialization database. For Phase II
projects awards in 2005-2009 (the corresponding years for the subsequent PII-E award, which comes at the end of Phase II), the 155 PII-E accounts come from 400 Phase II projects, or 38.8 percent, in the EHB database. This suggests that Phase II-E is to some degree aligned with commercialization.
These numbers should, however, be treated with some caution. The outcomes data collected so far from the EHB are incomplete, and there is no way to determine the biases that affect this reporting. During the same period, NASA initiated a total of 789 Phase II SBIR awards, so only about 51 percent of all NASA awards reported outcomes through the EHB. Even with these caveats, it is still striking that 20 percent of Phase II awards accounted for about 39 percent of the records in the new commercialization database. Further exploration of these data—viewed against the backdrop of the Great Recession and the recovery—would be potentially fruitful. With better data in the EHB, such broader analysis would be possible.
Phase II-E funding is limited in several ways, aside from the relatively short time horizon (4 months). First, new work proposed under Phase II-E must “build upon and demonstrably advance” the technology developed under Phase II, which suggests that more ambitious expansions of the technology may not be funded. On the other hand, Phase II-E is supposed to “lead to new outcomes not achievable with Phase II funding alone.”8 Second, there is a tight window for Phase II-E applications, between the 12th and the 15th month of the Phase II award.
External funding can come from a range of investors, including not only private investors, but also a NASA program, a NASA contractor, or a non-SBIR/non-STTR government program. Government matching funds from outside the SBIR program are not limited because they are regarded as a Phase III event and hence not subject to Phase II limits. Discussions with NASA staff indicate that NASA does not include sales revenues as a potential match. A precise match may change by solicitation (i.e., annually).
PHASE II-EXPANDED (PII-X)
Launched in 2014, the Phase II-Expanded (Phase II-X) program represents a new effort to create better bridges between the SBIR/STTR program and the NASA Mission Directorates (MDs) who have acquisition funds. Effectively, the program operates in ways quite similar to the Phase II.5 (or Phase 2.5) program recently implemented in some components at DoD9: the SBIR program provides additional funding against a match specifically provided by the NASA Mission Directorates as a bridge toward commercialization and use by NASA MDs.
The PII-X program aims to “establish a strong and direct partnership between the NASA SBIR/STTR program and other NASA projects undertaking the development of new technologies or innovations for future use.”10 The program adds an option to all NASA Phase II contracts, which can be exercised under appropriate circumstances.11 The option provides for a 2:1 match by the SBIR program against funding from other NASA sources. To participate, eligible firms must secure a NASA program or project (other than the NASA SBIR/STTR program) as an investment partner funding further research or infusion activities. A minimum of $75,000 in NASA non-SBIR/non-STTR funding is required, and the SBIR program will match up to a maximum expenditure of $500,000. If fully exercised, then this funding would take the maximum award (Phase II + PII-X) to $1.25 million for standard topics and $2 million for Select Topics (see Table 3-5). Contributions from other NASA programs or projects are not limited, because they are not regulated under SBIR guidelines.
The PII-X program started too recently for outcomes to be available for analysis. However, this effort does indicate that NASA continues to explore initiatives that will connect the SBIR program more effectively to other programs at NASA. Agency staff indicated that Phase II-X funds account for about 5 percent of SBIR/STTR program funds.
Starting in 2012, NASA has identified a small number of topics as Select Topics. Similar to standard topics in many respects, these topics attract additional SBIR/STTR funding. In 2015, these Select Topics were limited to the Human Exploration and Operations Mission Directorate and the Science Mission Directorate.12
TABLE 3-5 NASA Phase II-X Program
Minimum NASA Non-SBIR/STTR Funding Required
Corresponding SBIR/STTR Program Contribution
Maximum Cumulative Award (Phase II + Phase II-X Match)
2:1 match to a maximum of $500,000
$1,250,000 (SBIR and STTR) $2,000,000 (SBIR Select)
SOURCE: NASA, “Post Phase II Initiatives and Opportunities, Phase II-X”, http://sbir.gsfc.nasa.gov/content/post-phase-ii-initiatives#Phase-II-x, accessed July 23, 2015.
10 NASA, Post Phase II initiatives, http://sbir.gsfc.nasa.gov/content/post-phase-ii-initiatives, accessed March 8, 2015.
11 Note: companies may apply for Phase II-E or Phase II-X but not both (presumably because Phase II-X is a subset of Phase II-E, excluding non-NASA partners). Applications windows for both programs are tight. Companies must provide notice of intent to apply by the 13th month of the Phase II award. The submission window is the 4th month of the second year of a Phase II award.
12 Discussion of Select Topics draws in part on the NASA 2015 Select Topic Solicitation, http://sbir.gsfc.nasa.gov/solicit/54565/detail?l1=55463, accessed March 8, 2015.
Select Topics are run as a separate solicitation in parallel with the standard solicitation. Awards are capped at a higher level—$125,000 for Phase I and $1.5 million for Phase II. The performance time period is unchanged at 6 months for Phase I and 2 years for Phase II. NASA also imposed additional company limits on the FY 2015 Select SBIR Solicitation: acceptance of no more than three proposals from any one firm and award of no more than two Select Topic SBIR contracts to any applicant under the solicitation.13
As shown in Table 3-6, NASA made 26 Phase I Select awards and 10 Phase II Select awards totaling about $18.7 million in 2012. The number of Phase I awards increased by almost 40 percent in 2014, from 26 to 36 (data for Phase II awards had not yet been tabulated for 2014 at the time of writing).
Select Topics are a NASA initiative to place additional funding on topics that MDs consider to be of particular significance or priority for their operations. It remains to be seen whether this approach is more successful than standard solicitations, but it does represent another potentially important initiative aimed at improving program outcomes. NASA expects that end-of-project Technology Readiness Levels (TRLs) will be significantly higher than those for standard topics.14
COMMERCIALIZATION READINESS PILOT PROGRAM (CRP)
NASA is now engaging with opportunities presented under reauthorization through the Commercialization Readiness Pilot (CRP) program.15 A limited pilot in 2014 expanded in 2015, aimed at increasing the infusion of SBIR-/STTR-developed technology into NASA’s broader programs. Under the CRP program, other NASA programs will act as sponsors, who will show how proposed activities result in risk reduction and bridge the “TRL gap” discussed in Box 3-1. Increasing TRL levels will help commercialization. The SBIR/STTR program will then offer matching funds to support these activities.
TABLE 3-6 NASA SBIR Awards Made Under Select Topics, 2012-2014
|Year||Number of Select Topic Phase I SBIR Awards||Select Topic Phase I SBIR Funding ($)||Number of Select Topic Phase II SBIR Awards||Select Topic Phase II SBIR Funding ($)|
SOURCE: Awards data provided by NASA.
13 NASA 2015 Select Topic Solicitation, http://sbir.gsfc.nasa.gov/solicit/54565/detail?l1=55463, accessed July 23, 2015.
14 Carol Lewis, FAQ: NASA SBIR Technology Infusion and Post Phase II Opportunities, NASA internal memo, November 2013.
15 Information on the CRP program is drawn from Joseph Grant, SBIR/STTR, Presentation to the National SBIR/STTR conference, 2014, and discussions with agency staff.
The CRP program is designed to focus specifically on bringing technologies through TRL 5-6 so that they can be funded by NASA
development contracts, because that is the stage where many small companies fail (the so-called “Valley of Death”).16 The program requires that the related Phase II SBIR contract be awarded no earlier than 2008 and provides up to $1.5 million from the SBIR/STTR program over 24 to 36 months.
The NASA SBIR program has put in place several promising initiatives. The EHB is an important and potentially powerful electronic management tool that provides a mechanism for enhanced data collection.17 TIMs can play a useful role in connecting SBIR companies, potential customers, and Center priorities. The Phase II-E, Phase II-X, and CRP initiatives all have potential to help link the SBIR program to downstream agency programs. The concept of Select Topics seems an appropriate mechanism for identifying and funding projects that are especially important from the agency’s perspective to bring them to the desired higher TRL level, but the results of its employment should be monitored for effects on the broader SBIR program at NASA.
The larger questions for the NASA SBIR program revolve around implementation, follow-through, and tracking of these initiatives. NASA has developed enhanced data collection tools for a data-driven approach—one that can permit ongoing evaluation of the current initiatives and the identification and implementation of appropriate adjustments—however, previous NASA initiatives such as the NASA Alliance for Small Business Opportunities (NASBO)—which sought to link NASA SBIR companies with commerce-ready technologies to small business service providers, large contractor firms, and investors—lacked consistent implementation. Although the EHB has great potential, our research shows further effort is needed for this potential to be realized.
16 Other studies in science have noted the TRL gap and the need to address it. See, e.g., National Research Council, New Worlds, New Horizons in Astronomy and Astrophysics, Washington, DC: The National Academies Press, 2010, Table ES.1 and Chapter 5.
17 Our analysis focused only on the use of the Electronic Handbook in relation to the NASA SBIR Program, but it has great potential outside the program and is used elsewhere.